Lambda and Kappa architectures

Lambda runs a batch layer for accuracy next to a speed layer for freshness; Kappa uses a single streaming pipeline and replays the log to recompute. Kappa is simpler when the stream processor is reliable.

Where it shows up in real systems

  • Stream processor

    Windows such as the last five minutes turn an endless stream into results: counts, fraud scores, trending topics. Checkpoints let the job restart without losing or double-counting events.

    In production: Apache Flink, Kafka Streams, Spark Structured Streaming, Amazon Managed Service for Apache Flink

  • Batch processing cluster

    Spark-style jobs split data into partitions processed by many executors. Throughput is enormous but results arrive hours later; feed it from a queue, a log or object storage, never from the user request path.

    In production: Apache Spark (Databricks, EMR), Google Dataflow, AWS Glue, Hadoop MapReduce

See it break, then fix it

Build an architecture, send it a day of traffic and outages, and learn why it holds or fails. Free to start.

Open the learning path Open the sandbox