Batch processing

Process large bounded datasets in one go, for example every night, with frameworks like Spark or MapReduce. High throughput and simple recovery, but results are hours old.

Where it shows up in real systems

  • Data lake

    Stores everything first and decides the schema at read time. Table formats like Iceberg add transactions and time travel; without governance a lake quickly becomes a swamp.

    In production: Amazon S3 + Apache Iceberg, Delta Lake, Azure Data Lake Storage, Apache Hudi

  • Batch processing cluster

    Spark-style jobs split data into partitions processed by many executors. Throughput is enormous but results arrive hours later; feed it from a queue, a log or object storage, never from the user request path.

    In production: Apache Spark (Databricks, EMR), Google Dataflow, AWS Glue, Hadoop MapReduce

See it break, then fix it

Build an architecture, send it a day of traffic and outages, and learn why it holds or fails. Free to start.

Open the learning path Open the sandbox