Data pipelines

Chains of jobs that move and reshape data between systems. They need idempotent steps, monitoring of freshness and a way to backfill history.

Where it shows up in real systems

  • Data warehouse

    Built for large analytical queries that take seconds, not for serving users. Placing it in a request path destroys latency; feed it asynchronously through a pipeline instead.

    In production: Snowflake, Google BigQuery, Amazon Redshift, ClickHouse, Databricks SQL

  • Batch processing cluster

    Spark-style jobs split data into partitions processed by many executors. Throughput is enormous but results arrive hours later; feed it from a queue, a log or object storage, never from the user request path.

    In production: Apache Spark (Databricks, EMR), Google Dataflow, AWS Glue, Hadoop MapReduce

See it break, then fix it

Build an architecture, send it a day of traffic and outages, and learn why it holds or fails. Free to start.

Open the learning path Open the sandbox