Advanced
Partitions cap your parallelism
A clickstream pipeline writes 20,000 events per second into a log. The team added consumers, yet the lag keeps growing and dashboards are hours behind. Make the pipeline keep up.
Within a consumer group, each partition is read by exactly one consumer.
What your design must achieve
- Worst queue lag < 120 s
- Availability ≥ 99.9 %
- Writes p99 < 150 ms