systemdesign/concepts distributedsystem
Tldr
- Event streaming = the durable transport layer (the “stream” itself).
- Stream processing = the engine that computes over that stream continuously.
- Batch processing = a fundamentally different paradigm — processing data in bounded chunks rather than continuously — though it can consume from the same sources (e.g., reading a Kafka topic in batch mode periodically).
Batch Processing
Processing a finite, bounded set of data all at once, typically on a schedule (e.g., nightly, hourly).
- Data is collected over some period, then processed as a complete chunk.
- Optimized for throughput, not latency — it’s fine for results to be hours or days old.
- Example: running a nightly job that aggregates yesterday’s sales transactions into a report.
- Tools: AWS Batch, Hadoop MapReduce, Apache Spark (batch mode), SQL-based ETL jobs.
The key idea: you wait until you have “all” the data (or a defined slice of it), then run computation over the whole thing.
Examples: AWS Batch, Apache Spark
Stream Processing
The computation/transformation layer that operates continuously on data as it flows through the system — consuming a stream (often from an Event Streaming platform), applying logic, and producing results in (near) real time.
- Unlike batch, there’s no “complete” dataset — the data is unbounded and processing happens continuously, record by record or in small windows.
- Handles complexities like state (e.g., counters), time windows, out-of-order events, and exactly-once guarantees.
Commonly used for applications that require real-time insights. Example of this could be fraud detection, or real-time recommendations.
- Example: counting clicks per user every 5 minutes, detecting fraud as transactions occur.
Tools: Apache Flink, Kafka Streams, Spark Structured Streaming.
How they relate
A simple mental model:
[Event Streaming] → [Stream Processing] → [Sink/Storage]
(Kafka) (Flink) (DB, dashboard)
A useful rule of thumb: streaming answers “what’s happening right now?”, while batch answers “what happened over this period, looking back?”
When to use batching or event streaming?
- Like always, depends on your use case. Can your data processing tolerate some delay? Then maybe batching might be a better fit. Do you need to process the data right now? Will you have an advantage to processing information straight away? Then streaming events might make more sense.
- Remember to consider the costs between batching options and streaming options.
- Use patterns to help your downstream consumers. Remember with streaming and batching you can apply integrations patterns during that particular phase. Example would be enriching events/messages as they go through your stream for downstream consumers.
Introduction
The transport/delivery mechanism for continuously moving individual records (“Event(s)”) from producers to consumers as they happen, in order, often with persistence/replay capability.
- It’s about getting data from point A to point B reliably and in real time — not necessarily transforming it.
- Events are usually immutable facts: “user clicked button X at time T.”
Think of event streams as a flow of events traveling through a river. Each event is captured and can be processed by downstream consumers.
When we build applications there are times where the amount of data to process is unknown and we want to capture it (e.g. user interactions with a shopping cart online), we can collect this data and then process it in real-time downstream (e.g. for analytics or reporting for example).
Data loses value over time. Many companies want to process information in real-time to make decisions. Event streaming can help.
Link to original
