distributedsystem/microservice systemdesign
Apache Kafka Crash Course
Title Unavailable | Site Unreachable
[!video]- Kafka Deep Dive w/ a Ex-Meta Staff Engineer
How Kafka Works
- Kafka is a distributed log-based streaming platform.
- Data is written to topics.
- Each topic is divided into
npartitions. - A hash key (or round-robin) determines which partition a message goes to.
- Within a partition, messages are strictly ordered.
- No global ordering across partitions.
Key Kafka Concepts
| Concept | Description |
|---|---|
| Topic | Logical stream/category of messages. |
| Partition | Unit of parallelism and storage; messages are appended in order. |
| Producer | Sends data to Kafka topic (partitioned via key). |
| Consumer | Reads messages from partitions (via consumer groups). |
| Offset | Unique ID for message position within a partition. |
Limitation of Kafka
No. of consumers = No. of partitions (within a consumer group)
- Each partition can be read by only one consumer in a consumer group.
- This limits horizontal scalability of consumers:
- E.g., 5 partitions → max 5 consumers in one group.
- More consumers than partitions? → Some consumers sit idle.