database/redis distributedsystem systemdesign
The Redis Streams We Have Known and Loved | by Oleksii (Alex) Zakharchenko | bitso.engineering | Medium
Redis Streams | Docs
Introduction
Redis Streams is a DS introduced Redis 5.0 that allows you to store, retrieve, and process Data Streams in a log-like, append-only fashion. It enables message queuing, real time event processing and distributed system coordination.
- Append-only: New messages are added at the end.
- Message ID: Each message has a unique, time-based ID.
- Consumer Groups: Multiple consumers can read messages in parallel.
- Persistence: Unlike Pub/Sub, messages are stored in Redis until explicitly deleted.
- Auto-trimming: Supports limiting stream length to save memory.
Understanding “Log-like” and “Append-Only” in Redis Streams
Tldr
- Log-like = Messages are stored sequentially and can be replayed.
- Append-Only = Messages are never modified, only added.
Info
What Does “Log-like” Mean ?
A log is a sequence of records stored in chronological order. Redis Streams behaves like a log, meaning:
- Data is recorded sequentially: New messages are appended at the end.
- Past messages remain available until explicitly deleted.
- Consumers can replay data by reading old messages (unlike Pub/Sub, where messages are lost if not read immediately).
What Does “Append-Only” Mean ?
- No Overwrites: Data is never updated in place—only new messages are added.
- Historical Data is Preserved (unless explicitly deleted).
- Ensures Durability: Redis Streams can be used as an event store for processing historical data.
Comparison with Other Redis Data Structures
Feature Redis Streams Redis Lists Redis Sets Append-Only ✅ Yes ✅ Yes ❌ No Unique IDs ✅ Yes ❌ No ❌ No Persistent Storage ✅ Yes ❌ No (if not persisted) ✅ Yes Replay Old Data ✅ Yes ✅ Yes ❌ No
Usage
Redis CLI
Add Messages ( XADD)
This command adds an entry to a stream and return generated Timestamp based ID.
XADD mystream * name "Alice" age 25
XADD mystream * name "Bob" age 30*—> Redis automatically generate an ID- ID —>
<milliseconds-time>-<sequence-id>
Reading Messages
After a message is read, it is NOT automatically deleted from Redis Streams. The message remains in the stream until explicitly deleted or trimmed.
Reading All Messages
XRANGE mystream - +
# Reverse Order
XRANGE mystream + -Reading new Messages
XREAD BLOCK 5000 STREAMS mystream $BLOCK 5000: Waits up to 5 seconds for new messages.STREAMS mystream $: Reads only new messages.
Other Commands
# Trim Old Messages
XTRIM mystream MAXLEN 1000
XDEL mystream <key>Consumer Groups
A Consumer Group in Redis Streams allows multiple consumers to share the workload of processing messages from a stream. Instead of all consumers receiving the same message (like in Pub/Sub), each message is delivered to only one consumer in the group.
# 1. Create a stream and add messages
XADD mystream * name "Alice"
XADD mystream * name "Bob"
# 2. Create a consumer group
# `mystream`: Stream name.
# `mygroup`: Consumer group name.
# `$`: Start reading new messages from now.
XGROUP CREATE mystream mygroup $
# 3. Consumers read messages (parallel processing)
# `COUNT 1`: Read one message at a time.
# `STREAMS mystream >`: Read only new messages.
XREADGROUP GROUP mygroup consumer1 COUNT 1 STREAMS mystream >
XREADGROUP GROUP mygroup consumer2 COUNT 1 STREAMS mystream >
# 4. Consumers acknowledge messages
XACK mystream mygroup "1710248714594-0"- If a consumer reads a message using
XREADGROUP, the message stays in the stream. - The message moves to the Pending Entries List (PEL) until acknowledged with
XACK. - If a consumer acknowledges the message using
XACK, it is removed from the PEL but remains in the stream. - The message is only removed if you explicitly delete (
XDEL) or trim (XTRIM) the stream.
Code Example (Go)
Redis List vs Redis Stream
| Feature | Redis List (LPUSH / RPUSH) | Redis Stream (XADD, XREAD) |
|---|---|---|
| Data Model | Ordered list of string values | Log-like append-only structure with key-value pairs |
| Message Storage | FIFO queue-like | Persistent log for event-driven systems |
| Access Pattern | Push (LPUSH, RPUSH), Pop (LPOP, RPOP), Range (LRANGE) | Append (XADD), Read (XREAD, XREADGROUP), Query (XRANGE, XREVRANGE) |
| Message Retention | Messages disappear after LPOP | Messages persist until explicitly deleted |
| Consumer Model | Simple consumer, no tracking of read state | Supports Consumer Groups (multiple consumers can read the same stream) |
| Multiple Consumers | Not supported (only one consumer can pop an element) | Supports multiple consumers reading from the same stream efficiently |
| Guaranteed Delivery | No tracking, message lost if not consumed | Yes, with Consumer Groups and Pending Entries List (PEL) |
| Use Case | Simple queue (e.g., task queue, job scheduling) | Event-driven architectures, logs, real-time data processing |