systemdesign distributedsystem/microservice distributedsystem/microservice/messagequeue
What is a Message Queue and When should you use Messaging Queue Systems Like RabbitMQ and Kafka
🌟 Distributed Message Queue | Book Notes
[!video]- Publish-Subscribe Pattern vs Message Queues vs Request Response (Detailed Discussions with Examples)
Message Queues | System Design
GitHub - donnemartin/system-design-primer: Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.
architecture - What is the difference between message queue and message broker? - Stack Overflow
Redis Message Broker | Redis Enterprise
ByteByteGo | IBM MQ -> RabbitMQ -> Kafka -> Pulsar: Message Queue Evolution
Intro
A message queue is a form of service-to-service communication that facilitates asynchronous communication. It asynchronously receives messages from producers and sends them to consumers
Message Queue and Message Broker
A message queue is a data structure, or a container - a way to hold messages for eventual consumption. A message broker is a separate component that manages queues.
architecture - What is the difference between message queue and message broker? - Stack Overflow
Messages are stored in the queue until they are processed and deleted. Each message is processed only once by a single consumer. Here’s how it works:
- A producer publishes a job to the queue, then notifies the user of the job status.
- A consumer picks up the job from the queue, processes it, then signals that the job is complete.
Features
Push or Pull Delivery
Pull Delivery - Consumer continuously querying the queue for new messages.
Push Delivery - Consumer notified when a message is available.(by message broker)
FIFO Queues
In these queues, the oldest (or first) entry, sometimes called the “head” of the queue, is processed first.
Schedule or Delay Delivery
Many message queues support setting a specific delivery time for a message. If we need to have a common delay for all messages, we can set up a delay queue.
At-Least-Once Delivery
Message queues may store multiple copies of messages for redundancy and high availability, and resend messages in the event of communication failures or errors to ensure they are delivered at least once.
Exactly-Once Delivery
When duplicates can’t be tolerated, FIFO (first-in-first-out) message queues will make sure that each message is delivered exactly once (and only once) by filtering out duplicates automatically.
Dead-letter Queues
A dead-letter queue is a queue to which other queues can send messages that can’t be processed successfully. This makes it easy to set them aside for further inspection without blocking the queue processing or spending CPU cycles on a message that might never be consumed successfully.
Ordering
Most message queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they’re sent and that a message is delivered at least once.
Poison-pill Messages
Poison pills are special messages that can be received, but not processed. They are a mechanism used in order to signal a consumer to end its work so it is no longer waiting for new inputs, and are similar to closing a socket in a client/server model.
Security
Message queues will authenticate applications that try to access the queue, this allows us to encrypt messages over the network as well as in the queue itself.
Task Queues
Tasks queues receive tasks and their related data, run them, then deliver their results. They can support scheduling and can be used to run computationally-intensive jobs in the background.
Message Queue vs Task Queue
Message Queue
- Purpose: A message queue is a communication method used for sending messages between distributed systems or components. It allows different parts of an application to communicate asynchronously.
- Use Cases: Commonly used for event-driven architectures, where one part of a system produces messages (events) that other parts consume. Examples include logging, notifications, or any scenario where components need to communicate without being tightly coupled.
- Examples: RabbitMQ, Apache Kafka, AWS SQS.
Task Queue
- Purpose: A task queue is a specialized type of message queue that focuses on distributing tasks (jobs) to workers for processing. It is designed to manage the execution of asynchronous tasks.
- Use Cases: Used in scenarios where background processing is needed, such as sending emails, processing images, or performing long-running computations. Task queues often include features for scheduling, retrying failed tasks, and tracking task states.
- Examples: Celery (which uses message brokers like RabbitMQ or Redis), RQ (Redis Queue).