Nimendra's Notes 🪴

Redis Sentinal

4 min read

database/redis distributedsystem

Introduction

3.3 Understanding Sentinels
🌟 High availability with Redis Sentinel | Docs

Redis sentinel provides high availability for Redis with a system that automatically detects master node failure and replaces it with a slave node.

Redis Sentinels: Overview and Key Concepts

Redis Sentinel is a system designed to manage Redis instances, providing high availability and monitoring for Redis deployments. It is responsible for performing failover operations, monitoring the health of Redis servers, and notifying clients of changes in master-slave topology.

Key Components:

  1. Redis Master:
    • The primary Redis server that handles read and write operations.
    • Sentinel monitors the health of the master.
  2. Redis Replicas:
    • Secondary Redis instances that replicate the master.
    • They serve read-only queries and can be promoted to master in case of failure.
  3. Redis Sentinel:
    • A separate Redis process that monitors Redis servers (both master and replicas) and performs automatic failover when the master goes down.
    • Sentinels communicate with each other to agree on failover decisions.

Core Functions

  1. Monitoring:
    • Sentinels monitor the Redis master and replicas to check their health.
    • They also monitor the network and can detect when a Redis server is not reachable.
  2. Failover:
    • If the master Redis server becomes unavailable, Sentinels work together to elect a new master from the available replicas.
    • Once a replica is promoted to master, Sentinels notify all other Redis nodes to adjust their roles.
  3. Configuration Provider:
    • Sentinels provide clients with the current master’s address.
    • Clients can query a Sentinel to get the current master’s location, especially after a failover.
  4. Notifications:
    • Sentinels can notify clients and administrators about changes in the Redis cluster, such as the promotion of a replica to master or a server becoming unreachable.

Sentinel Configuration

  • Sentinel Monitor: Sentinels need to be configured to monitor the Redis master and replicas. This is done using the sentinel monitor directive in the Sentinel configuration file (sentinel.conf), like:
sentinel monitor <master-name> <master-ip> <master-port> <quorum>

Example:

sentinel monitor mymaster 172.23.0.10 6379 2

This tells the Sentinel to monitor the Redis instance named mymaster running on IP 172.23.0.10 and port 6379, requiring a quorum of 2 Sentinels for failover.

  • Sentinel Auth-Password: If Redis instances require authentication, you can specify the password for each Redis server in the Sentinel configuration:
sentinel auth-pass mymaster <password>
  • Failover: The failover process is triggered when the master goes down, and the Sentinels collectively choose a replica to promote to master.

Redis Sentinel Setup:

  1. Sentinel Cluster:

    • Typically, you need at least 3 Sentinel instances to form a quorum and ensure proper failover decisions.
    • The Sentinels should be configured in separate processes or containers to avoid a single point of failure.

  2. Failover Process:

    • When a master node fails, Sentinels check the health of the replicas.
    • After a quorum is reached, a replica is promoted to master.
    • Other replicas and clients are updated with the new master information.

  3. Client Communication:

    • Clients can be configured to query Sentinels for the current master node’s IP and port. This allows clients to automatically adjust if the master changes.

Example Sentinel Command:

To start a Sentinel instance, you can use the following command (assuming you have the sentinel.conf file configured):

redis-server /path/to/sentinel.conf --sentinel

Troubleshooting Tips:

  1. Sentinel Logs:

    • Check Sentinel logs to see if any errors or issues with failover or monitoring are reported.
    • You can check logs by running docker logs redis-sentinel-<number>.

  2. Ensure Network Connectivity:

    • Sentinels need to communicate with the Redis master and replicas. Make sure there are no network issues preventing this.

  3. Check Quorum:

    • If there is no quorum (i.e., not enough Sentinels), failover won’t happen. Ensure that the quorum size is set correctly.

Benefits of Using Redis Sentinel:

  • High Availability: Sentinel provides automated failover, ensuring that your Redis deployment is always available even if the master node goes down.
  • Automatic Failover: When the master goes down, Sentinel automatically promotes a replica to be the new master without manual intervention.
  • Monitoring: Redis Sentinel continuously monitors your Redis nodes and alerts you to any issues.
  • Minimal Downtime: Clients experience minimal downtime during failover as they can be reconfigured to the new master by querying the Sentinels.

Conclusion:

Redis Sentinel is an essential component for ensuring high availability and fault tolerance in a Redis-based system. By using Sentinel, you can automatically handle failovers, monitor Redis instances, and ensure that your Redis infrastructure remains robust and responsive.

Redis Insight Demo

Backlinks

Graph View