Redis Sentinel provides high availability for Redis. If you ever ran SQL Server mirroring or Oracle Golden Gate the concept should be somewhat familiar to you. To start you need to have Redis replication configured with master and N number slaves. From there, you have Sentinel daemons running, be it on your application servers or on the servers Redis is running on. These keep track of the master’s health.
Redis Sentinel provides high availability for Redis. If you ever ran SQL Server mirroring or Oracle Golden Gate the concept should be somewhat familiar to you. To start you need to have Redis replication configured with master and N number slaves. From there, you have Sentinel daemons running, be it on your application servers or on the servers Redis is running on. These keep track of the master’s health.
How does the failover work? Sentinel actually failover by rewriting configuration (conf) files for Redis instances that are running, I already mentioned SLAVEOF command before – https://gennadny.wordpress.com/2015/01/06/meet-redis-masters-slaves-and-scaling-out/, so by rewriting this command failover is achieved
Say we have a master “A” replicating to slaves “B” and “C”. We have three Sentinels (s1, s2, s3) running on our application servers, which write to Redis. At this point “A”, our current master, goes offline. Our sentinels all see “A” as offline, and send SDOWN messages to each other. Then they all agree that “A” is down, so “A” is set to be in ODOWN status. From here, an election happens to see who is most ahead, and in this case “B” is chosen as the new master.
The config file for “B” is set so that it is no longer the slave of anyone. Meanwhile, the config file for “C” is rewritten so that it is no longer the slave of “A” but rather “B.” From here, everything continues on as normal. Should “A” come back online, the Sentinels will recognize this, and rewrite the configuration file for “A” to be the slave of “B,” since “B” is the current master.
The current version of Sentinel is called Sentinel 2. It is a rewrite of the initial Sentinel implementation using stronger and simpler to predict algorithms (that are explained in this documentation).
A stable release of Redis Sentinel is shipped since Redis 2.8. Redis Sentinel version 1, shipped with Redis 2.6, is deprecated and should not be used.
When configuring Sentinel you need to take time and decide where you want to run Sentinel processes. Many folks recommend running those on your application servers. Presumably if you’re setting this up, you’re concerned about write availability to your master. As such, Sentinels provide insight to whether or not your application server can talk to the master. However a lot of folks decide to run Sentinel processes in their Redis instance servers amd that makes sense as well.
If you are using the redis-sentinel executable (or if you have a symbolic link with that name to the redis-server executable) you can run Sentinel with the following command line:
redis-sentinel /path/to/sentinel.conf
Otherwise you can use directly the redis-server executable starting it in Sentinel mode:
redis-server /path/to/sentinel.conf --sentinel
You have to use configuration file when running Sentinel (sentinel.conf) which is separate from Redis configuration file (redis.conf) and this file this file will be used by the system in order to save the current state that will be reloaded in case of restarts. Sentinel will simply refuse to start if no configuration file is given or if the configuration file path is not writable.
By default , Sentinel listens on TCP port 26379, so for Sentinels to work, port 26379 of your servers must be open to receive connections from the IP addresses of the other Sentinel instances. Otherwise Sentinels can’t talk and can’t agree about what to do, so failover will never be performed.
Some important items to remember on Sentinel
1. You need at least three Sentinel instances for a robust deployment.
2. As per Redis docs, three Sentinel instances should be placed into computers or virtual machines that are believed to fail in an independent way. So for example different physical servers or Virtual Machines executed on different availability zones or application fault domains
3. Sentinel + Redis distributed system does not guarantee that acknowledged writes are retained during failures, since Redis uses asynchronous replication. However there are ways to deploy Sentinel that make the window to lose writes limited to certain moments, while there are other less secure ways to deploy it.
4. You need Sentinel support in your clients. Popular client libraries have Sentinel support, but not all.
5. Test your setup so you know it works. Otherwise you cannot be sure in its performance
Basically. Initial setup expects all nodes running as a master with replication on, with manual set slaveof ip port in redis-cli on futire redis slaves. Then run sentinel and it does the rest.
Minimal redis.conf configuration file looks like this
daemonize yes pidfile /usr/local/var/run/redis-master.pid port 6379 bind 10.0.0.1 timeout 0 loglevel notice logfile /opt/redis/redis.log databases 1 save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename master.rdb dir /usr/local/var/db/redis/ slave-serve-stale-data yes slave-read-only no slave-priority 100 maxclients 2048 maxmemory 256mb # act as binary log with transactions appendonly yes appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 128 hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-entries 512 list-max-ziplist-value 64 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60
Minimal sentinel.conf configuration file looks like this
port 17700 daemonize yes logfile "/opt/redis/sentinel.log" sentinel monitor master 10.0.0.1 6379 2 sentinel down-after-milliseconds master 4000 sentinel failover-timeout master 180000 sentinel parallel-syncs master 4
Start all of your redis nodes with redis config and choose master. Then run redis console and set all other nodes as a slave of given master, using command slaveof <ip address 6379>
Start all of your redis nodes with redis config and choose master. Then run redis console and set all other nodes as a slave of given master, using command slaveof <ip address 6379>. Then you can connect to your master and verify, if there are all of your slave nodes, connected and syncing – run info command in your master redis console. Output should show you something like this
role:master connected_slaves:3 slave0:ip=10.0.0.2,port=6379,state=online,offset=17367254333,lag=1 slave1:ip=10.0.0.3,port=6379,state=online,offset=17367242971,lag=1 slave2:ip=10.0.0.4,port=6379,state=online,offset=17367222381,lag=1
To test, if your sentinel works, just shutdown your redis master and watch sentinel log. You should see something like this
[17240] 04 Dec 07:56:16.289 # +sdown master master 10.24.37.144 6379 [17240] 04 Dec 07:56:16.551 # +new-epoch 1386165365 [17240] 04 Dec 07:56:16.551 # +vote-for-leader 185301a20bdfdf1d5316f95bae0fe1eb544edc58 1386165365 [17240] 04 Dec 07:56:17.442 # +odown master master 10.0.0.1 6379 #quorum 4/2 [17240] 04 Dec 07:56:18.489 # +switch-master master 10.0.0.1 6379 10.0.0.2 6379 [17240] 04 Dec 07:56:18.489 * +slave slave 10.0.0.3:6379 10.0.0.3 6379 @ master 10.0.0.2 6379 [17240] 04 Dec 07:56:18.490 * +slave slave 10.0.0.4:6379 10.0.0.4 6379 @ master 10.0.0.2 6379 [17240] 04 Dec 07:56:28.680 * +convert-to-slave slave 10.0.0.1:6379 10.0.0.1 6379 @ master 10.0.0.2 6379
What is also important to note that latest builds on MSOpenStack Redis for Windows have implemented Sentinel as well. As per http://grokbase.com/t/gg/redis-db/147ezmad89/installing-redis-sentinel-as-windows-service , You could use the following command line to install a sentinel
instance as a service:
redis-server --service-install --service-name Sentinel1 sentinel.1.conf --sentinel*
In this case the arguments passed to the service instance will be “*sentinel.1.conf
–sentinel*”.
Make sure of following
1. The configuration file must be the last parameter of the command line. If another parameter was last, such as –service-name, it would run fine when invoked the command line but would consistently fail went started as a service.
2. Since the service installs a Network Service by default, ensure that it has access to the directory where the log file will be written.
For more on Sentinel see official Redis docs – http://redis.io/topics/sentinel, https://discuss.pivotal.io/hc/en-us/articles/205309388-How-to-setup-HAProxy-and-Redis-Sentinel-for-automatic-failover-between-Redis-Master-and-Slave-servers, http://opentodo.net/2014/05/setup-redis-failover-with-redis-sentinel/, http://tech.3scale.net/2014/06/18/redis-sentinel-failover-no-downtime-the-hard-way/ ,https://seanmcgary.com/posts/how-to-build-a-fault-tolerant-redis-cluster-with-sentinel
A posteriori 