My Great Guardian – Watching Redis With Sentinel

 

redis

 

Redis Sentinel provides high availability for Redis. If you ever ran SQL Server mirroring or Oracle Golden Gate the concept should be somewhat familiar to you. To start you need to have Redis replication configured with master and N number slaves. From there, you have Sentinel daemons running, be it on your application servers or on the servers Redis is running on. These keep track of the master’s health.

Redis Sentinel provides high availability for Redis. If you ever ran SQL Server mirroring or Oracle Golden Gate the concept should be somewhat familiar to you. To start you need to have Redis replication configured with master and N number slaves. From there, you have Sentinel daemons running, be it on your application servers or on the servers Redis is running on. These keep track of the master’s health.

redis_sent

 

How does the failover work? Sentinel actually failover by rewriting configuration (conf) files for Redis instances that are running, I already mentioned SLAVEOF command before – https://gennadny.wordpress.com/2015/01/06/meet-redis-masters-slaves-and-scaling-out/, so by rewriting this command failover is achieved

Say we have a master “A” replicating to slaves “B” and “C”. We have three Sentinels (s1, s2, s3) running on our application servers, which write to Redis. At this point “A”, our current master, goes offline. Our sentinels all see “A” as offline, and send SDOWN messages to each other. Then they all agree that “A” is down, so “A” is set to be in ODOWN status. From here, an election happens to see who is most ahead, and in this case “B” is chosen as the new master.

The config file for “B” is set so that it is no longer the slave of anyone. Meanwhile, the config file for “C” is rewritten so that it is no longer the slave of “A” but rather “B.” From here, everything continues on as normal. Should “A” come back online, the Sentinels will recognize this, and rewrite the configuration file for “A” to be the slave of “B,” since “B” is the current master.

The current version of Sentinel is called Sentinel 2. It is a rewrite of the initial Sentinel implementation using stronger and simpler to predict algorithms (that are explained in this documentation).

A stable release of Redis Sentinel is shipped since Redis 2.8. Redis Sentinel version 1, shipped with Redis 2.6, is deprecated and should not be used.

When configuring Sentinel you need to take time and decide where you want to run Sentinel processes. Many folks recommend running those on your application servers. Presumably if you’re setting this up, you’re concerned about write availability to your master. As such, Sentinels provide insight to whether or not your application server can talk to the master. However a lot of folks decide to run Sentinel processes in their Redis instance servers amd that makes sense as well.

If you are using the redis-sentinel executable (or if you have a symbolic link with that name to the redis-server executable) you can run Sentinel with the following command line:

redis-sentinel /path/to/sentinel.conf

Otherwise you can use directly the redis-server executable starting it in Sentinel mode:

redis-server /path/to/sentinel.conf --sentinel

You have to use configuration file when running Sentinel (sentinel.conf) which is separate from Redis configuration file (redis.conf) and this file this file will be used by the system in order to save the current state that will be reloaded in case of restarts. Sentinel will simply refuse to start if no configuration file is given or if the configuration file path is not writable.

By default , Sentinel listens on TCP port 26379, so for Sentinels to work, port 26379 of your servers must be open to receive connections from the IP addresses of the other Sentinel instances. Otherwise Sentinels can’t talk and can’t agree about what to do, so failover will never be performed.

Redis-Sentinel

 

Some important items to remember on Sentinel

1. You need at least three Sentinel instances for a robust deployment.

2. As per Redis docs, three Sentinel instances should be placed into computers or virtual machines that are believed to fail in an independent way. So for example different physical servers or Virtual Machines executed on different availability zones or application fault domains

3. Sentinel + Redis distributed system does not guarantee that acknowledged writes are retained during failures, since Redis uses asynchronous replication. However there are ways to deploy Sentinel that make the window to lose writes limited to certain moments, while there are other less secure ways to deploy it.

4. You need Sentinel support in your clients. Popular client libraries have Sentinel support, but not all.

5. Test your setup so you know it works. Otherwise you cannot be sure in its performance

Basically. Initial setup expects all nodes running as a master with replication on, with manual set slaveof ip port in redis-cli on futire redis slaves. Then run sentinel and it does the rest.

Minimal redis.conf configuration file looks like this

daemonize yes
pidfile /usr/local/var/run/redis-master.pid
port 6379
bind 10.0.0.1
timeout 0
loglevel notice
logfile /opt/redis/redis.log
databases 1
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename master.rdb
 
dir /usr/local/var/db/redis/
slave-serve-stale-data yes
slave-read-only no
slave-priority 100
maxclients 2048
maxmemory 256mb
 
# act as binary log with transactions
appendonly yes
 
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
 
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

Minimal sentinel.conf configuration file looks like this

port 17700
daemonize yes
logfile "/opt/redis/sentinel.log"
 
sentinel monitor master 10.0.0.1 6379 2
sentinel down-after-milliseconds master 4000
sentinel failover-timeout master 180000
sentinel parallel-syncs master 4

Start all of your redis nodes with redis config and choose master. Then run redis console and set all other nodes as a slave of given master, using command slaveof <ip address 6379>

Start all of your redis nodes with redis config and choose master. Then run redis console and set all other nodes as a slave of given master, using command slaveof <ip address 6379>. Then you can connect to your master and verify, if there are all of your slave nodes, connected and syncing – run info command in your master redis console. Output should show you something like this

role:master
connected_slaves:3
slave0:ip=10.0.0.2,port=6379,state=online,offset=17367254333,lag=1
slave1:ip=10.0.0.3,port=6379,state=online,offset=17367242971,lag=1
slave2:ip=10.0.0.4,port=6379,state=online,offset=17367222381,lag=1

To test, if your sentinel works, just shutdown your redis master and watch sentinel log. You should see something like this

[17240] 04 Dec 07:56:16.289 # +sdown master master 10.24.37.144 6379
[17240] 04 Dec 07:56:16.551 # +new-epoch 1386165365
[17240] 04 Dec 07:56:16.551 # +vote-for-leader 185301a20bdfdf1d5316f95bae0fe1eb544edc58 1386165365
[17240] 04 Dec 07:56:17.442 # +odown master master 10.0.0.1 6379 #quorum 4/2
[17240] 04 Dec 07:56:18.489 # +switch-master master 10.0.0.1 6379 10.0.0.2 6379
[17240] 04 Dec 07:56:18.489 * +slave slave 10.0.0.3:6379 10.0.0.3 6379 @ master 10.0.0.2 6379
[17240] 04 Dec 07:56:18.490 * +slave slave 10.0.0.4:6379 10.0.0.4 6379 @ master 10.0.0.2 6379
[17240] 04 Dec 07:56:28.680 * +convert-to-slave slave 10.0.0.1:6379 10.0.0.1 6379 @ master 10.0.0.2 6379

What is also important to note that latest builds on MSOpenStack Redis for Windows have implemented Sentinel as well. As per http://grokbase.com/t/gg/redis-db/147ezmad89/installing-redis-sentinel-as-windows-service , You could use the following command line to install a sentinel
instance as a service:

redis-server --service-install --service-name Sentinel1
sentinel.1.conf --sentinel*

In this case the arguments passed to the service instance will be “*sentinel.1.conf
–sentinel*”.

Make sure of following

1. The configuration file must be the last parameter of the command line. If another parameter was last, such as –service-name, it would run fine when invoked the command line but would consistently fail went started as a service.

2. Since the service installs a Network Service by default, ensure that it has access to the directory where the log file will be written.

For more on Sentinel see official Redis docs – http://redis.io/topics/sentinel, https://discuss.pivotal.io/hc/en-us/articles/205309388-How-to-setup-HAProxy-and-Redis-Sentinel-for-automatic-failover-between-Redis-Master-and-Slave-servers, http://opentodo.net/2014/05/setup-redis-failover-with-redis-sentinel/, http://tech.3scale.net/2014/06/18/redis-sentinel-failover-no-downtime-the-hard-way/ ,https://seanmcgary.com/posts/how-to-build-a-fault-tolerant-redis-cluster-with-sentinel