Meet Redis – Masters, Slaves and Scaling Out

redis

In my previous posts I introduced Redis and attempted to show how it can work with advanced data structures , as well as persistence options. Another important Redis feature is master –slave asynchronous replication. Data from any Redis server can replicate to any number of slaves. A slave may be a master to another slave. This allows Redis to implement a single-rooted replication tree. Redis slaves can be configured to accept writes, permitting intentional and unintentional inconsistency between instances. Replication is useful for read (but not write) scalability or data redundancy.

redis_master_slave

How Redis replication works. According to Redis docs this is workflow description for Redis asynchronous replication:

  • If you set up a slave, upon connection it sends a SYNC command. It doesn’t matter if it’s the first time it has connected or if it’s a reconnection.
  • The master then starts background saving, and starts to buffer all new commands received that will modify the dataset. When the background saving is complete, the master transfers the database file to the slave, which saves it on disk, and then loads it into memory. The master will then send to the slave all buffered commands. This is done as a stream of commands and is in the same format of the Redis protocol itself.
  • Slaves are able to automatically reconnect when the master <-> slave link goes down for some reason. If the master receives multiple concurrent slave synchronization requests, it performs a single background save in order to serve all of them.
  • When a master and a slave reconnects after the link went down, a full resync is always performed. However, starting with Redis 2.8, a partial resynchronization is also possible.

So Redis master-slave replication can be useful in number of scenarios here:

  • Scaling performance by using the replicas for intensive read operations.
  • Data redundancy in multiple locations
  • Offloading data persistency costs in terms of expensive Disk IO (covered in last post) from the master by delegating it to the slaves

So, if replication is pretty useful as far as read-only scale out – how do I configure it? To configure replication is trivial: just add the following line to the slave configuration file (slave instance redis.conf) :

slaveof 

Example:

slaveof 10.84.16.18 6379

More importantly you can use SLAVEOF command in Redis CLI to switch replication on the fly – http://redis.io/commands/slaveof.  If a Redis server is already acting as slave, the command SLAVEOF NO ONE will turn off the replication, turning the Redis server into a MASTER. In the proper form SLAVEOF hostname port will make the server a slave of another server listening at the specified hostname and port.

Since Redis 2.6, slaves support a read-only mode that is enabled by default. This behavior is controlled by the slave-read-only option in the redis.conf file, and can be enabled and disabled at runtime using CONFIG SET.

That’s great, but what if for HA purposes I need an automated failover here from master to slave? Enter Redis Sentinel – system designed to help managing Redis instances.  It does following:

  • Sentinel constantly checks if your master and slave instances are working as expected
  • Sentinel can notify the system administrator, or another computer program, via an API, that something is wrong with one of the monitored Redis instances.
  • If a master is not working as expected, Sentinel can start a failover process where a slave is promoted to master, the other additional slaves are reconfigured to use the new master, and the applications using the Redis server informed about the new address to use when connecting.
  • Sentinel acts as a source of authority for clients service discovery: clients connect to Sentinels in order to ask for the address of the current Redis master responsible for a given service. If a failover occurs, Sentinels will report the new address.

For more on Redis Sentinel see – http://redis.io/topics/sentinel. Unfortunately MSOpenTech port of Redis on Windows doesn’t support this feature so I couldn’t easily test it here, hope that in future blog entry testing Redis on Linux flavor I can show you Sentinel configuration and failover.

However, even through above are great features, there is one item that is missing here that for example was present in AppFabric Cache  – distributed cluster capable of linear scale out for write traffic. Yes, theoretically I can have multiple masters in Redis as well, however you would have to build some sort of sharding mechanism as multiple folks did in Silicon Valley (Instagram and Facebook I believe done so) to scale out. Fortunately, there is a new Redis Cluster Project. Redis Cluster provides a way to run a Redis installation where data is automatically sharded across multiple Redis nodes.

Commands dealing with multiple keys are not supported by the cluster, because this would require moving data between Redis nodes, making Redis Cluster not able to provide Redis-alike performances and predictable behavior under load. Redis Cluster also provides some degree of availability during partitions, that is in practical terms the ability to continue the operations when some nodes fail or are not able to communicate. So here is what you get with Redis Cluster:

  • The ability to automatically split your dataset among multiple nodes (true scale out)
  • The ability to continue operations when a subset of the nodes are experiencing failures or are unable to communicate with the rest of the cluster.

Every Redis Cluster node requires two TCP connections open. The normal Redis TCP port used to serve clients, for example 6379, plus the port obtained by adding 10000 to the data port, so 16379 in the example.This second high port is used for the Cluster bus, that is a node-to-node communication channel using a binary protocol. The Cluster bus is used by nodes for failure detection, configuration update, failover authorization and so forth. Clients should never try to communicate with the cluster bus port, but always with the normal Redis command port, however make sure you open both ports in your firewall, otherwise Redis cluster nodes will be not able to communicate.

To create a cluster, the first thing we need is to have a few empty Redis instances running in cluster mode. This basically means that clusters are not created using normal Redis instances, but a special mode needs to be configured so that the Redis instance will enable the Cluster specific features and commands. Therefore we will add following to configuration (redis.conf):

port 6379
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

As you can see what enables the cluster mode is simply the cluster-enabled directive. Every instance also contains the path of a file where the configuration for this node is stored, that by default is nodes.conf. This file is never touched by humans, it is simply generated at startup by the Redis Cluster instances, and updated every time it is needed.

Note that the minimal cluster that works as expected requires to contain at least three master nodes.

When instances are setup and running cluster node, next you need to create a cluster using Redis Cluster command line utility – redis-trib. The redis-trib utility is in the src directory of the Redis source code distribution. Example of use would be something like:

./redis-trib.rb create host1.domain.com:6379 host2.domain.com:6379

As Redis Cluster is still work in progress check out Redis Cluster Spec – http://redis.io/topics/cluster-spec and doc pages – http://redis.io/topics/cluster-tutorial . For internals and details on Redis Cluster also see this presentation from Redis – http://redis.io/presentation/Redis_Cluster.pdf

Hope this helps.

Advertisements

3 thoughts on “Meet Redis – Masters, Slaves and Scaling Out

  1. Pingback: Meet Redis- Connection Limits, Benchmarking And Partitioning | A posteriori
  2. Pingback: My Great Guardian – Watching Redis With Sentinel | A posteriori

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s