Meet Redis- Connection Limits, Benchmarking And Partitioning

redis

In my previous posts on Redis  I went through basic tutorial for MSOpenTech Redis fork, master\slave setup, configuration, Azure PaaS version and finally monitoring with INFO command. In this post I want to touch on some questions that I ran into while working with Redis at some scale.

Redis 10000 concurrent client limit.

In Redis 2.6 and above and so is in MSOpenTech Redis fork there is a default 10000 client limit set in configuration file (.conf) .Here is my setting on MSOpenTech redis.windows.conf:

# Set the max number of connected clients at the same time. By default
# this limit is set to 10000 clients, however if the Redis server is not
# able to configure the process file limit to allow for the specified limit
# the max number of allowed clients is set to the current file limit
# minus 32 (as Redis reserves a few file descriptors for internal uses).
#
# Once the limit is reached Redis will close all the new connections sending
# an error 'max number of clients reached'.
#
 maxclients 10000

However Redis checks with the kernel what is the maximum number of file descriptors that we are able to open (the soft limit is checked), if the limit is smaller than the maximum number of clients we want to handle, plus 32 (that is the number of file descriptors Redis reserves for internal uses), then the number of maximum clients is modified by Redis to match the amount of clients we are really able to handle under the current operating system limit.

Can I edit that limit higher? Yes, but as above both max number of file descriptors and maxmemory configuration setting then become throttling factors.

Interesting that looking at Redis Azure PaaS , which is based I believe on MSOpenStack fork, I see that 10000 default is present there as well – https://msdn.microsoft.com/en-us/library/azure/dn793612.aspx. Moreover appears that it cannot be changed as per statement on that page – “

The settings in this section cannot be changed using the StackExchange.Redis.IServer.ConfigSet method. If this method is called with one of the commands in this section, an exception similar to the following is thrown:StackExchange.Redis.RedisServerException: ERR unknown command 'CONFIG'.

Any values that are configurable, such as max-memory-policy, are configurable through the portal.”

The Redis client command allows to inspect the state of every connected client, to kill a specific client, to set names to connections. It is a very powerful debugging tool if you use Redis at scale.. Example:

Client List

image

Please timeout your clients for typical activity. By default recent versions of Redis don’t close the connection with the client if the client is idle for many seconds: the connection will remain open forever.
However if you don’t like this behavior, you can configure a timeout, so that if the client is idle for more than the specified number of seconds, the client connection will be closed.
You can configure this limit via configuration in redis.conf or simply using CONFIG SET timeout .

For more see – http://grokbase.com/t/gg/redis-db/127cd55pgv/redis-connection-limit and docs at – http://redis.io/topics/clients

Benchmarking Redis with redis-benchmark utility.

Redis includes the redis-benchmark utility that simulates running commands done by N clients at the same time sending M total queries (it is similar to the Apache’s ab utility). MsOpenTech retains utility and here I will launch it against my local Redis on Windows using n parameter for 100,000 requests.

image

I piped the output into log and here is what I get, I guess I am doing great with huge majority of test requests running under or at 1 ms:

PING_INLINE: -1.#J
PING_INLINE: 132516.80
PING_INLINE: 136782.78
PING_INLINE: 135029.77
====== PING_INLINE ======
  100000 requests completed in 0.74 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.39% <= 1 milliseconds
100.00% <= 1 milliseconds
135135.14 requests per second

PING_BULK: 142731.48
PING_BULK: 143487.34
====== PING_BULK ======
  100000 requests completed in 0.71 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.57% <= 1 milliseconds
100.00% <= 1 milliseconds
140056.03 requests per second

SET: -1.#J
SET: 130784.55
SET: 128515.09
SET: 129911.53
====== SET ======
  100000 requests completed in 0.77 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.46% <= 1 milliseconds
100.00% <= 1 milliseconds
129870.13 requests per second

GET: 130646.16
GET: 133253.94
GET: 124021.30
====== GET ======
  100000 requests completed in 0.81 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.43% <= 1 milliseconds
100.00% <= 1 milliseconds
123152.71 requests per second

INCR: 127509.44
INCR: 130615.16
INCR: 129090.76
====== INCR ======
  100000 requests completed in 0.77 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.47% <= 1 milliseconds
100.00% <= 1 milliseconds
129533.68 requests per second

LPUSH: 118755.10
LPUSH: 126949.84
LPUSH: 129224.04
====== LPUSH ======
  100000 requests completed in 0.77 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.40% <= 1 milliseconds
100.00% <= 1 milliseconds
129701.68 requests per second

LPOP: -1.#J
LPOP: 115243.90
LPOP: 123830.65
LPOP: 123105.90
====== LPOP ======
  100000 requests completed in 0.81 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.37% <= 1 milliseconds
100.00% <= 1 milliseconds
123762.38 requests per second

SADD: 130807.69
SADD: 127049.38
SADD: 127387.79
====== SADD ======
  100000 requests completed in 0.78 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.52% <= 1 milliseconds
100.00% <= 1 milliseconds
128205.13 requests per second

SPOP: 140923.91
SPOP: 127520.47
SPOP: 129139.97
====== SPOP ======
  100000 requests completed in 0.81 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.42% <= 1 milliseconds
100.00% <= 1 milliseconds
123915.74 requests per second

LPUSH (needed to benchmark LRANGE): 94333.34
LPUSH (needed to benchmark LRANGE): 126600.80
LPUSH (needed to benchmark LRANGE): 126906.74
LPUSH (needed to benchmark LRANGE): 120746.34
====== LPUSH (needed to benchmark LRANGE) ======
  100000 requests completed in 0.82 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.30% <= 1 milliseconds
99.95% <= 2 milliseconds
100.00% <= 2 milliseconds
121212.13 requests per second

LRANGE_100 (first 100 elements): 41164.38
LRANGE_100 (first 100 elements): 42671.72
LRANGE_100 (first 100 elements): 38488.65
LRANGE_100 (first 100 elements): 36693.74
LRANGE_100 (first 100 elements): 37774.33
LRANGE_100 (first 100 elements): 38873.14
LRANGE_100 (first 100 elements): 39933.21
LRANGE_100 (first 100 elements): 39962.87
LRANGE_100 (first 100 elements): 40345.52
LRANGE_100 (first 100 elements): 40035.25
====== LRANGE_100 (first 100 elements) ======
  100000 requests completed in 2.50 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

88.73% <= 1 milliseconds
99.77% <= 2 milliseconds
99.85% <= 13 milliseconds
99.90% <= 14 milliseconds
99.95% <= 103 milliseconds
99.95% <= 104 milliseconds
100.00% <= 104 milliseconds
39984.01 requests per second

LRANGE_300 (first 300 elements): 17172.13
LRANGE_300 (first 300 elements): 18250.00
LRANGE_300 (first 300 elements): 18229.90
LRANGE_300 (first 300 elements): 18384.88
LRANGE_300 (first 300 elements): 18076.72
LRANGE_300 (first 300 elements): 17673.47
LRANGE_300 (first 300 elements): 17755.71
LRANGE_300 (first 300 elements): 17886.75
LRANGE_300 (first 300 elements): 17716.30
LRANGE_300 (first 300 elements): 17873.05
LRANGE_300 (first 300 elements): 17899.70
LRANGE_300 (first 300 elements): 17905.61
LRANGE_300 (first 300 elements): 17972.45
LRANGE_300 (first 300 elements): 18034.70
LRANGE_300 (first 300 elements): 18103.56
LRANGE_300 (first 300 elements): 18112.83
LRANGE_300 (first 300 elements): 18114.05
LRANGE_300 (first 300 elements): 18169.71
LRANGE_300 (first 300 elements): 18151.45
LRANGE_300 (first 300 elements): 18005.95
LRANGE_300 (first 300 elements): 18032.99
LRANGE_300 (first 300 elements): 18086.19
====== LRANGE_300 (first 300 elements) ======
  100000 requests completed in 5.52 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

0.27% <= 1 milliseconds
93.96% <= 2 milliseconds
99.91% <= 3 milliseconds
99.95% <= 11 milliseconds
99.97% <= 12 milliseconds
99.98% <= 13 milliseconds
100.00% <= 13 milliseconds
18115.94 requests per second

LRANGE_500 (first 450 elements): 9384.62
LRANGE_500 (first 450 elements): 12189.87
LRANGE_500 (first 450 elements): 11904.25

Docs on this utility are available here – http://redis.io/topics/benchmarks

Partitioning.,

I briefly touched on difficulties scaling Redis out in my previous post, and until promised redis cluster is available in earnest (and how soon will it be available on Windows and Azure? ), best way to scale Redis out remains partitioning aka sharding. Partitioning is the process of splitting your data into multiple Redis instances, so that every instance will only contain a subset of your keys. .

Partitioning will allow for following:

  • Much larger databases\Redis stores, using the sum of the memory of many computers. Without partitioning you are limited to the amount of memory a single instance can support.
  • It allows scaling the computational power to multiple cores and multiple computers, and the network bandwidth to multiple computers and network adapters.

Twitter, Instagram and other heavy Redis users have implemented custom partitioning which allowed these companies to scale Redis to their needs. As I started thinking of how to do this couple of methods came to my mind:

  • Classic Range Partitioning. This is accomplished by mapping ranges of objects into specific Redis instances. For example, I could say users from ID 0 to ID 10000 will go into instance R0, while users form ID 10001 to ID 20000 will go into instance R1 and so forth.This system works and is actually used in practice, however, it has the disadvantage of requiring a table that maps ranges to instances. This table needs to be managed and a table is needed for every kind of object, so therefore range partitioning in Redis is often undesirable because it is much more inefficient than other alternative partitioning approaches.
  • Hash Partitioning . Lets say we take the key name and use a hash function (e.g., the crc32 hash function) to turn it into a number. For example, if the key is foobar, crc32(foobar) will output something like 93024922. Then we can use a modulo operation with this number in order to turn it into a number between 0 and 3, so that this number can be mapped to one of our Redis instances.  Although there are few client in Redis that implement consistent hashing out of the box unfortunately some of the most popular do not.

As you may know many features of Redis, such as operations and transactions involving multiple or intersecting keys will not work and adding or removing capacity from Redis will be tricky to say the least. hen partitioning is used, data handling is more complex, for instance you have to handle multiple RDB / AOF files, and to make a backup of your data you need to aggregate the persistence files from multiple instances and hosts.  Moreover what is upsetting with C# StackExchange client connecting to MSOpenTech Redis on Windows or Azure there isn’t anything already built in for you, so you will have to build your own.   Also partitioning may be ok for Redis that is used as a cache store, but for data store may be an issue.  For more see – http://redis.io/topics/partitioning. Some interesting examples are here – http://petrohi.me/post/6323289515/scaling-redis, Twitter proxy implementation – http://antirez.com/news/44

Hope this helps

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s