Spreading The Magic Dust -SQL Server Row Versioning With Read Committed Snapshot Isolation Level

Working with many customers on very intense database\application tuning engagements in the past many times I had to go back to my “bag of magic dust” to provide for significant SQL Server backed performance enhancements under load without recoding application or queries. One of such items in many cases  was Read Committed Snapshot Isolation Level feature introduced while ago in SQL Server 2005.

So as of SQL Server 2005 and later versions, SQL Server provides two physical implementations of the read committed isolation level defined by the SQL standard, “classic locking” read committed and read committed snapshot isolation (RCSI).

What is transaction isolation level? Transactions specify an isolation level that defines the degree to which one transaction must be isolated from resource or data modifications made by other transactions. Isolation levels are described in terms of which concurrency side-effects, such as dirty reads or phantom reads, are allowed.

Choosing a transaction isolation level does not affect the locks acquired to protect data modifications. A transaction always gets an exclusive lock on any data it modifies, and holds that lock until the transaction completes, regardless of the isolation level set for that transaction. For read operations, transaction isolation levels primarily define the level of protection from the effects of modifications made by other transactions. A lower isolation level increases the ability of many users to access data at the same time, but increases the number of concurrency effects (such as dirty reads or lost updates) users might encounter. Conversely, a higher isolation level reduces the types of concurrency effects that users may encounter, but requires more system resources and increases the chances that one transaction will block another. Choosing the appropriate isolation level depends on balancing the data integrity requirements of the application against the overhead of each isolation level.

Lets start with SQL Server default “classic locking” read committed isolation level. When SQL Server executes a statement at the read committed isolation level, it acquires short lived share locks on a row by row basis, although depending on operation, indexing, statistics and number of rows that need to be scanned lock can be escalated to page level or even table level. The duration of these share locks is just long enough to read and process data. .  The sole purpose of these locks is to ensure that the statement only reads and returns committed data, i.e. avoid uncommitted\”dirty” reads.  The locks work because updates always acquire an exclusive lock which blocks any readers trying to acquire a share lock.

This is well known and default behavior and most of the time it works just fine under load, however sometimes with heavy OLTP loads excessive read blocking is noted.

Enter – Read Committed Snapshot utilizing Row Versioning. When transactions running under row versioning-based isolation read data, the read operations do not acquire shared (S) locks on the data being read, and therefore do not block transactions that are modifying data. Also, the overhead of locking resources is minimized as the number of locks acquired is reduced. Read committed isolation using row versioning and snapshot isolation are designed to provide statement-level or transaction-level read consistencies of versioned data.

When Read Committed Snapshot is  on , logical copies (versions) are maintained for all data modifications performed in the database. Every time a row is modified by a specific transaction, the instance of the Database Engine stores a version of the previously committed image of the row in TempDB. Each version is marked with the transaction sequence number of the transaction that made the change. The versions of modified rows are chained using a link list. The newest row value is always stored in the current database and chained to the versioned rows stored in TempDB.  So basically there is a version store located in TempDB that allows read queries to avoid shared locks and be served last committed value.

That’s whole bunch of theory, but how does it work in practice?  Lets test it out really quick.  First I am going to create a little database on my “legacy” SQL 2008 R2 instance:

( NAME = N'IsolationTests', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL10_50.GENNADYK_LEGACY\MSSQL\DATA\IsolationTests.mdf' , SIZE = 50MB , FILEGROWTH = 4MB )
( NAME = N'IsolationTests_log', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL10_50.GENNADYK_LEGACY\MSSQL\DATA\IsolationTests_log.ldf' , SIZE = 5MB , FILEGROWTH = 10%)

Next I will create a little table and insert some data into it.

USE IsolationTests
CREATE TABLE IsolationTest
INSERT INTO IsolationTest (VALUE1,VALUE2,VALUE3) VALUES (10,11,12)
INSERT INTO IsolationTest (VALUE1,VALUE2,VALUE3) VALUES (13,14,15)

Note I am not enabling Snapshot and currently will start transactions under default Read Committed “classic locking” isolation level. Here comes my uncommitted transaction that will require locks on update:

UPDATE IsolationTest SET VALUE1 = 25  
--Simulate long running transaction with long wait
WAITFOR DELAY '00:00:30'  

Next I will generate read transaction that will be blocked here

SELECT * FROM dbo.IsolationTest

Finally I will see my read being blocked via DMV query like this:

  wt.session_id as waiting_session_id,
  db_name(tl.resource_database_id)as DatabaseName,
  (select substring (st.text,(er.statement_start_offset/2)+1,
    ((case er.statement_end_offset
      when -1 then datalength(st.text)
      else er.statement_end_offset
      end - er.statement_start_offset)/2)+1)
   from sys.dm_exec_requests as er
   cross apply sys.dm_exec_sql_text(er.sql_handle) as st
   where er.session_id=tl.request_session_id)
   as waiting_query_text,
wt.resource_description as blocking_resource_description,
case when wt.blocking_session_id>0 then
(select st2.text from sys.sysprocesses as sp
  cross apply sys.dm_exec_sql_text(sp.sql_handle) as st2
   where sp.spid=wt.blocking_session_id)
else null
as blocking_query_text
from sys.dm_os_waiting_tasks as wt join sys.dm_tran_locks as tl
on wt.resource_address =tl.lock_owner_address
where wt.wait_duration_ms>5000
and wt.session_id>50

And I can clearly see my read transaction blocked here:


Now lets setup snapshot on our database and change read isolation level to read committed snapshot from default read committed.

USE master

Next lets check what isolation level is on our test database.

Use master

SELECT name, is_read_committed_snapshot_on FROM sys.databases

Looking good:


Finally lets “rinse and repeat” previous exercise with long running transactional update and read operations. Here is result as far as blocking via same DMV query:


Magic? Well not so fast, there are serious consequences

For more see – http://blogs.technet.com/b/sql_server_isv/archive/2010/12/21/using-read-committed-snapshot-isolation-mini-lab.aspx, http://www.brentozar.com/archive/2013/01/implementing-snapshot-or-read-committed-snapshot-isolation-in-sql-server-a-guide/, https://www.sqlpassion.at/archive/2014/01/21/myths-and-misconceptions-about-transaction-isolation-levels/,

Forecast Cloudy – Developing Simple Java Servlet Application for Google App Engine


I realize that this post is pretty basic, however I think it may be useful to someone starting to develop Web applications on Google App Engine. In my previous post I shown you  how you can use NetBeans to develop for App Engine, this time however I will use Eclipse as with real Google plug-in it’s a lot more slick, despite my preference for NetBeans as Java IDE. In general I am a server side vs. UI guy so my servlet will be very basic, but that I don’t think is the point here as servlets in general are quite easy to do and make more more functional.

So you decided to build Web App on App Engine?  First you need to install Google Plug-In for Eclipse for AppEngine. Installation instructions are available here. In Eclipse Go To Help Menu –>Install Software and add proper link as per table in Google doc: If you are using latest Eclipse (Luna) full instructions are here – https://developers.google.com/eclipse/docs/install-eclipse-4.4

After you install Plug-In you will see new icon on your toolbar in Eclipse:


Next I will create a Dynamic Web Project in Eclipse to house my Servlet. Go To File –> New –>Dynamic Web Project. Resulting dialog looks like this:


After you complete this step, the project-creation wizard will create a simple servlet-based application featuring a Hello World-type servlet.  After you set that up you should see your project structure:


Now on to Java Servlets.  Java servlet is a Java programming language program that extends the capabilities of a server. Although servlets can respond to any types of requests, they most commonly implement applications hosted on Web servers. Such Web servlets are the Java counterpart to other dynamic Web content technologies such as PHP and ASP.NET.

servlet is a Java class that implements the Servlet interface. This interface has three methods that define the servlet’s life cycle:

public void init(ServletConfig config) throws ServletException
This method is called once when the servlet is loaded into the servlet engine, before the servlet is asked to process its first request.

public void service(ServletRequest request, ServletResponse response) throws ServletException, IOException
This method is called to process a request. It can be called zero, one or many times until the servlet is unloaded. Multiple threads (one per request) can execute this method in parallel so it must be thread safe.

public void destroy()
This method is called once just before the servlet is unloaded and taken out of service.

The init method has a ServletConfig attribute. The servlet can read its initialization arguments through the ServletConfig object. How the initialization arguments are set is servlet engine dependent but they are usually defined in a configuration file.

The doGet method has two interesting parameters: HttpServletRequest and HttpServletResponse. These two objects give you full access to all information about the request and let you control the output sent to the client as the response to the request.

My Servlet as stated is pretty basic:

package com.example.myproject;

import java.io.IOException;
import javax.servlet.http.*;

public class HelloAppEngineServletServlet extends HttpServlet {
	public void doGet(HttpServletRequest req, HttpServletResponse resp)
			throws IOException {
		resp.getWriter().println("Hello, AppEngine from GennadyK");

The servlet gets mapped under the URI /simpleservletapp in the web.xml, as you can see below:


The project-creation wizard also provides an index.html file that has a link to the new servlet.

Before I deploy this to App Engine I probably want to debug it locally. Lets find Debug button on toolbar

Hit dropdown arrow and you can setup local server for debugging. If your application uses App Engine but not GWT, the only indication that the App Engine development server is running will be output in the Console view. App Engine-only launches will not appear in the development mode view. The console output includes the URL of the server, which by default is http://localhost:8888/. You can change the port number via Eclipse’s launch configuration dialog by selecting your Web Application launch and editing the Port value on the Main tab.


Simple result here:


Now lets upload this “application” to App Engine. Since I already configured development server I used Google Engine WPT Deploy. Right click on the project and you can pick that option:


I can now watch my status in the status bar:


And read logged messages in Eclipse Console:

May 29, 2015 7:34:53 PM java.util.prefs.WindowsPreferences 
WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.
Reading application configuration data...
May 29, 2015 7:34:54 PM com.google.apphosting.utils.config.AppEngineWebXmlReader readAppEngineWebXml
INFO: Successfully processed C:/Users/gennadyk/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/helloAppEngineServlet\WEB-INF/appengine-web.xml
May 29, 2015 7:34:54 PM com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
INFO: Successfully processed C:/Users/gennadyk/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/helloAppEngineServlet\WEB-INF/web.xml

Beginning interaction for module default...
0% Created staging directory at: 'C:\Users\gennadyk\AppData\Local\Temp\appcfg8550091122421213739.tmp'
5% Scanning for jsp files.
20% Scanning files on local disk.
25% Initiating update.
28% Cloning 3 static files.
31% Cloning 13 application files.
40% Uploading 4 files.
52% Uploaded 1 files.
61% Uploaded 2 files.
68% Uploaded 3 files.
73% Uploaded 4 files.
77% Initializing precompilation...
80% Sending batch containing 4 file(s) totaling 5KB.
90% Deploying new version.
95% Closing update: new version is ready to start serving.
98% Uploading index definitions.

Update for module default completed successfully.
Cleaning up temporary files for module default

Now let me open my browser and go to App Engine Console at https://appengine.google.com/.


Something I learned along the way – At this time you cannot run Java 8 on GAE. My IT department is a stickler for latest Java installed on my work laptop. Therefore I am running Java 8 SDK and JRE. However when I first build this application in Java 8 and deployed to App Engine trying to run it I got http 500 result. Looking at the logs in dashboard I can see following error:

2015-05-29 16:49:22.442  
Uncaught exception from servlet
java.lang.UnsupportedClassVersionError: com/example/myproject/HelloAppEngineServletServlet : Unsupported major.minor version 52.0
	at com.google.appengine.runtime.Request.process-ee32bdc226f79da9(Request.java)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:817)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:375)
	at org.mortbay.util.Loader.loadClass(Loader.java:91)
	at org.mortbay.util.Loader.loadClass(Loader.java:71)
	at org.mortbay.jetty.servlet.Holder.doStart(Holder.java:73)
	at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:242)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:437)
	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:444)
	at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:230)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:308)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:300)
	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:441)
	at java.lang.Thread.run(Thread.java:745)

Did a bit of digging and here is what I found out.

I believe these are the current version numbers:

J2SE 7 = 51,  //Note this one

J2SE 6.0 = 50,

J2SE 5.0 = 49,

JDK 1.4 = 48,

JDK 1.3 = 47,

JDK 1.2 = 46,

JDK 1.1 = 45

51.0 appears to be Java 7, which would mean in my case I am using 8 as its seeing 52. So issue here is that I compiled on higher version JDK (52 = Java 8) and then execute it on lower JRE version (GAE uses java 7). Repointing PATH to Java 7 did the trick, moreover as it was suggested elsewhere on the web I just created a batch file for this purpose as

SET JAVA_HOME="C:\Program Files\Java\jdk1.7.0_55"
START eclipse.exe

For more information see – https://cloud.google.com/appengine/docs/java/tools/eclipse, https://developers.google.com/eclipse/docs/appengine_deploy, https://cloud.google.com/appengine/docs/java/tools/maven, https://cloud.google.com/appengine/docs/java/webtoolsplatform

Forecast Cloudy -Developing for Google App Engine in NetBeans IDE


Although I am fairly Microsoft centric and Azure centric in my interests I have decided to give a quick try to a competing alternative using language that I am familiar with – Java and IDE that I know fairly well for it – NetBeans. I wanted to try something quick and easy, but my first task was making sure that NetBeans actually can deploy code to AppEngine.

Good news there is a plug-in for Google AppEngine available from this Kenai Project – https://kenai.com/projects/nbappengine/pages/Home.

First things first. Navigate to developers.google.com  and sign up to enable your Google Account for use with Google App Engine.  Create a project, make sure you write down Project ID and name. Download and unzip the Google App Engine SDK to a folder on your hard drive. Obviously here I am using Java SDK.

Now, I will assume that you have NetBeans already installed. If not it can be installed from here – https://netbeans.org/ . It is my favorite IDE when it comes to non-Microsoft or Java IDEs at this time.

Next, you will need to install plug-in. Go to https://kenai.com/projects/nbappengine/pages/Home and follow these instructions:

  • In NetBeans click Tools –> Plugins
  • Go To Settings Tab
  • Click Add Button
  • Type “App Engine” (without the quotes) into the Name field
  • If using NetBeans 6.9 and above paste http://kenai.com/downloads/nbappengine/NetBeans69/updates.xml into the URL field
  • Click the OK button
  • Click on Available Plugins
  • Select all Google App Engine plugins
  • If you’re using NetBeans 6.9 you must also install the  Java Web Applications plugin for NetBeans.

After you are done you should see plugins in your installed tab like this:


To install the Google App Engine service in NetBeans, follow these instructions:

  • Start NetBeans
  • Click on the Services tab next to Projects and Files
  • Right-click on Servers and click Add
  • Select Google App Engine and Click Next
  • Select the location you unzipped the Google App Engine SDK and Click Next
  • Unless you have another service running on port 8080 and port 8765 leave the default port values
  • Finally Click Finish

After you are done you should see Google AppEngine in your Servers”


Next I created really easy small JSP file that I want to deploy. First thing lets take a look at appengine-web.xml settings file. As per Google docs –“ An App Engine Java app must have a file named appengine-web.xml in its WAR, in the directory WEB-INF/. This is an XML file whose root element is .” My config looks like this:



Once you are ready to deploy just go to project and right click and pick Deploy to Google App Engine option. It will prompt you for user name and password. That’s where I ran into first issue with error that my password and email don’t match detailed here – https://groups.google.com/forum/#!topic/opendatakit/5hpDkvrd_WQ. After working through this and another issue noted here with JSPs and AppEngine deploy – https://code.google.com/p/googleappengine/issues/detail?id=1226#makechanges I finally deployed and ran my little application generating traffic I can see on console:


Frankly the experience for me wasn’t as seamless as I would like (perhaps I am spoiled a bit by Visual Studio Azure SDK integration for Azure PaaS), I may have had an easier time with official Google plugin and Eclipse, but as stated I like NetBeans more.

For more see – http://rocky.developerblogs.com/tutorials/getting-started-google-app-engine-netbeans/, https://techtavern.wordpress.com/2014/05/13/google-app-engine-on-netbeans-8/, https://cloud.google.com/appengine/docs/java/gettingstarted/introduction

Well this was different and going “back to regularly scheduled programming”

Meet Redis- Connection Limits, Benchmarking And Partitioning


In my previous posts on Redis  I went through basic tutorial for MSOpenTech Redis fork, master\slave setup, configuration, Azure PaaS version and finally monitoring with INFO command. In this post I want to touch on some questions that I ran into while working with Redis at some scale.

Redis 10000 concurrent client limit.

In Redis 2.6 and above and so is in MSOpenTech Redis fork there is a default 10000 client limit set in configuration file (.conf) .Here is my setting on MSOpenTech redis.windows.conf:

# Set the max number of connected clients at the same time. By default
# this limit is set to 10000 clients, however if the Redis server is not
# able to configure the process file limit to allow for the specified limit
# the max number of allowed clients is set to the current file limit
# minus 32 (as Redis reserves a few file descriptors for internal uses).
# Once the limit is reached Redis will close all the new connections sending
# an error 'max number of clients reached'.
 maxclients 10000

However Redis checks with the kernel what is the maximum number of file descriptors that we are able to open (the soft limit is checked), if the limit is smaller than the maximum number of clients we want to handle, plus 32 (that is the number of file descriptors Redis reserves for internal uses), then the number of maximum clients is modified by Redis to match the amount of clients we are really able to handle under the current operating system limit.

Can I edit that limit higher? Yes, but as above both max number of file descriptors and maxmemory configuration setting then become throttling factors.

Interesting that looking at Redis Azure PaaS , which is based I believe on MSOpenStack fork, I see that 10000 default is present there as well – https://msdn.microsoft.com/en-us/library/azure/dn793612.aspx. Moreover appears that it cannot be changed as per statement on that page – “

The settings in this section cannot be changed using the StackExchange.Redis.IServer.ConfigSet method. If this method is called with one of the commands in this section, an exception similar to the following is thrown:StackExchange.Redis.RedisServerException: ERR unknown command 'CONFIG'.

Any values that are configurable, such as max-memory-policy, are configurable through the portal.”

The Redis client command allows to inspect the state of every connected client, to kill a specific client, to set names to connections. It is a very powerful debugging tool if you use Redis at scale.. Example:

Client List


Please timeout your clients for typical activity. By default recent versions of Redis don’t close the connection with the client if the client is idle for many seconds: the connection will remain open forever.
However if you don’t like this behavior, you can configure a timeout, so that if the client is idle for more than the specified number of seconds, the client connection will be closed.
You can configure this limit via configuration in redis.conf or simply using CONFIG SET timeout .

For more see – http://grokbase.com/t/gg/redis-db/127cd55pgv/redis-connection-limit and docs at – http://redis.io/topics/clients

Benchmarking Redis with redis-benchmark utility.

Redis includes the redis-benchmark utility that simulates running commands done by N clients at the same time sending M total queries (it is similar to the Apache’s ab utility). MsOpenTech retains utility and here I will launch it against my local Redis on Windows using n parameter for 100,000 requests.


I piped the output into log and here is what I get, I guess I am doing great with huge majority of test requests running under or at 1 ms:

PING_INLINE: 132516.80
PING_INLINE: 136782.78
PING_INLINE: 135029.77
====== PING_INLINE ======
  100000 requests completed in 0.74 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.39% <= 1 milliseconds
100.00% <= 1 milliseconds
135135.14 requests per second

PING_BULK: 142731.48
PING_BULK: 143487.34
====== PING_BULK ======
  100000 requests completed in 0.71 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.57% <= 1 milliseconds
100.00% <= 1 milliseconds
140056.03 requests per second

SET: -1.#J
SET: 130784.55
SET: 128515.09
SET: 129911.53
====== SET ======
  100000 requests completed in 0.77 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.46% <= 1 milliseconds
100.00% <= 1 milliseconds
129870.13 requests per second

GET: 130646.16
GET: 133253.94
GET: 124021.30
====== GET ======
  100000 requests completed in 0.81 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.43% <= 1 milliseconds
100.00% <= 1 milliseconds
123152.71 requests per second

INCR: 127509.44
INCR: 130615.16
INCR: 129090.76
====== INCR ======
  100000 requests completed in 0.77 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.47% <= 1 milliseconds
100.00% <= 1 milliseconds
129533.68 requests per second

LPUSH: 118755.10
LPUSH: 126949.84
LPUSH: 129224.04
====== LPUSH ======
  100000 requests completed in 0.77 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.40% <= 1 milliseconds
100.00% <= 1 milliseconds
129701.68 requests per second

LPOP: -1.#J
LPOP: 115243.90
LPOP: 123830.65
LPOP: 123105.90
====== LPOP ======
  100000 requests completed in 0.81 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.37% <= 1 milliseconds
100.00% <= 1 milliseconds
123762.38 requests per second

SADD: 130807.69
SADD: 127049.38
SADD: 127387.79
====== SADD ======
  100000 requests completed in 0.78 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.52% <= 1 milliseconds
100.00% <= 1 milliseconds
128205.13 requests per second

SPOP: 140923.91
SPOP: 127520.47
SPOP: 129139.97
====== SPOP ======
  100000 requests completed in 0.81 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.42% <= 1 milliseconds
100.00% <= 1 milliseconds
123915.74 requests per second

LPUSH (needed to benchmark LRANGE): 94333.34
LPUSH (needed to benchmark LRANGE): 126600.80
LPUSH (needed to benchmark LRANGE): 126906.74
LPUSH (needed to benchmark LRANGE): 120746.34
====== LPUSH (needed to benchmark LRANGE) ======
  100000 requests completed in 0.82 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.30% <= 1 milliseconds
99.95% <= 2 milliseconds
100.00% <= 2 milliseconds
121212.13 requests per second

LRANGE_100 (first 100 elements): 41164.38
LRANGE_100 (first 100 elements): 42671.72
LRANGE_100 (first 100 elements): 38488.65
LRANGE_100 (first 100 elements): 36693.74
LRANGE_100 (first 100 elements): 37774.33
LRANGE_100 (first 100 elements): 38873.14
LRANGE_100 (first 100 elements): 39933.21
LRANGE_100 (first 100 elements): 39962.87
LRANGE_100 (first 100 elements): 40345.52
LRANGE_100 (first 100 elements): 40035.25
====== LRANGE_100 (first 100 elements) ======
  100000 requests completed in 2.50 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

88.73% <= 1 milliseconds
99.77% <= 2 milliseconds
99.85% <= 13 milliseconds
99.90% <= 14 milliseconds
99.95% <= 103 milliseconds
99.95% <= 104 milliseconds
100.00% <= 104 milliseconds
39984.01 requests per second

LRANGE_300 (first 300 elements): 17172.13
LRANGE_300 (first 300 elements): 18250.00
LRANGE_300 (first 300 elements): 18229.90
LRANGE_300 (first 300 elements): 18384.88
LRANGE_300 (first 300 elements): 18076.72
LRANGE_300 (first 300 elements): 17673.47
LRANGE_300 (first 300 elements): 17755.71
LRANGE_300 (first 300 elements): 17886.75
LRANGE_300 (first 300 elements): 17716.30
LRANGE_300 (first 300 elements): 17873.05
LRANGE_300 (first 300 elements): 17899.70
LRANGE_300 (first 300 elements): 17905.61
LRANGE_300 (first 300 elements): 17972.45
LRANGE_300 (first 300 elements): 18034.70
LRANGE_300 (first 300 elements): 18103.56
LRANGE_300 (first 300 elements): 18112.83
LRANGE_300 (first 300 elements): 18114.05
LRANGE_300 (first 300 elements): 18169.71
LRANGE_300 (first 300 elements): 18151.45
LRANGE_300 (first 300 elements): 18005.95
LRANGE_300 (first 300 elements): 18032.99
LRANGE_300 (first 300 elements): 18086.19
====== LRANGE_300 (first 300 elements) ======
  100000 requests completed in 5.52 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

0.27% <= 1 milliseconds
93.96% <= 2 milliseconds
99.91% <= 3 milliseconds
99.95% <= 11 milliseconds
99.97% <= 12 milliseconds
99.98% <= 13 milliseconds
100.00% <= 13 milliseconds
18115.94 requests per second

LRANGE_500 (first 450 elements): 9384.62
LRANGE_500 (first 450 elements): 12189.87
LRANGE_500 (first 450 elements): 11904.25

Docs on this utility are available here – http://redis.io/topics/benchmarks


I briefly touched on difficulties scaling Redis out in my previous post, and until promised redis cluster is available in earnest (and how soon will it be available on Windows and Azure? ), best way to scale Redis out remains partitioning aka sharding. Partitioning is the process of splitting your data into multiple Redis instances, so that every instance will only contain a subset of your keys. .

Partitioning will allow for following:

  • Much larger databases\Redis stores, using the sum of the memory of many computers. Without partitioning you are limited to the amount of memory a single instance can support.
  • It allows scaling the computational power to multiple cores and multiple computers, and the network bandwidth to multiple computers and network adapters.

Twitter, Instagram and other heavy Redis users have implemented custom partitioning which allowed these companies to scale Redis to their needs. As I started thinking of how to do this couple of methods came to my mind:

  • Classic Range Partitioning. This is accomplished by mapping ranges of objects into specific Redis instances. For example, I could say users from ID 0 to ID 10000 will go into instance R0, while users form ID 10001 to ID 20000 will go into instance R1 and so forth.This system works and is actually used in practice, however, it has the disadvantage of requiring a table that maps ranges to instances. This table needs to be managed and a table is needed for every kind of object, so therefore range partitioning in Redis is often undesirable because it is much more inefficient than other alternative partitioning approaches.
  • Hash Partitioning . Lets say we take the key name and use a hash function (e.g., the crc32 hash function) to turn it into a number. For example, if the key is foobar, crc32(foobar) will output something like 93024922. Then we can use a modulo operation with this number in order to turn it into a number between 0 and 3, so that this number can be mapped to one of our Redis instances.  Although there are few client in Redis that implement consistent hashing out of the box unfortunately some of the most popular do not.

As you may know many features of Redis, such as operations and transactions involving multiple or intersecting keys will not work and adding or removing capacity from Redis will be tricky to say the least. hen partitioning is used, data handling is more complex, for instance you have to handle multiple RDB / AOF files, and to make a backup of your data you need to aggregate the persistence files from multiple instances and hosts.  Moreover what is upsetting with C# StackExchange client connecting to MSOpenTech Redis on Windows or Azure there isn’t anything already built in for you, so you will have to build your own.   Also partitioning may be ok for Redis that is used as a cache store, but for data store may be an issue.  For more see – http://redis.io/topics/partitioning. Some interesting examples are here – http://petrohi.me/post/6323289515/scaling-redis, Twitter proxy implementation – http://antirez.com/news/44

Hope this helps

Forecast Cloudy – Azure Event Hub Introductory Tutorial

Imagine a scenario that rolling out new IoT enabled device or getting telemetry from millions of mobile apps or games. Architecting for this type of solution poses many challenges.  How do you enable ingest of messages at this scale?  More importantly, how do you enable processing of the messages in a manner which does not create a performance bottleneck?  How do you independently scale the message ingest and message processing capabilities?  How do you provide for loose coupling between message ingress and egress and ability to scale out as needed?

Enter Azure Event Hubs.

Azure Event Hubs is a highly scalable publish-subscribe ingestor that can intake millions of events per second so that you can process and analyze the massive amounts of data produced by your connected devices and applications. Once collected into Event Hubs you can transform and store data using any real-time analytics provider or with batching/storage adapters.


Azure Event Hubs is a new member to join the Azure Service bus family adding to the existing topics and queues offering. Event Hubs offer the ability to process a very high volume of messages fast. Event Hubs provides FIFO (First In First Out) messaging much like queues and topics and also supports pub-sub messaging just like topics.

Planning for Azure Event Hubs.

You must put some effort in capacity planning before you create an Event Hub. In order to make the right decisions let’s go over a couple details about Event Hubs. Event Hubs are partitioned. The minimum number of partitions is 8 and the maximum (public) number of partitions is 32. If you need more partitions you must go through a support ticket. At that time, you can request up to 1024 (and higher) partitions for your Event Hub.

A partition is an ordered sequence of events that is held in an Event Hub. As newer events arrive, they are added to the end of this sequence. A partition can be thought of as a “commit log.”

Event Hubs

Partitions retain data for a configured retention time that is set at the Event Hub level and applies across all partitions in the Event Hub. Events expire on a time basis; you cannot explicitly delete them. Each partition is independent and contains its own sequence of data. As a result, partitions often grow at different rates.

Event Hubs

Each partition has a performance target of 1 MB ingress or 1000 operations and 2 MB egress per second. By default, each Event Hub is created with 16 partitions. This corresponds to 16 MB ingress or 16,000 operations and 32 MB egress per second.

Although this is a lot, there is also need to look at message size here. If messages were 1 KB in size we could technically hit the performance target of 1000 operations per second. This could technically represent 16,000 messages per second. Calculating the number of partitions you need starts by calculating potential throughput in megabytes. Then you need to calculate the number of messages per second. This will give you the first part of the equation. The second part of the equation is found by calculating the number of processing nodes required to meet your performance targets. Since a single partition cannot be processed concurrently by multiple nodes, you must have at least as many partitions as you have backend node instances. For example, a system that must process 30 messages simultaneously, where each process requires a dedicated processing node, requires the Event Hub to have at least 30 partitions. Number of partitions = MAX (cumulative throughput required for the stream – given 1 MB per partition, number of nodes needed by the backend processing application)

Changing the number of partitions once in production can cause quite a bit of headaches because it means that we need to create a new Event Hub and reconfigure publishers to use the new Event Hub. While events are piling up let the backend nodes empty the Event Hub. Once it’s empty, you can reconfigure your backend nodes to consumer events from the new Event Hub. Switching to the backend processing nodes to the new Event Hub too early would break the order of events. From a billing perspective, the number of partitions is irrelevant because there is not charge for partitions.

Once the application is deployed you can provision throughput units to scale the Event Hub’s throughput capacity. A single throughput unit  has the capacity of 1 MB per second of ingress events (events sent into an Event Hub), but no more than 1000 ingress events, management operations or control API calls per second. It has 2 MB per second of egress events (events consumed from an Event Hub) and 84 GB of event storage (sufficient for the default 24-hour retention period).  While partitions are a data organization concept, throughput units are purely a capacity concept. Throughput units are billed per hour and are purchased ahead of time. Once purchased, throughput units are billed for a minimum of one hour. Up to 20 throughput units can be purchased for a Service Bus namespace, and there is an Azure account limit of 20 throughput units. These throughput units are shared across all Event Hubs in a given namespace. Throughput units are provisioned on a best effort basis and may not always be available for immediate purchase. If you require a specific capacity, it is recommended that you purchase those throughput units ahead of time. If you require more than 20 throughput units, you can contact Microsoft Azure Service Bus support to purchase more throughput units on a commitment basis in blocks of 20, up to the first 100 throughput units. Beyond that, you can also purchase blocks of 100 throughput units. It is recommended that you carefully balance throughput units and partitions in order to achieve optimal scale with Event Hubs. A single partition has a maximum scale of one throughput unit. The number of throughput units should be less than or equal to the number of partitions in an Event Hub.

If the total ingress throughput or the total ingress event rate across all Event Hubs in a namespace exceeds the aggregate throughput unit allowances, senders will be throttled and receive errors indicating that the ingress quota has been exceeded. If the total egress throughput or the total event egress rate across all Event Hubs in a namespace exceeds the aggregate throughput unit allowances, receivers are throttled and receive errors indicating that the egress quota has been exceeded. Ingress and egress quotas are enforced separately, so that no sender can cause event consumption to slow down, nor can a receiver prevent events from being sent into Event Hubs.

Enough Theory – Lets Create Event Hub

An EventHub can be created from the Azure Management Portal, under the current Service Bus area.Click on the +NEW area at the bottom of the portal and navigate to the Event Hub service


If you have an existing Service Bus namespace, then you can reuse it as I have done. Otherwise, you will need a new namespace before you create Event Hub. A wizard will show on screen to assist you with the creation. Enter an Event Hub name, select a Region that hosts the services that will process the events and click next. Be sure that you are creating the resource on the correct subscription and desired Azure Service Bus Namespace.


The next panel prompts for important information. This is where you specify the number of partitions for your Event Hub. Then because this is a Standard Event Hub, you can specie the number of days that an event stays in the Event Hub. This is especially handy when you need to replay events from a week ago. The maximum number of days is set to 30, the minimum is set to 1 day.


Security and Connectivity

In order to publish events to an Event Hub, you must create Shared Access Policies. In order to publish an event we need a Shared Access Policy that allows us to send events. Navigate to the Service Bus namespace and click on the Event Hubs tab. From the list choose the newly created Event Hub.Then click on Configure. In this screen, you will be able to create a Send enabled Shared Access Policy.


Now we need to retrieve a SAS key connection strings to authenticate and interact with the Event Hub. Select the Dashboard tab and click on the View Connection String link. From the wizard copy the Connection String that is enabled for Send operations. Click the copy button to store the connection string in you clipboard. This will allow you to paste the connection string in your project’s configurations.

Create Event Sender in Code

As my sample will be pretty basic I will just create a simple C# console project in Visual Studio.


In the Solution Explorer, right-click on References and select Manage NuGet Packages…

In the Search Online box type Azure Service Bus.


Install Microsoft Azure Service Bus version 2.6.1 or later.

After that its simply a matter of code. Below sample synchronously  sends very simple messages to the hub:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Threading;
using Microsoft.ServiceBus.Messaging;
using System.Diagnostics;

namespace EventHubSender
    class Program
        static string eventHubName = "{put your hub name here}";
        static string connectionString = "Endpoint=sb:/{put your namespace here}/;SharedAccessKeyName=SendRule;SharedAccessKey={put your access key here}";
        static void Main(string[] args)
            Console.WriteLine("Press Ctrl-C to stop the sender process");
            Console.WriteLine("Press Enter to start now");
        static void SendingRandomMessages()
            var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, eventHubName);
            while (true)
                    var message = Guid.NewGuid().ToString();
                    Console.WriteLine("{0} > Sending message: {1}", DateTime.Now, message);
                    eventHubClient.Send(new EventData(Encoding.UTF8.GetBytes(message)));

                    Console.WriteLine("Duration: " + sw.ElapsedMilliseconds.ToString());
                catch (Exception exception)
                    Console.ForegroundColor = ConsoleColor.Red;
                    Console.WriteLine("{0} > Exception: {1}", DateTime.Now, exception.Message);


You will notice above that certain configuration items are marked {put your… here}. Replace {put your namespace here} with the name of the Service Bus Namespace you created in the Azure portal.  Replace your {put your hub name here} with name of the event hub you created. Finally you’ll see SharedAccessKeyName=. I want you to replace SendRule I used with the name of the data ingress shared access policy you created in your Event Hub.In order to replace {put your access key here} with the correct value, go to the Dashboard page of your Event Hub and click the Connection Information key at the bottom of the page. A dialog containing access connection information with connection strings will appear. Copy the connection string from the data ingress shared access policy you created and paste it into notepad because it contains too much information. Just copy the SharedAccessKey value at the end of the connection string into {put your access key here} and then save and close the file.

Finally running this application is really easy:




Back in Azure Portal in the dashboard for our event hub I can see events being received:


For more and deeper information see these excellent articles and blogs :

Hope this helps.

HotSpot JVM JIT Profiling Using JitWatch Tool

Enter Just In Time Compilation – JIT.

Java code is usually compiled into platform independent bytecode (class files) using “javac” command. This “javac” command is the Java programming language compiler.

The JVM is able to load the class files and execute the Java bytecode via the Java interpreter. Even though this bytecode is usually interpreted, it might also be compiled in to native machine code using the JVM’s Just-In-Time (JIT) compiler.

Unlike the normal compiler, the JIT compiler compiles the code (bytecode) only when required. With JIT compiler, the JVM monitors the methods executed by the interpreter and identifies the “hot methods” for compilation. After identifying the Java method calls, the JVM compiles the bytecode into a more efficient native code.

In this way, the JVM can avoid interpreting a method each time during the execution and thereby improves the run time performance of the application. For each method, the JVM maintains a call count, which is incremented every time the method is called. The JVM interprets a method until its call count exceeds a JIT compilation threshold. Therefore, often-used methods are compiled soon after the JVM has started, and less-used methods are compiled much later, or not at all. The JIT compilation threshold helps the JVM start quickly and still have improved performance. The threshold has been carefully selected to obtain an optimal balance between startup times and long term performance.After a method is compiled, its call count is reset to zero and subsequent calls to the method continue to increment its count. When the call count of a method reaches a JIT recompilation threshold, the JIT compiler compiles it a second time, applying a larger selection of optimizations than on the previous compilation. This process is repeated until the maximum optimization level is reached. The busiest methods of a Java program are always optimized most aggressively, maximizing the performance benefits of using the JIT compiler. The JIT compiler can also measure operational data at run time, and use that data to improve the quality of further recompilations.

To learn more about the JIT compilation process, see “Understanding the Java HotSpot VM Code Cache,” and “Introduction to JIT Compilation in Java HotSpot VM.”


Hotspot has a large number of different optimization techniques for JIT compilation – but one of the most important  is inlining. This is the process of removing virtual method calls, by effectively hoisting the body of methods into the caller’s scope. For example consider following:

 public int add(int x, int y) {
    return x + y;
  int result = add(a, b);

With inlining this code is effectively transformed just to:

 int result = a + b;  

The values a and b have been substituted for the method parameters, and the code comprising the body of the method has been copied into the caller’s scope. This speeds up frequently executed method calls.

For more on JIT optimization phases see – http://www-01.ibm.com/support/knowledgecenter/SSYKE2_7.0.0/com.ibm.java.win.70.doc/diag/understanding/jit_optimize.html

Ah, but is there a catch?

So the JIT process marks a method for compilation to machine instructions. After compilation, method invocations are generally pretty fast on a method since the JIT process done analysis ads to what happens during invocation of this method and can optimize accordingly. The thing is, when performing this operation of marking methods for compilation there’s a catch. An 8k byte code instructions catch to be exact. If a method in byte code contains more then 8000 byte code instructions it will never ever be marked for compilation, period.  You can attempt to use “-XX:-DontCompileHugeMethods”  parameter when invoking JVM, but I have read in many places that it’s a bad practice – it forces the JVM to dig through all that ugly code and try and do something with it, which can have a negative effect on performance rather than a positive one! Refactoring, or better still not writing methods that huge to start with would be the way forward.

Types of JIT Compilers

There are two different JIT compilers for client and server systems in Java. A server application needs to be run for a longer time and therefore it needs more optimizations. However a client application may not need lot of optimizations compared to a server application.

.Prior to some of the later Java SE 7 releases, these two modes were available using the -client and -server switches, respectively.

Very good internals explanation is available here – http://www.javaworld.com/article/2078635/enterprise-middleware/jvm-performance-optimization-part-2-compilers.html

A Window into World of Dynamic Compilation.

JITWatch is a log analyser for Java HotSpot JIT compiler. It consumes JIT log files and visualizes its activity.

JarScan is a tool included in JITWatch to analyze jar files and count the bytes of each method’s bytecode.

With this tool, we can identify the methods, which are too large to JIT.’

JitWatch can be downloaded from GitHub –https://github.com/AdoptOpenJDK/jitwatch

The JITWatch tool can analyze the compilation logs generated with the “-XX:+LogCompilation” flag.

The logs generated by LogCompilation are XML-based and has lot of information related to JIT compilation. Hence these files are very large.


What is great that download includes a sandbox folder with few examples of JIT code that can be used to test drive this tool.


Here is an example after running SandBoxTest.java provided sample looking at the logs:


Chris Newland who created this tool has excellent presentation on it and internals of Java HotSpot JIT on video here – https://skillsmatter.com/skillscasts/5243-chris-newland-hotspot-profiling-with-jit-watch, also there is a great article from Bill Evans on InfoQ that has great JarScan examples – http://www.infoq.com/articles/Java-Application-Hostile-to-JIT-Compilation