Forecast Cloudy – Troubleshooting and Debugging Azure PaaS Roles–Live Debug On The Cloud Is Possible

In previous post I went through Azure architecture and logging for troubleshooting. However, sometimes working with large Azure projects you may find it useful to go all in and debug role in production using Windows Debugger\WinDBG. My colleague Jeff Van Nortwick spent weeks debugging fairly tricky issues on rather huge Azure PaaS solution and came up with a way to use WinDBG to live debug Azure PaaS roles. I will be using his notes and attempt to attach debugger to my test Azure worker role.  In the same previous post I already created a sample worker and showed you how to connect to it via RDP.

My first step after connecting again to my role via RDP is to get Debugging Tools for Windows on it. I navigated in IE to Windows SDK download page, which includes Debugging Tools for Windows at https://msdn.microsoft.com/en-US/windows/desktop/bg162891. In order to download installer I will need to disable IE enhanced security on the role via Server Manager on that role:

image

After accepting all legal stuff and default path, I will pick only Debugging Tools for Windows for installation. Windows SDK is a rather large package with large number of tools, APIs and samples and all I need is Debugging Tools for Windows with less than 15 MB footprint:

image

Now that Debugging Tools for Windows is installed on the role we should be ready to proceed here:

image

Next items are to setup WinDBG to run as a server on the role for remote debugging. First we will configure process we need to debug – WaWorkerHost to start under Windows Debugger. We can use registry or our old friend GFLAGS for that purpose:

image

Once you have it running under Windows Debugger, next we need to configure remote debugger session. add the command line parameter -Server TCP:Port=<ANY PORT> to windbg.exe.  (e.g. “C:\debugger\Windbg.exe” -server tcp:port=5555). Doing so will not only start the service under debugger but also start a server session for windbg, that can be connected from user session. Now from any user session you can start windbg and connect to the session our WaWorkerHost is running in via debugger using ‘Connect to a remote session ‘ option in windbg.

Bounce the WindowsAzureGuestAgent service. When you do this, your remote desktop connection will get severed and reconnected.  And, if you use TLIST, you’ll see an instance of WinDbg running – attached to WaWorkerHost.exe. 

Next make sure you open Windows firewall so debugger can do its thing. Finally, start WinDbg, and choose Connect to Remote Session from the File menu:

image

Under Connection String, enter the following string.

tcp:Port=5005,Server=localhost

you’re connected to the Remote debugger and able to control the target. No you can set breakpoints with bp , break on exceptions with sx command and do usual live debug.

This is a pretty complex sequence and usually you don’t need to live debug in Azure production systems vs. usual need for postmortem debugging.  That’s where I will concentrate on my next post in this series.  This would be impossible without a pioneer who managed to figure out these steps in a desperate situation first – my colleague Jeff Van Northwick, who graciously shared these steps with me.

Hope this was interesting.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s