Save Our Souls- Troubleshooting Heap Corruption With DebugDiag, AppVerifier and Windows Debugger (WINDBG)–PART 2

In Part 1 of this post I went into basics of setting up a memory dump to troubleshoot heap corruption with App Verifier and DebugDiag tool.  Now I will finish by using Windows Debugger\WinDBG to take a quick look at these dumps to see a culprit that causes heap corruption issue to begin with. To protect the innocent I will rename module names in the application , as the point of this post isn’t to blame anyone, but to illustrate easy technique to troubleshoot heap corruption.

As stated the App Verifier hit breakpoint on invalid memory operation and DebugDiag captured a dump on that breakpoint. Next I will open this dump in Windows Debugger\WinDBG.

WinDbg is a debugger that wraps NTSD and KD with a better UI. It provides command-line options like starting minimized (-m), attach to a process by pid (-p) and auto-open crash files (-z)..

It supports three types of commands:

  • regular commands (e.g.: k). The regular commands are to debug processes.
  • dot commands (e.g.: .sympath). The dot commands are to control the debugger.
  • extension commands (e.g.: !handle) – these are custom commands that you can add to WinDbg; they are implemented as exported functions in extension DLLs.

PDB files are program database files generated by the linker. Private PDB files contain information about private and public symbols, source lines, types, locals and globals. Public PDB files do not contain types, local and source line information.

You need symbols in order to be able to do effective debugging. Symbol files could be in an older COFF format or the PDB format. PDBs are program database files and contain public symbols. These debuggers allow you to mention a list of URIs where they would look for symbols for loaded binaries.

OS symbols are usually installed in the %SYSTEMDIR%Symbols directory. Driver symbols (.DBG or .PDB files) are usually in the same folder as the driver (.sys file). Private symbol files contain information about functions, local and global variables, and line information to correlate assembly code to source code; symbol files that are usually made available to customers are public symbol files – these files contain information about public members only.

You can set symbol directories through File->Symbol File Path, or using .sympath from the WinDbg command window. To add reference to a symbol server on the web, add:

SRV*downstream_store*http://msdl.microsoft.com/download/symbols

to your .sympath, thus:

.sympath+ SRV*c:\tmp*http://msdl.microsoft.com/download/symbols

 

Where c:\tmp is the download_store where necessary symbols will be downloaded and stored. Note that this particular symbol server exposes public symbols only.

The debugger matches information like filename, timestamp and checksum when matching a PDB with a binary (DLL or exe). If you have symbol information, you’d be able to see function names and their arguments in your call stack. If the binaries and PDBs are from your application, you’d additionally have information about private functions, local variables and type information.

The sympath can consist of multiple URIs. Sympath is initialized from the _NT_SYMBOL_PATH system environment variable.

So lets open dump we received in debugger:and run very simple stack command on faulting\active thread:

 

kb

1870e42c 10003b68 0869a580 01151000 1870e6f8 ntdll!DbgBreakPoint

1870e634 100078c9 1000c540 0000000f 01151000 vrfcore!VerifierStopMessageEx+0x4d1 1870e658 7c878689 0000000f 7c8789a0 01151000 vrfcore!VfCoreRedirectedStopMessage+0x81 1870e6d4 7c878d0c 01151000 00000009 65cd2108 ntdll!RtlpDphReportCorruptedBlock+0x1cf 1870e738 7c879894 7f660020 00000000 01151000 ntdll!RtlpDphAddToDelayedFreeQueue+0x120 1870e75c 7c879ab4 01151000 01250000 01000002 ntdll!RtlpDphNormalHeapFree+0x73

1870e7b4 7c87c98b 01150000 01000002 7f660040 ntdll!RtlpDebugPageHeapFree+0x146

1870e81c 7c85567a 01150000 01000002 7f660040 ntdll!RtlDebugFreeHeap+0x2c

1870e8f4 7c83e448 01150000 01000002 7f660040 ntdll!RtlFreeHeapSlowly+0x37

1870e9d8 776bcf60 01150000 01000002 7f660040 ntdll!RtlFreeHeap+0x11a

1870e9ec 776bcf7c 7779777c 7f660040 fffffff6 ole32!CRetailMalloc_Free+0x1c

*** WARNING: Unable to verify checksum for OraOps10.dll

*** ERROR: Symbol file could not be found. Defaulted to export symbols for OraOps10.dll -

1870e9fc 07cd8c1f 7f660040 07cd7805 7f660040 ole32!CoTaskMemFree+0x13

1870ea04 07cd7805 7f660040 00000000 00000100 OraOps10!ssmem_free+0xf

1870ea20 07cd25b4 1274afd8 77def3d8 77def338 OraOps10+0x7805

Noted prohibited operation by Oracle.NET Provider (OraOps.dll) that reports heap corruption and hits Verifier breakpoint. Also note below that OraDRResultSet does PInvoke into unmanaged space from managed:

0:015> !clrstack

OS Thread Id: 0xfd0 (15)

Child SP IP Call Site

1870eaf0 7c81a229 [InlinedCallFrame: 1870eaf0] 1870ead0 0b4911b0 DomainBoundILStubClass.IL_STUB_PInvoke(IntPtr, IntPtr, IntPtr, IntPtr ByRef, Oracle.DataAccess.Client.OpoSqlValCtx*, Oracle.DataAccess.Client.OpoMetValCtx*, Oracle.DataAccess.Client.OpoDacValCtx*)

1870eaf0 0b490ea0 [InlinedCallFrame: 1870eaf0] Oracle.DataAccess.Client.OpsDac.Read(IntPtr, IntPtr, IntPtr, IntPtr ByRef, Oracle.DataAccess.Client.OpoSqlValCtx*, Oracle.DataAccess.Client.OpoMetValCtx*, Oracle.DataAccess.Client.OpoDacValCtx*) 1870eb48 0b490ea0 Oracle.DataAccess.Client.OracleDataReader.Read() 1

870eb78 0b499979 v5Oracle.OraDRResultSet.Fetch(Boolean)

Customer was using older version of Oracle .NET Provider and from what I understand may have run into the issue that was fixed already some time ago.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s