Verifier and Empty Registry key

Well first blog from the Land of 10,000 lakes. Somebody had a server with a continuous boot cycle due to a crash… NICE Smile

Upon checking the bugcheck code 0xC4 – Verifier crash… It has been a long time I did not had a pool corruption to look at. However the stack was very interesting

fffff880`02393488 fffff800`021604ec nt!KeBugCheckEx
fffff880`02393490 fffff800`02160f2b nt!VerifierBugCheckIfAppropriate+0x3c
fffff880`023934d0 fffff800`02171bc8 nt!ExAllocatePoolSanityChecks+0xcb
fffff880`02393510 fffff800`02171e37 nt!VeAllocatePoolWithTagPriority+0x88
fffff880`02393580 fffff880`05bc35a1 nt!VerifierExAllocatePoolWithTagPriority+0x17
fffff880`023935c0 fffff880`05bc27bb tcpipreg!InterfaceAddressRegKeyChangeHandler+0x109
fffff880`023936f0 fffff880`05bc1a59 tcpipreg!TcpipRegQueryAndUpdateKeyValue+0x363
fffff880`02393780 fffff880`00e55754 tcpipreg!TcpipRegStartRegistryKeyNotification+0xbd
fffff880`023937d0 fffff880`05bc2293 NETIO!RtlInvokeStartRoutines+0x34
fffff880`02393810 fffff800`020bdba7 tcpipreg!DriverEntry+0x257
fffff880`02393860 fffff800`020bdfa5 nt!IopLoadDriver+0xa07
fffff880`02393b30 fffff800`01cdaa95 nt!IopLoadUnloadDriver+0x55
fffff880`02393b70 fffff800`01f6fb8a nt!ExpWorkerThread+0x111
fffff880`02393c00 fffff800`01cc28e6 nt!PspSystemThreadStartup+0x5a
fffff880`02393c40 00000000`00000000 nt!KxStartSystemThread+0x16

Pointing to tcpipreg module. However it did not make much sense and I request the technician why verifier was enabled in the first place just to check if we had 0x19 , 0xC2 crashes. But not of those were happening and by having a little experience with people being trigger happy with verifier and not understanding the consequences of it, we just disable it and main issue of reboot cycle was fixed.

Since I had a fresh dump and plenty of time after running outside in zero degree weather nothing like a debug session to warm up the spirit.

First !analyze pointed to tcpipreg – km:x64_0xc4_0_vrf_tcpipreg!interfaceaddressregkeychangehandler+109

Well by checking the output of !verifier , it was doing automatic check and Special Pool :

 

9: kd> !verifier

Verify Flags Level 0x00000001

  STANDARD FLAGS:
    [X] (0x00000000) Automatic Checks
    [X] (0x00000001) Special pool
    [ ] (0x00000002) Force IRQL checking
    [ ] (0x00000008) Pool tracking
    [ ] (0x00000010) I/O verification
    [ ] (0x00000020) Deadlock detection
    [ ] (0x00000080) DMA checking
    [ ] (0x00000100) Security checks
    [ ] (0x00000800) Miscellaneous checks

  ADDITIONAL FLAGS:
    [ ] (0x00000004) Randomized low resources simulation
    [ ] (0x00000200) Force pending I/O requests
    [ ] (0x00000400) IRP logging

    [X] Indicates flag is enabled

Summary of All Verifier Statistics

  RaiseIrqls           0x0
  AcquireSpinLocks     0x1616566
  Synch Executions     0x3508
  Trims                0x0

  Pool Allocations Attempted             0x58f15
  Pool Allocations Succeeded             0x58f15
  Pool Allocations Succeeded SpecialPool 0x58f15
  Pool Allocations With NO TAG           0x4
  Pool Allocations Failed                0x0

  Current paged pool allocations         0x0 for 00000000 bytes
  Peak paged pool allocations            0x0 for 00000000 bytes
  Current nonpaged pool allocations      0x0 for 00000000 bytes
  Peak nonpaged pool allocations         0x0 for 00000000 bytes

Well checking the stack and looking for TAG allocation

nt!VeAllocatePoolWithTagPriority+0x88

nt!VerifierExAllocatePoolWithTagPriority+0x17

In the complete arguments we find the POOL allocation

fffff880`02393580 fffff880`05bc35a1 : 00000000`00000000 00000000`00000000 fffff980`12d82fd0 fffff800`0216d15c : nt!VerifierExAllocatePoolWithTagPriority+0x17

9: kd> !pool fffff980`12d82fd0
Pool page fffff98012d82fd0 region is Special pool
Address fffff98012d82000 does not belong to any pool
*fffff98012d82000 size:   22 data: fffff98012d82fd0 (NonPaged) *LeGe
       Pooltag LeGe : Legacy Registry Mapping Module Buffers, Binary : tcpip.sys

It points to tcpip.sys driver owning the tag. Whenever you don’t have the pooltag mapped correctly you can search for it in the dump by using !for_each_module

9: kd> !for_each_module s -a @#Base @#End “LeGe” –> FOUR characters for TAG (fill with spaces if tag less than )

fffff880`05bc56fd  4c 65 47 65 49 8b d6 33-c9 44 89 5c 24 40 ff 15  LeGeI..3.D.\$@..
fffff880`05bc5744  4c 65 47 65 45 33 c9 49-8b d6 33 c9 44 8b c3 ff  LeGeE3.I..3.D…
fffff880`05bc5b73  4c 65 47 65 44 89 bc 24-f8 00 00 00 ff 15 c3 15  LeGeD..$…..

 

Then you need to use lm a <base address>

9: kd> lm a fffff880`05bc5b73
start             end                 module name
fffff880`05bbf000 fffff880`05bd1000   tcpipreg   (pdb symbols)

In this case you confirm that it belongs to tcpipreg one of drivers using the LeGe TAG.

But it not make much sense in the case the tcpip.sys and tcpipreg.sys where in the latest public available for Windows Server 2008 R2.

By searching the web we come across the following article :

http://support.microsoft.com/kb/2550576/en-us

“When Driver Verifier (Verifier.exe) is enabled on the tcpipreg.sys driver, the STOP 0xC4 occurs following a NULL length memory allocation.”

“This problem has only been observed after the machine’s TCP/IP configuration has been changed from a static IP address to a dynamic address with DHCP, which leaves an empty value for the IPAddress registry key for the network interface.”

Well it matched the issue we were having. But since we had time and a good dump is always good to look and retrieve more information and create a bag of tricks. In this case it will be registry information with !reg extension.

First we need to find which key is being manipulated, lets look again at the stack

9: kd> k
Child-SP          RetAddr           Call Site
fffff880`02393488 fffff800`021604ec nt!KeBugCheckEx
fffff880`02393490 fffff800`02160f2b nt!VerifierBugCheckIfAppropriate+0x3c
fffff880`023934d0 fffff800`02171bc8 nt!ExAllocatePoolSanityChecks+0xcb
fffff880`02393510 fffff800`02171e37 nt!VeAllocatePoolWithTagPriority+0x88
fffff880`02393580 fffff880`05bc35a1 nt!VerifierExAllocatePoolWithTagPriority+0x17
fffff880`023935c0 fffff880`05bc27bb tcpipreg!InterfaceAddressRegKeyChangeHandler+0x109
fffff880`023936f0 fffff880`05bc1a59 tcpipreg!TcpipRegQueryAndUpdateKeyValue+0x363
fffff880`02393780 fffff880`00e55754 tcpipreg!TcpipRegStartRegistryKeyNotification+0xbd
fffff880`023937d0 fffff880`05bc2293 NETIO!RtlInvokeStartRoutines+0x34
fffff880`02393810 fffff800`020bdba7 tcpipreg!DriverEntry+0x257
fffff880`02393860 fffff800`020bdfa5 nt!IopLoadDriver+0xa07
fffff880`02393b30 fffff800`01cdaa95 nt!IopLoadUnloadDriver+0x55
fffff880`02393b70 fffff800`01f6fb8a nt!ExpWorkerThread+0x111
fffff880`02393c00 fffff800`01cc28e6 nt!PspSystemThreadStartup+0x5a
fffff880`02393c40 00000000`00000000 nt!KxStartSystemThread+0x16

The first function resembling calling registry

fffff880`02393780 fffff880`00e55754 : fffff880`05bc7a60 00000000`00000004 00000000`00000000 fffff880`05bca318 : tcpipreg!TcpipRegStartRegistryKeyNotification+0xbd

Dump the memory location and you see a registry key path over there.

9: kd> dc fffff880`05bc7a60+100
fffff880`05bc7b60  006f0043 0074006e 006f0072 0053006c  C.o.n.t.r.o.l.S.
fffff880`05bc7b70  00740065 0053005c 00720065 00690076  e.t.\.S.e.r.v.i.
fffff880`05bc7b80  00650063 005c0073 00630054 00690070  c.e.s.\.T.c.p.i.
fffff880`05bc7b90  005c0070 00610050 00610072 0065006d  p.\.P.a.r.a.m.e.
fffff880`05bc7ba0  00650074 00730072 0049005c 0074006e  t.e.r.s.\.I.n.t.
fffff880`05bc7bb0  00720065 00610066 00650063 00000073  e.r.f.a.c.e.s…
fffff880`05bc7bc0  0052005c 00670065 00730069 00720074  \.R.e.g.i.s.t.r.
fffff880`05bc7bd0  005c0079 0061004d 00680063 006e0069  y.\.M.a.c.h.i.n.

Now it is time to check our registry hive with !reg

First do a hive list to check the HiveAddr for SYSTEM

9: kd> !reg hivelist

——————————————————————————————————————————————————-
|     HiveAddr     |Stable Length|    Stable Map    |Volatile Length|    Volatile Map    |MappedViews|PinnedViews|U(Cnt)|     BaseBlock     | FileName
——————————————————————————————————————————————————-
| fffff8a000014010 |       1000  | fffff8a0000140c8 |       1000    |  fffff8a000014340  |        0  |        0  |     0| fffff8a000015000  | <NONAME>
| fffff8a000020410 |     e3b000  | fffff8a000027000 |      42000    |  fffff8a000020740  |        0  |        0  |     0| fffff8a000021000  | SYSTEM
| fffff8a000058320 |       f000  | fffff8a0000583d8 |      11000    |  fffff8a000058650  |        0  |        0  |     0| fffff8a000068000  | <NONAME>

 

 

After that list the open keys under SYSTEM and look for keys under the TCPIP\Parameters

!reg openkeys fffff8a000020410

Index 50:      43de5ac1 kcb=fffff8a000d6f380 cell=00075420 f=00200000 \REGISTRY\MACHINE\SYSTEM\CONTROLSET001\SERVICES\TCPIP\PARAMETERS

With that you can query the value of the keys and see that some of the keys in the TCPIP\INTERFACES were empty. Which most likely matches the symptom in the article.

9: kd> !reg querykey \REGISTRY\MACHINE\SYSTEM\CONTROLSET001\SERVICES\TCPIP\PARAMETERS\INTERFACES

Found KCB = fffff8a00184a5d0 :: \REGISTRY\MACHINE\SYSTEM\CONTROLSET001\SERVICES\TCPIP\PARAMETERS\INTERFACES

Hive         fffff8a000020410
KeyNode      fffff8a0053a93ac

[SubKeyAddr]         [SubKeyName]
fffff8a000b910cc     {065f0c42-703a-11de-9954-806e6f6e6963}
fffff8a000b910cc     {065f0c42-703a-11de-9954-806e6f6e6963}
fffff8a000b910cc     {065f0c42-703a-11de-9954-806e6f6e6963}

Use ‘!reg keyinfo fffff8a000020410 <SubKeyAddr>’ to dump the subkey details

[ValueType]         [ValueName]                   [ValueData]
Key has no Values
9: kd> !reg keyinfo fffff8a000020410 fffff8a000b910cc

KeyPath     \REGISTRY\MACHINE\SYSTEM\ControlSet001\services\Tcpip\Parameters\Interfaces\{065f0c42-703a-11de-9954-806e6f6e6963}

[ValueType]         [ValueName]                   [ValueData]
 Key has no Values

 

Well that ends this blog. Just make sure every time you request that something as intrusive as driver verifier, you know the consequences and why you are actually requesting this.

Good hunting,

Alessandro

 

Advertisements

About smartwindows

Support professional for Microsoft technologies with interest in Performance and Debugging
This entry was posted in !verifier, WinDBG Trick and tagged , . Bookmark the permalink.

2 Responses to Verifier and Empty Registry key

  1. Always very useful! thanks for sharing the 0xC4 😉

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s