Hi Friends,
I need your help in one of the Windows server unexpected reboot. Please find the below details.
We have an Windows Cluster 2008 R2 servers (N1 & N2). All of sudden, N2 node is getting unexpected reboots from past couple of weeks. We found that PSP and NIC drivers were old versions. Performed the change to update the PSP, NIC drivers, Firmware and
HP NCU. We can able to update successfully all components except NCU driver. When we try to upgrade this driver getting BSOD while server reboot. During the server we got memory dump. Here are technical information for further understood.
I am very sorry it is not correct Forum for this issue but please help me
OS version: Windows Server 2008 R2
Cluster: Yes
Network Cards Model: hp ethernet 1gb 4-port 331flr adapter
Current CPQTEAM.SYS version: 10.65.0.0
Upgrading to : 10.90.0.0 (B)
Please find the below Memory dump to understand the issue.
Need resolution Who can i update the cpqteam.sys without BSOD update.
Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [D:\Users\a076901x\Desktop\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available
Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Kernel Version 7601 (Service Pack 1) MP (16 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Built by: 7601.22908.amd64fre.win7sp1_ldr.141211-1743
Machine Name:
Kernel base = 0xfffff800`0181a000 PsLoadedModuleList = 0xfffff800`01a5d890
Debug session time: Thu Jan 29 03:00:56.188 2015 (GMT+1)
System Uptime: 4 days 16:43:20.962
Loading Kernel Symbols
...............................................................
................................................................
...................................
Loading User Symbols
PEB is paged out (Peb.Ldr = 000007ff`fffd9018). Type ".hh dbgerr001" for details
Loading unloaded module list
........................................
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck C5, {ffffffff3bbc8128, 2, 1, fffff800019c3fad}
*** ERROR: Module load completed but symbols could not be loaded for cpqteam.sys
*** ERROR: Module load completed but symbols could not be loaded for b57nd60a.sys
PEB is paged out (Peb.Ldr = 000007ff`fffd9018). Type ".hh dbgerr001" for details
PEB is paged out (Peb.Ldr = 000007ff`fffd9018). Type ".hh dbgerr001" for details
Probably caused by : hardware ( cpqteam+8168 )
Followup: MachineOwner
---------
4: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
* *
*******************************************************************************
DRIVER_CORRUPTED_EXPOOL (c5)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is
caused by drivers that have corrupted the system pool. Run the driver
verifier against any new (or suspect) drivers, and if that doesn't turn up
the culprit, then use gflags to enable special pool.
Arguments:
Arg1: ffffffff3bbc8128, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, value 0 = read operation, 1 = write operation
Arg4: fffff800019c3fad, address which referenced memory
Debugging Details:
------------------
PEB is paged out (Peb.Ldr = 000007ff`fffd9018). Type ".hh dbgerr001" for details
PEB is paged out (Peb.Ldr = 000007ff`fffd9018). Type ".hh dbgerr001" for details
BUGCHECK_STR: 0xC5_2
CURRENT_IRQL: 2
FAULTING_IP:
nt!ExFreePoolWithTag+20d
fffff800`019c3fad 1080080000ff adc byte ptr [rax-0FFFFF8h],al
DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
PROCESS_NAME: Volvo.PLS.Prep
TRAP_FRAME: fffff880023843a0 -- (.trap 0xfffff880023843a0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=ffffffff3cbc8120 rbx=0000000000000000 rcx=0000000000002224
rdx=fffff88002363000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff800019c3fad rsp=fffff88002384530 rbp=0000000000000000
r8=0000000000011120 r9=0000000000000050 r10=fffff8000181a000
r11=00000000000006d4 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei ng nz na pe cy
nt!ExFreePoolWithTag+0x20d:
fffff800`019c3fad 1080080000ff adc byte ptr [rax-0FFFFF8h],al ds:0050:ffffffff`3bbc8128=??
Resetting default scope
MISALIGNED_IP:
nt!ExFreePoolWithTag+20d
fffff800`019c3fad 1080080000ff adc byte ptr [rax-0FFFFF8h],al
LAST_CONTROL_TRANSFER: from fffff8000188c769 to fffff8000188d1c0
STACK_TEXT:
fffff880`02384258 fffff800`0188c769 : 00000000`0000000a ffffffff`3bbc8128 00000000`00000002 00000000`00000001 : nt!KeBugCheckEx
fffff880`02384260 fffff800`0188b3e0 : fffffa80`285cf020 fffff880`01c16dd7 00000000`00000000 fffffa80`217a5510 : nt!KiBugCheckDispatch+0x69
fffff880`023843a0 fffff800`019c3fad : fffffa80`217a5510 00000000`00000050 00000000`00000002 00000000`00000000 : nt!KiPageFault+0x260
fffff880`02384530 fffff880`00edf568 : fffffa80`1b5b8370 fffffa80`28481000 fffffa80`4449544e fffffa80`000006cc : nt!ExFreePoolWithTag+0x20d
fffff880`023845e0 fffff880`04b29168 : fffffa80`1b77e790 fffffa80`1b6afd50 00000000`00000000 fffff880`00802008 : NDIS!NdisFreeNetBufferListContext+0x58
fffff880`02384610 fffff880`00f5b8d4 : fffffa80`1b52f1a0 00000000`00000000 00000000`00000001 00000000`000005d0 : cpqteam+0x8168
fffff880`02384660 fffff880`00f9217b : fffffa80`1b641ba0 fffffa80`1b52f1a0 00000000`00000001 fffff880`00ed61a7 : NDIS!ndisReturnNetBufferListsInternal+0x94
fffff880`023846a0 fffff880`01c520c6 : 00000000`00000000 00000000`00000001 00000000`00000000 fffffa80`21295b90 : NDIS!NdisReturnNetBufferLists+0x3b
fffff880`023846e0 fffff880`00e60c58 : fffffa80`20db4e20 00000000`0000000d 00000000`0000000f fffff8a0`108c2000 : tcpip!FlpReturnNetBufferListChain+0x96
fffff880`02384730 fffff880`01c8998f : 00000000`00000000 fffff880`02384a01 fffff880`02384700 fffffa80`00000014 : NETIO!NetioDereferenceNetBufferListChain+0x518
fffff880`02384800 fffff880`01c5dde5 : fffff880`049bbe00 00000000`00000000 00000000`00000000 00000000`00000000 : tcpip!TcpFlushDelay+0x13f
fffff880`023848e0 fffff880`01c56137 : fffffa80`1a977510 fffffa80`1a910910 fffffa80`00007af4 00000000`0000001d : tcpip!TcpPreValidatedReceive+0x3e5
fffff880`023849b0 fffff880`01c55caa : 00000000`00000000 fffff880`01d6ca10 fffff880`02384b70 fffff8a0`03953000 : tcpip!IppDeliverListToProtocol+0x97
fffff880`02384a70 fffff880`01c55261 : 00000000`00009918 00000000`00000002 00000000`0000000d fffff880`02384b60 : tcpip!IppProcessDeliverList+0x5a
fffff880`02384b10 fffff880`01c52eef : 00000000`bb3e7099 fffff880`01d6ca10 00000000`00000000 00000000`00000000 : tcpip!IppReceiveHeaderBatch+0x232
fffff880`02384c10 fffff880`01c524c2 : fffffa80`1b5f8a40 00000000`00000000 fffffa80`1b641b01 fffff800`0000001e : tcpip!IpFlcReceivePackets+0x64f
fffff880`02384e10 fffff880`01c518ea : fffffa80`1b641ba0 fffff880`02384f40 fffffa80`1b641ba0 00000000`00000000 : tcpip!FlpReceiveNonPreValidatedNetBufferListChain+0x2b2
fffff880`02384ef0 fffff800`01899ad8 : fffffa80`1b5c6c10 00000000`00004800 fffffa80`223ecb50 00000000`00000000 : tcpip!FlReceiveNetBufferListChainCalloutRoutine+0xda
fffff880`02384f40 fffff880`01c51fe2 : fffff880`01c51810 00000000`00000000 fffffa80`1b4bc102 fffff880`02386000 : nt!KeExpandKernelStackAndCalloutEx+0xd8
fffff880`02385020 fffff880`00f920eb : fffffa80`1b645010 00000000`00000000 fffffa80`1b52f1a0 fffff880`02386001 : tcpip!FlReceiveNetBufferListChain+0xb2
fffff880`02385090 fffff880`00f5bad6 : fffff880`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : NDIS!ndisMIndicateNetBufferListsToOpen+0xdb
fffff880`02385100 fffff880`00ed5ac1 : fffffa80`1b52f1a0 00000000`00000002 00000000`0000001e fffff800`01d2a37c : NDIS!ndisMDispatchReceiveNetBufferLists+0x1d6
fffff880`02385580 fffff880`04b2a746 : fffffa80`224da9b0 fffffa80`1b5c6c10 fffffa80`1b7c6000 00000000`00000001 : NDIS!NdisMIndicateReceiveNetBufferLists+0xc1
fffff880`023855d0 fffff880`04b2a64e : 00000000`00000000 00000000`00000000 fffffa80`00000000 fffff800`00000001 : cpqteam+0x9746
fffff880`02385640 fffff880`00f920eb : 00000000`0000000d fffff800`0195334b 00000000`00000003 fffffa80`1b6518d0 : cpqteam+0x964e
fffff880`02385720 fffff880`00f5bc75 : 00000000`00000000 00000000`00000000 00000000`00000000 fffffa80`1b641ba0 : NDIS!ndisMIndicateNetBufferListsToOpen+0xdb
fffff880`02385790 fffff880`00ed5ac1 : fffffa80`1b4bc1a0 00001f80`00380200 00000000`00000200 fffff880`04a2b499 : NDIS!ndisMDispatchReceiveNetBufferLists+0x375
fffff880`02385c10 fffff880`04a5ff34 : fffffa80`1b653000 fffff880`02385d00 00000000`0000001e fffffa80`1b5c6c10 : NDIS!NdisMIndicateReceiveNetBufferLists+0xc1
fffff880`02385c60 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : b57nd60a+0x48f34
STACK_COMMAND: kb
FOLLOWUP_IP:
cpqteam+8168
fffff880`04b29168 4d85e4 test r12,r12
SYMBOL_STACK_INDEX: 5
SYMBOL_NAME: cpqteam+8168
FOLLOWUP_NAME: MachineOwner
IMAGE_NAME: hardware
DEBUG_FLR_IMAGE_TIMESTAMP: 0
MODULE_NAME: hardware
FAILURE_BUCKET_ID: X64_IP_MISALIGNED
BUCKET_ID: X64_IP_MISALIGNED
Followup: MachineOwner
---------
4: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************
DRIVER_CORRUPTED_EXPOOL (c5)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is
caused by drivers that have corrupted the system pool. Run the driver
verifier against any new (or suspect) drivers, and if that doesn't turn up
the culprit, then use gflags to enable special pool.
Arguments:
Arg1: ffffffff3bbc8128, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, value 0 = read operation, 1 = write operation
Arg4: fffff800019c3fad, address which referenced memory
Debugging Details:
------------------
PEB is paged out (Peb.Ldr = 000007ff`fffd9018). Type ".hh dbgerr001" for details
PEB is paged out (Peb.Ldr = 000007ff`fffd9018). Type ".hh dbgerr001" for details
BUGCHECK_STR: 0xC5_2
CURRENT_IRQL: 2
FAULTING_IP:
nt!ExFreePoolWithTag+20d
fffff800`019c3fad 1080080000ff adc byte ptr [rax-0FFFFF8h],al
DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
PROCESS_NAME: Volvo.PLS.Prep
TRAP_FRAME: fffff880023843a0 -- (.trap 0xfffff880023843a0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=ffffffff3cbc8120 rbx=0000000000000000 rcx=0000000000002224
rdx=fffff88002363000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff800019c3fad rsp=fffff88002384530 rbp=0000000000000000
r8=0000000000011120 r9=0000000000000050 r10=fffff8000181a000
r11=00000000000006d4 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei ng nz na pe cy
nt!ExFreePoolWithTag+0x20d:
fffff800`019c3fad 1080080000ff adc byte ptr [rax-0FFFFF8h],al ds:0050:ffffffff`3bbc8128=??
Resetting default scope
MISALIGNED_IP:
nt!ExFreePoolWithTag+20d
fffff800`019c3fad 1080080000ff adc byte ptr [rax-0FFFFF8h],al
LAST_CONTROL_TRANSFER: from fffff8000188c769 to fffff8000188d1c0
STACK_TEXT:
fffff880`02384258 fffff800`0188c769 : 00000000`0000000a ffffffff`3bbc8128 00000000`00000002 00000000`00000001 : nt!KeBugCheckEx
fffff880`02384260 fffff800`0188b3e0 : fffffa80`285cf020 fffff880`01c16dd7 00000000`00000000 fffffa80`217a5510 : nt!KiBugCheckDispatch+0x69
fffff880`023843a0 fffff800`019c3fad : fffffa80`217a5510 00000000`00000050 00000000`00000002 00000000`00000000 : nt!KiPageFault+0x260
fffff880`02384530 fffff880`00edf568 : fffffa80`1b5b8370 fffffa80`28481000 fffffa80`4449544e fffffa80`000006cc : nt!ExFreePoolWithTag+0x20d
fffff880`023845e0 fffff880`04b29168 : fffffa80`1b77e790 fffffa80`1b6afd50 00000000`00000000 fffff880`00802008 : NDIS!NdisFreeNetBufferListContext+0x58
fffff880`02384610 fffff880`00f5b8d4 : fffffa80`1b52f1a0 00000000`00000000 00000000`00000001 00000000`000005d0 : cpqteam+0x8168
fffff880`02384660 fffff880`00f9217b : fffffa80`1b641ba0 fffffa80`1b52f1a0 00000000`00000001 fffff880`00ed61a7 : NDIS!ndisReturnNetBufferListsInternal+0x94
fffff880`023846a0 fffff880`01c520c6 : 00000000`00000000 00000000`00000001 00000000`00000000 fffffa80`21295b90 : NDIS!NdisReturnNetBufferLists+0x3b
fffff880`023846e0 fffff880`00e60c58 : fffffa80`20db4e20 00000000`0000000d 00000000`0000000f fffff8a0`108c2000 : tcpip!FlpReturnNetBufferListChain+0x96
fffff880`02384730 fffff880`01c8998f : 00000000`00000000 fffff880`02384a01 fffff880`02384700 fffffa80`00000014 : NETIO!NetioDereferenceNetBufferListChain+0x518
fffff880`02384800 fffff880`01c5dde5 : fffff880`049bbe00 00000000`00000000 00000000`00000000 00000000`00000000 : tcpip!TcpFlushDelay+0x13f
fffff880`023848e0 fffff880`01c56137 : fffffa80`1a977510 fffffa80`1a910910 fffffa80`00007af4 00000000`0000001d : tcpip!TcpPreValidatedReceive+0x3e5
fffff880`023849b0 fffff880`01c55caa : 00000000`00000000 fffff880`01d6ca10 fffff880`02384b70 fffff8a0`03953000 : tcpip!IppDeliverListToProtocol+0x97
fffff880`02384a70 fffff880`01c55261 : 00000000`00009918 00000000`00000002 00000000`0000000d fffff880`02384b60 : tcpip!IppProcessDeliverList+0x5a
fffff880`02384b10 fffff880`01c52eef : 00000000`bb3e7099 fffff880`01d6ca10 00000000`00000000 00000000`00000000 : tcpip!IppReceiveHeaderBatch+0x232
fffff880`02384c10 fffff880`01c524c2 : fffffa80`1b5f8a40 00000000`00000000 fffffa80`1b641b01 fffff800`0000001e : tcpip!IpFlcReceivePackets+0x64f
fffff880`02384e10 fffff880`01c518ea : fffffa80`1b641ba0 fffff880`02384f40 fffffa80`1b641ba0 00000000`00000000 : tcpip!FlpReceiveNonPreValidatedNetBufferListChain+0x2b2
fffff880`02384ef0 fffff800`01899ad8 : fffffa80`1b5c6c10 00000000`00004800 fffffa80`223ecb50 00000000`00000000 : tcpip!FlReceiveNetBufferListChainCalloutRoutine+0xda
fffff880`02384f40 fffff880`01c51fe2 : fffff880`01c51810 00000000`00000000 fffffa80`1b4bc102 fffff880`02386000 : nt!KeExpandKernelStackAndCalloutEx+0xd8
fffff880`02385020 fffff880`00f920eb : fffffa80`1b645010 00000000`00000000 fffffa80`1b52f1a0 fffff880`02386001 : tcpip!FlReceiveNetBufferListChain+0xb2
fffff880`02385090 fffff880`00f5bad6 : fffff880`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : NDIS!ndisMIndicateNetBufferListsToOpen+0xdb
fffff880`02385100 fffff880`00ed5ac1 : fffffa80`1b52f1a0 00000000`00000002 00000000`0000001e fffff800`01d2a37c : NDIS!ndisMDispatchReceiveNetBufferLists+0x1d6
fffff880`02385580 fffff880`04b2a746 : fffffa80`224da9b0 fffffa80`1b5c6c10 fffffa80`1b7c6000 00000000`00000001 : NDIS!NdisMIndicateReceiveNetBufferLists+0xc1
fffff880`023855d0 fffff880`04b2a64e : 00000000`00000000 00000000`00000000 fffffa80`00000000 fffff800`00000001 : cpqteam+0x9746
fffff880`02385640 fffff880`00f920eb : 00000000`0000000d fffff800`0195334b 00000000`00000003 fffffa80`1b6518d0 : cpqteam+0x964e
fffff880`02385720 fffff880`00f5bc75 : 00000000`00000000 00000000`00000000 00000000`00000000 fffffa80`1b641ba0 : NDIS!ndisMIndicateNetBufferListsToOpen+0xdb
fffff880`02385790 fffff880`00ed5ac1 : fffffa80`1b4bc1a0 00001f80`00380200 00000000`00000200 fffff880`04a2b499 : NDIS!ndisMDispatchReceiveNetBufferLists+0x375
fffff880`02385c10 fffff880`04a5ff34 : fffffa80`1b653000 fffff880`02385d00 00000000`0000001e fffffa80`1b5c6c10 : NDIS!NdisMIndicateReceiveNetBufferLists+0xc1
fffff880`02385c60 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : b57nd60a+0x48f34
STACK_COMMAND: kb
FOLLOWUP_IP:
cpqteam+8168
fffff880`04b29168 4d85e4 test r12,r12
SYMBOL_STACK_INDEX: 5
SYMBOL_NAME: cpqteam+8168
FOLLOWUP_NAME: MachineOwner
IMAGE_NAME: hardware
DEBUG_FLR_IMAGE_TIMESTAMP: 0
MODULE_NAME: hardware
FAILURE_BUCKET_ID: X64_IP_MISALIGNED
BUCKET_ID: X64_IP_MISALIGNED
Followup: MachineOwner
---------
4: kd> lmvm hardware
start end module name
Please let me know if you need any other information on this