Athlon64 + Nforce4 MCE panic

Athlon64 + Nforce4 MCE panic

[Rumi Szabolcs]

Hello!

Tonight I had a kernel panic (full hang) with the following on the console:

CPU 0: Machine Check Exception: 0000000000000004
Bank 4: b200000000070f0f
Kernel panic - not syncing: CPU context corrupt

I tried to decode this:

# ./parsemce -e 0000000000000004 -b 4 -s b200000000070f0f -a 0
Status: (4) Machine Check in progress.
Restart IP invalid.
parsebank(4): b200000000070f0f @ 0
        External tag parity error
        CPU state corrupt. Restart not possible
        Error enabled in control register
        Error not corrected.
        Bus and interconnect error
        Participation: Generic
        Timeout:
        Request: Generic error
        Transaction type : Invalid
        Memory/IO : Other

# echo 'CPU 0: Machine Check Exception: 0000000000000004 Bank 4: b200000000070f0f' | mcelog --ascii --k8
HARDWARE ERROR. This is *NOT* a software problem!
Please contact your hardware vendor
CPU 0 4 northbridge   Northbridge Watchdog error
       bit57 = processor context corrupt
       bit61 = error uncorrected
  bus error 'generic participation, request timed out
      generic error mem transaction
      generic access, level generic'
STATUS b200000000070f0f MCGSTATUS 4

I've been searching for and found some additional info, it
looks like I'm not the first experiencing this problem:

http://kerneltrap.org/node/4993

Here ^^^ it is suggested that there was some thread about A64 MCEs
on LKML but I failed to find it so I decided to post... sorry if
it's redundant.

The kernel is 2.6.16-gentoo-r9 and the CPU is an A64 "Venice" 3500+
I can post further hw/sw environment information if req'd.

Thanks!

Regards,

Sab

Re: Athlon64 + Nforce4 MCE panic

[Avuton Olrich]

On 7/13/06, Rumi Szabolcs <rumi_ml@rtfm.hu> wrote:
> Hello!
>
> Tonight I had a kernel panic (full hang) with the following on the console:
>
> CPU 0: Machine Check Exception: 0000000000000004
> Bank 4: b200000000070f0f
> Kernel panic - not syncing: CPU context corrupt
>
> I tried to decode this:
>
> # ./parsemce -e 0000000000000004 -b 4 -s b200000000070f0f -a 0
> Status: (4) Machine Check in progress.
> Restart IP invalid.
> parsebank(4): b200000000070f0f @ 0
>         External tag parity error
>         CPU state corrupt. Restart not possible
>         Error enabled in control register
>         Error not corrected.
>         Bus and interconnect error
>         Participation: Generic
>         Timeout:
>         Request: Generic error
>         Transaction type : Invalid
>         Memory/IO : Other
>
> # echo 'CPU 0: Machine Check Exception: 0000000000000004 Bank 4: b200000000070f0f' | mcelog --ascii --k8
> HARDWARE ERROR. This is *NOT* a software problem!
> Please contact your hardware vendor
> CPU 0 4 northbridge   Northbridge Watchdog error
>        bit57 = processor context corrupt
>        bit61 = error uncorrected
>   bus error 'generic participation, request timed out
>       generic error mem transaction
>       generic access, level generic'
> STATUS b200000000070f0f MCGSTATUS 4
>
> I've been searching for and found some additional info, it
> looks like I'm not the first experiencing this problem:
>
> http://kerneltrap.org/node/4993
>
> Here ^^^ it is suggested that there was some thread about A64 MCEs
> on LKML but I failed to find it so I decided to post... sorry if
> it's redundant.
>
> The kernel is 2.6.16-gentoo-r9 and the CPU is an A64 "Venice" 3500+
> I can post further hw/sw environment information if req'd.

I had this same problem, search the lkml if you're interested in
investigating this further, I had the problem, was told it was
hardware, replaced my motherboard with the same model, the MCE
continued to occur, replaced the motherboard with a different model
and the MCE went away. It was some Biostar IIRC, but you can check the
archives if you really want to find out.

Good luck
-- 
avuton
--
 Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: Athlon64 + Nforce4 MCE panic

[Alan Cox]

Ar Iau, 2006-07-13 am 09:01 +0200, ysgrifennodd Rumi Szabolcs:
> # echo 'CPU 0: Machine Check Exception: 0000000000000004 Bank 4: b200000000070f0f' | mcelog --ascii --k8
> HARDWARE ERROR. This is *NOT* a software problem!
> Please contact your hardware vendor

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


It prints this bit for a reason

MCE almost always occurs because your processor detected an internal
inconsistency, in this case a parity error. So you almost certainly have
a real hardware problem.

Fixing it tends to depend on the system. With bigger servers you usually
find that the MCE info produces the correct response because the server
people actually understand use and care about MCEs even in other-os.

For random cheap desktop PC systems it can be more fun, reporting an MCE
is as likely to cause them to send you a new monitor as any other part
unfortunately 8)

RE: Athlon64 + Nforce4 MCE panic

[Allen Martin]

> CPU 0: Machine Check Exception: 0000000000000004
> Bank 4: b200000000070f0f
> Kernel panic - not syncing: CPU context corrupt
> 
> I tried to decode this:
> 
> # ./parsemce -e 0000000000000004 -b 4 -s b200000000070f0f -a 0
> Status: (4) Machine Check in progress.
> Restart IP invalid.
> parsebank(4): b200000000070f0f @ 0
>         External tag parity error
>         CPU state corrupt. Restart not possible
>         Error enabled in control register
>         Error not corrected.
>         Bus and interconnect error
>         Participation: Generic
>         Timeout:
>         Request: Generic error
>         Transaction type : Invalid
>         Memory/IO : Other

I'm not sure why parsemce says this is a parity error, it's a K8 virtual
northbridge watchdog timeout on an I/O transaction.   In other words a
CPU PIO transaction to some device timed out, the timeout is usually set
to 10s.  Prior to K8 these types of errors would usually be system
hardlocks.

I've debugged a few of these before and they usually end up being IDE
device issues.  The IDE controller is really thin (the ATA registers
actually reside in the device) so for example if the CPU goes to read
the ATA status register of some device and the device doesn't respond,
the PCI transaction won't be terminated and the watchdog will fire.

-Allen

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Re: Athlon64 + Nforce4 MCE panic

[Rumi Szabolcs]

On Thu, 13 Jul 2006 21:30:28 -0700
"Allen Martin" <AMartin@nvidia.com> wrote:

> I'm not sure why parsemce says this is a parity error, it's a K8 virtual
> northbridge watchdog timeout on an I/O transaction.   In other words a
> CPU PIO transaction to some device timed out, the timeout is usually set
> to 10s.  Prior to K8 these types of errors would usually be system
> hardlocks.
> 
> I've debugged a few of these before and they usually end up being IDE
> device issues.  The IDE controller is really thin (the ATA registers
> actually reside in the device) so for example if the CPU goes to read
> the ATA status register of some device and the device doesn't respond,
> the PCI transaction won't be terminated and the watchdog will fire.
> 
> -Allen

Hmm, this is informative, thanks a lot!

Actually what I've g00gled about the issue so far it seems that
only Nforce4 + A64 users used to get exactly the same error, some
are indeed suggesting NF4 is buggy when it comes to IDE DMA...
Isn't it possible that this is some NF4 specific problem, a chipset
bug or a motherboard design issue (some sporadic signal distortion
problem) that is specific to some but not all NF4-based motherboards?

In fact this particular system I'm having the problem with
contains the following parts:

1x Asrock NF4G-SATA2 motherboard (http://www.asrock.com/product/939NF4G-SATA2.htm)
1x Athlon64 "Venice" 3500+ with a huge Arctic cooler
1x Corsair kit of 2 matched 512MB DDR400 modules
1x Seagate 160GB SATA drive
1x well ventilated Chieftec rackmount chassis w/PSU

The former three are brand new, none of them was ever overclocked
or otherwise tortured, the latter two are reliably working since
years. The environmental temperatures were optimal at the time the
panic occurred, so it probably was not a heat related issue. In fact
the system was rather unloaded (idling around 3AM). The uptime was
about two weeks without any problem until the panic occurred.
So I have a reason to believe that this could be a chipset specific
problem which not only affects me but quite a number of NF4 users,
most of which (using Windo$$$) will probably never know why their
system suddenly hung after some weeks or months of use...

Or maybe just a neutrino hit to the CPU?
What do you think?

Regards,

Sab

RE: Athlon64 + Nforce4 MCE panic

[Allen Martin]

> 1x Asrock NF4G-SATA2 motherboard 
> (http://www.asrock.com/product/939NF4G-SATA2.htm)
> 1x Athlon64 "Venice" 3500+ with a huge Arctic cooler
> 1x Corsair kit of 2 matched 512MB DDR400 modules
> 1x Seagate 160GB SATA drive
> 1x well ventilated Chieftec rackmount chassis w/PSU

You don't have any PATA devices at all?  SATA is a lot more resilient to
this type of problem.

> So I have a reason to believe that this could be a chipset specific
> problem which not only affects me but quite a number of NF4 users,
> most of which (using Windo$$$) will probably never know why their
> system suddenly hung after some weeks or months of use...

Windows will generate a bugcheck on an MCA exception just like Linux.
We have really detailed statistics on Windows bugchecks due to OCA, so I
know this is not a widespread issue at least on Windows.


> Or maybe just a neutrino hit to the CPU?
> What do you think?

The stack trace will almost always tell you exactly what device timed
out the PIO, you should start there.
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Re: Athlon64 + Nforce4 MCE panic

[Rumi Szabolcs]

On Fri, 14 Jul 2006 11:18:38 -0700
"Allen Martin" <AMartin@nvidia.com> wrote:

> > 1x Asrock NF4G-SATA2 motherboard 
> > (http://www.asrock.com/product/939NF4G-SATA2.htm)
> > 1x Athlon64 "Venice" 3500+ with a huge Arctic cooler
> > 1x Corsair kit of 2 matched 512MB DDR400 modules
> > 1x Seagate 160GB SATA drive
> > 1x well ventilated Chieftec rackmount chassis w/PSU
> 
> You don't have any PATA devices at all?  SATA is a lot more resilient to
> this type of problem.

Exactly. I hooked up a PATA CDROM temporarily when I changed the mobo,
but otherwise the only storage it has is that SATA disk.

> > So I have a reason to believe that this could be a chipset specific
> > problem which not only affects me but quite a number of NF4 users,
> > most of which (using Windo$$$) will probably never know why their
> > system suddenly hung after some weeks or months of use...
> 
> Windows will generate a bugcheck on an MCA exception just like Linux.
> We have really detailed statistics on Windows bugchecks due to OCA, so I
> know this is not a widespread issue at least on Windows.

Do you think that this problem is caused by or at least triggered by
Linux or it's disk usage patterns? At least if you type "b200000000070f0f"
into g00gle you get a lot of hits all of which has something to do with
Linux.

> The stack trace will almost always tell you exactly what device timed
> out the PIO, you should start there.

Well I have to admit I'm not a kernel hacker but a simple user without a clue
so I have no idea how I could have got a stack trace from a kernel that is
paniced, frozen hard, and couldn't even sync it's disks so after I realized
there is no reaction to any input devices on the system I've just hit the
reset button and hoped it will come up again. I guess this is exactly what
Windo$$$ users used to do when they get a blue screen... ;)

Thanks!

Regards,

Sab