Lockup/reboots on multiple dual core Opteron systems

Lockup/reboots on multiple dual core Opteron systems

[Michael Madore]

Hi,

I have several dozen systems based on the AMD 8111/8131 chipset which
are experiencing lockups or reboots while under heavy stress.  All
systems are configured identically:

(2) dual core Opteron 285 processors
8 GB RAM
1 IDE hard disk

When the lockup occurs, there is no information printed to the screen,
and magic sysrq does not work.   The systems will often lockup within
a matter of hours, but sometimes they can run for several days before
locking up.

The testing involves:

1) Allocate several GB of memory from each node and read/write to it.
2) Perform lots of disk I/O
3) Keep all cores busy

I see the lockup no matter which kernel I try:

2.6.5 (SLES 9)
2.6.9 (RHEL 4)
2.6.16 (kernel.org)

The systems will run perfectly stable under the following conditions:

1) Use a 32-bit kernel
or
2) Boot 64-bit kernel with mem=3000M
or
3) Don't perform any disk I/O during the test

I have also tested the dual core 8GB combination on both NForce and
Serverworks based motherboards with the same results.  In this case
both systems were using SATA hard disks instead of IDE.

Any ideas what the culprit could be?

Mike

Re: Lockup/reboots on multiple dual core Opteron systems

[David Lang]

On Fri, 14 Apr 2006, Michael Madore wrote:

> Hi,
>
> I have several dozen systems based on the AMD 8111/8131 chipset which
> are experiencing lockups or reboots while under heavy stress.  All
> systems are configured identically:
>
> (2) dual core Opteron 285 processors
> 8 GB RAM
> 1 IDE hard disk
>
> When the lockup occurs, there is no information printed to the screen,
> and magic sysrq does not work.   The systems will often lockup within
> a matter of hours, but sometimes they can run for several days before
> locking up.
>
> The testing involves:
>
> 1) Allocate several GB of memory from each node and read/write to it.
> 2) Perform lots of disk I/O
> 3) Keep all cores busy
>
> I see the lockup no matter which kernel I try:
>
> 2.6.5 (SLES 9)
> 2.6.9 (RHEL 4)
> 2.6.16 (kernel.org)
>
> The systems will run perfectly stable under the following conditions:
>
> 1) Use a 32-bit kernel
> or
> 2) Boot 64-bit kernel with mem=3000M
> or
> 3) Don't perform any disk I/O during the test
>
> I have also tested the dual core 8GB combination on both NForce and
> Serverworks based motherboards with the same results.  In this case
> both systems were using SATA hard disks instead of IDE.
>
> Any ideas what the culprit could be?
>
> Mike

I'm fighting similar problems on one machine at home (single dual core, 4G 
ram nforce 939 motherboard, 1 ide, 10 3ware, 1 adaptec controlled drives). 
It's locked up under the ubuntu and gentoo install disk kernels. I did 
have it running for a day and a half under 2.6.17-rc1, but I managed to 
corrupt the install and haven't gotten back to that kernel yet to confirm 
it's long-term stability.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: Lockup/reboots on multiple dual core Opteron systems

[Michael Madore]

On 4/14/06, David Lang <dlang@digitalinsight.com> wrote:
> On Fri, 14 Apr 2006, Michael Madore wrote:
>
> > Hi,
> >
> > I have several dozen systems based on the AMD 8111/8131 chipset which
> > are experiencing lockups or reboots while under heavy stress.  All
> > systems are configured identically:
> >
> > (2) dual core Opteron 285 processors
> > 8 GB RAM
> > 1 IDE hard disk
> >
> > When the lockup occurs, there is no information printed to the screen,
> > and magic sysrq does not work.   The systems will often lockup within
> > a matter of hours, but sometimes they can run for several days before
> > locking up.
> >
> > The testing involves:
> >
> > 1) Allocate several GB of memory from each node and read/write to it.
> > 2) Perform lots of disk I/O
> > 3) Keep all cores busy
> >
> > I see the lockup no matter which kernel I try:
> >
> > 2.6.5 (SLES 9)
> > 2.6.9 (RHEL 4)
> > 2.6.16 (kernel.org)
> >
> > The systems will run perfectly stable under the following conditions:
> >
> > 1) Use a 32-bit kernel
> > or
> > 2) Boot 64-bit kernel with mem=3000M
> > or
> > 3) Don't perform any disk I/O during the test
> >
> > I have also tested the dual core 8GB combination on both NForce and
> > Serverworks based motherboards with the same results.  In this case
> > both systems were using SATA hard disks instead of IDE.
> >
> > Any ideas what the culprit could be?
> >
> > Mike
>
> I'm fighting similar problems on one machine at home (single dual core, 4G
> ram nforce 939 motherboard, 1 ide, 10 3ware, 1 adaptec controlled drives).
> It's locked up under the ubuntu and gentoo install disk kernels. I did
> have it running for a day and a half under 2.6.17-rc1, but I managed to
> corrupt the install and haven't gotten back to that kernel yet to confirm
> it's long-term stability.

Who makes your motherboard?  Also, how do you trigger the lockup?

Mike

Re: Lockup/reboots on multiple dual core Opteron systems

[David Lang]

On Fri, 14 Apr 2006, Michael Madore wrote:

>>
>> I'm fighting similar problems on one machine at home (single dual core, 4G
>> ram nforce 939 motherboard, 1 ide, 10 3ware, 1 adaptec controlled drives).
        ^^^^^^^^^^^^^^^^^^^^^^

>> It's locked up under the ubuntu and gentoo install disk kernels. I did
>> have it running for a day and a half under 2.6.17-rc1, but I managed to
>> corrupt the install and haven't gotten back to that kernel yet to confirm
>> it's long-term stability.
>
> Who makes your motherboard?  Also, how do you trigger the lockup?

I've triggered the lockups by doing large compile sessions, but the lockup 
seems to happen much quicker if I'm running under X (more memory 
allocated)

David Lang

> Mike
>

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare