Re: BUG: soft lockup in 2.6.25.5

[Brett Pemberton <brett@vpac.org>]

----- "Andrew Morton" <akpm@linux-foundation.org> wrote:

> On Fri, 19 Sep 2008 13:49:16 +1000 Brett Pemberton <brett@vpac.org>
> wrote:
> 
> > I'm getting about 3-5 machines in a cluster of 95 hanging with 
> > 
> > BUG: soft lockup - CPU#7 stuck for 61s! [pdflush:321]
> > 
> > per week.  Nothing in common each time, different users running
> > different jobs on different nodes.
> > 
> > The most recent is at the end of this email, .config is attached.
> > 
> > Googling is scary.  Many people reporting these, but never any
> response.
> > It's happening on enough separate nodes that I can't believe it's
> > hardware, although they are identical machines:
> > 
> > - 2x Quad-Core AMD Opteron(tm) Processor 2356
> > - 32gb ram
> > - 4 x sata drives
> > 
> > Running CentOS 5.2 with a kernel.org kernel
> > Has been happening with a variety of kernels from 2.6.25 - present.
> 
> Yes, it's a false positive.  With a lot of memory and a random-access
> or lot-of-files writing behaviour, it can take tremendous amounts of
> time to get everything stored on the disk.
> 
> Not sure what to do about it, really.  Perhaps touch the softlockup
> detector somewhere in the writeback code.
> 
> > I'd love any advice on where to turn to next and what avenues to
> pursue.
> 
> Set /proc/sys/kernel/softlockup_thresh to zero to shut it up :(
> 

Hmm,

I'd love to believe it's a false positive, but I guess I didn't mention that
once a machine gets one of these, it incrementally increases load until it
falls over a few (hours, days) later.

When I've noticed, and have been able to log in and run top before it falls over,
I can see that one or two cores have their processes stuck in a wait state, while
the others continue as normal.

This is consistent.  I've never had a node hit this BUG: and not eventually
die (lose all network connectivity, sit at console waiting for login but
not registering keystrokes) within a week.  The node today hit lockup around
2 hours after registering this BUG:

Surely turning off the detection via the proc file will just mean this will
happen silently in the future?

cheers,

     / Brett