Re: PROBLEM: All CPUs in soft lockup

[Robert Norris <robn@opera.com>]

On Wed, Mar 27, 2013 at 12:55:41PM +1100, Robert Norris wrote:
> The console shows a new "BUG: soft lockup" line every few seconds

Looking closer, the whole thing starts with a _hard_ lockup.

  2013-03-26T08:33:39.921834-04:00 imap30 kernel: [185090.090328] Watchdog detected hard LOCKUP on cpu 3

(also in the logs of the other two servers I mentioned).

Looking down to where the watchdog interrupt comes in:

  2013-03-26T08:33:39.921870-04:00 imap30 kernel: [185090.090426] <<EOE>>  <IRQ>  [<ffffffff8112a5b1>] ? end_buffer_async_read+0x79/0xff

Disassembling:

  0xffffffff8112a57a <+66>:    mov    %rbx,%rdi
  0xffffffff8112a57d <+69>:    callq  0xffffffff81129265 <buffer_io_error>
  0xffffffff8112a582 <+74>:    lock orb $0x2,0x0(%rbp)
  0xffffffff8112a587 <+79>:    mov    0x0(%rbp),%rax
  0xffffffff8112a58b <+83>:    test   $0x8,%ah
  0xffffffff8112a58e <+86>:    jne    0xffffffff8112a594 <end_buffer_async_read+92>
  0xffffffff8112a590 <+88>:    ud2    
  0xffffffff8112a592 <+90>:    jmp    0xffffffff8112a592 <end_buffer_async_read+90>

That lock at +74 is presumably the offender here. Which is line 275 of fs/buffer.c:

  275                 SetPageError(page);

So another CPU has these page flags locked right now, and isn't keen to
release that lock?

I don't know how to debug this further. What's the next step?

Thanks,
Rob.