Re: rcu_bh stalls on 3.2.28

[Henrique de Moraes Holschuh <hmh@hmh.eng.br>]

On Mon, 03 Sep 2012, Michael Wang wrote:
> On 09/01/2012 07:02 AM, Henrique de Moraes Holschuh wrote:
> > Just got one of these:
> > 
> > kernel: INFO: rcu_bh detected stall on CPU 2 (t=0 jiffies)
> > kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.28+ #2
> > kernel: Call Trace:
> > kernel: <IRQ>  [<ffffffff810d1609>] __rcu_pending+0x159/0x400
> > kernel: [<ffffffff810d20bb>] rcu_check_callbacks+0x9b/0x120
> > kernel: [<ffffffff81089673>] update_process_times+0x43/0x80
> > kernel: [<ffffffff810a836f>] tick_sched_timer+0x5f/0xb0
> > kernel: [<ffffffff8109c097>] __run_hrtimer.isra.30+0x57/0x100
> > kernel: [<ffffffff8109c8f5>] hrtimer_interrupt+0xe5/0x220
> > kernel: [<ffffffff8104ce14>] smp_apic_timer_interrupt+0x64/0xa0
> > kernel: [<ffffffff8159b5cb>] apic_timer_interrupt+0x6b/0x70
> > kernel: <EOI>  [<ffffffff81315645>] ? intel_idle+0xe5/0x140
> > kernel: [<ffffffff81315623>] ? intel_idle+0xc3/0x140
> > kernel: [<ffffffff814420ee>] cpuidle_idle_call+0x8e/0xf0
> > kernel: [<ffffffff81032425>] cpu_idle+0xa5/0x110
> > kernel: [<ffffffff8158a9ac>] start_secondary+0x1e5/0x1ec
> 
> Hi, Henrique
> 
> rsp->gp_start and rsp->jiffies_stall should already set before we start
> check stall for this gp, but the INFO show that we have a current
> jiffies which bigger then rsp->jiffies_stall but equal to rsp->gp_start,
> really strange...
> 
> Could you please have a try on the latest kernel and confirm whether
> this issue still exist?

It is a production box, it is difficult to run a -rc kernel there.

And the stalls are very rare, too.  That's the only one I got, so at this
point I cannot tell you whether something fixed the problem or not, just try
to give you clues if a stall does happen.

> BTW:
> 	Is this stall info comes from a virtual machine?

No, it runs on baremetal.  The box has one Xeon X5550 processor, 4 cores, 8
threads, and it is allowed to go into C1, C3 and C6 (which it does very very
often).

It might be some sort of race related to SMIs?  The worst-case SMM-induced
delay on this box is quite high (I don't recall if that means 150ms or
150us), as measured by the Intel BITS[1].

[1] http://biosbits.org/

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh