Re: corruption causing crash in __queue_work

[Tejun Heo <tj@kernel.org>]

Hello, Nikolay.

On Wed, Dec 09, 2015 at 02:08:56PM +0200, Nikolay Borisov wrote:
> 73309.529940] BUG: unable to handle kernel NULL pointer dereference at           (null)
> [73309.530238] IP: [<ffffffff8106b663>] __queue_work+0xb3/0x390
...
> [73309.537319]  <IRQ> 
> [73309.537373]  [<ffffffff8106b940>] ? __queue_work+0x390/0x390
> [73309.537714]  [<ffffffff8106b958>] delayed_work_timer_fn+0x18/0x20
> [73309.537891]  [<ffffffff810ad1d7>] call_timer_fn+0x47/0x110
> [73309.538071]  [<ffffffff810be302>] ? tick_sched_timer+0x52/0xa0
> [73309.538249]  [<ffffffff810adb6f>] run_timer_softirq+0x17f/0x2b0
> [73309.538425]  [<ffffffff8106b940>] ? __queue_work+0x390/0x390
> [73309.538604]  [<ffffffff81057f40>] __do_softirq+0xe0/0x290
> [73309.538778]  [<ffffffff810581e6>] irq_exit+0xa6/0xb0
> [73309.538952]  [<ffffffff8159413a>] smp_apic_timer_interrupt+0x4a/0x59
> [73309.539128]  [<ffffffff815926bb>] apic_timer_interrupt+0x6b/0x70
...
> The gist is that this fail on the following line: 
> 
> if (last_pool && last_pool != pwq->pool) {

That's new.

> Since the pointer 'pwq' is wrong (it is loaded in %rdx) which in this 
> case is 0000000000000000. Looking at the function's source pwq should 
> be loaded by per_cpu_ptr since the  if (!(wq->flags & WQ_UNBOUND)) 
> check should evaluate to false. So pwq is loaded as the result from 
> unbound_pwq_by_node(wq, cpu_to_node(cpu));
> 
> Here are the flags of the workqueue: 
> crash> struct workqueue_struct.flags 0xffff8803df464c00
>   flags = 131082

That's ordered unbound workqueue w/ a rescuer.

> (0xffff8803df464c00 is indeed the pointer to the workqueue struct, 
> so the flags aren't bogus).
> 
> So reading the numa_pwq_tbl it seems that it's uninitialised: 
> 
> crash> struct workqueue_struct.numa_pwq_tbl 0xffff8803df464c00
>   numa_pwq_tbl = 0xffff8803df464d10
> crash> rd -64 0xffff8803df464d10 3
> ffff8803df464d10:  0000000000000000 0000000000000000   ................
> ffff8803df464d20:  0000000000000000                    ........
> 
> The machine where the crash occurred has a single NUMA node, so at the 
> very least I would have expected to have a pointer, rather than NULL ptr. 
> 
> Also this crash is not isolated in that I have observed it on multiple
> other nodes running vanilla 4.2.5/4.2.6 kernels. 
> 
> Any advice how to further debug that?

Adding printk or tracepoints at numa_pwq_tbl_install() to dump what's
being installed would be helpful.  It should at least tell us whether
it's the table being corrupted by something else or workqueue failing
to set it up correctly to begin with.  How reproducible is the
problem?

Thanks.

-- 
tejun