Re: [tip:sched/core] sched/numa: Move task_numa_free() to __put_task_struct()

[Mike Galbraith <bitbucket@online.de>]

On Sun, 2014-04-06 at 15:17 -0400, Sasha Levin wrote: 
> On 03/11/2014 08:40 AM, tip-bot for Mike Galbraith wrote:
> > Commit-ID:  156654f491dd8d52687a5fbe1637f472a52ce75b
> > Gitweb:     http://git.kernel.org/tip/156654f491dd8d52687a5fbe1637f472a52ce75b
> > Author:     Mike Galbraith <bitbucket@online.de>
> > AuthorDate: Fri, 28 Feb 2014 07:23:11 +0100
> > Committer:  Ingo Molnar <mingo@kernel.org>
> > CommitDate: Tue, 11 Mar 2014 12:05:43 +0100
> > 
> > sched/numa: Move task_numa_free() to __put_task_struct()
> > 
> > Bad idea on -rt:
> > 
> > [  908.026136]  [<ffffffff8150ad6a>] rt_spin_lock_slowlock+0xaa/0x2c0
> > [  908.026145]  [<ffffffff8108f701>] task_numa_free+0x31/0x130
> > [  908.026151]  [<ffffffff8108121e>] finish_task_switch+0xce/0x100
> > [  908.026156]  [<ffffffff81509c0a>] thread_return+0x48/0x4ae
> > [  908.026160]  [<ffffffff8150a095>] schedule+0x25/0xa0
> > [  908.026163]  [<ffffffff8150ad95>] rt_spin_lock_slowlock+0xd5/0x2c0
> > [  908.026170]  [<ffffffff810658cf>] get_signal_to_deliver+0xaf/0x680
> > [  908.026175]  [<ffffffff8100242d>] do_signal+0x3d/0x5b0
> > [  908.026179]  [<ffffffff81002a30>] do_notify_resume+0x90/0xe0
> > [  908.026186]  [<ffffffff81513176>] int_signal+0x12/0x17
> > [  908.026193]  [<00007ff2a388b1d0>] 0x7ff2a388b1cf
> > 
> > and since upstream does not mind where we do this, be a bit nicer ...
> > 
> > Signed-off-by: Mike Galbraith <bitbucket@online.de>
> > Signed-off-by: Peter Zijlstra <peterz@infradead.org>
> > Cc: Mel Gorman <mgorman@suse.com>
> > Cc: Linus Torvalds <torvalds@linux-foundation.org>
> > Cc: Andrew Morton <akpm@linux-foundation.org>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Link: http://lkml.kernel.org/r/1393568591.6018.27.camel@marge.simpson.net
> > Signed-off-by: Ingo Molnar <mingo@kernel.org>
> 
> As it seems, upstream does mind:
> 
> [ 2590.260734] ======================================================
> [ 2590.261695] [ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]
> [ 2590.262748] 3.14.0-next-20140403-sasha-00022-g10224c0 #377 Tainted: G        W
> [ 2590.263846] ------------------------------------------------------
> [ 2590.264730] trinity-c244/1210 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
> [ 2590.265783] (&(&grp->lock)->rlock){+.+...}, at: task_numa_free (kernel/sched/fair.c:1714)
> [ 2590.267179]
> [ 2590.267179] and this task is already holding:
> [ 2590.267996] (&(&new_timer->it_lock)->rlock){-.....}, at: exit_itimers (kernel/posix-timers.c:971 kernel/posix-timers.c:998)
> [ 2590.269381] which would create a new lock dependency:
> [ 2590.270067]  (&(&new_timer->it_lock)->rlock){-.....} -> (&(&grp->lock)->rlock){+.+...}

I'm not getting it.

I moved task_numa_free() from one interrupts enabled spot to another.
But, with numa=fake=4 and lockdep enabled, not only does lockdep gripe,
my little box locks up on splat.  Saying spin_lock/unlock_irq() did the
expected, just moved lockdep gripe to task_numa_fault().


> [ 2590.270067]  Possible interrupt unsafe locking scenario:
> [ 2590.270067]
> [ 2590.270067]        CPU0                    CPU1
> [ 2590.270067]        ----                    ----
> [ 2590.270067]   lock(&(&grp->lock)->rlock);
> [ 2590.270067]                                local_irq_disable();
> [ 2590.270067]                                lock(&(&new_timer->it_lock)->rlock);
> [ 2590.270067]                                lock(&(&grp->lock)->rlock);
> [ 2590.270067]   <Interrupt>
> [ 2590.270067]     lock(&(&new_timer->it_lock)->rlock);
> [ 2590.270067]
> [ 2590.270067]  *** DEADLOCK ***

Ok, so how did I manage that HARDIRQ-safe -> HARDIRQ-unsafe?

-Mike