mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Steven Rostedt <rostedt@goodmis.org>
To: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Neeraj Upadhyay <neeraju@codeaurora.org>,
	josh@joshtriplett.org, mathieu.desnoyers@efficios.com,
	jiangshanlai@gmail.com, linux-kernel@vger.kernel.org,
	sramana@codeaurora.org, prsood@codeaurora.org,
	pkondeti@codeaurora.org, markivx@codeaurora.org,
	peterz@infradead.org, byungchul.park@lge.com
Subject: Re: Query regarding synchronize_sched_expedited and resched_cpu
Date: Mon, 18 Sep 2017 21:23:01 -0400	[thread overview]
Message-ID: <20170918212301.417c8b36@gandalf.local.home> (raw)
In-Reply-To: <20170918235311.GA20177@linux.vnet.ibm.com>

On Mon, 18 Sep 2017 16:53:11 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> On Mon, Sep 18, 2017 at 09:55:27AM -0700, Paul E. McKenney wrote:
> > On Mon, Sep 18, 2017 at 12:29:31PM -0400, Steven Rostedt wrote:  
> > > On Mon, 18 Sep 2017 09:24:12 -0700
> > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > 
> > >   
> > > > As soon as I work through the backlog of lockdep complaints that
> > > > appeared in the last merge window...  :-(
> > > > 
> > > > sparse_irq_lock, I am looking at you!!!  ;-)  
> > > 
> > > I just hit one too, and decided to write a patch to show a chain of 3
> > > when applicable.
> > > 
> > > For example:
> > > 
> > >  Chain exists of:
> > >    cpu_hotplug_lock.rw_sem --> smpboot_threads_lock --> (complete)&self->parked
> > > 
> > >   Possible unsafe locking scenario by crosslock:
> > > 
> > >         CPU0                    CPU1                    CPU2
> > >         ----                    ----                    ----
> > >    lock(smpboot_threads_lock);
> > >    lock((complete)&self->parked);
> > >                                 lock(cpu_hotplug_lock.rw_sem);
> > >                                 lock(smpboot_threads_lock);
> > >                                                        lock(cpu_hotplug_lock.rw_sem);
> > >                                                        unlock((complete)&self->parked);
> > > 
> > >   *** DEADLOCK ***
> > > 
> > > :-)  
> > 
> > Nice!!!
> > 

Note, the above lockdep splat does discover a bug.

> > My next step is reverting 12ac1d0f6c3e ("genirq: Make sparse_irq_lock
> > protect what it should protect") to see if that helps.  
> 
> No joy, but it is amazing how much nicer "git bisect" is when your
> failure happens deterministically within 35 seconds.  ;-)
> 
> The bisection converged to the range starting with 7a46ec0e2f48
> ("locking/refcounts, x86/asm: Implement fast refcount overflow
> protection") and ending with 0c2364791343 ("Merge branch 'x86/asm'
> into locking/core").  All of these failed with an unrelated build
> error, but there was a fix that could be merged.  This flagged
> d0541b0fa64b ("locking/lockdep: Make CONFIG_LOCKDEP_CROSSRELEASE part
> of CONFIG_PROVE_LOCKING"), which unfortunately does not revert cleanly.
> However, the effect of a reversion can be obtained by removing the
> selects of LOCKDEP_CROSSRELEASE and LOCKDEP_COMPLETE from
> PROVE_LOCKING, which allows recent commits to complete a short
> rcutorture test successfully.

I don't think you want to remove those. It appears that lockdep now
covers completions, and it is uncovering a lot of bugs.

> 
> So, Byungchul, any enlightenment?  Please see lockdep splat below.

Did you discover the below by reverting lockdep patches? It doesn't
really make sense. It looks to me to be about completions but not
fully covering it.

-- Steve
 
> 							Thanx, Paul
> 
> ------------------------------------------------------------------------
> 
> [   35.310179] ======================================================
> [   35.310749] WARNING: possible circular locking dependency detected
> [   35.310749] 4.13.0-rc4+ #1 Not tainted
> [   35.310749] ------------------------------------------------------
> [   35.310749] torture_onoff/766 is trying to acquire lock:
> [   35.313943]  ((complete)&st->done){+.+.}, at: [<ffffffffb905f5a6>] takedown_cpu+0x86/0xf0
> [   35.313943] 
> [   35.313943] but task is already holding lock:
> [   35.313943]  (sparse_irq_lock){+.+.}, at: [<ffffffffb90c5e42>] irq_lock_sparse+0x12/0x20
> [   35.313943] 
> [   35.313943] which lock already depends on the new lock.
> [   35.313943] 
> [   35.313943] 
> [   35.313943] the existing dependency chain (in reverse order) is:
> [   35.313943] 
> [   35.313943] -> #1 (sparse_irq_lock){+.+.}:
> [   35.313943]        __mutex_lock+0x65/0x960
> [   35.313943]        mutex_lock_nested+0x16/0x20
> [   35.313943]        irq_lock_sparse+0x12/0x20
> [   35.313943]        irq_affinity_online_cpu+0x13/0xd0
> [   35.313943]        cpuhp_invoke_callback+0xa7/0x8b0
> [   35.313943] 
> [   35.313943] -> #0 ((complete)&st->done){+.+.}:
> [   35.313943]        check_prev_add+0x401/0x800
> [   35.313943]        __lock_acquire+0x1100/0x11a0
> [   35.313943]        lock_acquire+0x9e/0x1e0
> [   35.313943]        wait_for_completion+0x36/0x130
> [   35.313943]        takedown_cpu+0x86/0xf0
> [   35.313943]        cpuhp_invoke_callback+0xa7/0x8b0
> [   35.313943]        cpuhp_down_callbacks+0x3d/0x80
> [   35.313943]        _cpu_down+0xbb/0xf0
> [   35.313943]        do_cpu_down+0x39/0x50
> [   35.313943]        cpu_down+0xb/0x10
> [   35.313943]        torture_offline+0x75/0x140
> [   35.313943]        torture_onoff+0x102/0x1e0
> [   35.313943]        kthread+0x142/0x180
> [   35.313943]        ret_from_fork+0x27/0x40
> [   35.313943] 
> [   35.313943] other info that might help us debug this:
> [   35.313943] 
> [   35.313943]  Possible unsafe locking scenario:
> [   35.313943] 
> [   35.313943]        CPU0                    CPU1
> [   35.313943]        ----                    ----
> [   35.313943]   lock(sparse_irq_lock);
> [   35.313943]                                lock((complete)&st->done);
> [   35.313943]                                lock(sparse_irq_lock);
> [   35.313943]   lock((complete)&st->done);
> [   35.313943] 
> [   35.313943]  *** DEADLOCK ***
> [   35.313943] 
> [   35.313943] 3 locks held by torture_onoff/766:
> [   35.313943]  #0:  (cpu_add_remove_lock){+.+.}, at: [<ffffffffb9060be2>] do_cpu_down+0x22/0x50
> [   35.313943]  #1:  (cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffffb90acc41>] percpu_down_write+0x21/0xf0
> [   35.313943]  #2:  (sparse_irq_lock){+.+.}, at: [<ffffffffb90c5e42>] irq_lock_sparse+0x12/0x20
> [   35.313943] 
> [   35.313943] stack backtrace:
> [   35.313943] CPU: 7 PID: 766 Comm: torture_onoff Not tainted 4.13.0-rc4+ #1
> [   35.313943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
> [   35.313943] Call Trace:
> [   35.313943]  dump_stack+0x67/0x97
> [   35.313943]  print_circular_bug+0x21d/0x330
> [   35.313943]  ? add_lock_to_list.isra.31+0xc0/0xc0
> [   35.313943]  check_prev_add+0x401/0x800
> [   35.313943]  ? wake_up_q+0x70/0x70
> [   35.313943]  __lock_acquire+0x1100/0x11a0
> [   35.313943]  ? __lock_acquire+0x1100/0x11a0
> [   35.313943]  ? add_lock_to_list.isra.31+0xc0/0xc0
> [   35.313943]  lock_acquire+0x9e/0x1e0
> [   35.313943]  ? takedown_cpu+0x86/0xf0
> [   35.313943]  wait_for_completion+0x36/0x130
> [   35.313943]  ? takedown_cpu+0x86/0xf0
> [   35.313943]  ? stop_machine_cpuslocked+0xb9/0xd0
> [   35.313943]  ? cpuhp_invoke_callback+0x8b0/0x8b0
> [   35.313943]  ? cpuhp_complete_idle_dead+0x10/0x10
> [   35.313943]  takedown_cpu+0x86/0xf0
> [   35.313943]  cpuhp_invoke_callback+0xa7/0x8b0
> [   35.313943]  cpuhp_down_callbacks+0x3d/0x80
> [   35.313943]  _cpu_down+0xbb/0xf0
> [   35.313943]  do_cpu_down+0x39/0x50
> [   35.313943]  cpu_down+0xb/0x10
> [   35.313943]  torture_offline+0x75/0x140
> [   35.313943]  torture_onoff+0x102/0x1e0
> [   35.313943]  kthread+0x142/0x180
> [   35.313943]  ? torture_kthread_stopping+0x70/0x70
> [   35.313943]  ? kthread_create_on_node+0x40/0x40
> [   35.313943]  ret_from_fork+0x27/0x40

  reply	other threads:[~2017-09-19  1:23 UTC|newest]

Thread overview: 35+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-09-15 11:14 Neeraj Upadhyay
2017-09-17  1:00 ` Paul E. McKenney
2017-09-17  6:07   ` Neeraj Upadhyay
2017-09-18 15:11     ` Steven Rostedt
2017-09-18 16:01       ` Paul E. McKenney
2017-09-18 16:12         ` Steven Rostedt
2017-09-18 16:24           ` Paul E. McKenney
2017-09-18 16:29             ` Steven Rostedt
2017-09-18 16:55               ` Paul E. McKenney
2017-09-18 23:53                 ` Paul E. McKenney
2017-09-19  1:23                   ` Steven Rostedt [this message]
2017-09-19  2:26                     ` Paul E. McKenney
2017-09-19  1:50                   ` Byungchul Park
2017-09-19  2:06                     ` Byungchul Park
2017-09-19  2:33                       ` Paul E. McKenney
2017-09-19  2:48                         ` Byungchul Park
2017-09-19  4:04                           ` Paul E. McKenney
2017-09-19  5:37                             ` Boqun Feng
2017-09-19  6:11                               ` Mike Galbraith
2017-09-19  6:53                                 ` Byungchul Park
2017-09-19 13:40                                 ` Paul E. McKenney
2017-09-21 13:57                 ` Peter Zijlstra
2017-09-21 15:33                   ` Paul E. McKenney
2017-09-19  1:55               ` Byungchul Park
2017-09-19 15:31             ` Paul E. McKenney
2017-09-19 15:58               ` Steven Rostedt
2017-09-19 16:12                 ` Paul E. McKenney
2017-09-21 13:59               ` Peter Zijlstra
2017-09-21 16:00                 ` Paul E. McKenney
2017-09-21 16:30                   ` Peter Zijlstra
2017-09-21 16:47                     ` Paul E. McKenney
2017-09-21 13:55       ` Peter Zijlstra
2017-09-21 15:31         ` Paul E. McKenney
2017-09-21 16:18           ` Peter Zijlstra
2017-09-21 15:46         ` Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170918212301.417c8b36@gandalf.local.home \
    --to=rostedt@goodmis.org \
    --cc=byungchul.park@lge.com \
    --cc=jiangshanlai@gmail.com \
    --cc=josh@joshtriplett.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=markivx@codeaurora.org \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=neeraju@codeaurora.org \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=peterz@infradead.org \
    --cc=pkondeti@codeaurora.org \
    --cc=prsood@codeaurora.org \
    --cc=sramana@codeaurora.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®