mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Petr Mladek <pmladek@suse.com>
To: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	Dmitry Vyukov <dvyukov@google.com>,
	paulmck@kernel.org, Christian Brauner <christian@brauner.io>,
	prsood@codeaurora.org, avagin@gmail.com,
	syzkaller-bugs <syzkaller-bugs@googlegroups.com>,
	Tejun Heo <tj@kernel.org>, Thomas Gleixner <tglx@linutronix.de>,
	Andrew Morton <akpm@linux-foundation.org>,
	Andrea Arcangeli <aarcange@redhat.com>,
	Oleg Nesterov <oleg@redhat.com>,
	dbueso@suse.de,
	syzbot <syzbot+18379f2a19bc62c12565@syzkaller.appspotmail.com>,
	LKML <linux-kernel@vger.kernel.org>,
	John Ogness <john.ogness@linutronix.de>
Subject: Re: INFO: rcu detected stall in sys_exit_group
Date: Tue, 24 Sep 2019 16:02:41 +0200	[thread overview]
Message-ID: <20190924140241.be77u2jne3melzte@pathway.suse.cz> (raw)
In-Reply-To: <46269171-ff57-8afd-f36b-40e6519b6557@i-love.sakura.ne.jp>

On Fri 2019-09-20 19:22:10, Tetsuo Handa wrote:
> Calling printk() people.
> 
> On 2019/09/20 16:50, Dmitry Vyukov wrote:
> >>>                                   How it runs on top of an interrupt?
> >>
> >> It is not running on top of an interrupt.  Its stack was dumped
> >> separately.
> > 
> > I see. Usually the first stack is the traceback of the current stack.
> > So I was confused.
> > 
> >>> And why one cpu tracebacks another one?
> >>
> >> The usual reason is because neither CPU's quiescent state was reported
> >> to the RCU core, so the stall-warning code dumped both stacks.
> > 
> > But should the other CPU traceback _itself_? Rather than being traced
> > back by another CPU?
> > E.g. see this report:
> > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/350#L61-L83
> > Here the overall problem was detected by C2, but then C1 traces back itself.
> > 
> > ... however even in that case C0 and C3 are traced by C2:
> > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/350#L84-L149
> > I can't understand this...
> > This makes understanding what happened harder because it's not easy to
> > exclude things on other CPUs.
> 
> I think this should be
> https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/350#L84-L172
> than #L84-L149 .
> 
> Is the reason these lines have "[    C2]" is that these lines were flushed (printk_caller_id()
> was called) from log_output() from vprintk_store() from vprintk_emit() from vprintk_deferred()
>  from printk_deferred() from printk_safe_flush_line() from __printk_safe_flush() from
> printk_safe_flush() from printk_safe_flush_on_panic() from panic() ?

It seems to be the case. CPU2 is clearly flushing per-CPU buffers
from NMI context, for example:

[ 1098.703114][    C2] NMI backtrace for cpu 0
[...]
[ 1098.703295][    C2] NMI backtrace for cpu 3

A solution would be to store all these metadata (timestamp, caller
info) already into the per-CPU buffers. I think that it would be
doable.

But much better solution is a lockless ring buffer. John Ogness is
working hard on it. The plan is to have it ready for 5.5 or 5.6.
I would prefer to concentrate on this solution for the moment.

Best Regards,
Petr

  reply	other threads:[~2019-09-24 14:02 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-18 11:19 syzbot
2019-09-18 15:05 ` Dmitry Vyukov
2019-09-19 17:07   ` Paul E. McKenney
2019-09-19 17:39     ` Dmitry Vyukov
2019-09-19 20:12       ` Paul E. McKenney
2019-09-20  7:50         ` Dmitry Vyukov
2019-09-20  7:54           ` Dmitry Vyukov
2019-09-20  8:03             ` Dmitry Vyukov
2019-09-20  8:21               ` Dmitry Vyukov
2019-09-20 13:50                 ` Paul E. McKenney
2019-09-20 10:22           ` Tetsuo Handa
2019-09-24 14:02             ` Petr Mladek [this message]
2019-09-20 13:36           ` Paul E. McKenney

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190924140241.be77u2jne3melzte@pathway.suse.cz \
    --to=pmladek@suse.com \
    --cc=aarcange@redhat.com \
    --cc=akpm@linux-foundation.org \
    --cc=avagin@gmail.com \
    --cc=christian@brauner.io \
    --cc=dbueso@suse.de \
    --cc=dvyukov@google.com \
    --cc=john.ogness@linutronix.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=oleg@redhat.com \
    --cc=paulmck@kernel.org \
    --cc=penguin-kernel@i-love.sakura.ne.jp \
    --cc=prsood@codeaurora.org \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=syzbot+18379f2a19bc62c12565@syzkaller.appspotmail.com \
    --cc=syzkaller-bugs@googlegroups.com \
    --cc=tglx@linutronix.de \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®