From: Petr Mladek <pmladek@suse.com>
To: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
Dmitry Vyukov <dvyukov@google.com>,
paulmck@kernel.org, Christian Brauner <christian@brauner.io>,
prsood@codeaurora.org, avagin@gmail.com,
syzkaller-bugs <syzkaller-bugs@googlegroups.com>,
Tejun Heo <tj@kernel.org>, Thomas Gleixner <tglx@linutronix.de>,
Andrew Morton <akpm@linux-foundation.org>,
Andrea Arcangeli <aarcange@redhat.com>,
Oleg Nesterov <oleg@redhat.com>,
dbueso@suse.de,
syzbot <syzbot+18379f2a19bc62c12565@syzkaller.appspotmail.com>,
LKML <linux-kernel@vger.kernel.org>,
John Ogness <john.ogness@linutronix.de>
Subject: Re: INFO: rcu detected stall in sys_exit_group
Date: Tue, 24 Sep 2019 16:02:41 +0200 [thread overview]
Message-ID: <20190924140241.be77u2jne3melzte@pathway.suse.cz> (raw)
In-Reply-To: <46269171-ff57-8afd-f36b-40e6519b6557@i-love.sakura.ne.jp>
On Fri 2019-09-20 19:22:10, Tetsuo Handa wrote:
> Calling printk() people.
>
> On 2019/09/20 16:50, Dmitry Vyukov wrote:
> >>> How it runs on top of an interrupt?
> >>
> >> It is not running on top of an interrupt. Its stack was dumped
> >> separately.
> >
> > I see. Usually the first stack is the traceback of the current stack.
> > So I was confused.
> >
> >>> And why one cpu tracebacks another one?
> >>
> >> The usual reason is because neither CPU's quiescent state was reported
> >> to the RCU core, so the stall-warning code dumped both stacks.
> >
> > But should the other CPU traceback _itself_? Rather than being traced
> > back by another CPU?
> > E.g. see this report:
> > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/350#L61-L83
> > Here the overall problem was detected by C2, but then C1 traces back itself.
> >
> > ... however even in that case C0 and C3 are traced by C2:
> > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/350#L84-L149
> > I can't understand this...
> > This makes understanding what happened harder because it's not easy to
> > exclude things on other CPUs.
>
> I think this should be
> https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/350#L84-L172
> than #L84-L149 .
>
> Is the reason these lines have "[ C2]" is that these lines were flushed (printk_caller_id()
> was called) from log_output() from vprintk_store() from vprintk_emit() from vprintk_deferred()
> from printk_deferred() from printk_safe_flush_line() from __printk_safe_flush() from
> printk_safe_flush() from printk_safe_flush_on_panic() from panic() ?
It seems to be the case. CPU2 is clearly flushing per-CPU buffers
from NMI context, for example:
[ 1098.703114][ C2] NMI backtrace for cpu 0
[...]
[ 1098.703295][ C2] NMI backtrace for cpu 3
A solution would be to store all these metadata (timestamp, caller
info) already into the per-CPU buffers. I think that it would be
doable.
But much better solution is a lockless ring buffer. John Ogness is
working hard on it. The plan is to have it ready for 5.5 or 5.6.
I would prefer to concentrate on this solution for the moment.
Best Regards,
Petr
next prev parent reply other threads:[~2019-09-24 14:02 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-18 11:19 syzbot
2019-09-18 15:05 ` Dmitry Vyukov
2019-09-19 17:07 ` Paul E. McKenney
2019-09-19 17:39 ` Dmitry Vyukov
2019-09-19 20:12 ` Paul E. McKenney
2019-09-20 7:50 ` Dmitry Vyukov
2019-09-20 7:54 ` Dmitry Vyukov
2019-09-20 8:03 ` Dmitry Vyukov
2019-09-20 8:21 ` Dmitry Vyukov
2019-09-20 13:50 ` Paul E. McKenney
2019-09-20 10:22 ` Tetsuo Handa
2019-09-24 14:02 ` Petr Mladek [this message]
2019-09-20 13:36 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190924140241.be77u2jne3melzte@pathway.suse.cz \
--to=pmladek@suse.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=avagin@gmail.com \
--cc=christian@brauner.io \
--cc=dbueso@suse.de \
--cc=dvyukov@google.com \
--cc=john.ogness@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=oleg@redhat.com \
--cc=paulmck@kernel.org \
--cc=penguin-kernel@i-love.sakura.ne.jp \
--cc=prsood@codeaurora.org \
--cc=sergey.senozhatsky@gmail.com \
--cc=syzbot+18379f2a19bc62c12565@syzkaller.appspotmail.com \
--cc=syzkaller-bugs@googlegroups.com \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®