From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Ross Green <rgkernel@gmail.com>
Cc: "Peter Zijlstra" <peterz@infradead.org>,
"Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>,
"Chatre, Reinette" <reinette.chatre@intel.com>,
"Jacob Pan" <jacob.jun.pan@linux.intel.com>,
"Josh Triplett" <josh@joshtriplett.org>,
"John Stultz" <john.stultz@linaro.org>,
"Thomas Gleixner" <tglx@linutronix.de>,
lkml <linux-kernel@vger.kernel.org>,
"Ingo Molnar" <mingo@kernel.org>,
"Lai Jiangshan" <jiangshanlai@gmail.com>,
dipankar@in.ibm.com, "Andrew Morton" <akpm@linux-foundation.org>,
rostedt <rostedt@goodmis.org>,
"David Howells" <dhowells@redhat.com>,
"Eric Dumazet" <edumazet@google.com>,
"Darren Hart" <dvhart@linux.intel.com>,
"Frédéric Weisbecker" <fweisbec@gmail.com>,
"Oleg Nesterov" <oleg@redhat.com>,
"pranith kumar" <bobby.prani@gmail.com>
Subject: Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17
Date: Sat, 7 May 2016 08:25:01 -0700 [thread overview]
Message-ID: <20160507152501.GW3593@linux.vnet.ibm.com> (raw)
In-Reply-To: <CANfgCY1iqWJ+Kj7R5TwFrNDbD+1bvreQSCoCYuGB8SkksYDuFw@mail.gmail.com>
On Fri, May 06, 2016 at 04:25:16PM +1000, Ross Green wrote:
> On Sun, Apr 3, 2016 at 6:18 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Thu, Mar 31, 2016 at 08:42:55AM -0700, Paul E. McKenney wrote:
> >> On Wed, Mar 30, 2016 at 07:55:47AM -0700, Paul E. McKenney wrote:
> >> > On Tue, Mar 29, 2016 at 06:49:08AM -0700, Paul E. McKenney wrote:
> >> > > On Mon, Mar 28, 2016 at 05:28:14PM -0700, Paul E. McKenney wrote:
> >> > > > On Mon, Mar 28, 2016 at 05:25:18PM -0700, Paul E. McKenney wrote:
> >> > > > > On Mon, Mar 28, 2016 at 06:08:41AM -0700, Paul E. McKenney wrote:
> >> > > > > > On Mon, Mar 28, 2016 at 08:25:47AM +0200, Peter Zijlstra wrote:
> >> > > > > > > On Sun, Mar 27, 2016 at 02:06:41PM -0700, Paul E. McKenney wrote:
> >> > > > >
> >> > > > > [ . . . ]
> >> > > > >
> >> > > > > > > > OK, so I should instrument migration_call() if I get the repro rate up?
> >> > > > > > >
> >> > > > > > > Can do, maybe try the below first. (yes I know how long it all takes :/)
> >> > > > > >
> >> > > > > > OK, will run this today, then run calibration for last night's run this
> >> > > > > > evening.
> >> > >
> >> > > And of 18 two-hour runs, there were five failures, or about 28%.
> >> > > That said, I don't have even one significant digit on the failure rate,
> >> > > as 5 of 18 is within the 95% confidence limits for a failure probability
> >> > > as low as 12.5% and as high as 47%.
> >> >
> >> > And after last night's run, this is narrowed down to between 23% and 38%,
> >> > which is close enough. Average is 30%, 18 failures in 60 runs.
> >> >
> >> > Next step is to test Peter's patch some more. Might take a couple of
> >> > night's worth of runs to get statistical significance. After which
> >> > it will be time to rebase to 4.6-rc1.
> >>
> >> And the first night was not so good: 6 failures out of 24 runs. Adding
> >> this to the 1-of-10 earlier gets 7 failures of 34. Here are how things
> >> stack up given the range of base failure estimates:
> >>
> >> Low 95% bound of 23%: 84% confidence.
> >>
> >> Actual measurement of 30%: 92% confidence.
> >>
> >> High 95% bound of 38%: 98% confidence.
> >>
> >> So there is still some chance that Peter's patch is helping. I will
> >> run for one more evening, after which it will be time to move forward
> >> to 4.6-rc1.
> >
> > And no luck reducing bounds. However, moving to 4.6-rc1 did get some
> > of the trace_printk() to print. The ftrace_dump()s resulted in RCU
> > CPU stall warnings, and the dumps were truncated due to test timeouts
> > in my scripting. (I need to make my scripts more patient when they
> > see an ftrace dump in progress, I guess.)
> >
> > Here are the results:
> >
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.1.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.2.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.3.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.4.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.5.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.6.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.7.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.8.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.9.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.11.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.12.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.13.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.14.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.15.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.16.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.17.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.18.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.19.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.20.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.21.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.22.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.23.console.log.tgz
> > http://www2.rdrop.com/users/paulmck/submission/TREE03.24.console.log.tgz
> >
> > The config is here:
> >
> > http://www2.rdrop.com/users/paulmck/submission/config.tgz
> >
> > More runs to measure 4.6-rc1 base error rate...
> >
> > Thanx, Paul
> >
> G'day Paul,
>
>
> Thought i would update a few runs with the linux-4.6-rc kernels.
>
> I have attached log outputs through dmesg showing rcu_preempt stall warnings.
>
>
> Thought it might be interesting for someone else to look at.
>
> Currently running linux-4.6-rc6 in testing.
Thank you for sending these, I will look them over!
Still working to reproduce this quickly enough to do real debug... :-/
Thanx, Paul
next prev parent reply other threads:[~2016-05-07 15:25 UTC|newest]
Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-09 10:11 Ross Green
2016-02-17 5:45 ` Paul E. McKenney
2016-02-17 19:28 ` Paul E. McKenney
2016-02-17 19:45 ` Peter Zijlstra
2016-02-17 20:28 ` Paul E. McKenney
2016-02-17 23:19 ` Paul E. McKenney
2016-02-18 11:51 ` Ross Green
2016-02-18 23:14 ` Mathieu Desnoyers
2016-02-19 3:56 ` Ross Green
2016-02-19 4:13 ` John Stultz
2016-02-19 17:33 ` Paul E. McKenney
2016-02-20 4:34 ` Ross Green
2016-02-20 6:32 ` Paul E. McKenney
2016-02-21 5:04 ` Ross Green
2016-02-21 18:15 ` Ross Green
2016-02-23 20:34 ` Mathieu Desnoyers
2016-02-23 20:55 ` Paul E. McKenney
2016-02-23 21:28 ` Ross Green
2016-02-25 5:13 ` Ross Green
2016-02-26 0:56 ` Paul E. McKenney
2016-02-26 1:35 ` Paul E. McKenney
2016-03-04 5:30 ` Ross Green
2016-03-04 15:18 ` Paul E. McKenney
2016-03-18 21:00 ` Josh Triplett
2016-03-18 23:56 ` Paul E. McKenney
2016-03-21 16:22 ` Jacob Pan
2016-03-21 17:26 ` Paul E. McKenney
2016-03-22 16:35 ` Chatre, Reinette
2016-03-22 17:40 ` Paul E. McKenney
2016-03-22 21:04 ` Chatre, Reinette
2016-03-22 21:19 ` Paul E. McKenney
2016-03-23 17:15 ` Chatre, Reinette
2016-03-23 18:20 ` Paul E. McKenney
2016-03-23 18:25 ` Chatre, Reinette
2016-03-23 19:50 ` Paul E. McKenney
2016-03-25 21:24 ` Chatre, Reinette
2016-03-25 21:46 ` Paul E. McKenney
2016-03-26 12:29 ` Mathieu Desnoyers
2016-03-26 15:28 ` Paul E. McKenney
2016-03-26 18:49 ` Paul E. McKenney
2016-03-26 22:22 ` Mathieu Desnoyers
2016-03-27 1:34 ` Paul E. McKenney
2016-03-27 13:48 ` Mathieu Desnoyers
2016-03-27 15:40 ` Paul E. McKenney
2016-03-27 20:00 ` Paul E. McKenney
2016-03-27 20:45 ` Peter Zijlstra
2016-03-27 21:06 ` Paul E. McKenney
2016-03-28 6:25 ` Peter Zijlstra
2016-03-28 13:08 ` Paul E. McKenney
2016-03-29 0:25 ` Paul E. McKenney
2016-03-29 0:28 ` Paul E. McKenney
2016-03-29 13:49 ` Paul E. McKenney
2016-03-30 14:55 ` Paul E. McKenney
2016-03-31 15:42 ` Paul E. McKenney
2016-04-03 8:18 ` Paul E. McKenney
2016-05-06 6:25 ` Ross Green
2016-05-07 15:25 ` Paul E. McKenney [this message]
2016-05-10 2:36 ` Ross Green
2016-06-30 17:52 ` Paul E. McKenney
2016-03-28 1:44 ` Mathieu Desnoyers
2016-03-28 2:23 ` Mathieu Desnoyers
2016-03-28 6:13 ` Peter Zijlstra
2016-03-28 13:50 ` Paul E. McKenney
2016-03-28 14:15 ` Mathieu Desnoyers
2016-03-27 20:53 ` Peter Zijlstra
2016-03-27 21:07 ` Paul E. McKenney
2016-03-27 20:54 ` Peter Zijlstra
2016-03-27 21:09 ` Paul E. McKenney
2016-03-28 6:28 ` Peter Zijlstra
2016-03-28 13:29 ` Paul E. McKenney
2016-03-28 15:07 ` Mathieu Desnoyers
2016-03-28 15:56 ` Paul E. McKenney
2016-03-28 16:12 ` Mathieu Desnoyers
2016-03-28 16:29 ` Paul E. McKenney
2016-03-30 12:58 ` Boqun Feng
2016-03-30 13:30 ` Paul E. McKenney
2016-03-30 14:15 ` Boqun Feng
2016-02-19 4:22 ` Paul E. McKenney
2016-02-19 5:59 ` Ross Green
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160507152501.GW3593@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=akpm@linux-foundation.org \
--cc=bobby.prani@gmail.com \
--cc=dhowells@redhat.com \
--cc=dipankar@in.ibm.com \
--cc=dvhart@linux.intel.com \
--cc=edumazet@google.com \
--cc=fweisbec@gmail.com \
--cc=jacob.jun.pan@linux.intel.com \
--cc=jiangshanlai@gmail.com \
--cc=john.stultz@linaro.org \
--cc=josh@joshtriplett.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mingo@kernel.org \
--cc=oleg@redhat.com \
--cc=peterz@infradead.org \
--cc=reinette.chatre@intel.com \
--cc=rgkernel@gmail.com \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome