From: Yinghai Lu <yinghai@kernel.org>
To: Frederic Weisbecker <fweisbec@gmail.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Ingo Molnar <mingo@elte.hu>,
linux-kernel@vger.kernel.org
Subject: Re: [GIT PULL rcu/next] rcu commits for 2.6.40
Date: Wed, 18 May 2011 14:10:26 -0700 [thread overview]
Message-ID: <4DD435C2.6040305@kernel.org> (raw)
In-Reply-To: <20110517024000.GA5026@nowhere>
On 05/16/2011 07:40 PM, Frederic Weisbecker wrote:
> On Mon, May 16, 2011 at 02:24:49PM -0700, Paul E. McKenney wrote:
>> On Mon, May 16, 2011 at 02:23:29PM +0200, Ingo Molnar wrote:
>>>
>>> * Ingo Molnar <mingo@elte.hu> wrote:
>>>
>>>>> In the meantime, would you be willing to try out the patch at
>>>>> https://lkml.org/lkml/2011/5/14/89? This patch helped out Yinghai in
>>>>> several configurations.
>>>>
>>>> Wasn't this the one i tested - or is it a new iteration?
>>>>
>>>> I'll try it in any case.
>>>
>>> oh, this was a new iteration, mea culpa!
>>>
>>> And yes, it solves all problems for me as well. Mind pushing it as a fix? :-)
>>
>> ;-)
>>
>> Unfortunately, the only reason I can see that it works is (1) there
>> is some obscure bug in my code or (2) someone somewhere is failing to
>> call irq_exit() on some interrupt-exit path. Much as I might be tempted
>> to paper this one over, I believe that we do need to find whatever the
>> underlying bug is.
>>
>> Oh, yes, there is option (3) as well: maybe if an interrupt deschedules
>> a process, the final irq_exit() is omitted in favor of rcu_enter_nohz()?
>> But I couldn't see any evidence of this in my admittedly cursory scan
>> of the x86 interrupt-handling code.
>>
>> So until I learn differently, I am assuming that each and every
>> irq_enter() has a matching call to irq_exit(), and that rcu_enter_nohz()
>> is called after the final irq_exit() of a given burst of interrupts.
>>
>> If my assumptions are mistaken, please do let me know!
>
> So it would be nice to have a trace of the calls to rcu_irq_*() / rcu_*_nohz()
> before the unpairing happened.
>
> I have tried to reproduce it but couldn't trigger anything.
>
> So it would be nice if Yinghai can test the patch below, since he was able
> to trigger the warning.
>
> This is essentially Paul's patch but with stacktrace of the calls recorded.
> Then the whole trace is dumped on the console when one of the WARN_ON_ONCE
> sanity check is positive. Beware as the trace will be dumped everytime
> WARN_ON_ONCE() is positive. So the first dump is enough, you can ignore the
> rest.
>
> This requires CONFIG_TRACING. May be a good thing to boot with
> "ftrace=nop" parameter, so that ftrace will set up a long enough buffer
> to have an interesting trace.
with this patches if the kernel is compiled from opensuse 11.3 no delay anymore, but have one warning:
[ 82.895182] ------------[ cut here ]------------
[ 82.895189] WARNING: at kernel/rcutree.c:352 rcu_enter_nohz+0x49/0x8b()
[ 82.895193] Switched to NOHz mode on CPU #90
[ 82.895199] Switched to NOHz mode on CPU #8
[ 82.895202] Modules linked in:
[ 82.895206] Switched to NOHz mode on CPU #28
[ 82.895211] Pid: 0, comm: swapper Not tainted 2.6.39-rc7-tip-yh-05234-g3a108a0-dirty #1016
[ 82.895213] Call Trace:
[ 82.895233] [<ffffffff81080144>] warn_slowpath_common+0x85/0x9d
[ 82.895238] [<ffffffff81080176>] warn_slowpath_null+0x1a/0x1c
[ 82.895242] [<ffffffff810d32cc>] rcu_enter_nohz+0x49/0x8b
[ 82.895250] [<ffffffff810ab121>] tick_nohz_stop_sched_tick+0x27d/0x366
[ 82.895255] [<ffffffff810391bc>] cpu_idle+0x7a/0xcc
[ 82.895261] [<ffffffff81bda6e3>] rest_init+0xb7/0xbe
[ 82.895266] [<ffffffff81bda62c>] ? csum_partial_copy_generic+0x16c/0x16c
[ 82.895272] [<ffffffff82742e39>] start_kernel+0x3b2/0x3bd
[ 82.895276] [<ffffffff827422cc>] x86_64_start_reservations+0x9c/0xa0
[ 82.895281] [<ffffffff827424a8>] x86_64_start_kernel+0x1d8/0x1e3
[ 82.895290] ---[ end trace 2cfc591bf7de931f ]---
[ 82.895310] Switched to NOHz mode on CPU #72
[ 82.895315] Dumping ftrace buffer:
[ 82.895328] ---------------------------------
[ 82.895340] CPU:0 [LOST 35328 EVENTS]
[ 82.895341] <idle>-0 0d... 82735399us : Unknown type 4
[ 82.895347] <idle>-0 0dN.. 82735431us : Unknown type 4
[ 82.895353] <idle>-0 0d... 82739390us : Unknown type 4
[ 82.895358] <idle>-0 0dN.. 82739415us : Unknown type 4
[ 82.895364] <idle>-0 0d... 82743384us : Unknown type 4
[ 82.895369] <idle>-0 0dN.. 82743408us : Unknown type 4
[ 82.895375] <idle>-0 0d... 82747376us : Unknown type 4
[ 82.895379] Switched to NOHz mode on CPU #53
[ 82.895385] <idle>-0 0dN.. 82747403us : Unknown type 4
[ 82.895390] <idle>-0 0d... 82751370us : Unknown type 4
[ 82.895395] <idle>-0 0dN.. 82751391us : Unknown type 4
[ 82.895400] Switched to NOHz mode on CPU #60
[ 82.895405] <idle>-0 0d... 82755364us : Unknown type 4
[ 82.895411] <idle>-0 0dN.. 82755386us : Unknown type 4
[ 82.895416] <idle>-0 0d... 82759355us : Unknown type 4
[ 82.895421] <idle>-0 0dN.. 82759378us : Unknown type 4
[ 82.895428] Switched to NOHz mode on CPU #102
[ 82.895431] <idle>-0 0d... 82763350us : Unknown type 4
[ 82.895436] <idle>-0 0dN.. 82763372us : Unknown type 4
[ 82.895441] <idle>-0 0d... 82767341us : Unknown type 4
[ 82.895448] Switched to NOHz mode on CPU #155
[ 82.895453] <idle>-0 0dN.. 82767364us : Unknown type 4
[ 82.895459] <idle>-0 0d... 82771334us : Unknown type 4
[ 82.895464] <idle>-0 0dN.. 82771357us : Unknown type 4
[ 82.895469] <idle>-0 0d... 82775328us : Unknown type 4
[ 82.895474] <idle>-0 0dN.. 82775355us : Unknown type 4
[ 82.895480] <idle>-0 0d... 82779321us : Unknown type 4
[ 82.895485] <idle>-0 0dN.. 82779345us : Unknown type 4
[ 82.895490] <idle>-0 0d... 82783313us : Unknown type 4
[ 82.895495] <idle>-0 0dN.. 82783340us : Unknown type 4
[ 82.895501] <idle>-0 0d... 82787308us : Unknown type 4
[ 82.895506] <idle>-0 0dN.. 82787331us : Unknown type 4
[ 82.895511] <idle>-0 0d... 82791300us : Unknown type 4
[ 82.895516] <idle>-0 0dN.. 82791322us : Unknown type 4
[ 82.895522] <idle>-0 0d... 82795293us : Unknown type 4
[ 82.895527] <idle>-0 0dN.. 82795320us : Unknown type 4
[ 82.895532] <idle>-0 0d... 82799287us : Unknown type 4
[ 82.895537] <idle>-0 0dN.. 82799310us : Unknown type 4
[ 82.895542] <idle>-0 0d... 82803279us : Unknown type 4
[ 82.895547] <idle>-0 0dN.. 82803302us : Unknown type 4
[ 82.895552] <idle>-0 0d... 82807272us : Unknown type 4
[ 82.895558] <idle>-0 0dN.. 82807294us : Unknown type 4
[ 82.895563] <idle>-0 0d... 82811264us : Unknown type 4
[ 82.895568] <idle>-0 0dN.. 82811288us : Unknown type 4
[ 82.895573] <idle>-0 0d... 82815258us : Unknown type 4
[ 82.895578] <idle>-0 0dN.. 82815281us : Unknown type 4
[ 82.895583] <idle>-0 0d... 82819250us : Unknown type 4
[ 82.895588] <idle>-0 0dN.. 82819277us : Unknown type 4
[ 82.895594] <idle>-0 0d... 82823244us : Unknown type 4
[ 82.895599] <idle>-0 0dN.. 82823266us : Unknown type 4
[ 82.895604] <idle>-0 0d... 82827237us : Unknown type 4
[ 82.895609] <idle>-0 0dN.. 82827261us : Unknown type 4
[ 82.895615] <idle>-0 0d... 82831230us : Unknown type 4
[ 82.895620] <idle>-0 0dN.. 82831250us : Unknown type 4
[ 82.895625] <idle>-0 0d... 82835223us : Unknown type 4
[ 82.895631] <idle>-0 0dN.. 82835249us : Unknown type 4
[ 82.895635] Switched to NOHz mode on CPU #54
[ 82.895640] <idle>-0 0d... 82839217us : Unknown type 4
[ 82.895645] <idle>-0 0dN.. 82839243us : Unknown type 4
[ 82.895651] <idle>-0 0d... 82843208us : Unknown type 4
[ 82.895656] <idle>-0 0dN.. 82843231us : Unknown type 4
[ 82.895661] <idle>-0 0d... 82847201us : Unknown type 4
[ 82.895666] <idle>-0 0dN.. 82847224us : Unknown type 4
[ 82.895671] <idle>-0 0d... 82851195us : Unknown type 4
[ 82.895677] <idle>-0 0dN.. 82851219us : Unknown type 4
[ 82.895682] <idle>-0 0d... 82855188us : Unknown type 4
[ 82.895686] Switched to NOHz mode on CPU #42
[ 82.895693] Switched to NOHz mode on CPU #46
[ 82.895699] Switched to NOHz mode on CPU #49
[ 82.895705] <idle>-0 0dN.. 82855213us : Unknown type 4
[ 82.895709] Switched to NOHz mode on CPU #109
[ 82.895715] Switched to NOHz mode on CPU #111
[ 82.895720] <idle>-0 0d... 82859183us : Unknown type 4
[ 82.895724] Switched to NOHz mode on CPU #101
[ 82.895729] <idle>-0 0dN.. 82859211us : Unknown type 4
[ 82.895733] Switched to NOHz mode on CPU #98
[ 82.895739] Switched to NOHz mode on CPU #96
[ 82.895744] <idle>-0 0d... 82863174us : Unknown type 4
[ 82.895749] <idle>-0 0dN.. 82863198us : Unknown type 4
[ 82.895754] <idle>-0 0d... 82867167us : Unknown type 4
[ 82.895759] <idle>-0 0dN.. 82867191us : Unknown type 4
[ 82.895765] <idle>-0 0d... 82871161us : Unknown type 4
[ 82.895770] <idle>-0 0dN.. 82871185us : Unknown type 4
[ 82.895775] <idle>-0 0d... 82875153us : Unknown type 4
[ 82.895780] Switched to NOHz mode on CPU #17
[ 82.895784] <idle>-0 0dN.. 82875174us : Unknown type 4
[ 82.895790] Switched to NOHz mode on CPU #14
[ 82.895793] <idle>-0 0d... 82879147us : Unknown type 4
[ 82.895799] Switched to NOHz mode on CPU #3
[ 82.895801] <idle>-0 0dN.. 82879171us : Unknown type 4
[ 82.895806] <idle>-0 0d... 82883139us : Unknown type 4
[ 82.895811] <idle>-0 0dN.. 82883164us : Unknown type 4
[ 82.895814] Switched to NOHz mode on CPU #4
[ 82.895817] <idle>-0 0d... 82887132us : Unknown type 4
[ 82.895822] Switched to NOHz mode on CPU #153
[ 82.895827] <idle>-0 0dN.. 82887154us : Unknown type 4
[ 82.895833] <idle>-0 0d... 82891124us : Unknown type 4
[ 82.895838] <idle>-0 0dN.. 82891148us : Unknown type 4
[ 82.895843] <idle>-0 0d... 82893612us : Unknown type 4
[ 82.895848] <idle>-0 0d... 82893643us : Unknown type 4
[ 82.895853] <idle>-0 0d... 82895118us : Unknown type 4
[ 82.895859] <idle>-0 0dN.. 82895147us : Unknown type 4
[ 82.895864] <idle>-0 0d... 82895177us : Unknown type 4
[ 82.895865] ---------------------------------
but if compile from fedora 14 gcc, will still have some delay.
[ 33.464561] cpu_dev_init done
[ 51.953005] memory_dev_init done
and it also have that warning...
Thanks
Yinghai
next prev parent reply other threads:[~2011-05-18 21:10 UTC|newest]
Thread overview: 76+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-05-08 15:18 Paul E. McKenney
2011-05-09 7:36 ` Ingo Molnar
2011-05-09 21:09 ` Yinghai Lu
2011-05-10 8:56 ` Paul E. McKenney
2011-05-10 9:37 ` Ingo Molnar
2011-05-10 18:04 ` Yinghai Lu
2011-05-10 19:32 ` Paul E. McKenney
2011-05-10 20:52 ` Yinghai Lu
2011-05-11 4:54 ` Paul E. McKenney
2011-05-11 6:03 ` Yinghai Lu
2011-05-11 6:42 ` Yinghai Lu
2011-05-11 20:13 ` Paul E. McKenney
2011-05-11 16:54 ` Yinghai Lu
2011-05-11 16:56 ` Yinghai Lu
2011-05-11 20:18 ` Paul E. McKenney
2011-05-11 20:59 ` Yinghai Lu
2011-05-11 21:30 ` Yinghai Lu
2011-05-11 23:02 ` Yinghai Lu
2011-05-12 6:03 ` Ingo Molnar
2011-05-12 7:27 ` Yinghai Lu
2011-05-12 7:42 ` Yinghai Lu
2011-05-12 9:20 ` Paul E. McKenney
2011-05-12 17:31 ` Yinghai Lu
2011-05-12 21:36 ` Yinghai Lu
2011-05-13 1:28 ` Yinghai Lu
2011-05-13 8:42 ` Ingo Molnar
2011-05-13 12:19 ` Ingo Molnar
2011-05-13 13:04 ` Ingo Molnar
2011-05-13 13:12 ` Ingo Molnar
2011-05-13 14:14 ` Paul E. McKenney
2011-05-13 15:07 ` Ingo Molnar
2011-05-13 16:26 ` Paul E. McKenney
2011-05-16 7:08 ` Ingo Molnar
2011-05-16 7:48 ` Paul E. McKenney
2011-05-16 11:51 ` Ingo Molnar
2011-05-16 12:23 ` Ingo Molnar
2011-05-16 14:30 ` Ingo Molnar
2011-05-16 21:33 ` Paul E. McKenney
2011-05-16 22:07 ` Paul E. McKenney
2011-05-16 21:24 ` Paul E. McKenney
2011-05-16 23:52 ` Frederic Weisbecker
2011-05-17 2:40 ` Frederic Weisbecker
2011-05-17 7:53 ` Paul E. McKenney
2011-05-17 12:43 ` Frederic Weisbecker
2011-05-17 22:21 ` Paul E. McKenney
2011-05-18 21:10 ` Yinghai Lu [this message]
2011-05-18 23:13 ` Frederic Weisbecker
2011-05-19 4:33 ` Yinghai Lu
2011-05-19 14:47 ` Frederic Weisbecker
2011-05-19 19:51 ` Yinghai Lu
2011-05-19 21:15 ` Frederic Weisbecker
2011-05-19 21:45 ` Yinghai Lu
2011-05-20 0:09 ` [PATCH] rcu: Fix unpaired rcu_irq_enter() from locking selftests Frederic Weisbecker
2011-05-20 8:36 ` Ingo Molnar
2011-05-20 15:12 ` Paul E. McKenney
2011-05-20 15:11 ` Paul E. McKenney
2011-05-20 0:14 ` [GIT PULL rcu/next] rcu commits for 2.6.40 Frederic Weisbecker
2011-05-13 14:40 ` Ingo Molnar
2011-05-13 16:38 ` Paul E. McKenney
2011-05-16 7:10 ` Ingo Molnar
2011-05-13 21:08 ` Yinghai Lu
2011-05-14 14:26 ` Paul E. McKenney
2011-05-14 15:31 ` Paul E. McKenney
2011-05-14 18:34 ` Paul E. McKenney
2011-05-15 3:59 ` Yinghai Lu
2011-05-15 4:14 ` Yinghai Lu
2011-05-15 5:41 ` Yinghai Lu
2011-05-15 5:49 ` Yinghai Lu
2011-05-15 6:04 ` Paul E. McKenney
2011-05-15 6:59 ` Paul E. McKenney
2011-05-16 7:08 ` Paul E. McKenney
2011-05-16 7:39 ` Ingo Molnar
2011-05-15 6:01 ` Paul E. McKenney
2011-05-15 22:01 ` Frederic Weisbecker
2011-05-16 5:56 ` Paul E. McKenney
2011-05-16 22:40 ` Frederic Weisbecker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4DD435C2.6040305@kernel.org \
--to=yinghai@kernel.org \
--cc=fweisbec@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=paulmck@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome