From: Jiaying Zhang <jiayingz@google.com>
To: Steven Rostedt <srostedt@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>, Michael Rubin <mrubin@google.com>,
David Sharp <dhsharp@google.com>,
linux-kernel@vger.kernel.org,
Steven Rostedt <rostedt@goodmis.org>
Subject: Re: lockup in rb_get_reader_page
Date: Tue, 30 Mar 2010 21:01:08 -0700 [thread overview]
Message-ID: <q2o5df78e1d1003302101q585143aan32e8872fcdb05c8a@mail.gmail.com> (raw)
In-Reply-To: <1269995724.22564.58.camel@localhost.localdomain>
Thanks a lot for your quick reply!
On Tue, Mar 30, 2010 at 5:35 PM, Steven Rostedt <srostedt@redhat.com> wrote:
> Hi Jiaying,
>
>
> On Tue, 2010-03-30 at 16:27 -0700, Jiaying Zhang wrote:
>> Hi Steven,
>>
>> We recently saw some cpu lockups while running kernel tracing.
>> The problem started to happen after we synced up our ring buffer
>> code with the upstream lockless ring buffer change. It usually
>> took many hours and heavy trace load to hit this problem. When
>> the lockup happens, the problematic cpu seemed to be in an infinite
>> loop of trying to grab the head_page in rb_get_reader_page().
>>
>> We would like to check with you on whether this is a known issue.
>> If so, do we have a bug fix? If not, do you have any suggestions on
>> where we should check?
>
> I'm unaware of any problems with the ring buffer. I've been running the
> lockless version for over a year now, and hammering it with very
> intensive tracing.
>
> Now, you see this on the reader side. There has been a few recent fixes
> that could cause problems when we have multiple readers. Ftrace usage
> does not usually encounter multiple readers so I have not had issues.
> But Li Zefan had a stress test that did find and trigger the problems.
>
I looked at Li Zefan's patch but I think that is to fix a lockup issue in
ftrace. We are not using ftrace right now but still use our own kernel
trace wrapper built on top of ring buffer. We also only use one reader.
I guess it is more likely a race between trace reader and trace writer.
> Are you using the latest ring buffer that is in Linus's tree? Are you
> resetting the ring buffer while reading it?
>
No. We don't allow resetting buffer when there is any active reader.
I don't think we ever reset the ring buffer on those machines that hit
this problem.
> I guess I need to know more exactly what you are doing to understand the
> problem.
>
I am going to patch rb_get_reader_page() to print out some debugging
message when it enters into an infinitely loop. I will keep you updated
if I find any interesting info.
Thanks a lot!
Jiaying
> Thanks,
>
> -- Steve
>
>
>
next prev parent reply other threads:[~2010-03-31 4:01 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-03-30 23:27 Jiaying Zhang
2010-03-31 0:35 ` Steven Rostedt
2010-03-31 4:01 ` Jiaying Zhang [this message]
2010-04-01 0:28 ` Steven Rostedt
2010-04-01 1:14 ` Jiaying Zhang
2010-04-01 1:23 ` Steven Rostedt
2010-04-02 23:10 ` Jiaying Zhang
2010-04-03 12:45 ` Steven Rostedt
2010-04-06 21:12 ` Jiaying Zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=q2o5df78e1d1003302101q585143aan32e8872fcdb05c8a@mail.gmail.com \
--to=jiayingz@google.com \
--cc=dhsharp@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=mrubin@google.com \
--cc=rostedt@goodmis.org \
--cc=srostedt@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®