From: Andi Kleen <ak@linux.intel.com>
To: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Cc: lkml <linux-kernel@vger.kernel.org>, Jiri Olsa <jolsa@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
linux-perf-users@vger.kernel.org,
Arnaldo Carvalho de Melo <acme@kernel.org>,
eranian@google.com, vincent.weaver@maine.edu,
"Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com>
Subject: Re: System crash with perf_fuzzer (kernel: 5.0.0-rc3)
Date: Thu, 31 Jan 2019 05:00:30 -0800 [thread overview]
Message-ID: <20190131130030.GW6118@tassilo.jf.intel.com> (raw)
In-Reply-To: <33d300b5-f8b0-4987-4d4c-b5175b6b6c60@linux.ibm.com>
On Thu, Jan 31, 2019 at 01:28:34PM +0530, Ravi Bangoria wrote:
> Hi Andi,
>
> On 1/25/19 9:30 PM, Andi Kleen wrote:
> >> [Fri Jan 25 10:28:53 2019] perf: interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
> >> [Fri Jan 25 10:29:08 2019] perf: interrupt took too long (3136 > 3126), lowering kernel.perf_event_max_sample_rate to 63750
> >> [Fri Jan 25 10:29:11 2019] perf: interrupt took too long (4140 > 3920), lowering kernel.perf_event_max_sample_rate to 48250
> >> [Fri Jan 25 10:29:11 2019] perf: interrupt took too long (5231 > 5175), lowering kernel.perf_event_max_sample_rate to 38000
> >> [Fri Jan 25 10:29:11 2019] perf: interrupt took too long (6736 > 6538), lowering kernel.perf_event_max_sample_rate to 29500
> >
> > These are fairly normal.
>
> I understand that throttling mechanism is designed exactly to do this.
> But I've observed that, everytime I run the fuzzer, max_sample_rates is
> been throttled down to 250 (which is CONFIG_HZ I guess). Doesn't this
> mean the interrupt time is somehow increasing gradually? Is that fine?
It's more like the throttling mechanism is an controller
and it takes multiple tries to zoom in on the truely
needed value.
You can measure the PMI time by enabling the nmi:nmi_handler
trace point. It directly reports it. From what I've seen
it's a long tail distribution with regular large outliers.
Most of the PMIs are not that slow, just an occassional
few are.
When I did some investigation on this a couple years back
the outliers were either due to call stack processing,
or due to flushing the perf ring buffer. There were some
fixes on the the call stack case back then, but I'm sure more could
be done.
For the call stack processing there isn't much more we can do I think
(other than switching to call stack LBR only),
but I suspect the buffer flushing problem could be improved more.
It's relatively easy to investigate with a variant of the ftrace
recipe I posted earlier (but you need to fix the Makefile first
to enable ftrace for all of perf) Just add a ftrace trigger on the
nmi_handler trace point to stop tracing when the nmi_handler
time exceeds a threshold and look at the traces.
-Andi
next prev parent reply other threads:[~2019-01-31 13:00 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-25 6:46 Ravi Bangoria
2019-01-25 15:11 ` Vince Weaver
2019-01-25 16:00 ` Andi Kleen
2019-01-31 7:58 ` Ravi Bangoria
2019-01-31 13:00 ` Andi Kleen [this message]
2019-01-31 20:27 ` Cong Wang
2019-01-31 20:39 ` Andi Kleen
2019-03-06 23:09 ` Pavel Machek
2019-01-30 18:36 ` Jiri Olsa
2019-01-30 20:39 ` Andi Kleen
2019-01-30 22:33 ` Jiri Olsa
2019-01-31 7:36 ` Jiri Olsa
2019-01-31 8:27 ` Jiri Olsa
2019-02-01 7:43 ` Jiri Olsa
2019-02-01 7:54 ` Ravi Bangoria
2019-02-02 3:24 ` Ravi Bangoria
2019-02-02 10:34 ` Jiri Olsa
2019-02-01 16:27 ` Vince Weaver
2019-02-01 17:38 ` Jiri Olsa
2019-02-02 17:58 ` Vince Weaver
2019-02-04 12:35 ` [PATCH] perf: Add check_period pmu callback Jiri Olsa
2019-02-11 10:11 ` Peter Zijlstra
2019-02-11 13:22 ` [tip:perf/urgent] perf/x86: Add check_period PMU callback tip-bot for Jiri Olsa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190131130030.GW6118@tassilo.jf.intel.com \
--to=ak@linux.intel.com \
--cc=acme@kernel.org \
--cc=eranian@google.com \
--cc=jolsa@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=naveen.n.rao@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=ravi.bangoria@linux.ibm.com \
--cc=vincent.weaver@maine.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome