From: Zhen Lei <thunder.leizhen@huawei.com>
To: "Paul E . McKenney" <paulmck@kernel.org>,
Frederic Weisbecker <frederic@kernel.org>,
Neeraj Upadhyay <quic_neeraju@quicinc.com>,
"Josh Triplett" <josh@joshtriplett.org>,
Steven Rostedt <rostedt@goodmis.org>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Lai Jiangshan <jiangshanlai@gmail.com>,
Joel Fernandes <joel@joelfernandes.org>, <rcu@vger.kernel.org>,
<linux-kernel@vger.kernel.org>
Cc: Zhen Lei <thunder.leizhen@huawei.com>, Robert Elliott <elliott@hpe.com>
Subject: [PATCH v7 5/6] doc: Document CONFIG_RCU_CPU_STALL_CPUTIME=y stall information
Date: Fri, 11 Nov 2022 21:07:08 +0800 [thread overview]
Message-ID: <20221111130709.247-6-thunder.leizhen@huawei.com> (raw)
In-Reply-To: <20221111130709.247-1-thunder.leizhen@huawei.com>
This commit doucments how to quickly determine the bug causing a given
RCU CPU stall fault warning based on the output information provided
by CONFIG_RCU_CPU_STALL_CPUTIME=y.
[ paulmck: Apply wordsmithing. ]
Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
Documentation/RCU/stallwarn.rst | 88 +++++++++++++++++++++++++++++++++
1 file changed, 88 insertions(+)
diff --git a/Documentation/RCU/stallwarn.rst b/Documentation/RCU/stallwarn.rst
index dfa4db8c0931eaf..5e24e849290a286 100644
--- a/Documentation/RCU/stallwarn.rst
+++ b/Documentation/RCU/stallwarn.rst
@@ -390,3 +390,91 @@ for example, "P3421".
It is entirely possible to see stall warnings from normal and from
expedited grace periods at about the same time during the same run.
+
+RCU_CPU_STALL_CPUTIME
+=====================
+
+In kernels built with CONFIG_RCU_CPU_STALL_CPUTIME=y or booted with
+rcupdate.rcu_cpu_stall_cputime=1, the following additional information
+is supplied with each RCU CPU stall warning::
+
+rcu: hardirqs softirqs csw/system
+rcu: number: 624 45 0
+rcu: cputime: 69 1 2425 ==> 2500(ms)
+
+These statistics are collected during the sampling period. The values
+in row "number:" are the number of hard interrupts, number of soft
+interrupts, and number of context switches on the stalled CPU. The
+first three values in row "cputime:" indicate the CPU time in
+milliseconds consumed by hard interrupts, soft interrupts, and tasks
+on the stalled CPU. The last number is the measurement interval, again
+in milliseconds. Because user-mode tasks normally do not cause RCU CPU
+stalls, these tasks are typically kernel tasks, which is why only the
+system CPU time are considered.
+
+The sampling period is shown as follows:
+|<------------first timeout---------->|<-----second timeout----->|
+|<--half timeout-->|<--half timeout-->| |
+| |<--first period-->| |
+| |<-----------second sampling period---------->|
+| | | |
+| sampling time point 1st-stall 2nd-stall
+
+
+The following describes four typical scenarios:
+
+1. A CPU looping with interrupts disabled.::
+
+ rcu: hardirqs softirqs csw/system
+ rcu: number: 0 0 0
+ rcu: cputime: 0 0 0 ==> 2500(ms)
+
+ Because interrupts have been disabled throughout the measurement
+ interval, there are no interrupts and no context switches.
+ Furthermore, because CPU time consumption was measured using interrupt
+ handlers, the system CPU consumption is misleadingly measured as zero.
+ This scenario will normally also have "(0 ticks this GP)" printed on
+ this CPU's summary line.
+
+2. A CPU looping with bottom halves disabled.
+
+ This is similar to the previous example, but with non-zero number of
+ and CPU time consumed by hard interrupts, along with non-zero CPU
+ time consumed by in-kernel execution.::
+
+ rcu: hardirqs softirqs csw/system
+ rcu: number: 624 0 0
+ rcu: cputime: 49 0 2446 ==> 2500(ms)
+
+ The fact that there are zero softirqs gives a hint that these were
+ disabled, perhaps via local_bh_disable(). It is of course possible
+ that there were no softirqs, perhaps because all events that would
+ result in softirq execution are confined to other CPUs. In this case,
+ the diagnosis should continue as shown in the next example.
+
+3. A CPU looping with preemption disabled.
+
+ Here, only the number of context switches is zero.::
+
+ rcu: hardirqs softirqs csw/system
+ rcu: number: 624 45 0
+ rcu: cputime: 69 1 2425 ==> 2500(ms)
+
+ This situation hints that the stalled CPU was looping with preemption
+ disabled.
+
+4. No looping, but massive hard and soft interrupts.::
+
+ rcu: hardirqs softirqs csw/system
+ rcu: number: xx xx 0
+ rcu: cputime: xx xx 0 ==> 2500(ms)
+
+ Here, the number and CPU time of hard interrupts are all non-zero,
+ but the number of context switches and the in-kernel CPU time consumed
+ are zero. The number and cputime of soft interrupts will usually be
+ non-zero, but could be zero, for example, if the CPU was spinning
+ within a single hard interrupt handler.
+
+ If this type of RCU CPU stall warning can be reproduced, you can
+ narrow it down by looking at /proc/interrupts or by writing code to
+ trace each interrupt, for example, by referring to show_interrupts().
--
2.25.1
next prev parent reply other threads:[~2022-11-11 13:08 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-11 13:07 [PATCH v7 0/6] rcu: Add RCU stall diagnosis information Zhen Lei
2022-11-11 13:07 ` [PATCH v7 1/6] genirq: Fix the return type of kstat_cpu_irqs_sum() Zhen Lei
2022-11-11 13:07 ` [PATCH v7 2/6] sched: Add helper kstat_cpu_softirqs_sum() Zhen Lei
2022-11-14 11:42 ` Frederic Weisbecker
2022-11-14 12:45 ` Leizhen (ThunderTown)
2022-11-14 12:50 ` Frederic Weisbecker
2022-11-14 14:26 ` Leizhen (ThunderTown)
2022-11-11 13:07 ` [PATCH v7 3/6] sched: Add helper nr_context_switches_cpu() Zhen Lei
2022-11-11 13:07 ` [PATCH v7 4/6] rcu: Add RCU stall diagnosis information Zhen Lei
2022-11-14 11:24 ` Frederic Weisbecker
2022-11-14 12:32 ` Leizhen (ThunderTown)
2022-11-14 12:46 ` Frederic Weisbecker
2022-11-16 22:39 ` Frederic Weisbecker
2022-11-17 1:57 ` Leizhen (ThunderTown)
2022-11-17 12:22 ` Frederic Weisbecker
2022-11-17 13:25 ` Leizhen (ThunderTown)
2022-11-17 14:26 ` Frederic Weisbecker
2022-11-18 2:03 ` Leizhen (ThunderTown)
2022-11-11 13:07 ` Zhen Lei [this message]
2022-11-16 22:55 ` [PATCH v7 5/6] doc: Document CONFIG_RCU_CPU_STALL_CPUTIME=y stall information Frederic Weisbecker
2022-11-17 2:03 ` Leizhen (ThunderTown)
2022-11-17 12:23 ` Frederic Weisbecker
2022-11-11 13:07 ` [PATCH v7 6/6] rcu: Align the output of RCU stall Zhen Lei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221111130709.247-6-thunder.leizhen@huawei.com \
--to=thunder.leizhen@huawei.com \
--cc=elliott@hpe.com \
--cc=frederic@kernel.org \
--cc=jiangshanlai@gmail.com \
--cc=joel@joelfernandes.org \
--cc=josh@joshtriplett.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=paulmck@kernel.org \
--cc=quic_neeraju@quicinc.com \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®