From: "Paul E. McKenney" <paulmck@kernel.org>
To: rcu@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com,
rostedt@goodmis.org,
"Joel Fernandes (Google)" <joel@joelfernandes.org>,
Paul McKenney <paulmck@kernel.org>,
Frederic Weisbecker <fweisbec@gmail.com>,
Zhouyi Zhou <zhouzhouyi@gmail.com>,
stable@vger.kernel.org, Davidlohr Bueso <dave@stgolabs.net>
Subject: [PATCH rcu 7/7] torture: Fix hang during kthread shutdown phase
Date: Wed, 4 Jan 2023 16:45:01 -0800 [thread overview]
Message-ID: <20230105004501.1771332-13-paulmck@kernel.org> (raw)
In-Reply-To: <20230105004454.GA1771168@paulmck-ThinkPad-P17-Gen-1>
From: "Joel Fernandes (Google)" <joel@joelfernandes.org>
During rcutorture shutdown, the rcu_torture_cleanup() function calls
torture_cleanup_begin(), which sets the fullstop global variable to
FULLSTOP_RMMOD. This causes the rcutorture threads for readers and
fakewriters to exit all of their "while" loops and start shutting down.
They then call torture_kthread_stopping(), which in turn waits for
kthread_stop() to be called. However, rcu_torture_cleanup() has
not yet called kthread_stop() on those threads, and before it gets a
chance to do so, multiple instances of torture_kthread_stopping() invoke
schedule_timeout_interruptible(1) in a tight loop. Tracing confirms that
TIMER_SOFTIRQ can then continuously execute timer callbacks. If that
TIMER_SOFTIRQ preempts the task executing rcu_torture_cleanup(), that
task might never invoke kthread_stop().
This commit improves this situation by increasing the timeout passed to
schedule_timeout_interruptible() from one jiffy to 1/20th of a second.
This change prevents TIMER_SOFTIRQ from monopolizing its CPU, thus
allowing rcu_torture_cleanup() to carry out the needed kthread_stop()
invocations. Testing has shown 100 runs of TREE07 passing reliably,
as oppose to the tens-of-percent failure rates seen beforehand.
Cc: Paul McKenney <paulmck@kernel.org>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Zhouyi Zhou <zhouzhouyi@gmail.com>
Cc: <stable@vger.kernel.org> # 6.0.x
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Tested-by: Zhouyi Zhou <zhouzhouyi@gmail.com>
Reviewed-by: Davidlohr Bueso <dave@stgolabs.net>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/torture.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/torture.c b/kernel/torture.c
index 29afc62f2bfec..1a0519b836ac9 100644
--- a/kernel/torture.c
+++ b/kernel/torture.c
@@ -915,7 +915,7 @@ void torture_kthread_stopping(char *title)
VERBOSE_TOROUT_STRING(buf);
while (!kthread_should_stop()) {
torture_shutdown_absorb(title);
- schedule_timeout_uninterruptible(1);
+ schedule_timeout_uninterruptible(HZ / 20);
}
}
EXPORT_SYMBOL_GPL(torture_kthread_stopping);
--
2.31.1.189.g2e36527f23
prev parent reply other threads:[~2023-01-05 0:49 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-05 0:44 [PATCH rcu 0/7] Torture-test updates for v6.3 Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 1/6] rcu-tasks: Use accurate runstart time for RCU Tasks boot-time testing Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 1/7] torture: Seed torture_random_state on CPU Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 2/6] rcu-tasks: Improve comments explaining tasks_rcu_exit_srcu purpose Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 2/7] refscale: Provide for initialization failure Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 3/6] rcu-tasks: Remove preemption disablement around srcu_read_[un]lock() calls Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 3/7] refscale: Add tests using SLAB_TYPESAFE_BY_RCU Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 4/7] locktorture: Allow non-rtmutex lock types to be boosted Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 4/6] rcu-tasks: Fix synchronize_rcu_tasks() VS zap_pid_ns_processes() Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 5/7] locktorture: Make the rt_boost factor a tunable Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 5/6] rcu-tasks: Make rude RCU-Tasks work well with CPU hotplug Paul E. McKenney
2023-01-05 0:44 ` [PATCH rcu 6/6] rcu-tasks: Handle queue-shrink/callback-enqueue race condition Paul E. McKenney
2023-01-05 0:45 ` [PATCH rcu 6/7] rcutorture: Drop sparse lock-acquisition annotations Paul E. McKenney
2023-01-05 0:45 ` Paul E. McKenney [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230105004501.1771332-13-paulmck@kernel.org \
--to=paulmck@kernel.org \
--cc=dave@stgolabs.net \
--cc=fweisbec@gmail.com \
--cc=joel@joelfernandes.org \
--cc=kernel-team@meta.com \
--cc=linux-kernel@vger.kernel.org \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=stable@vger.kernel.org \
--cc=zhouzhouyi@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®