From: Costa Shulyupin <costa.shul@redhat.com>
To: Thomas Gleixner <tglx@linutronix.de>,
Peter Zijlstra <peterz@infradead.org>,
Yury Norov <yury.norov@gmail.com>,
Andrew Morton <akpm@linux-foundation.org>,
Costa Shulyupin <costa.shul@redhat.com>,
Valentin Schneider <vschneid@redhat.com>,
Frederic Weisbecker <frederic@kernel.org>,
Neeraj Upadhyay <neeraj.upadhyay@kernel.org>,
linux-kernel@vger.kernel.org, Waiman Long <longman@redhat.com>,
x86@kernel.org
Subject: [RFC PATCH v1] stop_machine: Add stop_housekeeping_cpuslocked()
Date: Wed, 18 Dec 2024 19:15:31 +0200 [thread overview]
Message-ID: <20241218171531.2217275-1-costa.shul@redhat.com> (raw)
CPU hotplug interferes with CPU isolation and introduces latency to
real-time tasks.
The test:
rtla timerlat hist -c 1 -a 500 &
echo 0 > /sys/devices/system/cpu/cpu2/online
The RTLA tool reveals the following blocking thread stack trace:
-> multi_cpu_stop
-> cpu_stopper_thread
-> smpboot_thread_fn
This happens because multi_cpu_stop() disables interrupts for EACH online
CPU since takedown_cpu() indirectly invokes take_cpu_down() through
stop_machine_cpuslocked(). I'm omitting the detailed description of the
call chain.
Proposal: Limit the stop operation to housekeeping CPUs.
take_cpu_down() invokes with cpuhp_invoke_callback_range_nofail:
- tick_cpu_dying()
- hrtimers_cpu_dying()
- smpcfd_dying_cpu()
- x86_pmu_dying_cpu()
- rcutree_dying_cpu()
- sched_cpu_dying()
- cache_ap_offline()
Which synchronizations do these functions require instead of stop_machine?
Passed standard regression tests:
- https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/6042
- https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-10/-/merge_requests/89
Passes primitive CPU hotplug test without warnings.
What tests and test suites do you recommend?
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
---
include/linux/stop_machine.h | 12 ++++++++++++
kernel/cpu.c | 2 +-
kernel/stop_machine.c | 23 +++++++++++++++++++++++
3 files changed, 36 insertions(+), 1 deletion(-)
diff --git a/include/linux/stop_machine.h b/include/linux/stop_machine.h
index 3132262a404dc..4c9e709f174c8 100644
--- a/include/linux/stop_machine.h
+++ b/include/linux/stop_machine.h
@@ -140,6 +140,18 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data, const struct cpumask *
*/
int stop_core_cpuslocked(unsigned int cpu, cpu_stop_fn_t fn, void *data);
+/**
+ * stop_housekeeping_cpuslocked: freeze housekeeping CPUs and run specified function.
+ * Unlike stop_machine_cpuslocked, it doesn't stop isolated CPUs.
+ * @fn: the function to run
+ * @data: the data ptr for the @fn()
+ * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
+ *
+ * Must be called from with in a cpus_read_lock() protected
+ * region. Avoids nested calls to cpus_read_lock().
+ */
+int stop_housekeeping_cpuslocked(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
+
int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
const struct cpumask *cpus);
#else /* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 6d7c51e7c366c..052f48bebf816 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1310,7 +1310,7 @@ static int takedown_cpu(unsigned int cpu)
/*
* So now all preempt/rcu users must observe !cpu_active().
*/
- err = stop_machine_cpuslocked(take_cpu_down, NULL, cpumask_of(cpu));
+ err = stop_housekeeping_cpuslocked(take_cpu_down, NULL, cpumask_of(cpu));
if (err) {
/* CPU refused to die */
irq_unlock_sparse();
diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index da821ce258ea7..1f36e44d42476 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -22,6 +22,7 @@
#include <linux/atomic.h>
#include <linux/nmi.h>
#include <linux/sched/wake_q.h>
+#include <linux/sched/isolation.h>
/*
* Structure to determine completion condition and record errors. May
@@ -619,6 +620,28 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
return stop_cpus(cpu_online_mask, multi_cpu_stop, &msdata);
}
+int stop_housekeeping_cpuslocked(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
+{
+ struct multi_stop_data msdata = { .fn = fn, .data = data, .active_cpus = cpus };
+ cpumask_var_t stop_mask;
+ int ret;
+
+ lockdep_assert_cpus_held();
+
+ if (!alloc_cpumask_var(&stop_mask, GFP_KERNEL))
+ return -ENOMEM;
+ cpumask_and(stop_mask, cpu_online_mask, housekeeping_cpumask(HK_TYPE_MANAGED_IRQ));
+ cpumask_or(stop_mask, stop_mask, cpus);
+ msdata.num_threads = cpumask_weight(stop_mask);
+
+ /* Set the initial state and stop online housekeeping cpus. */
+ set_state(&msdata, MULTI_STOP_PREPARE);
+ ret = stop_cpus(stop_mask, multi_cpu_stop, &msdata);
+ free_cpumask_var(stop_mask);
+
+ return ret;
+}
+
int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
{
int ret;
--
2.47.0
next reply other threads:[~2024-12-18 17:15 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-12-18 17:15 Costa Shulyupin [this message]
2024-12-18 17:27 ` Peter Zijlstra
2024-12-18 17:32 ` Frederic Weisbecker
2025-01-06 13:08 ` Costa Shulyupin
2025-01-06 21:24 ` Frederic Weisbecker
2025-01-07 14:40 ` Costa Shulyupin
2024-12-19 21:38 ` Thomas Gleixner
2025-01-04 18:37 ` Srivatsa S. Bhat
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20241218171531.2217275-1-costa.shul@redhat.com \
--to=costa.shul@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=frederic@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=neeraj.upadhyay@kernel.org \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=vschneid@redhat.com \
--cc=x86@kernel.org \
--cc=yury.norov@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®