From: Michael Kelley <mhklinux@outlook.com>
To: Frederic Weisbecker <frederic@kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
Naman Jain <namjain@linux.microsoft.com>
Cc: Thomas Gleixner <tglx@kernel.org>,
Marco Crivellari <marco.crivellari@suse.com>,
Phil Auld <pauld@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Waiman Long <longman@redhat.com>,
"Paul E. McKenney" <paulmck@kernel.org>
Subject: RE: [PATCH 00/47 v2] cpu/hotplug: Remove cpu_down() global stop_machine() except for CPUHP_TEARDOWN_CPU itself
Date: Sun, 22 Nov 2026 21:26:13 +0000 [thread overview]
Message-ID: <MN2PR02MB6863080FA24079F8F2F1879AD4462@MN2PR02MB6863.namprd02.prod.outlook.com> (raw)
In-Reply-To: <20261105163045.38862-1-frederic@kernel.org>
From: Frederic Weisbecker <frederic@kernel.org> Sent: Thursday, November 5, 2026 8:30 AM
>
> Hi,
>
> As discussed in LPC, cpu_down()'s stop_machine() is a problem for
> running real time and nohz_full workloads, for example if we want to
> provide a cpuset interface for nohz_full and isolated partitions are
> created at the same time as those critical workloads.
I'm looking at this patch set specifically from the standpoint of Hyper-V
and Patch 39 of the series. But the top level "why" in this paragraph
seems vague to me. I presume a key goal for RT workloads is to reduce
the interval that stop_machine() has all CPUs paused, since during that
interval RT workloads can't do anything and might exceed their response
time commitments. And this goal applies when the system is staying up,
but one or more CPUs are being taken down. If the whole system is being
shut down or doing a kernel kexec, is the stop_machine() interval an issue?
Do you have any sense of much this patch set reduces the duration
of the stop_machine() pause? Just wondering if it has been measured
and is significant enough to matter.
Your paragraph above mentions a cpuset interface for nohz_full and
creation of isolated partitions, and I don't know enough about those
topics to be able to map them to the duration of stop_machine() as
invoked by cpu_down(). Can you clarify?
And are there any other problems with cpu_down()/stop_machine()
other than how long all CPUs are paused?
>
> Here is a first step to start. stop_machine() gets a new "post" callback
> that is executed after the main one such that:
>
> * As before the main fn executes while all the other CPUs spin.
> * While other CPUs can exit stop_machine(), the target keeps IRQs
> disabled and execute the post_fn.
>
> All the states between CPUHP_AP_ONLINE and CPUHP_AP_OFFLINE are moved
> to the post callback because they can eventually run concurrently with
> the other CPUs. I made a dedicated commit for each of them with
> individual justification analysis and tweaked a bit CPUHP_AP_TICK_DYING
> to adapt it.
Maybe it's obvious, but perhaps you could explain the "why" for embedding
the post_fn in the stop_machine() mechanism vs. doing something separate
after stop_machine() finishes. I presume it's because you want to preserve
the locking and interrupt masking that stop_machine() does. Having
stop_machine() release everything, and then re-obtaining introduces a gap
that might be problematic. Or is there some other reason?
Also, I presume the order in which the CPUHP_STATE_* states are executed
during the "CPU down" path does not change. States between
CPUHP_AP_ONLINE and CPUHP_AP_OFFLINE are executed as before,
just with difference that other online CPUs might no longer be paused.
>
> Please double check CPUHP_AP_CPU_PM_STARTING with which I'm not too
> confident.
>
> In the end only CPUHP_TEARDOWN_CPU (which is __cpu_disable()) remains in
Does "In the end" mean "upon adoption of this patch set"? Or is it referring
to follow-on patches that haven't been submitted yet?
> the global stop machine, this will be the next step. The plan will be to
> (all with IRQs disabled)
The phrase "this will be the next step" seemed tacked on here in a
way that the meaning isn't clear to me. Can you crisp-up the language so
it is clear?
>
> * Clear the CPU from cpu_online_mask
>
> * Start an RCU grace period and poll on it (while doing momentary eqs in
> the loop), this *should work* with IRQs disabled. So all CPUs observe
> the CPU as offline.
>
> * Check if the rest of x86 __cpu_disable() can be accomodated to run
> without making other CPUs spin.
>
> Changes since v2:
>
> * Remove the irq work kick because nohz full kick now has content in its
> irq_work callback.
>
> * Fix a build issues
>
> * Kick the new timekeeper with the more appropriate wake_up_nohz_cpu()
> function.
>
> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> core/hotplug-v2
>
> HEAD: 12f90ee6b38dbe86a3c8b9eccd8a0b509bfc005b
> Thanks,
> Frederic
> ---
>
> Frederic Weisbecker (47):
> stop_machine: Add post function to execute while other CPUs can exit
> x86/cpu/cacheinfo: Move CPUHP_AP_CACHECTRL_STARTING to post stop machine
> sched: Move CPUHP_AP_SCHED_STARTING to post stop_machine()
> rcu: Move CPUHP_AP_RCUTREE_DYING to post stop_machine()
> cpuidle: Move CPUHP_AP_CPU_PM_STARTING post stop_machine()
> cpu: Move CPUHP_AP_IRQ_ARMADA_XP_STARTING..CPUHP_AP_IRQ_GIC_STARTING range post stop_machine()
> irqchip/bcm2836: Move CPUHP_AP_IRQ_BCM2836_STARTING post stop_machine()
> cpu: Move [CPUHP_AP_IRQ_EIOINTC_STARTING..CPUHP_AP_IRQ_MIPS_GIC_STARTING] range post stop_machine()
> irqchip/loongarch-avec: Move CPUHP_AP_IRQ_AVECINTC_STARTING post stop_machine()
> irqchip/sifive-plic: Move CPUHP_AP_IRQ_SIFIVE_PLIC_STARTING post stop_machine()
> irqchip/aslint-sswi: Move CPUHP_AP_IRQ_ACLINT_SSWI_STARTING post stop_machine()
> irqchip/irq-riscv-imsic-early: Move CPUHP_AP_IRQ_RISCV_IMSIC_STARTING post stop_machine()
> cpu: Move [CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING..CPUHP_AP_IRQ_RISCV_SBI_IPI_STARTING] range post stop_machine()
> perf/x86: Move CPUHP_AP_PERF_X86_STARTING post stop_machine()
> perf/amd/ibs: Move CPUHP_AP_PERF_X86_AMD_IBS_STARTING post stop_machine()
> cpu: Move CPUHP_AP_PERF_XTENSA_STARTING post stop_machine()
> ARM: vfp: Move CPUHP_AP_ARM_VFP_STARTING post stop_machine()
> cpu: Move [CPUHP_AP_PERF_ARM_ACPI_STARTING..CPUHP_AP_ARM64_DEBUG_MONITORS_STARTING] range post stop_machine()
> perf/arm_pmu: Move CPUHP_AP_PERF_ARM_STARTING post stop_machine()
> perf: RISC-V: Move CPUHP_AP_PERF_RISCV_STARTING post stop_machine()
> ARM: l2c: Move CPUHP_AP_ARM_L2X0_STARTING post stop_machine()
> clocksource/drivers/exynos_mct: Move CPUHP_AP_EXYNOS4_MCT_TIMER_STARTING post stop_machine()
> clocksource/drivers/arm_arch_timer: Move CPUHP_AP_ARM_ARCH_TIMER_STARTING post stop_machine()
> clocksource/drivers/arm_arch_timer: Move CPUHP_AP_ARM_ARCH_TIMER_EVTSTRM_STARTING post stop_machine()
> clocksource/drivers/arm_global_timer: Move CPUHP_AP_ARM_GLOBAL_TIMER_STARTING post stop_machine()
> clocksource/drivers/jcore: Move CPUHP_AP_JCORE_TIMER_STARTING post stop_machine()
> ARM: smp_twd: Move CPUHP_AP_ARM_TWD_STARTING post stop_machine()
> clocksource/drivers/qcom: Move CPUHP_AP_QCOM_TIMER_STARTING post stop_machine()
> clocksource/drivers/timer-tegra: Move CPUHP_AP_TEGRA_TIMER_STARTING post stop_machine()
> clocksource/drivers/armada: Move CPUHP_AP_ARMADA_TIMER_STARTING post stop_machine()
> LoongArch: Move CPUHP_AP_LOONGARCH_ARCH_TIMER_STARTING post stop_machine()
> clocksource/drivers/mips-gic-timer: Move CPUHP_AP_MIPS_GIC_TIMER_STARTING post stop_machine()
> clocksource/drivers/arc_timer: Move CPUHP_AP_ARC_TIMER_STARTING post stop_machine()
> clocksource/drivers/rtl-otto: Move CPUHP_AP_REALTEK_TIMER_STARTING post stop_machine()
> clocksource/drivers/timer-riscv: Move CPUHP_AP_RISCV_TIMER_STARTING post stop_machine()
> clocksource: clint: Move CPUHP_AP_CLINT_TIMER_STARTING post stop_machine()
> clocksource/drivers/c-sky: Move CPUHP_AP_CSKY_TIMER_STARTING post stop_machine()
> clocksource/drivers/timer-ti-dm: Move CPUHP_AP_TI_GP_TIMER_STARTING post stop_machine()
> clocksource: hyper-v: Move CPUHP_AP_HYPERV_TIMER_STARTING post stop_machine()
> clocksource/dummy_timer: Move CPUHP_AP_DUMMY_TIMER_STARTING post stop_machine()
> ARM: xen: Move CPUHP_AP_ARM_XEN_STARTING post stop_machine()
> ARM: xen: Move CPUHP_AP_ARM_XEN_RUNSTATE_STARTING and CPUHP_AP_ARM64_ISNDEP_STARTING post stop_machine()
> smp: Move CPUHP_AP_SMPCFD_DYING post stop_machine()
> hrtimer: Move CPUHP_AP_HRTIMERS_DYING post stop_machine()
> tick: Move CPUHP_AP_TICK_DYING post stop_machine()
> x86/boot/tboot: Move CPUHP_AP_X86_TBOOT_DYING post stop_machine()
> ARM: mm: Move CPUHP_AP_ARM_CACHE_B15_RAC_DYING post stop_machine()
I have a general but limited understanding of CPUHP states, and of how
stop_machine() works. Within the limits of that understanding, the
core code changes look reasonable to me. But I can't really give a
"Reviewed-by:".
I tested this patch set in a VM running on Hyper-V, which uses the
CPUHP_AP_HYPERV_TIMER_STARTING state that is moved in Patch 39 of
the series. Taking single CPUs offline and back online works. Taking a
panic and kexec'ing into the kdump kernel works. Normal shutdown of
Linux and the VM works. Are there other scenarios I should try?
See also a couple of comments that I've put in Patch 39.
Michael
Tested-by: Michael Kelley <mhklinux@outlook.com>
>
> arch/arm/kernel/patch.c | 2 +-
> arch/arm/probes/kprobes/core.c | 2 +-
> arch/arm64/kernel/patching.c | 4 ++--
> arch/csky/kernel/probes/kprobes.c | 3 ++-
> arch/loongarch/kernel/inst.c | 3 ++-
> arch/parisc/kernel/patch.c | 4 ++--
> arch/powerpc/mm/book3s64/hash_pgtable.c | 4 ++--
> arch/powerpc/platforms/powernv/subcore.c | 2 +-
> arch/powerpc/platforms/pseries/lpar.c | 2 +-
> arch/riscv/kernel/patch.c | 3 ++-
> arch/s390/kernel/kprobes.c | 4 ++--
> arch/s390/kernel/time.c | 3 ++-
> arch/x86/kernel/cpu/microcode/core.c | 3 ++-
> arch/x86/kernel/cpu/mtrr/mtrr.c | 3 ++-
> arch/x86/virt/vmx/tdx/seamldr.c | 2 +-
> arch/xtensa/kernel/jump_label.c | 4 ++--
> include/linux/stop_machine.h | 12 ++++++++----
> kernel/cpu.c | 21 +++++++++++++++++++--
> kernel/sched/core.c | 1 -
> kernel/stop_machine.c | 14 +++++++++++---
> kernel/time/tick-common.c | 15 +++++++++------
> 21 files changed, 74 insertions(+), 37 deletions(-)
prev parent reply other threads:[~2026-11-22 21:28 UTC|newest]
Thread overview: 52+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-05 16:29 Frederic Weisbecker
2026-11-05 16:29 ` [PATCH 01/47] stop_machine: Add post function to execute while other CPUs can exit Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 02/47] x86/cpu/cacheinfo: Move CPUHP_AP_CACHECTRL_STARTING to post stop machine Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 03/47] sched: Move CPUHP_AP_SCHED_STARTING to post stop_machine() Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 04/47] rcu: Move CPUHP_AP_RCUTREE_DYING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 05/47] cpuidle: Move CPUHP_AP_CPU_PM_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 06/47] cpu: Move CPUHP_AP_IRQ_ARMADA_XP_STARTING..CPUHP_AP_IRQ_GIC_STARTING range " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 07/47] irqchip/bcm2836: Move CPUHP_AP_IRQ_BCM2836_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 08/47] cpu: Move [CPUHP_AP_IRQ_EIOINTC_STARTING..CPUHP_AP_IRQ_MIPS_GIC_STARTING] range " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 09/47] irqchip/loongarch-avec: Move CPUHP_AP_IRQ_AVECINTC_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 10/47] irqchip/sifive-plic: Move CPUHP_AP_IRQ_SIFIVE_PLIC_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 11/47] irqchip/aslint-sswi: Move CPUHP_AP_IRQ_ACLINT_SSWI_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 12/47] irqchip/irq-riscv-imsic-early: Move CPUHP_AP_IRQ_RISCV_IMSIC_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 13/47] cpu: Move [CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING..CPUHP_AP_IRQ_RISCV_SBI_IPI_STARTING] range " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 14/47] perf/x86: Move CPUHP_AP_PERF_X86_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 15/47] perf/amd/ibs: Move CPUHP_AP_PERF_X86_AMD_IBS_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 16/47] cpu: Move CPUHP_AP_PERF_XTENSA_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 17/47] ARM: vfp: Move CPUHP_AP_ARM_VFP_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 18/47] cpu: Move [CPUHP_AP_PERF_ARM_ACPI_STARTING..CPUHP_AP_ARM64_DEBUG_MONITORS_STARTING] range " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 19/47] perf/arm_pmu: Move CPUHP_AP_PERF_ARM_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 20/47] perf: RISC-V: Move CPUHP_AP_PERF_RISCV_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 21/47] ARM: l2c: Move CPUHP_AP_ARM_L2X0_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 22/47] clocksource/drivers/exynos_mct: Move CPUHP_AP_EXYNOS4_MCT_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 23/47] clocksource/drivers/arm_arch_timer: Move CPUHP_AP_ARM_ARCH_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 24/47] clocksource/drivers/arm_arch_timer: Move CPUHP_AP_ARM_ARCH_TIMER_EVTSTRM_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 25/47] clocksource/drivers/arm_global_timer: Move CPUHP_AP_ARM_GLOBAL_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 26/47] clocksource/drivers/jcore: Move CPUHP_AP_JCORE_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 27/47] ARM: smp_twd: Move CPUHP_AP_ARM_TWD_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 28/47] clocksource/drivers/qcom: Move CPUHP_AP_QCOM_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 29/47] clocksource/drivers/timer-tegra: Move CPUHP_AP_TEGRA_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 30/47] clocksource/drivers/armada: Move CPUHP_AP_ARMADA_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 31/47] LoongArch: Move CPUHP_AP_LOONGARCH_ARCH_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 32/47] clocksource/drivers/mips-gic-timer: Move CPUHP_AP_MIPS_GIC_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 33/47] clocksource/drivers/arc_timer: Move CPUHP_AP_ARC_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 34/47] clocksource/drivers/rtl-otto: Move CPUHP_AP_REALTEK_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 35/47] clocksource/drivers/timer-riscv: Move CPUHP_AP_RISCV_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 36/47] clocksource: clint: Move CPUHP_AP_CLINT_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 37/47] clocksource/drivers/c-sky: Move CPUHP_AP_CSKY_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 38/47] clocksource/drivers/timer-ti-dm: Move CPUHP_AP_TI_GP_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 39/47] clocksource: hyper-v: Move CPUHP_AP_HYPERV_TIMER_STARTING " Frederic Weisbecker
2026-11-06 10:07 ` Naman Jain
2026-11-06 12:38 ` Frederic Weisbecker
2026-11-22 21:26 ` Michael Kelley
2026-11-05 16:30 ` [PATCH 40/47] clocksource/dummy_timer: Move CPUHP_AP_DUMMY_TIMER_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 41/47] ARM: xen: Move CPUHP_AP_ARM_XEN_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 42/47] ARM: xen: Move CPUHP_AP_ARM_XEN_RUNSTATE_STARTING and CPUHP_AP_ARM64_ISNDEP_STARTING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 43/47] smp: Move CPUHP_AP_SMPCFD_DYING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 44/47] hrtimer: Move CPUHP_AP_HRTIMERS_DYING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 45/47] tick: Move CPUHP_AP_TICK_DYING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 46/47] x86/boot/tboot: Move CPUHP_AP_X86_TBOOT_DYING " Frederic Weisbecker
2026-11-05 16:30 ` [PATCH 47/47] ARM: mm: Move CPUHP_AP_ARM_CACHE_B15_RAC_DYING " Frederic Weisbecker
2026-11-22 21:26 ` Michael Kelley [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=MN2PR02MB6863080FA24079F8F2F1879AD4462@MN2PR02MB6863.namprd02.prod.outlook.com \
--to=mhklinux@outlook.com \
--cc=frederic@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=marco.crivellari@suse.com \
--cc=namjain@linux.microsoft.com \
--cc=pauld@redhat.com \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=tglx@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®