[RFC PATCH 0/2] cpu/hotplug: use cpu_enabled_mask for SMT bringup

[RFC PATCH 0/2] cpu/hotplug: use cpu_enabled_mask for SMT bringup

[salil.mehta]

From: Salil Mehta <salil.mehta@opnsrc.net>

Hi,

For context, I had been away from the QEMU/kernel mailing-list work for an
extended period due to exceptional personal circumstances. I have only
recently started catching up with the Arm vCPU hotplug work and the related
changes that have landed upstream. While doing so, I came across the change
discussed below.

============
I. The Story
============

This is an RFC and, in particular, an open question about the interaction
between cpu_present_mask, cpu_enabled_mask and cpuhp_smt_enable(). I may have
missed a later constraint or discussion while catching up, so I would very
much appreciate a sanity check on the direction proposed here.

Commit f9a82544c717 ("cpu/hotplug: Fix NULL kobject warning in
cpuhp_smt_enable()") fixed a real warning on arm64 by changing the arm64
present-mask semantics: an ACPI Online-Capable CPU which is not MADT
Enabled is no longer initially marked present, and acpi_map_cpu() /
acpi_unmap_cpu() now add and remove it from cpu_present_mask.

I wondered whether the generic enabled mask gives us a narrower way to fix
the original problem.

cpu_enabled_mask was introduced by 4e1a7df45480 ("cpumask: Add enabled
cpumask for present CPUs that can be brought online") specifically for the
case where a CPU can be present but is not currently allowed to be brought
online.  cpuhp_smt_enable() is itself trying to bring offline CPUs online,
so patch 1 simply skips a CPU when !cpu_enabled(cpu).

If that is the intended meaning of cpu_enabled_mask, should
cpuhp_smt_enable() consume it rather than require arm64 to collapse the
present/not-enabled distinction?  Or is there another reason why changing
the arm64 present-mask semantics is preferred here?  I may be missing a
constraint in another architecture or in the SMT hotplug path, hence this
RFC.

Patch 2 reverts f9a82544c717 so that the question can be evaluated with the
original arm64 virtual CPU hotplug model restored.  The ordering is
intentional: the generic enabled check lands first, so reverting the arm64
present-mask change does not reintroduce the NULL-kobject warning.

===========
II. Testing
===========

The series is based on current upstream master, after Linux v7.3-rc5.

An arm64 Image build with CONFIG_ACPI=y, CONFIG_HOTPLUG_CPU=y and
CONFIG_HOTPLUG_SMT=y passes.

The runtime tests were performed on an NVIDIA Jetson Orin Nano system.
The host does not provide hardware SMT; QEMU's virtual SMT topology was
used to exercise the guest HOTPLUG_SMT paths described below.

===============
III. Reproducer
===============

The warning can be reproduced with an arm64 ACPI guest using a virtual SMT
topology, for example:

  qemu-system-aarch64 \
    -machine virt,gic-version=3,acpi=on \
    -accel tcg,thread=multi -cpu cortex-a57 \
    -smp cpus=3,maxcpus=6,sockets=1,clusters=1,cores=3,threads=2 \
    -m 1024M \
    -bios QEMU_EFI.fd \
    -kernel Image \
    -initrd rootfs.cpio.gz \
    -append "console=ttyAMA0 root=/dev/ram rdinit=/init acpi=force" \
    -nographic

After boot:

  cat /sys/devices/system/cpu/present
  cat /sys/devices/system/cpu/enabled
  cat /sys/devices/system/cpu/online

With the pre-f9a arm64 semantics these report:

  present=0-5
  enabled=0-2
  online=0-2

Then exercise SMT disable/enable:

  echo off > /sys/devices/system/cpu/smt/control
  cat /sys/devices/system/cpu/online

  echo on > /sys/devices/system/cpu/smt/control
  cat /sys/devices/system/cpu/online

With the original cpuhp_smt_enable() implementation, the second write
produces the following warning:

  WARNING: fs/sysfs/group.c:137 at internal_create_group+0x418/0x550,
  CPU#2: bash/1

with the relevant call trace:

  internal_create_group+0x418/0x550
  sysfs_create_group+0x20/0x38
  topology_add_dev+0x24/0x38
  cpuhp_invoke_callback+0x174/0x2c0
  __cpuhp_invoke_callback_range+0x98/0x128
  _cpu_up+0x150/0x280
  cpuhp_smt_enable+0xc4/0x128
  control_store+0xf8/0x1f0

With patch 1 followed by patch 2, the same sequence gives:

  SMT off: online=0,2
  SMT on:  online=0-2

and:

  dmesg | grep -E "WARNING:|internal_create_group|cpuhp_smt_enable"

produces no output.

=========================
IV. Additional Comparison
=========================

I also checked cpus=4,maxcpus=6: all four requested initial CPUs are brought
up with the restored present semantics.  By comparison, applying the
f9a82544c717 changes in isolation to the same firmware model resulted in
only the boot CPU being brought up during early SMP initialization, with
the remaining enabled CPUs appearing later through ACPI enumeration.

Both patches pass scripts/checkpatch.pl --strict with no warnings or
errors.

I also have an up-to-date QEMU branch containing the corresponding Arm vCPU
hotplug work used for the tests above. If that would be useful for reproducing
or testing this RFC, please shout and I can share the branch.

=================
V. 'The' Question
=================

The main question for review is therefore whether cpu_enabled_mask should be
the generic eligibility check in cpuhp_smt_enable(), preserving the
present/enabled distinction that motivated the mask, or whether there is a
reason that distinction should instead be removed from arm64.

Many thanks in anticipation!
Salil

Salil Mehta (2):
  cpu/hotplug: Skip disabled CPUs in cpuhp_smt_enable
  Revert "cpu/hotplug: Fix NULL kobject warning in cpuhp_smt_enable()"

 Documentation/arch/arm64/cpu-hotplug.rst | 28 ++++++++++--------------
 arch/arm64/kernel/acpi.c                 |  2 --
 arch/arm64/kernel/smp.c                  | 12 +---------
 kernel/cpu.c                             |  5 +++--
 4 files changed, 16 insertions(+), 31 deletions(-)

-- 
2.34.1

[RFC PATCH 1/2] cpu/hotplug: Skip disabled CPUs in cpuhp_smt_enable

[salil.mehta]

From: Salil Mehta <salil.mehta@opnsrc.net>

The CPU enabled mask describes whether a present CPU may currently be
brought online. cpuhp_smt_enable() is an online operation, but currently
walks all present CPUs and only filters CPUs that are already online or
belong to offline NUMA nodes.

This can make it attempt _cpu_up() for a present CPU which firmware has
not enabled and which has not yet been registered as a CPU device.

Use the enabled mask for the policy decision it was introduced to
represent. This keeps present-but-disabled CPUs out of the SMT bring-up
path while still allowing registered offline SMT threads to be brought
back online.

Present and enabled are separate generic CPU states, so callers which
intend to bring CPUs online should not assume that every present CPU is
enabled.

Signed-off-by: Salil Mehta <salil.mehta@opnsrc.net>
---
 kernel/cpu.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/cpu.c b/kernel/cpu.c
index b3c8553d7bd6..988b7a1e8298 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -2706,8 +2706,9 @@ int cpuhp_smt_enable(void)
 	cpu_maps_update_begin();
 	cpu_smt_control = CPU_SMT_ENABLED;
 	for_each_present_cpu(cpu) {
-		/* Skip online CPUs and CPUs on offline nodes */
-		if (cpu_online(cpu) || !node_online(cpu_to_node(cpu)))
+		/* Skip online/disabled CPUs and CPUs on offline nodes */
+		if (cpu_online(cpu) || !cpu_enabled(cpu) ||
+		    !node_online(cpu_to_node(cpu)))
 			continue;
 		if (!cpu_smt_thread_allowed(cpu) || !topology_is_core_online(cpu))
 			continue;
-- 
2.34.1

[RFC PATCH 2/2] Revert "cpu/hotplug: Fix NULL kobject warning in cpuhp_smt_enable()"

[salil.mehta]

From: Salil Mehta <salil.mehta@opnsrc.net>

This reverts commit f9a82544c7174851f5c7524622f5966dcafd3a47.

That commit fixed a NULL kobject warning in cpuhp_smt_enable() by changing
arm64 so that Online-Capable but MADT-disabled CPUs are not initially in
cpu_present_mask, and by adding/removing them from that mask through the
ACPI map/unmap path.

The original arm64 virtual CPU hotplug model deliberately allowed CPUs to
be present but not enabled. cpu_enabled_mask was introduced to represent
exactly whether a present CPU may currently be brought online.

With the preceding change, cpuhp_smt_enable() now checks cpu_enabled()
before attempting _cpu_up(). This addresses the original warning at the
caller while preserving the present/enabled distinction and the arm64
virtual CPU hotplug model.

Restore the previous arm64 present-mask handling and documentation so that
all enumerated possible CPUs remain present while firmware availability is
represented by the enabled state.

This is posted as an RFC because there may be another reason why changing
the arm64 present-mask semantics is preferred over consuming the generic
enabled mask in cpuhp_smt_enable().

Signed-off-by: Salil Mehta <salil.mehta@opnsrc.net>
---
 Documentation/arch/arm64/cpu-hotplug.rst | 28 ++++++++++--------------
 arch/arm64/kernel/acpi.c                 |  2 --
 arch/arm64/kernel/smp.c                  | 12 +---------
 3 files changed, 13 insertions(+), 29 deletions(-)

diff --git a/Documentation/arch/arm64/cpu-hotplug.rst b/Documentation/arch/arm64/cpu-hotplug.rst
index 7c3379b704aa..8fb438bf7781 100644
--- a/Documentation/arch/arm64/cpu-hotplug.rst
+++ b/Documentation/arch/arm64/cpu-hotplug.rst
@@ -47,12 +47,11 @@ ever have can be described at boot. There are no power-domain considerations
 as such devices are emulated.

 CPU Hotplug on virtual systems is supported. It is distinct from physical
-CPU Hotplug as all vCPU resources are statically described in the firmware
-configuration tables (e.g. MADT), meaning their maximum possible count is
-known at boot. However, vCPUs that are not enabled at boot are not marked
-as ``present`` by the kernel until they are hotplugged. An example is where
-a virtual machine boots with a single CPU, and additional CPUs are added
-once a cloud orchestrator deploys the workload.
+CPU Hotplug as all resources are described as ``present``, but CPUs may be
+marked as disabled by firmware. Only the CPU's online/offline behaviour is
+influenced by firmware. An example is where a virtual machine boots with a
+single CPU, and additional CPUs are added once a cloud orchestrator deploys
+the workload.

 For a virtual machine, the VMM (e.g. Qemu) plays the part of firmware.

@@ -61,19 +60,16 @@ brought online. Firmware can enforce its policy via PSCI's return codes. e.g.
 ``DENIED``.

 The ACPI tables must describe all the resources of the virtual machine. CPUs
-that are hot-pluggable must have the ``online capable`` bit set and the
-``enabled`` bit cleared in the MADT GICC structures to indicate they can be
-enabled later. The boot CPU must be marked as ``enabled`` with its
-``online capable`` bit cleared. The 'always on' GICR structure must be used
-to describe the redistributors.
+that firmware wishes to disable either from boot (or later) should not be
+``enabled`` in the MADT GICC structures, but should have the ``online capable``
+bit set, to indicate they can be enabled later. The boot CPU must be marked as
+``enabled``.  The 'always on' GICR structure must be used to describe the
+redistributors.

 CPUs described as ``online capable`` but not ``enabled`` can be set to enabled
 by the DSDT's Processor object's _STA method. On virtual systems the _STA method
-must always set the ``ACPI_STA_DEVICE_PRESENT`` bit, while toggling the
-``ACPI_STA_DEVICE_ENABLED`` bit to reflect its plug status. The kernel will
-then dynamically mark the vCPU as ``present`` within the OS when the
-``ACPI_STA_DEVICE_ENABLED`` bit becomes set during hot-add. Changes to the
-firmware policy can be notified to the OS via device-check or eject-request.
+must always report the CPU as ``present``. Changes to the firmware policy can
+be notified to the OS via device-check or eject-request.

 CPUs described as ``enabled`` in the static table, should not have their _STA
 modified dynamically by firmware. Soft-restart features such as kexec will
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 681aa2bbc399..5891f92c2035 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -448,14 +448,12 @@ int acpi_map_cpu(acpi_handle handle, phys_cpuid_t physid, u32 apci_id,
 		return *pcpu;
 	}

-	set_cpu_present(*pcpu, true);
 	return 0;
 }
 EXPORT_SYMBOL(acpi_map_cpu);

 int acpi_unmap_cpu(int cpu)
 {
-	set_cpu_present(cpu, false);
 	return 0;
 }
 EXPORT_SYMBOL(acpi_unmap_cpu);
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index a61dc3016a11..0c5292e4f4e0 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -558,11 +558,6 @@ struct acpi_madt_generic_interrupt *acpi_cpu_get_madt_gicc(int cpu)
 }
 EXPORT_SYMBOL_GPL(acpi_cpu_get_madt_gicc);

-static bool acpi_cpu_is_present(int cpu)
-{
-	return acpi_cpu_get_madt_gicc(cpu)->flags & ACPI_MADT_ENABLED;
-}
-
 /*
  * acpi_map_gic_cpu_interface - parse processor MADT entry
  *
@@ -667,10 +662,6 @@ static void __init acpi_parse_and_init_cpus(void)
 		early_map_cpu_to_node(i, acpi_numa_get_nid(i));
 }
 #else
-static bool acpi_cpu_is_present(int cpu)
-{
-	return false;
-}
 #define acpi_parse_and_init_cpus(...)	do { } while (0)
 #endif

@@ -815,8 +806,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
 		if (err)
 			continue;

-		if (acpi_disabled || acpi_cpu_is_present(cpu))
-			set_cpu_present(cpu, true);
+		set_cpu_present(cpu, true);
 		numa_store_cpu_info(cpu);
 	}
 }
-- 
2.34.1

Re: [RFC PATCH 1/2] cpu/hotplug: Skip disabled CPUs in cpuhp_smt_enable

[Bradley Morgan]

On 29 September 2026 20:41:29 BST, salil.mehta@opnsrc.net wrote:
>From: Salil Mehta <salil.mehta@opnsrc.net>
>
>The CPU enabled mask describes whether a present CPU may currently be
>brought online. cpuhp_smt_enable() is an online operation, but currently
>walks all present CPUs and only filters CPUs that are already online or
>belong to offline NUMA nodes.
>
>This can make it attempt _cpu_up() for a present CPU which firmware has
>not enabled and which has not yet been registered as a CPU device.
>
>Use the enabled mask for the policy decision it was introduced to
>represent. This keeps present-but-disabled CPUs out of the SMT bring-up
>path while still allowing registered offline SMT threads to be brought
>back online.
>
>Present and enabled are separate generic CPU states, so callers which
>intend to bring CPUs online should not assume that every present CPU is
>enabled.

Ok.

>
>Signed-off-by: Salil Mehta <salil.mehta@opnsrc.net>
>---
> kernel/cpu.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
>diff --git a/kernel/cpu.c b/kernel/cpu.c
>index b3c8553d7bd6..988b7a1e8298 100644
>--- a/kernel/cpu.c
>+++ b/kernel/cpu.c
>@@ -2706,8 +2706,9 @@ int cpuhp_smt_enable(void)
> 	cpu_maps_update_begin();
> 	cpu_smt_control = CPU_SMT_ENABLED;
> 	for_each_present_cpu(cpu) {
>-		/* Skip online CPUs and CPUs on offline nodes */
>-		if (cpu_online(cpu) || !node_online(cpu_to_node(cpu)))
>+		/* Skip online/disabled CPUs and CPUs on offline nodes */
>+		if (cpu_online(cpu) || !cpu_enabled(cpu) ||
>+		    !node_online(cpu_to_node(cpu)))

Ehh, have you had a issue with this code? Like something e.g: a splat.


> 			continue;
> 		if (!cpu_smt_thread_allowed(cpu) || !topology_is_core_online(cpu))
> 			continue;
>

--- Thanks!
"I'm not a very positive person" - Linus torvalds

Re: [RFC PATCH 1/2] cpu/hotplug: Skip disabled CPUs in cpuhp_smt_enable

[Salil Mehta]

[Sincere Apologies, sending it again as Plain Text; earlier reply got
filtered perhaps due to HTML content]

Hi Bradley,

Many thanks for taking a look.


On Tue, Sep 29, 2026 at 9:16 PM Bradley Morgan <brads@mainlining.org> wrote:
>
> On 29 September 2026 20:41:29 BST, salil.mehta@opnsrc.net wrote:
> >From: Salil Mehta <salil.mehta@opnsrc.net>
> >
> >The CPU enabled mask describes whether a present CPU may currently be
> >brought online. cpuhp_smt_enable() is an online operation, but currently
> >walks all present CPUs and only filters CPUs that are already online or
> >belong to offline NUMA nodes.
> >
> >This can make it attempt _cpu_up() for a present CPU which firmware has
> >not enabled and which has not yet been registered as a CPU device.
> >
> >Use the enabled mask for the policy decision it was introduced to
> >represent. This keeps present-but-disabled CPUs out of the SMT bring-up
> >path while still allowing registered offline SMT threads to be brought
> >back online.
> >
> >Present and enabled are separate generic CPU states, so callers which
> >intend to bring CPUs online should not assume that every present CPU is
> >enabled.
>
> Ok.
>
> >
> >Signed-off-by: Salil Mehta <salil.mehta@opnsrc.net>
> >---
> > kernel/cpu.c | 5 +++--
> > 1 file changed, 3 insertions(+), 2 deletions(-)
> >
> >diff --git a/kernel/cpu.c b/kernel/cpu.c
> >index b3c8553d7bd6..988b7a1e8298 100644
> >--- a/kernel/cpu.c
> >+++ b/kernel/cpu.c
> >@@ -2706,8 +2706,9 @@ int cpuhp_smt_enable(void)
> >       cpu_maps_update_begin();
> >       cpu_smt_control = CPU_SMT_ENABLED;
> >       for_each_present_cpu(cpu) {
> >-              /* Skip online CPUs and CPUs on offline nodes */
> >-              if (cpu_online(cpu) || !node_online(cpu_to_node(cpu)))
> >+              /* Skip online/disabled CPUs and CPUs on offline nodes */
> >+              if (cpu_online(cpu) || !cpu_enabled(cpu) ||
> >+                  !node_online(cpu_to_node(cpu)))
>
> Ehh, have you had a issue with this code? Like something e.g: a splat.

No, I am not reporting a new splat or crash on current upstream. This RFC
is a proactive polite enquiry about the design choice, together with a
proposed alternative. If you check the the cover letter it provides the
context, including reproduction of the original warning with the earlier
present-mask semantics restored and the results with the proposed fix.

Just for the context, there is also an outstanding QEMU Arm vCPU hotplug
series awaiting upstream acceptance. The distinction between a CPU being
present and being enabled has been an important part of the model we have
been explaining to the QEMU community. Changing those semantics now could
complicate that work by changing the assumptions on which the interface
and its explanation have been based.

The underlying CPU architectural requirement remains that all resources
associated with the possible vCPUs are described at boot; virtual CPU
hotplug does not dynamically add or remove those resources. The patch in
contention is not changing that assumption even now but are we changing
the contract between the ACPI and the kernel or misrepresenting what has
been discovered already?

My understanding from the earlier design discussions related to support
of the vCPU Hotplug on ARM was that toggling the present mask to represent
firmware enablement was considered and ultimately rejected. The intention
was to keep the kernel’s representation consistent with the ACPI/firmware
model: a CPU can remain present while firmware controls whether it is
enabled. The separate cpu_enabled_mask was introduced to represent that
distinction.

My concern is therefore about the compatibility implications of changing
this established, userspace-visible meaning. I cannot currently point to
a specific upper-layer consumer that breaks, but neither is it
straightforward to establish that no consumers depend on it.

Catalin also initially raised the possibility of breaking other things by
no longer marking these CPUs present in the discussion of Jinjie’s
original patch, although he subsequently proposed a present-mask change
himself:

https://lore.kernel.org/lkml/aeNxKpHzTQX4_kId@arm.com/

What I would like to understand is why changing the present-mask semantics
is preferable to using the existing enabled mask in cpuhp_smt_enable().
To me, checking eligibility at this caller appears to address the original
warning more directly while preserving the present/enabled distinction.

However, I may be missing a deeper constraint or a trade-off discussed
while I was away from this work. That is why I posted this as an RFC, and
I would appreciate understanding the reasoning.

Hope this explanation helps.

Many thanks,
Salil.