[PATCH v3 0/3] cgroup/cpuset: Keep the top scheduling domain populated

[PATCH v3 0/3] cgroup/cpuset: Keep the top scheduling domain populated

[Guopeng Zhang]

From: Guopeng Zhang <zhangguopeng@kylinos.cn>

Boot-isolated CPUs stay in top_cpuset.effective_cpus, but are excluded
from its scheduling domain. Allocating all boot domain housekeeping CPUs
to partitions can therefore leave the top scheduling domain empty even
though effective_cpus is nonempty, leading to warnings and an OOPS in
build_perf_domains(). CPU hotplug can reach the same state.

The patches are ordered as follows:

  1. Check local and remote partition creation and CPU mask updates.
  2. Allow top-level partition CPU replacements that return CPUs to the
     parent in the same update.
  3. Handle CPU hotplug and preserve partitions that cannot return an
     active boot domain housekeeping CPU.

Changes since v2:
  Keep patch 1's kernel changes unchanged. Update its selftests following
  Longman's and Sashiko's comments and cover recovery with a single
  top-level HK CPU. Add patch 2 to fix the CPU replacement rejection
  reported by Sashiko, with its own regression test. Update the hotplug
  fix, now patch 3, to avoid invalidating unrelated partitions, including
  partitions whose boot housekeeping CPUs are all offline. Drop v2's
  isolated CPU cleanup patch in favor of Ridong's fix:
  https://lore.kernel.org/all/20261107120100.3587595-4-ridong.chen@linux.dev/

  v2: https://lore.kernel.org/all/20261105130344.664336-1-guopeng.zhang@linux.dev/

Changes since v1:
  Rework the fix following Ridong's suggestion and add test cases.
  Add patch 2 for CPU hotplug and patch 3 for isolated CPU cleanup
  when the parent partition becomes invalid.

  v1: https://lore.kernel.org/all/20261104071600.474587-1-guopeng.zhang@linux.dev/

Guopeng Zhang (3):
  cgroup/cpuset: Reject updates that empty the top sched domain
  cgroup/cpuset: Allow top-level partition CPU replacement
  cgroup/cpuset: Keep the top sched domain non-empty on CPU hotplug

 kernel/cgroup/cpuset.c                        | 75 +++++++++++++++++--
 .../selftests/cgroup/test_cpuset_prs.sh       | 68 +++++++++++++++++
 2 files changed, 137 insertions(+), 6 deletions(-)

-- 
2.43.0

[PATCH v3 1/3] cgroup/cpuset: Reject updates that empty the top sched domain

[Guopeng Zhang]

From: Guopeng Zhang <zhangguopeng@kylinos.cn>

Boot-isolated CPUs remain in top_cpuset.effective_cpus, but are excluded
from its scheduling domain by generate_sched_domains(). Checking only
effective_cpus when distributing CPUs to partitions can therefore allow
all boot domain housekeeping CPUs to be taken from the top cpuset.

The resulting empty domain triggers a warning in build_sched_domains()
and can cause an out-of-bounds CPU access in build_perf_domains(). Both
root and isolated partitions can trigger this, whether local or remote.
For example, on a 32-CPU system booted with isolcpus=domain,10-31:

  echo 0-9 > A/cpuset.cpus
  echo root > A/cpuset.cpus.partition
  Oops: general protection fault
  RIP: 0010:build_perf_domains+...

Require local and remote partition creation and CPU mask updates to
leave an active boot domain housekeeping CPU in the top cpuset,
accounting for CPUs returned by the same update.

Add test cases to test_cpuset_prs.sh to verify that every partition
update leaves an active boot domain housekeeping CPU in the top cpuset.

Fixes: 9b496a8bbed9 ("cgroup/cpuset: Prevent leakage of isolated CPUs into sched domains")
Co-developed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
---
 kernel/cgroup/cpuset.c                        | 48 ++++++++++++++++++-
 .../selftests/cgroup/test_cpuset_prs.sh       | 46 ++++++++++++++++++
 2 files changed, 92 insertions(+), 2 deletions(-)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 3052413e41e7..c405719ccd12 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -1180,12 +1180,56 @@ static void update_partition_sd_lb(struct cpuset *cs, int old_prs)
 		cpuset_force_rebuild();
 }
 
+/*
+ * top_cpuset_sd_nocpu - Check if top_cpuset will have no sched domain CPU
+ * @add_cpus: CPUs to be taken away from top_cpuset
+ * @del_cpus: CPUs to be given back to top_cpuset, can be NULL
+ * Return: true if no active CPU will be left in the top_cpuset sched domain
+ *
+ * Note that @del_cpus may include CPUs that are also in @add_cpus and
+ * will stay in the partition. They are skipped by the @add_cpus check,
+ * so callers can pass the current effective_xcpus of a partition
+ * without masking it first.
+ *
+ * Boot time isolated CPUs stay in top_cpuset.effective_cpus, but they are
+ * excluded from the sched domain of top_cpuset in generate_sched_domains().
+ * So checking top_cpuset.effective_cpus alone is not enough to ensure that
+ * the top_cpuset sched domain won't become empty after @add_cpus are taken
+ * away from it. The sched domain of top_cpuset must keep at least one
+ * active boot domain housekeeping CPU.
+ */
+static bool top_cpuset_sd_nocpu(const struct cpumask *add_cpus,
+				const struct cpumask *del_cpus)
+{
+	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
+	int cpu;
+
+	for_each_cpu_and(cpu, hk_mask, cpu_active_mask) {
+		if (cpumask_test_cpu(cpu, add_cpus))
+			continue;
+		if (cpumask_test_cpu(cpu, top_cpuset.effective_cpus) ||
+		    (del_cpus && cpumask_test_cpu(cpu, del_cpus)))
+			return false;
+	}
+	return true;
+}
+
 /*
  * tasks_nocpu_error - Return true if tasks will have no effective_cpus
  */
 static bool tasks_nocpu_error(struct cpuset *parent, struct cpuset *cs,
 			      struct cpumask *xcpus)
 {
+	/*
+	 * The top_cpuset always has tasks (kthreads at least), so its sched
+	 * domain can't be empty. The current effective_xcpus of a valid
+	 * partition will be given back to top_cpuset if not in @xcpus.
+	 */
+	if (parent == &top_cpuset &&
+	    top_cpuset_sd_nocpu(xcpus, is_partition_valid(cs)
+				     ? cs->effective_xcpus : NULL))
+		return true;
+
 	/*
 	 * A populated partition (cs or parent) can't have empty effective_cpus
 	 */
@@ -1724,7 +1768,7 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,
 	 */
 	compute_excpus(cs, tmp->new_cpus);
 	if (!cpumask_intersects(tmp->new_cpus, cpu_active_mask) ||
-	    cpumask_subset(top_cpuset.effective_cpus, tmp->new_cpus))
+	    top_cpuset_sd_nocpu(tmp->new_cpus, NULL))
 		return PERR_INVCPUS;
 	if (cpumask_intersects(tmp->new_cpus, subpartitions_cpus))
 		return PERR_NOCPUS;
@@ -1828,7 +1872,7 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
 		if (!capable(CAP_SYS_ADMIN))
 			WRITE_ONCE(cs->prs_err, PERR_ACCESS);
 		else if (cpumask_intersects(tmp->addmask, subpartitions_cpus) ||
-			 cpumask_subset(top_cpuset.effective_cpus, tmp->addmask))
+			 top_cpuset_sd_nocpu(tmp->addmask, tmp->delmask))
 			WRITE_ONCE(cs->prs_err, PERR_NOCPUS);
 		else if (prstate_housekeeping_conflict(prs, PRS_ROOT,
 						       excpus, tmp->delmask))
diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
index bdca6c471f87..0288af1c2eb3 100755
--- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh
+++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
@@ -168,6 +168,13 @@ TEST_ISOLCPUS=
 		skip_test "Current isolated CPUs ($CURRENT_ISOLCPUS) don't match boot isolated CPUs!"
 }
 
+# These tests require boot domain housekeeping CPUs to be exactly 0-9.
+# Match the boot mask for isolcpus= with or without explicit domain flags.
+DOMAIN_ISOLCPUS="10-$LAST_PRESENT_CPU"
+[[ "$LAST_PRESENT_CPU" == 10 ]] && DOMAIN_ISOLCPUS=10
+RUN_ISOLCPUS_DOMAIN_TESTS=
+[[ "$BOOT_ISOLCPUS" == "$DOMAIN_ISOLCPUS" ]] && RUN_ISOLCPUS_DOMAIN_TESTS=1
+
 #
 # This test assumes that all the relevant CPUs being tested are online.
 # If some of them are offline, some of the tests may fail. So we are going to
@@ -602,6 +609,44 @@ ISOLCPUS_TEST_MATRIX=(
 	" CX1-2,I:X1,I:P1 . .      .     X1,2,I  .      .      .     0 A1:1-2,I|XA1:1-2,I A1:P-1 ."
 )
 
+# Test matrix for isolcpus=domain,10-<last> or isolcpus=10-<last> boots.
+# The top cpuset must retain an active boot domain housekeeping CPU.
+ISOLCPUS_DOMAIN_TEST_MATRIX=(
+	#  old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pstate ISOLCPUS
+	#  ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ------ --------
+	# Neither a root nor an isolated partition may take all boot HK CPUs.
+	" C0-9      .      .      .     P1      .      .      .     0 A1:0-9 A1:P-1 ."
+	" C0-9      .      .      .     P2      .      .      .     0 A1:0-9 A1:P-2 ."
+
+	# Growing cpuset.cpus must not take the last top-level HK CPU.
+	" C0-8:P1   .      .      .    C0-9     .      .      .     0 A1:0-9 A1:P-1 ."
+
+	# CPUs returned by the same update can keep the top sched domain alive.
+	" C2-9:P2   .      .      .    C0-4     .      .      .     0 A1:0-4 A1:P2 0-4"
+
+	# Root and isolated partitions together must leave a top-level HK CPU.
+	" C0-4:P1   .      .     C5-9    .      .      .     P2    0 A1:0-4|B1:5-9 A1:P1|B1:P-2 ."
+
+	# Remote partition creation must enforce the same constraint.
+	" X0-9    CX0-9    .      .      .     P1      .      .     0 A2:0-9 A2:P-1 ."
+
+	# Growing a remote partition through cpuset.cpus.exclusive is rejected.
+	" X0-9 C0-9:X0-8:P1 .     .      .    X0-9     .      .     0 A2:0-9 A2:P-1 ."
+
+	# Remote CPU replacements must account for the CPUs being returned.
+	" X0-9 C0-9:X2-9:P2 .     .      .    X0-4     .      .     0 A2:0-4 A2:P2 0-4"
+
+	# Local exclusive CPU updates must not consume the last top-level HK CPU.
+	# Keep the initial partition valid with nohz_full enabled.
+	" C0-9:X2-9:P2 .   .      .    X0-9     .      .      .     0 A1:0-9 A1:P-2 ."
+
+	# Ancestor updates must not grow a remote partition over the last HK CPU.
+	" X0-8 CX0-9:P1    .      .    X0-9     .      .      .     0 A2:0-9 A2:P-1 ."
+
+	# An invalid partition can recover once it leaves a top-level HK CPU.
+	" C0-9:P1   .      .      .    C0-6,8-9 .      .      .     0 A1:0-6,8-9 A1:P1 7|"
+)
+
 # Test matrix to be run when boot time nohz_full CPUs cover all CPUs except CPUs 0-7
 # This also means that CPU 8 is available and is a nohz_full CPU.
 NOHZ_FULL_TEST_MATRIX=(
@@ -1472,6 +1517,7 @@ test_inotify()
 run_state_test TEST_MATRIX
 run_remote_state_test REMOTE_TEST_MATRIX
 TEST_ISOLCPUS=1 run_state_test ISOLCPUS_TEST_MATRIX
+[[ -n "$RUN_ISOLCPUS_DOMAIN_TESTS" ]] && run_state_test ISOLCPUS_DOMAIN_TEST_MATRIX
 [[ -n "$RUN_NOHZ_FULL_TESTS" ]] && run_state_test NOHZ_FULL_TEST_MATRIX
 test_isolated
 test_boot_isolated
-- 
2.43.0

Re: [PATCH v3 1/3] cgroup/cpuset: Reject updates that empty the top sched domain

[Ridong Chen]

On 11/10/2026 1:03 PM, Guopeng Zhang wrote:
> From: Guopeng Zhang <zhangguopeng@kylinos.cn>
> 
> Boot-isolated CPUs remain in top_cpuset.effective_cpus, but are excluded
> from its scheduling domain by generate_sched_domains(). Checking only
> effective_cpus when distributing CPUs to partitions can therefore allow
> all boot domain housekeeping CPUs to be taken from the top cpuset.
> 
> The resulting empty domain triggers a warning in build_sched_domains()
> and can cause an out-of-bounds CPU access in build_perf_domains(). Both
> root and isolated partitions can trigger this, whether local or remote.
> For example, on a 32-CPU system booted with isolcpus=domain,10-31:
> 
>    echo 0-9 > A/cpuset.cpus
>    echo root > A/cpuset.cpus.partition
>    Oops: general protection fault
>    RIP: 0010:build_perf_domains+...
> 
> Require local and remote partition creation and CPU mask updates to
> leave an active boot domain housekeeping CPU in the top cpuset,
> accounting for CPUs returned by the same update.
> 
> Add test cases to test_cpuset_prs.sh to verify that every partition
> update leaves an active boot domain housekeeping CPU in the top cpuset.
> 
> Fixes: 9b496a8bbed9 ("cgroup/cpuset: Prevent leakage of isolated CPUs into sched domains")
> Co-developed-by: Ridong Chen <ridong.chen@linux.dev>
> Signed-off-by: Ridong Chen <ridong.chen@linux.dev>

It would be better with: Ridong Chen <chenridong@xiaomi.com>

-- 
Best regards
Ridong

[PATCH v3 2/3] cgroup/cpuset: Allow top-level partition CPU replacement

[Guopeng Zhang]

From: Guopeng Zhang <zhangguopeng@kylinos.cn>

When replacing a partition's CPUs, tasks_nocpu_error() checks whether
the parent's current effective_cpus is a subset of the new exclusive
CPU mask. The parent mask does not yet include CPUs being returned by
the update, so this can invalidate an otherwise valid partition.

For example, on an 8-CPU system without boot isolation:

  echo 2-7 > A/cpuset.cpus
  echo isolated > A/cpuset.cpus.partition
  echo 0-4 > A/cpuset.cpus
  cat A/cpuset.cpus.partition
  isolated invalid (Parent unable to distribute cpu downstream)

The update should return CPUs 5-7 to the parent and leave A as a
valid isolated partition.

Skip this subset check for top_cpuset. top_cpuset_sd_nocpu() already
accounts for returned CPUs and ensures that an active boot domain
housekeeping CPU remains.

Add a regression test for this case.

Fixes: 0c7f293efc87 ("cgroup/cpuset: Add cpuset.cpus.exclusive.effective for v2")
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
---
 kernel/cgroup/cpuset.c                            | 4 +++-
 tools/testing/selftests/cgroup/test_cpuset_prs.sh | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index c405719ccd12..8b6831983803 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -1232,8 +1232,10 @@ static bool tasks_nocpu_error(struct cpuset *parent, struct cpuset *cs,
 
 	/*
 	 * A populated partition (cs or parent) can't have empty effective_cpus
+	 * The top_cpuset check above also accounts for CPUs being returned.
 	 */
-	return (cpumask_subset(parent->effective_cpus, xcpus) &&
+	return (parent != &top_cpuset &&
+		cpumask_subset(parent->effective_cpus, xcpus) &&
 		partition_is_populated(parent, cs)) ||
 	       (!cpumask_intersects(xcpus, cpu_active_mask) &&
 		partition_is_populated(cs, NULL));
diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
index 0288af1c2eb3..ee4d57b7e35b 100755
--- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh
+++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
@@ -554,6 +554,9 @@ TEST_MATRIX=(
 	" C2-6:X2-5:P1 CX3-6:P2 \
 	                    .      .   T:X4-6    .      .      .     0 A1:4-6|A2:4-6 A1:P1|A2:P-2 ."
 
+	# A CPU replacement must account for CPUs returned to the top cpuset.
+	" C2-${CPULIST#0-}:P2 . . . C0-4 . . . 0 A1:0-4 A1:P2 0-4"
+
 	#
 	#  old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pstate ISOLCPUS
 	#  ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ------ --------
-- 
2.43.0

Re: [PATCH v3 2/3] cgroup/cpuset: Allow top-level partition CPU replacement

[Ridong Chen]

On 11/10/2026 1:03 PM, Guopeng Zhang wrote:
> From: Guopeng Zhang <zhangguopeng@kylinos.cn>
> 
> When replacing a partition's CPUs, tasks_nocpu_error() checks whether
> the parent's current effective_cpus is a subset of the new exclusive
> CPU mask. The parent mask does not yet include CPUs being returned by
> the update, so this can invalidate an otherwise valid partition.
> 
> For example, on an 8-CPU system without boot isolation:
> 
>    echo 2-7 > A/cpuset.cpus
>    echo isolated > A/cpuset.cpus.partition
>    echo 0-4 > A/cpuset.cpus
>    cat A/cpuset.cpus.partition
>    isolated invalid (Parent unable to distribute cpu downstream)
> 
> The update should return CPUs 5-7 to the parent and leave A as a
> valid isolated partition.
> 
> Skip this subset check for top_cpuset. top_cpuset_sd_nocpu() already
> accounts for returned CPUs and ensures that an active boot domain
> housekeeping CPU remains.
> 
> Add a regression test for this case.
> 
> Fixes: 0c7f293efc87 ("cgroup/cpuset: Add cpuset.cpus.exclusive.effective for v2")
> Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>

Hi Guopeng,

Patch 2 depends on patch 1. Skipping the cpumask_subset() check for
top_cpuset is only safe because top_cpuset_sd_nocpu() added by patch 1
now covers it. Without patch 1, patch 2 alone would let a partition
take away all the CPUs of top_cpuset. So patch 2 can't be backported
on its own, and the two have different Fixes tags. I think it would be
better to fold patch 2 into patch 1.

Also, the fix in patch 2 seems incomplete. The same false PERR_NOCPUS
happens when the parent is a local partition, as cpumask_subset()
there doesn't take the CPUs given back by the update into account
either. On an 8-CPU system:

	cd /sys/fs/cgroup
	echo +cpuset > cgroup.subtree_control
	mkdir A1
	echo 1-7 > A1/cpuset.cpus
	echo root > A1/cpuset.cpus.partition
	echo +cpuset > A1/cgroup.subtree_control
	mkdir A1/A2
	echo 3-7 > A1/A2/cpuset.cpus
	echo isolated > A1/A2/cpuset.cpus.partition
	sleep 1000 &
	echo $! > A1/cgroup.procs
	echo 1-4 > A1/A2/cpuset.cpus
	cat A1/A2/cpuset.cpus.partition
	isolated invalid (Parent unable to distribute cpu downstream)

A2 takes CPUs 1-2 from A1 and gives CPUs 5-7 back, so A1 still has
CPUs for its task. Instead of skipping the check for top_cpuset only,
how about making the check in tasks_nocpu_error() account for the CPUs
given back by cs for all parents? The hotplug caller would then need
to not count cs->effective_xcpus as given back, similar to what patch 3
does for top-level partitions.

> ---
>   kernel/cgroup/cpuset.c                            | 4 +++-
>   tools/testing/selftests/cgroup/test_cpuset_prs.sh | 3 +++
>   2 files changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index c405719ccd12..8b6831983803 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -1232,8 +1232,10 @@ static bool tasks_nocpu_error(struct cpuset *parent, struct cpuset *cs,
>   
>   	/*
>   	 * A populated partition (cs or parent) can't have empty effective_cpus
> +	 * The top_cpuset check above also accounts for CPUs being returned.
>   	 */
> -	return (cpumask_subset(parent->effective_cpus, xcpus) &&
> +	return (parent != &top_cpuset &&
> +		cpumask_subset(parent->effective_cpus, xcpus) &&
>   		partition_is_populated(parent, cs)) ||
>   	       (!cpumask_intersects(xcpus, cpu_active_mask) &&
>   		partition_is_populated(cs, NULL));
> diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> index 0288af1c2eb3..ee4d57b7e35b 100755
> --- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> +++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> @@ -554,6 +554,9 @@ TEST_MATRIX=(
>   	" C2-6:X2-5:P1 CX3-6:P2 \
>   	                    .      .   T:X4-6    .      .      .     0 A1:4-6|A2:4-6 A1:P1|A2:P-2 ."
>   
> +	# A CPU replacement must account for CPUs returned to the top cpuset.
> +	" C2-${CPULIST#0-}:P2 . . . C0-4 . . . 0 A1:0-4 A1:P2 0-4"
> +
>   	#
>   	#  old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pstate ISOLCPUS
>   	#  ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ------ --------

-- 
Best regards
Ridong

[PATCH v3 3/3] cgroup/cpuset: Keep the top sched domain non-empty on CPU hotplug

[Guopeng Zhang]

From: Guopeng Zhang <zhangguopeng@kylinos.cn>

In cpuset_hotplug_update_tasks(), the local partition check passes only
the partition's own effective CPUs to tasks_nocpu_error(). For a
top-level partition with children, top_cpuset_sd_nocpu() treats the
children's CPUs as being returned, although hotplug does not return
them. This can leave the top sched domain empty when its last boot
domain housekeeping CPU goes offline.

For example, on a 32-CPU system with isolcpus=domain,10-31:

  echo 0,2-9 > A/cpuset.cpus
  echo root > A/cpuset.cpus.partition
  echo 5-9 > A/B/cpuset.cpus
  echo root > A/B/cpuset.cpus.partition
  echo 0 > /sys/devices/system/cpu/cpu1/online
  Oops: general protection fault
  RIP: 0010:build_perf_domains+...

The remote hotplug path does not check whether the top sched domain
is empty.

Check the full effective_xcpus of top-level local and remote partitions
without assuming any CPUs have been returned. Only invalidate a
partition to restore the top domain if it can return an active boot
domain housekeeping CPU. Partitions holding only boot-isolated or
offline housekeeping CPUs cannot do that.

Add hotplug regression tests.

Fixes: 9b496a8bbed9 ("cgroup/cpuset: Prevent leakage of isolated CPUs into sched domains")
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
---
 kernel/cgroup/cpuset.c                        | 23 ++++++++++++++++---
 .../selftests/cgroup/test_cpuset_prs.sh       | 19 +++++++++++++++
 2 files changed, 39 insertions(+), 3 deletions(-)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 8b6831983803..0b392d50df27 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -4043,6 +4043,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 	static nodemask_t new_mems;
 	bool cpus_updated;
 	bool mems_updated;
+	bool nocpu;
 	int partcmd = -1;
 	struct cpuset *parent;
 
@@ -4054,9 +4055,24 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 	if (!tmp || !cs->partition_root_state)
 		goto update_tasks;
 
+	if (parent == &top_cpuset || is_remote_partition(cs)) {
+		/*
+		 * Hotplug does not return CPUs held by child partitions. Only
+		 * invalidate a partition for an empty top sched domain if it
+		 * can return an active boot domain housekeeping CPU.
+		 */
+		cpumask_and(tmp->addmask, cs->effective_xcpus,
+			    housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
+		nocpu = cpumask_intersects(tmp->addmask, cpu_active_mask) &&
+			top_cpuset_sd_nocpu(cs->effective_xcpus, NULL);
+		nocpu |= cpumask_empty(&new_cpus) &&
+			 partition_is_populated(cs, NULL);
+	} else {
+		nocpu = tasks_nocpu_error(parent, cs, &new_cpus);
+	}
+
 	if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) ||
-				       (cpumask_empty(&new_cpus) &&
-					partition_is_populated(cs, NULL)))) {
+				       nocpu)) {
 		WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
 		remote_partition_disable(cs, tmp);
 		compute_effective_cpumask(&new_cpus, cs, parent);
@@ -4069,10 +4085,11 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 	 * 2) parent is invalid or doesn't grant any cpus to child
 	 *    partitions.
 	 * 3) subpartitions_cpus is empty.
+	 * 4) top_cpuset needs an active boot domain housekeeping CPU from cs.
 	 */
 	if (is_local_partition(cs) &&
 	    (!is_partition_valid(parent) ||
-	     tasks_nocpu_error(parent, cs, &new_cpus) ||
+	     nocpu ||
 	     cpumask_empty(subpartitions_cpus)))
 		partcmd = partcmd_invalidate;
 	/*
diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
index ee4d57b7e35b..f345696e5b53 100755
--- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh
+++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
@@ -648,8 +648,27 @@ ISOLCPUS_DOMAIN_TEST_MATRIX=(
 
 	# An invalid partition can recover once it leaves a top-level HK CPU.
 	" C0-9:P1   .      .      .    C0-6,8-9 .      .      .     0 A1:0-6,8-9 A1:P1 7|"
+
+	# A local root must not count its child's CPUs as already returned.
+	" C0,2-9:P1 C5-9:P1 . . O1=0 . . . 0 A1:0,2-9|A2:5-9 A1:P-1|A2:P-1 1|"
+
+	# A remote root must return CPUs when the top domain loses its last HK CPU.
+	" X0,2-9 CX0,2-9:P1 . . O1=0 . . . 0 A2:0,2-9 A2:P-1 1|"
 )
 
+# Keep an online boot-isolated CPU at the top so hotplug does not
+# clear subpartitions_cpus before checking the individual partitions.
+if [[ "$ONLINE_LAST" -gt 10 ]]; then
+	ISOLCPUS_DOMAIN_TEST_MATRIX+=(
+		# An offline HK CPU cannot restore the top domain. Keep this local
+		# partition valid while its sibling returns active HK CPUs.
+		" O2=0:C2,10:P2 . . C0,3-9:P1 O1=0 . . . 0 A1:10|B1:0,3-9 A1:P2|B1:P-1 1-2|2"
+
+		# Check the same active-CPU rule for a remote partition.
+		" O2=0:X2,10 CX2,10:P2 . C0,3-9:P1 O1=0 . . . 0 A2:10|B1:0,3-9 A2:P2|B1:P-1 1-2|2"
+	)
+fi
+
 # Test matrix to be run when boot time nohz_full CPUs cover all CPUs except CPUs 0-7
 # This also means that CPU 8 is available and is a nohz_full CPU.
 NOHZ_FULL_TEST_MATRIX=(
-- 
2.43.0

Re: [PATCH v3 3/3] cgroup/cpuset: Keep the top sched domain non-empty on CPU hotplug

[Ridong Chen]

On 11/10/2026 1:03 PM, Guopeng Zhang wrote:
> From: Guopeng Zhang <zhangguopeng@kylinos.cn>
> 
> In cpuset_hotplug_update_tasks(), the local partition check passes only
> the partition's own effective CPUs to tasks_nocpu_error(). For a
> top-level partition with children, top_cpuset_sd_nocpu() treats the
> children's CPUs as being returned, although hotplug does not return
> them. This can leave the top sched domain empty when its last boot
> domain housekeeping CPU goes offline.
> 
> For example, on a 32-CPU system with isolcpus=domain,10-31:
> 
>    echo 0,2-9 > A/cpuset.cpus
>    echo root > A/cpuset.cpus.partition
>    echo 5-9 > A/B/cpuset.cpus
>    echo root > A/B/cpuset.cpus.partition
>    echo 0 > /sys/devices/system/cpu/cpu1/online
>    Oops: general protection fault
>    RIP: 0010:build_perf_domains+...
> 
> The remote hotplug path does not check whether the top sched domain
> is empty.
> 
> Check the full effective_xcpus of top-level local and remote partitions
> without assuming any CPUs have been returned. Only invalidate a
> partition to restore the top domain if it can return an active boot
> domain housekeeping CPU. Partitions holding only boot-isolated or
> offline housekeeping CPUs cannot do that.
> 
> Add hotplug regression tests.
> 
> Fixes: 9b496a8bbed9 ("cgroup/cpuset: Prevent leakage of isolated CPUs into sched domains")
> Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
> ---
>   kernel/cgroup/cpuset.c                        | 23 ++++++++++++++++---
>   .../selftests/cgroup/test_cpuset_prs.sh       | 19 +++++++++++++++
>   2 files changed, 39 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 8b6831983803..0b392d50df27 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -4043,6 +4043,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
>   	static nodemask_t new_mems;
>   	bool cpus_updated;
>   	bool mems_updated;
> +	bool nocpu;
>   	int partcmd = -1;
>   	struct cpuset *parent;
>   
> @@ -4054,9 +4055,24 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
>   	if (!tmp || !cs->partition_root_state)
>   		goto update_tasks;
>   
> +	if (parent == &top_cpuset || is_remote_partition(cs)) {
> +		/*
> +		 * Hotplug does not return CPUs held by child partitions. Only
> +		 * invalidate a partition for an empty top sched domain if it
> +		 * can return an active boot domain housekeeping CPU.
> +		 */
> +		cpumask_and(tmp->addmask, cs->effective_xcpus,
> +			    housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
> +		nocpu = cpumask_intersects(tmp->addmask, cpu_active_mask) &&
> +			top_cpuset_sd_nocpu(cs->effective_xcpus, NULL);
> +		nocpu |= cpumask_empty(&new_cpus) &&
> +			 partition_is_populated(cs, NULL);
> +	} else {
> +		nocpu = tasks_nocpu_error(parent, cs, &new_cpus);
> +	}
> +
>   	if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) ||
> -				       (cpumask_empty(&new_cpus) &&
> -					partition_is_populated(cs, NULL)))) {
> +				       nocpu)) {
>   		WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
>   		remote_partition_disable(cs, tmp);
>   		compute_effective_cpumask(&new_cpus, cs, parent);
> @@ -4069,10 +4085,11 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
>   	 * 2) parent is invalid or doesn't grant any cpus to child
>   	 *    partitions.
>   	 * 3) subpartitions_cpus is empty.
> +	 * 4) top_cpuset needs an active boot domain housekeeping CPU from cs.
>   	 */
>   	if (is_local_partition(cs) &&
>   	    (!is_partition_valid(parent) ||
> -	     tasks_nocpu_error(parent, cs, &new_cpus) ||
> +	     nocpu ||
>   	     cpumask_empty(subpartitions_cpus)))
>   		partcmd = partcmd_invalidate;
>   	/*
> diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> index ee4d57b7e35b..f345696e5b53 100755
> --- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> +++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> @@ -648,8 +648,27 @@ ISOLCPUS_DOMAIN_TEST_MATRIX=(
>   
>   	# An invalid partition can recover once it leaves a top-level HK CPU.
>   	" C0-9:P1   .      .      .    C0-6,8-9 .      .      .     0 A1:0-6,8-9 A1:P1 7|"
> +
> +	# A local root must not count its child's CPUs as already returned.
> +	" C0,2-9:P1 C5-9:P1 . . O1=0 . . . 0 A1:0,2-9|A2:5-9 A1:P-1|A2:P-1 1|"
> +
> +	# A remote root must return CPUs when the top domain loses its last HK CPU.
> +	" X0,2-9 CX0,2-9:P1 . . O1=0 . . . 0 A2:0,2-9 A2:P-1 1|"
>   )
>   
> +# Keep an online boot-isolated CPU at the top so hotplug does not
> +# clear subpartitions_cpus before checking the individual partitions.
> +if [[ "$ONLINE_LAST" -gt 10 ]]; then
> +	ISOLCPUS_DOMAIN_TEST_MATRIX+=(
> +		# An offline HK CPU cannot restore the top domain. Keep this local
> +		# partition valid while its sibling returns active HK CPUs.
> +		" O2=0:C2,10:P2 . . C0,3-9:P1 O1=0 . . . 0 A1:10|B1:0,3-9 A1:P2|B1:P-1 1-2|2"
> +
> +		# Check the same active-CPU rule for a remote partition.
> +		" O2=0:X2,10 CX2,10:P2 . C0,3-9:P1 O1=0 . . . 0 A2:10|B1:0,3-9 A2:P2|B1:P-1 1-2|2"
> +	)
> +fi
> +
>   # Test matrix to be run when boot time nohz_full CPUs cover all CPUs except CPUs 0-7
>   # This also means that CPU 8 is available and is a nohz_full CPU.
>   NOHZ_FULL_TEST_MATRIX=(

Hi Guopeng,

Looking at patches 1-3, the same check is changed back and forth in
several places, and the places don't do it in the same way. I'd prefer
a single helper that is unified and easy to understand, so that every
place can just call it.

As tasks_nocpu_error() is shared by the CPU mask update paths and the
hotplug path, patches 1-3 end up checking the same thing in several
places with different masks. How about making tasks_nocpu_error() take
the CPUs to be returned as an explicit argument, the same way as
top_cpuset_sd_nocpu() does, and use it for all parents? Something like
the following (not tested):

/*
* tasks_nocpu_error - Return true if tasks will have no effective_cpus
* @parent: parent partition, NULL for a remote partition
* @cs: the partition
* @xcpus: exclusive CPUs that @cs will hold after the update
* @del_cpus: exclusive CPUs to be returned to @parent, can be NULL
*
* @del_cpus may include CPUs that are also in @xcpus and will stay in
* @cs. They are skipped by the @xcpus check, so callers can pass the
* current effective_xcpus of @cs without masking it first.
*
* top_cpuset always has tasks, and its sched domain needs an active boot
* domain housekeeping CPU.
*/
static bool tasks_nocpu_error(struct cpuset *parent, struct cpuset *cs,
							const struct cpumask *xcpus,
							const struct cpumask *del_cpus)
{
	const struct cpumask *mask = cpu_possible_mask;
	int cpu;

	/* A populated @cs must have an active CPU in @xcpus */
	if (!cpumask_intersects(xcpus, cpu_active_mask) &&
		partition_is_populated(cs, NULL))
			return true;

	/*
	 * A populated @parent must keep an active CPU in @mask, either from
	 * its effective_cpus or from @del_cpus, that isn't taken by @cs.
	 */
	if (!parent)
		  parent = &top_cpuset;

	if (parent == &top_cpuset)
		  mask = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
	else if (!partition_is_populated(parent, cs))
		  return false;
		
	for_each_cpu_and(cpu, mask, cpu_active_mask) {
		  if (cpumask_test_cpu(cpu, xcpus))
				  continue;
		  if (cpumask_test_cpu(cpu, parent->effective_cpus) ||
			  (del_cpus && cpumask_test_cpu(cpu, del_cpus)))
				  return false;
	}
	return true;
}

The callers then become:

/* update_parent_effective_cpumask() */
nocpu = tasks_nocpu_error(parent, cs, tmp->delmask,
			is_partition_valid(cs) ? cs->effective_xcpus : NULL);
						
/* validate_partition() */
if (tasks_nocpu_error(is_remote_partition(cs) ? NULL : parent, cs,
			trialcs->effective_xcpus,
			is_partition_valid(cs) ? cs->effective_xcpus : NULL))
					
/* remote_partition_enable(), instead of top_cpuset_sd_nocpu() */
tasks_nocpu_error(NULL, cs, tmp->new_cpus, NULL)

/* remote_cpus_update(), instead of top_cpuset_sd_nocpu() */
tasks_nocpu_error(NULL, cs, excpus, cs->effective_xcpus)

/* cpuset_hotplug_update_tasks(), hotplug returns nothing */
nocpu = tasks_nocpu_error(is_remote_partition(cs) ? NULL : parent, cs,
						&new_cpus, NULL);
							
In the hotplug path, @del_cpus is NULL as nothing is returned to the
parent. new_cpus can be used as @xcpus for both checks, as the CPUs of
the child partitions of cs are not in the parent's effective_cpus
anyway. The check that cs can return an active boot domain housekeeping
CPU before invalidating it can stay there.

This also covers the local parent case I mentioned in patch 2, so
patch 2 won't be needed. Could you give it a try with your test cases
and that reproducer?

-- 
Best regards
Ridong