[PATCH] cgroup/cpuset: Fail if isolated CPUs don't leave any boot domain housekeeping CPU

[PATCH] cgroup/cpuset: Fail if isolated CPUs don't leave any boot domain housekeeping CPU

[Guopeng Zhang]

From: Guopeng Zhang <zhangguopeng@kylinos.cn>

With the "domain" flag of the isolcpus= boot option, the CPUs isolated
at boot are excluded from the sched domains and the remaining CPUs
become the boot domain housekeeping CPUs. An isolated partition is
allowed to use the boot isolated CPUs, but it must not take over all
the boot domain housekeeping CPUs.

prstate_housekeeping_conflict() checks that rule for root partitions
only. The 2nd check covers the nohz_full case as it is gated by
housekeeping_enabled(HK_TYPE_KERNEL_NOISE) which is not enabled by
isolcpus=domain. So on an isolcpus=domain,<cpus> boot, an isolated
partition can claim all the remaining boot domain housekeeping CPUs.
The state is committed, but the sched domain rebuild at the end of
the write finds no housekeeping CPU left for the boot domain and
hits WARNs and an OOPS in partition_sched_domains().
housekeeping_update() cannot accept the configuration either as it
cannot empty the boot domain housekeeping mask.
With an "isolcpus=domain,10-31" boot where CPUs 0-9 are the boot
domain housekeeping CPUs, for example:

  # echo 0-9 > A/cpuset.cpus
  # echo isolated > A/cpuset.cpus.partition
  WARNING: kernel/sched/topology.c:3089 at build_sched_domains+...
  WARNING: kernel/cgroup/cpuset.c:1478 at cpuset_update_sd_hk_unlock+...
  RIP: 0010:partition_sched_domains+0x3b8

Fix that by extending prstate_housekeeping_conflict() to reject an
isolated partition update that would leave no active boot domain
housekeeping CPU outside the isolated CPUs. The CPUs released back
to the boot domain by the same update are taken into account, so
replacing one of the last housekeeping CPUs with another one in a
single update remains allowed. The new check is only performed when
housekeeping_enabled(HK_TYPE_KERNEL_NOISE) is disabled so that the
existing nohz_full behavior is unchanged.

Add test cases in test_cpuset_prs.sh to verify that taking over all
the boot domain housekeeping CPUs is rejected while swapping one of
them with another one is allowed.

Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
---
 kernel/cgroup/cpuset.c                        | 36 +++++++++++++++++++
 .../selftests/cgroup/test_cpuset_prs.sh       | 30 ++++++++++++++++
 2 files changed, 66 insertions(+)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 6e136888fc8e..c1ed556ddb1a 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -1403,6 +1403,42 @@ static bool prstate_housekeeping_conflict(int new_prs, int parent_prs,
 	if (del_cpus && cpumask_empty(del_cpus))
 		del_cpus = NULL;
 
+	/*
+	 * With boot time domain isolation (isolcpus= with the domain flag),
+	 * the CPUs that are not isolated at boot become the boot domain
+	 * housekeeping CPUs. An isolated partition must not take over all
+	 * of them. Otherwise housekeeping_update() will fail later when
+	 * the isolated CPUs are handed over, leaving the sched domains in
+	 * an inconsistent state. This case isn't covered by the 2nd check
+	 * below which requires nohz_full to be enabled.
+	 */
+	if (!housekeeping_enabled(HK_TYPE_KERNEL_NOISE) &&
+	    housekeeping_enabled(HK_TYPE_DOMAIN_BOOT) && (new_prs == PRS_ISOLATED)) {
+		cpumask_var_t boot_hk_cpus;
+
+		if (!alloc_cpumask_var(&boot_hk_cpus, GFP_KERNEL))
+			return true;
+
+		/*
+		 * Compute the isolated CPUs after this update: the CPUs
+		 * released by it (@del_cpus) leave the isolated set and
+		 * become available to the boot domain again, unless they
+		 * are returned to an isolated parent instead.
+		 */
+		if (del_cpus && parent_prs != PRS_ISOLATED)
+			cpumask_andnot(boot_hk_cpus, isolated_cpus, del_cpus);
+		else
+			cpumask_copy(boot_hk_cpus, isolated_cpus);
+		cpumask_andnot(boot_hk_cpus,
+			       housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT),
+			       boot_hk_cpus);
+		cpumask_andnot(boot_hk_cpus, boot_hk_cpus, new_cpus);
+		res = !cpumask_intersects(boot_hk_cpus, cpu_active_mask);
+		free_cpumask_var(boot_hk_cpus);
+		if (res)
+			return true;
+	}
+
 	/*
 	 * Exit 2nd check early and return false if
 	 * 1) there is no boot time isolated nohz_full CPU
diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
index bdca6c471f87..8c7af34b51c8 100755
--- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh
+++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
@@ -133,6 +133,17 @@ LAST_PRESENT_CPU=$(cat /sys/devices/system/cpu/present | sed -e "s/.*[,-]//")
 [[ -n "$FIRST_NOHZ_FULL_CPU" && "$FIRST_NOHZ_FULL_CPU" = 8 &&
    "$LAST_NOHZ_FULL_CPU" = $LAST_PRESENT_CPU ]] && RUN_NOHZ_FULL_TESTS=1
 
+# The isolcpus domain test matrix assumes that the boot domain
+# housekeeping CPUs are exactly 0-9, i.e. an
+# "isolcpus=domain,10-<last>" boot.
+RUN_ISOLCPUS_DOMAIN_TESTS=
+if grep -q "isolcpus=.*domain" /proc/cmdline 2>/dev/null &&
+	[[ -z "$(cat /sys/devices/system/cpu/nohz_full 2>/dev/null)" ]]
+then
+	[[ "$(cat /sys/devices/system/cpu/isolated)" = "10-$LAST_PRESENT_CPU" ]] &&
+		RUN_ISOLCPUS_DOMAIN_TESTS=1
+fi
+
 #
 # If isolated CPUs have been reserved at boot time (as shown in
 # /sys/devices/system/cpu/isolated), these isolated CPUs should be outside of
@@ -602,6 +613,24 @@ ISOLCPUS_TEST_MATRIX=(
 	" CX1-2,I:X1,I:P1 . .      .     X1,2,I  .      .      .     0 A1:1-2,I|XA1:1-2,I A1:P-1 ."
 )
 
+# Test matrix to be run when the system is booted with
+# "isolcpus=domain,10-<last>" so that CPUs 0-9 are the boot domain
+# housekeeping CPUs.
+ISOLCPUS_DOMAIN_TEST_MATRIX=(
+	#  old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pstate ISOLCPUS
+	#  ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ------ --------
+	# Using all the boot domain housekeeping CPUs in an isolated
+	# partition is not allowed
+	" C0-9      .      .      .     P2      .      .      .     0 A1:0-9  A1:P-2  ."
+
+	# Replacing some of the boot domain housekeeping CPUs with other
+	# ones in a single update is allowed
+	" C2-9:P2   .      .      .    C0-4     .      .      .     0 A1:0-4  A1:P2   0-4"
+
+	# Taking over the last housekeeping CPU by an update is not allowed
+	" C0-8:P2   .      .      .    C0-9     .      .      .     0 A1:0-9  A1:P-2  ."
+)
+
 # Test matrix to be run when boot time nohz_full CPUs cover all CPUs except CPUs 0-7
 # This also means that CPU 8 is available and is a nohz_full CPU.
 NOHZ_FULL_TEST_MATRIX=(
@@ -1472,6 +1501,7 @@ test_inotify()
 run_state_test TEST_MATRIX
 run_remote_state_test REMOTE_TEST_MATRIX
 TEST_ISOLCPUS=1 run_state_test ISOLCPUS_TEST_MATRIX
+[[ -n "$RUN_ISOLCPUS_DOMAIN_TESTS" ]] && run_state_test ISOLCPUS_DOMAIN_TEST_MATRIX
 [[ -n "$RUN_NOHZ_FULL_TESTS" ]] && run_state_test NOHZ_FULL_TEST_MATRIX
 test_isolated
 test_boot_isolated
-- 
2.43.0

Re: [PATCH] cgroup/cpuset: Fail if isolated CPUs don't leave any boot domain housekeeping CPU

[Ridong Chen]

On 11/4/2026 3:16 PM, Guopeng Zhang wrote:
> From: Guopeng Zhang<zhangguopeng@kylinos.cn>
> 
> With the "domain" flag of the isolcpus= boot option, the CPUs isolated
> at boot are excluded from the sched domains and the remaining CPUs
> become the boot domain housekeeping CPUs. An isolated partition is
> allowed to use the boot isolated CPUs, but it must not take over all
> the boot domain housekeeping CPUs.
> 
> prstate_housekeeping_conflict() checks that rule for root partitions
> only. The 2nd check covers the nohz_full case as it is gated by
> housekeeping_enabled(HK_TYPE_KERNEL_NOISE) which is not enabled by
> isolcpus=domain. So on an isolcpus=domain,<cpus> boot, an isolated
> partition can claim all the remaining boot domain housekeeping CPUs.
> The state is committed, but the sched domain rebuild at the end of
> the write finds no housekeeping CPU left for the boot domain and
> hits WARNs and an OOPS in partition_sched_domains().
> housekeeping_update() cannot accept the configuration either as it
> cannot empty the boot domain housekeeping mask.
> With an "isolcpus=domain,10-31" boot where CPUs 0-9 are the boot
> domain housekeeping CPUs, for example:
> 
>    # echo 0-9 > A/cpuset.cpus
>    # echo isolated > A/cpuset.cpus.partition
>    WARNING: kernel/sched/topology.c:3089 at build_sched_domains+...
>    WARNING: kernel/cgroup/cpuset.c:1478 at cpuset_update_sd_hk_unlock+...
>    RIP: 0010:partition_sched_domains+0x3b8
> 
> Fix that by extending prstate_housekeeping_conflict() to reject an
> isolated partition update that would leave no active boot domain
> housekeeping CPU outside the isolated CPUs. The CPUs released back
> to the boot domain by the same update are taken into account, so
> replacing one of the last housekeeping CPUs with another one in a
> single update remains allowed. The new check is only performed when
> housekeeping_enabled(HK_TYPE_KERNEL_NOISE) is disabled so that the
> existing nohz_full behavior is unchanged.
> 
> Add test cases in test_cpuset_prs.sh to verify that taking over all
> the boot domain housekeeping CPUs is rejected while swapping one of
> them with another one is allowed.

Hi Guopeng,

Thanks for the report and the reproducer. I can reproduce the WARN and
the OOPS in partition_sched_domains() here.

However, I don't think prstate_housekeeping_conflict() is the right
place to fix it, and the fix doesn't seem complete. The same problem
can be triggered with a root partition, which this patch doesn't cover.
With an "isolcpus=domain,4-7" boot on an 8-CPU VM:

   # echo 0-3 > A/cpuset.cpus
   # echo root > A/cpuset.cpus.partition
   WARNING: kernel/sched/topology.c:3111 at build_sched_domains+0x2b96/0x5500
   Oops: general protection fault, ... [#1] SMP KASAN NOPTI
   RIP: 0010:partition_sched_domains+0x10a3/0x1a10

The faulting address is in build_perf_domains(), where cpumask_first()
on the empty cpu_map returns nr_cpu_ids and cpu_rq() goes out of
bounds.

The root cause looks like it isn't a housekeeping conflict. Boot time
isolated CPUs stay in top_cpuset.effective_cpus by design, and they are
only filtered out of the top_cpuset sched domain in
generate_sched_domains(). But the checks that make sure the parent
still has CPUs left, i.e. tasks_nocpu_error() for local partitions and
the cpumask_subset(top_cpuset.effective_cpus, ...) checks in
remote_partition_enable()/remote_cpus_update(), only look at
top_cpuset.effective_cpus. So any partition, root or isolated, local
or remote, can take away all the boot domain housekeeping CPUs as long
as some boot isolated CPUs are still left in top_cpuset, which leaves
the top_cpuset sched domain empty.

So I think the check should be done at the "parent has no CPU left"
level instead. Something like the diff below. It is not tested yet;
could you give it a try with your reproducer and the test cases?
Note that the root partition and remote partition cases should be
added to the selftest as well.

Without isolcpus=domain, HK_TYPE_DOMAIN_BOOT is cpu_possible_mask and
the new check is equivalent to the existing one, so no extra
housekeeping_enabled() gating is needed.

---
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 3219f927c35e..fb3e81405f5b 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -1180,12 +1180,51 @@ static void update_partition_sd_lb(struct cpuset *cs, 
int old_prs)
                 cpuset_force_rebuild();
  }

+/*
+ * top_cpuset_sd_nocpu - Check if top_cpuset will have no sched domain CPU
+ * @add_cpus: CPUs to be taken away from top_cpuset
+ * @del_cpus: CPUs to be given back to top_cpuset, can be NULL
+ * Return: true if no active CPU will be left in the top_cpuset sched domain
+ *
+ * Boot time isolated CPUs stay in top_cpuset.effective_cpus, but they are
+ * excluded from the sched domain of top_cpuset in generate_sched_domains().
+ * So checking top_cpuset.effective_cpus alone is not enough to ensure that
+ * the top_cpuset sched domain won't become empty after @add_cpus are taken
+ * away from it. The sched domain of top_cpuset must keep at least one
+ * active boot domain housekeeping CPU.
+ */
+static bool top_cpuset_sd_nocpu(const struct cpumask *add_cpus,
+                               const struct cpumask *del_cpus)
+{
+       const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
+       int cpu;
+
+       for_each_cpu_and(cpu, hk_mask, cpu_active_mask) {
+               if (cpumask_test_cpu(cpu, add_cpus))
+                       continue;
+               if (cpumask_test_cpu(cpu, top_cpuset.effective_cpus) ||
+                   (del_cpus && cpumask_test_cpu(cpu, del_cpus)))
+                       return false;
+       }
+       return true;
+}
+
  /*
   * tasks_nocpu_error - Return true if tasks will have no effective_cpus
   */
  static bool tasks_nocpu_error(struct cpuset *parent, struct cpuset *cs,
                               struct cpumask *xcpus)
  {
+       /*
+        * The top_cpuset always has tasks (kthreads at least), so its sched
+        * domain can't be empty. The current effective_xcpus of a valid
+        * partition will be given back to top_cpuset if not in @xcpus.
+        */
+       if ((parent == &top_cpuset) &&
+           top_cpuset_sd_nocpu(xcpus, is_partition_valid(cs)
+                                      ? cs->effective_xcpus : NULL))
+               return true;
+
         /*
          * A populated partition (cs or parent) can't have empty effective_cpus
          */
@@ -1724,7 +1763,7 @@ static int remote_partition_enable(struct cpuset *cs, int 
new_prs,
          */
         compute_excpus(cs, tmp->new_cpus);
         if (!cpumask_intersects(tmp->new_cpus, cpu_active_mask) ||
-           cpumask_subset(top_cpuset.effective_cpus, tmp->new_cpus))
+           top_cpuset_sd_nocpu(tmp->new_cpus, NULL))
                 return PERR_INVCPUS;
         if (cpumask_intersects(tmp->new_cpus, subpartitions_cpus))
                 return PERR_NOCPUS;
@@ -1828,7 +1867,7 @@ static void remote_cpus_update(struct cpuset *cs, struct 
cpumask *xcpus,
                 if (!capable(CAP_SYS_ADMIN))
                         WRITE_ONCE(cs->prs_err, PERR_ACCESS);
                 else if (cpumask_intersects(tmp->addmask, subpartitions_cpus) ||
-                        cpumask_subset(top_cpuset.effective_cpus, tmp->addmask))
+                        top_cpuset_sd_nocpu(tmp->addmask, tmp->delmask))
                         WRITE_ONCE(cs->prs_err, PERR_NOCPUS);
                 else if (prstate_housekeeping_conflict(prs, PRS_ROOT,
                                                        excpus, tmp->delmask))

-- 
Best regards
Ridong

Re: [PATCH] cgroup/cpuset: Fail if isolated CPUs don't leave any boot domain housekeeping CPU

[Guopeng Zhang]

在 2026/11/4 20:58, Ridong Chen 写道:
> 
> 
> On 11/4/2026 3:16 PM, Guopeng Zhang wrote:
>> From: Guopeng Zhang<zhangguopeng@kylinos.cn>
>>
>> With the "domain" flag of the isolcpus= boot option, the CPUs isolated
>> at boot are excluded from the sched domains and the remaining CPUs
>> become the boot domain housekeeping CPUs. An isolated partition is
>> allowed to use the boot isolated CPUs, but it must not take over all
>> the boot domain housekeeping CPUs.
>>
>> prstate_housekeeping_conflict() checks that rule for root partitions
>> only. The 2nd check covers the nohz_full case as it is gated by
>> housekeeping_enabled(HK_TYPE_KERNEL_NOISE) which is not enabled by
>> isolcpus=domain. So on an isolcpus=domain,<cpus> boot, an isolated
>> partition can claim all the remaining boot domain housekeeping CPUs.
>> The state is committed, but the sched domain rebuild at the end of
>> the write finds no housekeeping CPU left for the boot domain and
>> hits WARNs and an OOPS in partition_sched_domains().
>> housekeeping_update() cannot accept the configuration either as it
>> cannot empty the boot domain housekeeping mask.
>> With an "isolcpus=domain,10-31" boot where CPUs 0-9 are the boot
>> domain housekeeping CPUs, for example:
>>
>>    # echo 0-9 > A/cpuset.cpus
>>    # echo isolated > A/cpuset.cpus.partition
>>    WARNING: kernel/sched/topology.c:3089 at build_sched_domains+...
>>    WARNING: kernel/cgroup/cpuset.c:1478 at cpuset_update_sd_hk_unlock+...
>>    RIP: 0010:partition_sched_domains+0x3b8
>>
>> Fix that by extending prstate_housekeeping_conflict() to reject an
>> isolated partition update that would leave no active boot domain
>> housekeeping CPU outside the isolated CPUs. The CPUs released back
>> to the boot domain by the same update are taken into account, so
>> replacing one of the last housekeeping CPUs with another one in a
>> single update remains allowed. The new check is only performed when
>> housekeeping_enabled(HK_TYPE_KERNEL_NOISE) is disabled so that the
>> existing nohz_full behavior is unchanged.
>>
>> Add test cases in test_cpuset_prs.sh to verify that taking over all
>> the boot domain housekeeping CPUs is rejected while swapping one of
>> them with another one is allowed.
> 
> Hi Guopeng,
> 

Hi Ridong,

Thanks a lot for the analysis and the patch. 

> Thanks for the report and the reproducer. I can reproduce the WARN and
> the OOPS in partition_sched_domains() here.
> 
> However, I don't think prstate_housekeeping_conflict() is the right
> place to fix it, and the fix doesn't seem complete. The same problem
> can be triggered with a root partition, which this patch doesn't cover.
> With an "isolcpus=domain,4-7" boot on an 8-CPU VM:
> 
>   # echo 0-3 > A/cpuset.cpus
>   # echo root > A/cpuset.cpus.partition
>   WARNING: kernel/sched/topology.c:3111 at build_sched_domains+0x2b96/0x5500
>   Oops: general protection fault, ... [#1] SMP KASAN NOPTI
>   RIP: 0010:partition_sched_domains+0x10a3/0x1a10
> 

You are right. The sashiko review also reported this issue. I can
reproduce the WARN and the OOPS here as well.

> The faulting address is in build_perf_domains(), where cpumask_first()
> on the empty cpu_map returns nr_cpu_ids and cpu_rq() goes out of
> bounds.
> 
> The root cause looks like it isn't a housekeeping conflict. Boot time
> isolated CPUs stay in top_cpuset.effective_cpus by design, and they are
> only filtered out of the top_cpuset sched domain in
> generate_sched_domains(). But the checks that make sure the parent
> still has CPUs left, i.e. tasks_nocpu_error() for local partitions and
> the cpumask_subset(top_cpuset.effective_cpus, ...) checks in
> remote_partition_enable()/remote_cpus_update(), only look at
> top_cpuset.effective_cpus. So any partition, root or isolated, local
> or remote, can take away all the boot domain housekeeping CPUs as long
> as some boot isolated CPUs are still left in top_cpuset, which leaves
> the top_cpuset sched domain empty.
> 
> So I think the check should be done at the "parent has no CPU left"
> level instead. Something like the diff below. It is not tested yet;
> could you give it a try with your reproducer and the test cases?
> Note that the root partition and remote partition cases should be
> added to the selftest as well.
> 

That's a great idea, thanks for your guidance. I will test it and add
the relevant test cases.

Best regards,
Guopeng

> Without isolcpus=domain, HK_TYPE_DOMAIN_BOOT is cpu_possible_mask and
> the new check is equivalent to the existing one, so no extra
> housekeeping_enabled() gating is needed.
> 
> ---
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 3219f927c35e..fb3e81405f5b 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -1180,12 +1180,51 @@ static void update_partition_sd_lb(struct cpuset *cs, int old_prs)
>                 cpuset_force_rebuild();
>  }
> 
> +/*
> + * top_cpuset_sd_nocpu - Check if top_cpuset will have no sched domain CPU
> + * @add_cpus: CPUs to be taken away from top_cpuset
> + * @del_cpus: CPUs to be given back to top_cpuset, can be NULL
> + * Return: true if no active CPU will be left in the top_cpuset sched domain
> + *
> + * Boot time isolated CPUs stay in top_cpuset.effective_cpus, but they are
> + * excluded from the sched domain of top_cpuset in generate_sched_domains().
> + * So checking top_cpuset.effective_cpus alone is not enough to ensure that
> + * the top_cpuset sched domain won't become empty after @add_cpus are taken
> + * away from it. The sched domain of top_cpuset must keep at least one
> + * active boot domain housekeeping CPU.
> + */
> +static bool top_cpuset_sd_nocpu(const struct cpumask *add_cpus,
> +                               const struct cpumask *del_cpus)
> +{
> +       const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
> +       int cpu;
> +
> +       for_each_cpu_and(cpu, hk_mask, cpu_active_mask) {
> +               if (cpumask_test_cpu(cpu, add_cpus))
> +                       continue;
> +               if (cpumask_test_cpu(cpu, top_cpuset.effective_cpus) ||
> +                   (del_cpus && cpumask_test_cpu(cpu, del_cpus)))
> +                       return false;
> +       }
> +       return true;
> +}
> +
>  /*
>   * tasks_nocpu_error - Return true if tasks will have no effective_cpus
>   */
>  static bool tasks_nocpu_error(struct cpuset *parent, struct cpuset *cs,
>                               struct cpumask *xcpus)
>  {
> +       /*
> +        * The top_cpuset always has tasks (kthreads at least), so its sched
> +        * domain can't be empty. The current effective_xcpus of a valid
> +        * partition will be given back to top_cpuset if not in @xcpus.
> +        */
> +       if ((parent == &top_cpuset) &&
> +           top_cpuset_sd_nocpu(xcpus, is_partition_valid(cs)
> +                                      ? cs->effective_xcpus : NULL))
> +               return true;
> +
>         /*
>          * A populated partition (cs or parent) can't have empty effective_cpus
>          */
> @@ -1724,7 +1763,7 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,
>          */
>         compute_excpus(cs, tmp->new_cpus);
>         if (!cpumask_intersects(tmp->new_cpus, cpu_active_mask) ||
> -           cpumask_subset(top_cpuset.effective_cpus, tmp->new_cpus))
> +           top_cpuset_sd_nocpu(tmp->new_cpus, NULL))
>                 return PERR_INVCPUS;
>         if (cpumask_intersects(tmp->new_cpus, subpartitions_cpus))
>                 return PERR_NOCPUS;
> @@ -1828,7 +1867,7 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
>                 if (!capable(CAP_SYS_ADMIN))
>                         WRITE_ONCE(cs->prs_err, PERR_ACCESS);
>                 else if (cpumask_intersects(tmp->addmask, subpartitions_cpus) ||
> -                        cpumask_subset(top_cpuset.effective_cpus, tmp->addmask))
> +                        top_cpuset_sd_nocpu(tmp->addmask, tmp->delmask))
>                         WRITE_ONCE(cs->prs_err, PERR_NOCPUS);
>                 else if (prstate_housekeeping_conflict(prs, PRS_ROOT,
>                                                        excpus, tmp->delmask))
> 

Re: [PATCH] cgroup/cpuset: Fail if isolated CPUs don't leave any boot domain housekeeping CPU

[Waiman Long]

On 11/4/26 7:58 AM, Ridong Chen wrote:
>
>
> On 11/4/2026 3:16 PM, Guopeng Zhang wrote:
>> From: Guopeng Zhang<zhangguopeng@kylinos.cn>
>>
>> With the "domain" flag of the isolcpus= boot option, the CPUs isolated
>> at boot are excluded from the sched domains and the remaining CPUs
>> become the boot domain housekeeping CPUs. An isolated partition is
>> allowed to use the boot isolated CPUs, but it must not take over all
>> the boot domain housekeeping CPUs.
>>
>> prstate_housekeeping_conflict() checks that rule for root partitions
>> only. The 2nd check covers the nohz_full case as it is gated by
>> housekeeping_enabled(HK_TYPE_KERNEL_NOISE) which is not enabled by
>> isolcpus=domain. So on an isolcpus=domain,<cpus> boot, an isolated
>> partition can claim all the remaining boot domain housekeeping CPUs.
>> The state is committed, but the sched domain rebuild at the end of
>> the write finds no housekeeping CPU left for the boot domain and
>> hits WARNs and an OOPS in partition_sched_domains().
>> housekeeping_update() cannot accept the configuration either as it
>> cannot empty the boot domain housekeeping mask.
>> With an "isolcpus=domain,10-31" boot where CPUs 0-9 are the boot
>> domain housekeeping CPUs, for example:
>>
>>    # echo 0-9 > A/cpuset.cpus
>>    # echo isolated > A/cpuset.cpus.partition
>>    WARNING: kernel/sched/topology.c:3089 at build_sched_domains+...
>>    WARNING: kernel/cgroup/cpuset.c:1478 at 
>> cpuset_update_sd_hk_unlock+...
>>    RIP: 0010:partition_sched_domains+0x3b8
>>
>> Fix that by extending prstate_housekeeping_conflict() to reject an
>> isolated partition update that would leave no active boot domain
>> housekeeping CPU outside the isolated CPUs. The CPUs released back
>> to the boot domain by the same update are taken into account, so
>> replacing one of the last housekeeping CPUs with another one in a
>> single update remains allowed. The new check is only performed when
>> housekeeping_enabled(HK_TYPE_KERNEL_NOISE) is disabled so that the
>> existing nohz_full behavior is unchanged.
>>
>> Add test cases in test_cpuset_prs.sh to verify that taking over all
>> the boot domain housekeeping CPUs is rejected while swapping one of
>> them with another one is allowed.
>
> Hi Guopeng,
>
> Thanks for the report and the reproducer. I can reproduce the WARN and
> the OOPS in partition_sched_domains() here.
>
> However, I don't think prstate_housekeeping_conflict() is the right
> place to fix it, and the fix doesn't seem complete. The same problem
> can be triggered with a root partition, which this patch doesn't cover.
> With an "isolcpus=domain,4-7" boot on an 8-CPU VM:
>
>   # echo 0-3 > A/cpuset.cpus
>   # echo root > A/cpuset.cpus.partition
>   WARNING: kernel/sched/topology.c:3111 at 
> build_sched_domains+0x2b96/0x5500
>   Oops: general protection fault, ... [#1] SMP KASAN NOPTI
>   RIP: 0010:partition_sched_domains+0x10a3/0x1a10
>
> The faulting address is in build_perf_domains(), where cpumask_first()
> on the empty cpu_map returns nr_cpu_ids and cpu_rq() goes out of
> bounds.
>
> The root cause looks like it isn't a housekeeping conflict. Boot time
> isolated CPUs stay in top_cpuset.effective_cpus by design, and they are
> only filtered out of the top_cpuset sched domain in
> generate_sched_domains(). But the checks that make sure the parent
> still has CPUs left, i.e. tasks_nocpu_error() for local partitions and
> the cpumask_subset(top_cpuset.effective_cpus, ...) checks in
> remote_partition_enable()/remote_cpus_update(), only look at
> top_cpuset.effective_cpus. So any partition, root or isolated, local
> or remote, can take away all the boot domain housekeeping CPUs as long
> as some boot isolated CPUs are still left in top_cpuset, which leaves
> the top_cpuset sched domain empty.
>
> So I think the check should be done at the "parent has no CPU left"
> level instead. Something like the diff below. It is not tested yet;
> could you give it a try with your reproducer and the test cases?
> Note that the root partition and remote partition cases should be
> added to the selftest as well.
>
> Without isolcpus=domain, HK_TYPE_DOMAIN_BOOT is cpu_possible_mask and
> the new check is equivalent to the existing one, so no extra
> housekeeping_enabled() gating is needed.
>
> ---
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 3219f927c35e..fb3e81405f5b 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -1180,12 +1180,51 @@ static void update_partition_sd_lb(struct 
> cpuset *cs, int old_prs)
>                 cpuset_force_rebuild();
>  }
>
> +/*
> + * top_cpuset_sd_nocpu - Check if top_cpuset will have no sched 
> domain CPU
> + * @add_cpus: CPUs to be taken away from top_cpuset
> + * @del_cpus: CPUs to be given back to top_cpuset, can be NULL
> + * Return: true if no active CPU will be left in the top_cpuset sched 
> domain
> + *
> + * Boot time isolated CPUs stay in top_cpuset.effective_cpus, but 
> they are
> + * excluded from the sched domain of top_cpuset in 
> generate_sched_domains().
> + * So checking top_cpuset.effective_cpus alone is not enough to 
> ensure that
> + * the top_cpuset sched domain won't become empty after @add_cpus are 
> taken
> + * away from it. The sched domain of top_cpuset must keep at least one
> + * active boot domain housekeeping CPU.
> + */
> +static bool top_cpuset_sd_nocpu(const struct cpumask *add_cpus,
> +                               const struct cpumask *del_cpus)
> +{
> +       const struct cpumask *hk_mask = 
> housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
> +       int cpu;
> +
> +       for_each_cpu_and(cpu, hk_mask, cpu_active_mask) {
> +               if (cpumask_test_cpu(cpu, add_cpus))
> +                       continue;
> +               if (cpumask_test_cpu(cpu, top_cpuset.effective_cpus) ||
> +                   (del_cpus && cpumask_test_cpu(cpu, del_cpus)))
> +                       return false;
> +       }
> +       return true;
> +}
> +
>  /*
>   * tasks_nocpu_error - Return true if tasks will have no effective_cpus
>   */
>  static bool tasks_nocpu_error(struct cpuset *parent, struct cpuset *cs,
>                               struct cpumask *xcpus)
>  {
> +       /*
> +        * The top_cpuset always has tasks (kthreads at least), so its 
> sched
> +        * domain can't be empty. The current effective_xcpus of a valid
> +        * partition will be given back to top_cpuset if not in @xcpus.
> +        */
> +       if ((parent == &top_cpuset) &&
> +           top_cpuset_sd_nocpu(xcpus, is_partition_valid(cs)
> +                                      ? cs->effective_xcpus : NULL))

The del_cpus should normally be (cs->effective_xcpus & ~xcpus), but the 
top_cpuset_sd_nocpu() helper will skip CPUs that are in xcpus. So it 
will work in this case. It should be noted in the functional comment of 
top_cpuset_sd_nocpu() that @del_cpus can include common CPUs that will 
stay in the partition which will be skipped in the helper so that future 
change won't accidentally change that behavior.

Other than that, the code looks good to me.

Cheers,
Longman

> + return true;
> +
>         /*
>          * A populated partition (cs or parent) can't have empty 
> effective_cpus
>          */
> @@ -1724,7 +1763,7 @@ static int remote_partition_enable(struct cpuset 
> *cs, int new_prs,
>          */
>         compute_excpus(cs, tmp->new_cpus);
>         if (!cpumask_intersects(tmp->new_cpus, cpu_active_mask) ||
> -           cpumask_subset(top_cpuset.effective_cpus, tmp->new_cpus))
> +           top_cpuset_sd_nocpu(tmp->new_cpus, NULL))
>                 return PERR_INVCPUS;
>         if (cpumask_intersects(tmp->new_cpus, subpartitions_cpus))
>                 return PERR_NOCPUS;
> @@ -1828,7 +1867,7 @@ static void remote_cpus_update(struct cpuset 
> *cs, struct cpumask *xcpus,
>                 if (!capable(CAP_SYS_ADMIN))
>                         WRITE_ONCE(cs->prs_err, PERR_ACCESS);
>                 else if (cpumask_intersects(tmp->addmask, 
> subpartitions_cpus) ||
> -                        cpumask_subset(top_cpuset.effective_cpus, 
> tmp->addmask))
> +                        top_cpuset_sd_nocpu(tmp->addmask, tmp->delmask))
>                         WRITE_ONCE(cs->prs_err, PERR_NOCPUS);
>                 else if (prstate_housekeeping_conflict(prs, PRS_ROOT,
>                                                        excpus, 
> tmp->delmask))
>