[BUG] cpuset: WARNING in cpuset_update_sd_hk_unlock() when isolated partitions take every housekeeping CPU

[BUG] cpuset: WARNING in cpuset_update_sd_hk_unlock() when isolated partitions take every housekeeping CPU

[Mathieu Desnoyers]

Hi,

With isolcpus=domain, cpuset accepts isolated partitions that leave no
housekeeping CPU. housekeeping_update() then refuses the new mask, and
cpuset_update_sd_hk_unlock() warns.

I found this while testing a fix for another cpuset problem [1]. Its
sequence goes through that problem first, so I ran it only on a kernel
with that fix.

Trigger
=======

Boot 5 CPUs with isolcpus=domain,0,1. CPUs 2 to 4 are then the
housekeeping CPUs. With cgroup v2:

  cd /sys/fs/cgroup
  echo +cpuset > cgroup.subtree_control
  mkdir P Q
  echo 4 > P/cpuset.cpus
  echo isolated > P/cpuset.cpus.partition
  echo 2-3 > Q/cpuset.cpus
  echo root > Q/cpuset.cpus.partition
  echo isolated > Q/cpuset.cpus.partition    # WARNING

The last write is accepted. Q reads back as isolated.

  WARNING: kernel/cgroup/cpuset.c:1481 at cpuset_update_sd_hk_unlock+0x79/0x80, CPU#1: k2/93
  CPU: 1 UID: 0 PID: 93 Comm: k2 Not tainted 7.4.0-rc1-00268-ge99d0e004b05 #108 PREEMPT(lazy)
  RAX: 00000000ffffffea
  Call Trace:
   cpuset_partition_write+0x9e/0xe0
   kernfs_fop_write_iter+0x14d/0x210
   vfs_write+0x252/0x550
   ksys_write+0x6c/0xe0
   do_syscall_64+0x109/0x5d0

RAX is -EINVAL, the return of housekeeping_update().

Analysis
========

From reading tip sched/urgent (006991aa8997):
- Making Q isolated would remove CPUs 2 and 3 from HK_TYPE_DOMAIN.
  CPUs 0 and 1 are isolated at boot, and P holds CPU 4. So no
  housekeeping CPU would remain.
- housekeeping_update() refuses that mask with -EINVAL. Commit
  03ff73510169 ("cpuset: Update HK_TYPE_DOMAIN cpumask from cpuset")
  calls it under WARN_ON_ONCE() in cpuset_update_sd_hk_unlock().
- prstate_housekeeping_conflict() refuses such a partition only with
  nohz_full, since commit 103b08709e8a ("cgroup/cpuset: Fail if
  isolated and nohz_full don't leave any housekeeping").

So cpuset and the housekeeping code disagree on what is valid. The
write returns 0, and the isolated CPUs of the top cpuset read 0-4. But
by reading, HK_TYPE_DOMAIN keeps its old mask.

Should cpuset refuse an isolated partition that leaves no
HK_TYPE_DOMAIN housekeeping CPU, as it does for nohz_full? I did not
write a patch, since you may prefer another place for the check.

The analysis was written by an LLM coding assistant under my direction
and review.

Thanks,

Mathieu

[1] The empty sched domain report, sent along with this one.

Re: [BUG] cpuset: WARNING in cpuset_update_sd_hk_unlock() when isolated partitions take every housekeeping CPU

[Waiman Long]

On 11/8/26 9:57 AM, Mathieu Desnoyers wrote:
> Hi,
>
> With isolcpus=domain, cpuset accepts isolated partitions that leave no
> housekeeping CPU. housekeeping_update() then refuses the new mask, and
> cpuset_update_sd_hk_unlock() warns.
>
> I found this while testing a fix for another cpuset problem [1]. Its
> sequence goes through that problem first, so I ran it only on a kernel
> with that fix.
>
> Trigger
> =======
>
> Boot 5 CPUs with isolcpus=domain,0,1. CPUs 2 to 4 are then the
> housekeeping CPUs. With cgroup v2:
>
>    cd /sys/fs/cgroup
>    echo +cpuset > cgroup.subtree_control
>    mkdir P Q
>    echo 4 > P/cpuset.cpus
>    echo isolated > P/cpuset.cpus.partition
>    echo 2-3 > Q/cpuset.cpus
>    echo root > Q/cpuset.cpus.partition
>    echo isolated > Q/cpuset.cpus.partition    # WARNING
>
> The last write is accepted. Q reads back as isolated.
>
>    WARNING: kernel/cgroup/cpuset.c:1481 at cpuset_update_sd_hk_unlock+0x79/0x80, CPU#1: k2/93
>    CPU: 1 UID: 0 PID: 93 Comm: k2 Not tainted 7.4.0-rc1-00268-ge99d0e004b05 #108 PREEMPT(lazy)
>    RAX: 00000000ffffffea
>    Call Trace:
>     cpuset_partition_write+0x9e/0xe0
>     kernfs_fop_write_iter+0x14d/0x210
>     vfs_write+0x252/0x550
>     ksys_write+0x6c/0xe0
>     do_syscall_64+0x109/0x5d0
>
> RAX is -EINVAL, the return of housekeeping_update().
>
> Analysis
> ========
>
>  From reading tip sched/urgent (006991aa8997):
> - Making Q isolated would remove CPUs 2 and 3 from HK_TYPE_DOMAIN.
>    CPUs 0 and 1 are isolated at boot, and P holds CPU 4. So no
>    housekeeping CPU would remain.
> - housekeeping_update() refuses that mask with -EINVAL. Commit
>    03ff73510169 ("cpuset: Update HK_TYPE_DOMAIN cpumask from cpuset")
>    calls it under WARN_ON_ONCE() in cpuset_update_sd_hk_unlock().
> - prstate_housekeeping_conflict() refuses such a partition only with
>    nohz_full, since commit 103b08709e8a ("cgroup/cpuset: Fail if
>    isolated and nohz_full don't leave any housekeeping").
>
> So cpuset and the housekeeping code disagree on what is valid. The
> write returns 0, and the isolated CPUs of the top cpuset read 0-4. But
> by reading, HK_TYPE_DOMAIN keeps its old mask.
>
> Should cpuset refuse an isolated partition that leaves no
> HK_TYPE_DOMAIN housekeeping CPU, as it does for nohz_full? I did not
> write a patch, since you may prefer another place for the check.

As said in my previous email, cpuset will ensure that there will be at 
least one online housekeeping CPU left for generating sched domain for 
the top_cpuset.

Cheers,
Longman

>
> The analysis was written by an LLM coding assistant under my direction
> and review.
>
> Thanks,
>
> Mathieu
>
> [1] The empty sched domain report, sent along with this one.