[QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[sheviks]

Hi Frederic and maintainers,

I’m reaching out to discuss a change in kthread affinity behavior that
seems to be a regression for users relying on dynamic CPU isolation.
This started appearing after commit 041ee6f3727a ("kthread: Rely on
HK_TYPE_DOMAIN for preferred affinity management").

The Problem:
In my setup, I use nohz_full but intentionally avoid the deprecated
isolcpus= boot parameter. Instead, I use cgroup v2
(cpuset.cpus.partition=isolated) to dynamically isolate CPUs after the
system has booted.

The commit 041ee6f3727a changed kthreads to rely on HK_TYPE_DOMAIN.
However, since isolcpus= is not used, HK_TYPE_DOMAIN defaults to all
CPUs at boot time. Even after I later configure cgroups to isolate
CPUs 1-7, unbound kthreads (including kthreadd) remain on those
nohz_full CPUs.

It seems the assumption that "nohz_full implies domain isolation" only
holds true if isolation is statically defined at boot via isolcpus=.
For dynamic isolation via cgroups, HK_TYPE_KTHREAD and HK_TYPE_DOMAIN
no longer cover the same set of CPUs.

System Log:
Here is the state of my system after setting up the cgroup isolation:

$ uname -r
7.0.0-rc4-1-rt

# 1. Boot parameters (No isolcpus)
$ grep -oe "nohz_full=[^ ]*" -e "rcu_nocbs=[^ ]*" /proc/cmdline
nohz_full=1,2,3,4,5,6,7
rcu_nocbs=1,2,3,4,5,6,7

# 2. Cgroup v2 Isolation is active
$ cat /sys/fs/cgroup/isolated1.slice/cpuset.cpus.exclusive
1-7
$ cat /sys/fs/cgroup/isolated1.slice/cpuset.cpus.partition
isolated
$ cat /sys/fs/cgroup/cpuset.cpus.effective
0
$ cat /sys/fs/cgroup/cpuset.cpus.isolated
1-7

# 3. Unbound kthreads are still "trapped" on isolated/nohz_full CPUs
$ ps -eLo cpuid,comm | grep -e COMM -e "^ *[1-7] " | grep -ve
"/[1-7]$" -e "kworker/[1-7]:" | head
CPUID COMMAND
    4 pool_workqueue_release
    1 pr/legacy
    4 rcu_exp_gp_kthread_worker
    1 kdevtmpfs
    5 oom_reaper
    1 ksmd
    7 watchdogd
    7 kswapd0
    6 scsi_eh_0

Questions:
1. Is this an intended change that mandates the use of isolcpus= for
kthread exclusion?

2. If we prefer dynamic isolation via cgroup v2, is there a
recommended way to "refresh" or move these unbound kthreads once the
housekeeping mask changes at runtime?

3. Or should HK_TYPE_KTHREAD still be considered separately from
HK_TYPE_DOMAIN to account for nohz_full users without isolcpus=?

I would appreciate any insights or suggestions you might have.

Best regards,
Sheviks

<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
乾淨無病毒。www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[Waiman Long]

On 3/21/26 12:44 AM, sheviks wrote:
> Hi Frederic and maintainers,
>
> I’m reaching out to discuss a change in kthread affinity behavior that
> seems to be a regression for users relying on dynamic CPU isolation.
> This started appearing after commit 041ee6f3727a ("kthread: Rely on
> HK_TYPE_DOMAIN for preferred affinity management").
>
> The Problem:
> In my setup, I use nohz_full but intentionally avoid the deprecated
> isolcpus= boot parameter. Instead, I use cgroup v2
> (cpuset.cpus.partition=isolated) to dynamically isolate CPUs after the
> system has booted.
>
> The commit 041ee6f3727a changed kthreads to rely on HK_TYPE_DOMAIN.
> However, since isolcpus= is not used, HK_TYPE_DOMAIN defaults to all
> CPUs at boot time. Even after I later configure cgroups to isolate
> CPUs 1-7, unbound kthreads (including kthreadd) remain on those
> nohz_full CPUs.

Frederic's patch series is supposed to make HK_TYPE_DOMAIN cpumask to 
dynamically exclude cpuset isolated CPUs. Then those unbound kthreads 
are supposed to be modified to remove these CPUs from their cpumasks. If 
that is not happening, it will be a problem we need to look at.

Cheers,
Longman

>
> It seems the assumption that "nohz_full implies domain isolation" only
> holds true if isolation is statically defined at boot via isolcpus=.
> For dynamic isolation via cgroups, HK_TYPE_KTHREAD and HK_TYPE_DOMAIN
> no longer cover the same set of CPUs.
>
> System Log:
> Here is the state of my system after setting up the cgroup isolation:
>
> $ uname -r
> 7.0.0-rc4-1-rt
>
> # 1. Boot parameters (No isolcpus)
> $ grep -oe "nohz_full=[^ ]*" -e "rcu_nocbs=[^ ]*" /proc/cmdline
> nohz_full=1,2,3,4,5,6,7
> rcu_nocbs=1,2,3,4,5,6,7
>
> # 2. Cgroup v2 Isolation is active
> $ cat /sys/fs/cgroup/isolated1.slice/cpuset.cpus.exclusive
> 1-7
> $ cat /sys/fs/cgroup/isolated1.slice/cpuset.cpus.partition
> isolated
> $ cat /sys/fs/cgroup/cpuset.cpus.effective
> 0
> $ cat /sys/fs/cgroup/cpuset.cpus.isolated
> 1-7
>
> # 3. Unbound kthreads are still "trapped" on isolated/nohz_full CPUs
> $ ps -eLo cpuid,comm | grep -e COMM -e "^ *[1-7] " | grep -ve
> "/[1-7]$" -e "kworker/[1-7]:" | head
> CPUID COMMAND
>      4 pool_workqueue_release
>      1 pr/legacy
>      4 rcu_exp_gp_kthread_worker
>      1 kdevtmpfs
>      5 oom_reaper
>      1 ksmd
>      7 watchdogd
>      7 kswapd0
>      6 scsi_eh_0
>
> Questions:
> 1. Is this an intended change that mandates the use of isolcpus= for
> kthread exclusion?
>
> 2. If we prefer dynamic isolation via cgroup v2, is there a
> recommended way to "refresh" or move these unbound kthreads once the
> housekeeping mask changes at runtime?
>
> 3. Or should HK_TYPE_KTHREAD still be considered separately from
> HK_TYPE_DOMAIN to account for nohz_full users without isolcpus=?
>
> I would appreciate any insights or suggestions you might have.
>
> Best regards,
> Sheviks
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> 乾淨無病毒。www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[sheviks]

Hi Frederic, Waiman and maintainers,

A quick follow-up on my previous report. After leaving the system idle
for a longer period, I made an observation that pinpoints the issue
more precisely.

The cgroup v2 dynamic isolation does eventually work. I noticed that
the unbound kthreads are eventually migrated to the housekeeping CPU
(CPU 0), but only after they wake up from sleep and enter the running
state. This lazy migration highlights why commit 041ee6f3727a is
causing issues for setups using nohz_full= without isolcpus=:

1. At boot time, because isolcpus= is absent, the HK_TYPE_DOMAIN mask
includes the nohz_full CPUs.

2. When unbound kthreads are initially spawned or have their affinity
set, the new logic relies solely on HK_TYPE_DOMAIN. Consequently, they
are placed on the nohz_full CPUs and immediately go to sleep there.

3. They remain "trapped" on the isolated CPUs until a wake-up event
finally forces the scheduler to migrate them according to the updated
cgroup affinity.

This brings the focus back to HK_TYPE_KTHREAD vs HK_TYPE_DOMAIN. While
HK_TYPE_DOMAIN might default to all CPUs without isolcpus=,
HK_TYPE_KTHREAD correctly excludes the nohz_full CPUs from the very
beginning.

Is this "spawn on nohz_full and wait for wake-up to migrate" behavior
intended? To prevent these sleeping kthreads from polluting isolated
CPUs before cgroups can intervene, should the initial affinity check
still consider HK_TYPE_KTHREAD alongside or instead of HK_TYPE_DOMAIN?

Thanks again for your time.

Best regards,
Sheviks


Waiman Long <longman@redhat.com> 於 2026年3月23日週一 上午12:04寫道：
>
> On 3/21/26 12:44 AM, sheviks wrote:
> > Hi Frederic and maintainers,
> >
> > I’m reaching out to discuss a change in kthread affinity behavior that
> > seems to be a regression for users relying on dynamic CPU isolation.
> > This started appearing after commit 041ee6f3727a ("kthread: Rely on
> > HK_TYPE_DOMAIN for preferred affinity management").
> >
> > The Problem:
> > In my setup, I use nohz_full but intentionally avoid the deprecated
> > isolcpus= boot parameter. Instead, I use cgroup v2
> > (cpuset.cpus.partition=isolated) to dynamically isolate CPUs after the
> > system has booted.
> >
> > The commit 041ee6f3727a changed kthreads to rely on HK_TYPE_DOMAIN.
> > However, since isolcpus= is not used, HK_TYPE_DOMAIN defaults to all
> > CPUs at boot time. Even after I later configure cgroups to isolate
> > CPUs 1-7, unbound kthreads (including kthreadd) remain on those
> > nohz_full CPUs.
>
> Frederic's patch series is supposed to make HK_TYPE_DOMAIN cpumask to
> dynamically exclude cpuset isolated CPUs. Then those unbound kthreads
> are supposed to be modified to remove these CPUs from their cpumasks. If
> that is not happening, it will be a problem we need to look at.
>
> Cheers,
> Longman
>
> >
> > It seems the assumption that "nohz_full implies domain isolation" only
> > holds true if isolation is statically defined at boot via isolcpus=.
> > For dynamic isolation via cgroups, HK_TYPE_KTHREAD and HK_TYPE_DOMAIN
> > no longer cover the same set of CPUs.
> >
> > System Log:
> > Here is the state of my system after setting up the cgroup isolation:
> >
> > $ uname -r
> > 7.0.0-rc4-1-rt
> >
> > # 1. Boot parameters (No isolcpus)
> > $ grep -oe "nohz_full=[^ ]*" -e "rcu_nocbs=[^ ]*" /proc/cmdline
> > nohz_full=1,2,3,4,5,6,7
> > rcu_nocbs=1,2,3,4,5,6,7
> >
> > # 2. Cgroup v2 Isolation is active
> > $ cat /sys/fs/cgroup/isolated1.slice/cpuset.cpus.exclusive
> > 1-7
> > $ cat /sys/fs/cgroup/isolated1.slice/cpuset.cpus.partition
> > isolated
> > $ cat /sys/fs/cgroup/cpuset.cpus.effective
> > 0
> > $ cat /sys/fs/cgroup/cpuset.cpus.isolated
> > 1-7
> >
> > # 3. Unbound kthreads are still "trapped" on isolated/nohz_full CPUs
> > $ ps -eLo cpuid,comm | grep -e COMM -e "^ *[1-7] " | grep -ve
> > "/[1-7]$" -e "kworker/[1-7]:" | head
> > CPUID COMMAND
> >      4 pool_workqueue_release
> >      1 pr/legacy
> >      4 rcu_exp_gp_kthread_worker
> >      1 kdevtmpfs
> >      5 oom_reaper
> >      1 ksmd
> >      7 watchdogd
> >      7 kswapd0
> >      6 scsi_eh_0
> >
> > Questions:
> > 1. Is this an intended change that mandates the use of isolcpus= for
> > kthread exclusion?
> >
> > 2. If we prefer dynamic isolation via cgroup v2, is there a
> > recommended way to "refresh" or move these unbound kthreads once the
> > housekeeping mask changes at runtime?
> >
> > 3. Or should HK_TYPE_KTHREAD still be considered separately from
> > HK_TYPE_DOMAIN to account for nohz_full users without isolcpus=?
> >
> > I would appreciate any insights or suggestions you might have.
> >
> > Best regards,
> > Sheviks
> >
> > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> > 乾淨無病毒。www.avast.com
> > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
>

Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[Waiman Long]

On 3/22/26 1:48 PM, sheviks wrote:
> Hi Frederic, Waiman and maintainers,
>
> A quick follow-up on my previous report. After leaving the system idle
> for a longer period, I made an observation that pinpoints the issue
> more precisely.
>
> The cgroup v2 dynamic isolation does eventually work. I noticed that
> the unbound kthreads are eventually migrated to the housekeeping CPU
> (CPU 0), but only after they wake up from sleep and enter the running
> state. This lazy migration highlights why commit 041ee6f3727a is
> causing issues for setups using nohz_full= without isolcpus=:
>
> 1. At boot time, because isolcpus= is absent, the HK_TYPE_DOMAIN mask
> includes the nohz_full CPUs.
>
> 2. When unbound kthreads are initially spawned or have their affinity
> set, the new logic relies solely on HK_TYPE_DOMAIN. Consequently, they
> are placed on the nohz_full CPUs and immediately go to sleep there.
>
> 3. They remain "trapped" on the isolated CPUs until a wake-up event
> finally forces the scheduler to migrate them according to the updated
> cgroup affinity.
That makes much more sense now. Yes, this is what new behavior should be.
> This brings the focus back to HK_TYPE_KTHREAD vs HK_TYPE_DOMAIN. While
> HK_TYPE_DOMAIN might default to all CPUs without isolcpus=,
> HK_TYPE_KTHREAD correctly excludes the nohz_full CPUs from the very
> beginning.
>
> Is this "spawn on nohz_full and wait for wake-up to migrate" behavior
> intended? To prevent these sleeping kthreads from polluting isolated
> CPUs before cgroups can intervene, should the initial affinity check
> still consider HK_TYPE_KTHREAD alongside or instead of HK_TYPE_DOMAIN?

The plan is to make nohz_full also dynamically changeable at run time in 
the future. We are not there yet. Now HK_TYPE_KTHREAD is equivalent to 
HK_TYPE_DOMAIN.

A CPU cannot be considered fully isolated if it is in either 
HK_TYPE_DOMAIN or HK_TYPE_KERNEL_NOISE. Currently, the nohz_full kernel 
parameter can be used to put a set of partially isolated CPUs in the 
nohz_full reservoir. To fully isolate them, some or all of them will be 
need to be put in an isolated cpuset partition. BTW, HK_TYPE_MANAGED_IRQ 
will be made to be dynamic too.

That is my current view.

Cheers,
Longman

Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[sheviks]

Waiman Long <longman@redhat.com> 於 2026年3月23日 週一 05:18 寫道：
>
> The plan is to make nohz_full also dynamically changeable at run time in
> the future. We are not there yet. Now HK_TYPE_KTHREAD is equivalent to
> HK_TYPE_DOMAIN.
>
> A CPU cannot be considered fully isolated if it is in either
> HK_TYPE_DOMAIN or HK_TYPE_KERNEL_NOISE. Currently, the nohz_full kernel
> parameter can be used to put a set of partially isolated CPUs in the
> nohz_full reservoir. To fully isolate them, some or all of them will be
> need to be put in an isolated cpuset partition. BTW, HK_TYPE_MANAGED_IRQ
> will be made to be dynamic too.
>
> That is my current view.
>

That sounds like a great roadmap. I am very much looking forward to
seeing the final implementation, as making nohz_full and managed IRQs
dynamic would be a significant improvement for runtime isolation
flexibility.

However, regarding the current state in 7.0rc4, I observed that
HK_TYPE_KTHREAD still appears to be initialized as
HK_TYPE_KERNEL_NOISE rather than HK_TYPE_DOMAIN in the initialization
path.
Perhaps this part of the alignment hasn't been merged into the mainline yet?

Thanks for the clarification and for your work on this!
Best regards,
Sheviks

Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[Waiman Long]

On 3/23/26 12:58 AM, sheviks wrote:
> Waiman Long <longman@redhat.com> 於 2026年3月23日 週一 05:18 寫道：
>> The plan is to make nohz_full also dynamically changeable at run time in
>> the future. We are not there yet. Now HK_TYPE_KTHREAD is equivalent to
>> HK_TYPE_DOMAIN.
>>
>> A CPU cannot be considered fully isolated if it is in either
>> HK_TYPE_DOMAIN or HK_TYPE_KERNEL_NOISE. Currently, the nohz_full kernel
>> parameter can be used to put a set of partially isolated CPUs in the
>> nohz_full reservoir. To fully isolate them, some or all of them will be
>> need to be put in an isolated cpuset partition. BTW, HK_TYPE_MANAGED_IRQ
>> will be made to be dynamic too.
>>
>> That is my current view.
>>
> That sounds like a great roadmap. I am very much looking forward to
> seeing the final implementation, as making nohz_full and managed IRQs
> dynamic would be a significant improvement for runtime isolation
> flexibility.
>
> However, regarding the current state in 7.0rc4, I observed that
> HK_TYPE_KTHREAD still appears to be initialized as
> HK_TYPE_KERNEL_NOISE rather than HK_TYPE_DOMAIN in the initialization
> path.
> Perhaps this part of the alignment hasn't been merged into the mainline yet?

Yes, I notice that too. I am working on a patch to alias  it to 
HK_TYPE_DOMAIN instead.

Cheers,
Longman

>
> Thanks for the clarification and for your work on this!
> Best regards,
> Sheviks
>

Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[sheviks]

Waiman Long <longman@redhat.com> 於 2026年3月24日週二 上午5:36寫道：
> Yes, I notice that too. I am working on a patch to alias  it to
> HK_TYPE_DOMAIN instead.
>

Thanks for the confirmation. It is great to hear that a patch is
already in the works.

I will keep an eye on the mailing list and look forward to testing it
once it’s available. Thank you for your work on this and for the clear
explanation of the roadmap!

Best regards,
Sheviks

Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[Frederic Weisbecker]

Le Mon, Mar 23, 2026 at 01:48:07AM +0800, sheviks a écrit :
> Hi Frederic, Waiman and maintainers,
> 
> A quick follow-up on my previous report. After leaving the system idle
> for a longer period, I made an observation that pinpoints the issue
> more precisely.
> 
> The cgroup v2 dynamic isolation does eventually work. I noticed that
> the unbound kthreads are eventually migrated to the housekeeping CPU
> (CPU 0), but only after they wake up from sleep and enter the running
> state. This lazy migration highlights why commit 041ee6f3727a is
> causing issues for setups using nohz_full= without isolcpus=:
> 
> 1. At boot time, because isolcpus= is absent, the HK_TYPE_DOMAIN mask
> includes the nohz_full CPUs.
> 
> 2. When unbound kthreads are initially spawned or have their affinity
> set, the new logic relies solely on HK_TYPE_DOMAIN. Consequently, they
> are placed on the nohz_full CPUs and immediately go to sleep there.
> 
> 3. They remain "trapped" on the isolated CPUs until a wake-up event
> finally forces the scheduler to migrate them according to the updated
> cgroup affinity.
> 
> This brings the focus back to HK_TYPE_KTHREAD vs HK_TYPE_DOMAIN. While
> HK_TYPE_DOMAIN might default to all CPUs without isolcpus=,
> HK_TYPE_KTHREAD correctly excludes the nohz_full CPUs from the very
> beginning.
> 
> Is this "spawn on nohz_full and wait for wake-up to migrate" behavior
> intended?

Yes. The affinity is applied right after the first wake-up of the kthread.
And this wake-up is supposed to happen right after the kthread creation
because the only purpose of this first wake-up is to allow for calling
kthread_park(), kthread_bind() or kthread_affine_preferred() between
kthread_create() and wake_up_process().

So I'm wondering why you're facing such an issue. Because by the time
you create cgroup isolated partitions, all kthreads should have performed
their first wake-up already.

Which kthread did you observe after cgroup setting that didn't perform its
first wake-up?

> To prevent these sleeping kthreads from polluting isolated
> CPUs before cgroups can intervene, should the initial affinity check
> still consider HK_TYPE_KTHREAD alongside or instead of HK_TYPE_DOMAIN?

Well, domain isolation wants unbound kthreads to move away. And nohz_full
doesn't make sense without domain isolation. So we should focus on making
things work for HK_TYPE_DOMAIN.

Thanks.

-- 
Frederic Weisbecker
SUSE Labs

Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[sheviks]

Frederic Weisbecker <frederic@kernel.org> 於 2026年3月23日週一 上午6:33寫道：
> Yes. The affinity is applied right after the first wake-up of the kthread.
> And this wake-up is supposed to happen right after the kthread creation
> because the only purpose of this first wake-up is to allow for calling
> kthread_park(), kthread_bind() or kthread_affine_preferred() between
> kthread_create() and wake_up_process().
>
> So I'm wondering why you're facing such an issue. Because by the time
> you create cgroup isolated partitions, all kthreads should have performed
> their first wake-up already.
>
> Which kthread did you observe after cgroup setting that didn't perform its
> first wake-up?
>

Thank you for the response. I have performed a more detailed
observation to track the migration behavior of these kthreads over
time.

Setup and Procedure:
Booted with: nohz_full=1-7 rcu_nocbs=1-7 irqaffinity=0 (No isolcpus).
Monitored kthreads on CPUs 1-7 using: ps -eLo cpuid,comm | grep -e
COMM -e "^ *[1-7] " | grep -ve "/[1-7]$" -e "kworker/[1-7]:" -e nvme0q

Initially, there were 30 kthreads residing on CPUs 1-7 right after boot.
Manually created /sys/fs/cgroup/isolated1.slice and configured
cpuset.cpus.exclusive and cpuset.cpus.partition=isolated.

Migration Timeline:
Within 1 minute: rcuog/0 and rcuog/4 migrated back to CPU 0.
At 2 minutes: khungtaskd and jbd2/zram0-8 migrated.
At 8 minutes: kthreadd migrated.
At 17 minutes: pr/legacy migrated. The count dropped to 24 kthreads.
After 9 hours: 24 kthreads remain on CPUs 1-7.

The 24 kthreads remaining on CPUs 1-7 after 9 hours:
CPUID COMMAND
    1 card0-crtc3
    1 ksmd
    1 scsi_eh_4
    1 scsi_eh_9
    3 pool_workqueue_release
    4 card0-crtc0
    4 kdevtmpfs
    4 rcu_exp_gp_kthread_worker
    4 scsi_eh_5
    5 oom_reaper
    5 rcub/0
    5 scsi_eh_0
    5 scsi_eh_1
    5 scsi_eh_2
    5 scsi_eh_8
    6 kswapd0
    6 psimon
    6 scsi_eh_6
    6 watchdogd
    7 card0-crtc1
    7 card0-crtc2
    7 psimon
    7 scsi_eh_3
    7 scsi_eh_7

> Well, domain isolation wants unbound kthreads to move away. And nohz_full
> doesn't make sense without domain isolation. So we should focus on making
> things work for HK_TYPE_DOMAIN.

I understand your point. Then let’s focus on why these kthreads are
not being re-migrated when the housekeeping mask for HK_TYPE_DOMAIN
effectively changes at runtime via cgroups.

Thanks,
Sheviks

Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[Frederic Weisbecker]

Le Mon, Mar 23, 2026 at 10:00:32AM +0800, sheviks a écrit :
> Frederic Weisbecker <frederic@kernel.org> 於 2026年3月23日週一 上午6:33寫道：
> > Yes. The affinity is applied right after the first wake-up of the kthread.
> > And this wake-up is supposed to happen right after the kthread creation
> > because the only purpose of this first wake-up is to allow for calling
> > kthread_park(), kthread_bind() or kthread_affine_preferred() between
> > kthread_create() and wake_up_process().
> >
> > So I'm wondering why you're facing such an issue. Because by the time
> > you create cgroup isolated partitions, all kthreads should have performed
> > their first wake-up already.
> >
> > Which kthread did you observe after cgroup setting that didn't perform its
> > first wake-up?
> >
> 
> Thank you for the response. I have performed a more detailed
> observation to track the migration behavior of these kthreads over
> time.
> 
> Setup and Procedure:
> Booted with: nohz_full=1-7 rcu_nocbs=1-7 irqaffinity=0 (No isolcpus).
> Monitored kthreads on CPUs 1-7 using: ps -eLo cpuid,comm | grep -e
> COMM -e "^ *[1-7] " | grep -ve "/[1-7]$" -e "kworker/[1-7]:" -e nvme0q
> 
> Initially, there were 30 kthreads residing on CPUs 1-7 right after boot.
> Manually created /sys/fs/cgroup/isolated1.slice and configured
> cpuset.cpus.exclusive and cpuset.cpus.partition=isolated.
> 
> Migration Timeline:
> Within 1 minute: rcuog/0 and rcuog/4 migrated back to CPU 0.
> At 2 minutes: khungtaskd and jbd2/zram0-8 migrated.
> At 8 minutes: kthreadd migrated.
> At 17 minutes: pr/legacy migrated. The count dropped to 24 kthreads.
> After 9 hours: 24 kthreads remain on CPUs 1-7.
> 
> The 24 kthreads remaining on CPUs 1-7 after 9 hours:
> CPUID COMMAND
>     1 card0-crtc3
>     1 ksmd
>     1 scsi_eh_4
>     1 scsi_eh_9
>     3 pool_workqueue_release
>     4 card0-crtc0
>     4 kdevtmpfs
>     4 rcu_exp_gp_kthread_worker
>     4 scsi_eh_5
>     5 oom_reaper
>     5 rcub/0
>     5 scsi_eh_0
>     5 scsi_eh_1
>     5 scsi_eh_2
>     5 scsi_eh_8
>     6 kswapd0
>     6 psimon
>     6 scsi_eh_6
>     6 watchdogd
>     7 card0-crtc1
>     7 card0-crtc2
>     7 psimon
>     7 scsi_eh_3
>     7 scsi_eh_7

"ps -o cpuid" tells where the task is currently running or, if sleeping, where
it ran last.

The cpuids that belong to isolated CPUs you're observing on some kthreads are there
because those tasks have slept the whole time since the cpuset isolated
partition was created. Yet they have been correctly migrated to CPU 0 and that
will be displayed on "ps -o cpuid" the next time those kthreads are woken up.

Use taskset for more accurate information as to where a task is allowed to run.

Thanks.

-- 
Frederic Weisbecker
SUSE Labs

Re: [QUESTION/REGRESSION] Unbound kthreads scheduled on nohz_full CPUs after commit 041ee6f3727a

[sheviks]

Frederic Weisbecker <frederic@kernel.org> 於 2026年3月24日週二 上午5:32寫道：
> "ps -o cpuid" tells where the task is currently running or, if sleeping, where
> it ran last.
>
> The cpuids that belong to isolated CPUs you're observing on some kthreads are there
> because those tasks have slept the whole time since the cpuset isolated
> partition was created. Yet they have been correctly migrated to CPU 0 and that
> will be displayed on "ps -o cpuid" the next time those kthreads are woken up.
>
> Use taskset for more accurate information as to where a task is allowed to run.
>

Thank you very much for the clarification regarding ps vs taskset.

I have re-tested the system using taskset as you suggested.
I can confirm that the affinity masks are indeed updated correctly and
immediately across all kthreads as soon as the cgroup v2 isolated
partition is toggled.

My previous observation was indeed misled by the "last ran CPU" shown
in ps while the threads were sleeping.
This perfectly clears up my confusion.
I appreciate you taking the time to explain this technical nuance!

Best regards,
Sheviks