Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Adam Li <adamli@os.amperecomputing.com>]

Hi Valentin,

Thanks for your comments.
On 8/19/2025 10:00 PM, Valentin Schneider wrote:
> On 19/08/25 02:57, Adam Li wrote:
>> A qualified CPU to run NOHZ idle load balancing (ILB) has to be:
>> 1) housekeeping CPU in housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
>> 2) and in nohz.idle_cpus_mask
>> 3) and idle
>> 4) and not current CPU
>>
>> If most CPUs are in nohz_full CPU list there is few housekeeping CPU left.
>> In the worst case if all CPUs are in nohz_full only the boot CPU is used
>> for housekeeping. And the housekeeping CPU is usually busier so it will
>> be unlikely added to nohz.idle_cpus_mask.
>>
>> Therefore if there is few housekeeping CPUs, find_new_ilb() may likely
>> failed to find any CPU to do NOHZ idle load balancing. Some NOHZ CPUs may
>> stay idle while other CPUs are busy.
>>
>> This patch adds fallback options when looking for ILB CPU if there is
>> no CPU meeting above requirements. Then it searches in bellow order:
>> 1) Try looking for the first idle housekeeping CPU
>> 2) Try looking for the first idle CPU in nohz.idle_cpus_mask if no SMT.
>> 3) Select the first housekeeping CPU even if it is busy.
>>
>> With this patch the NOHZ idle balancing happens more frequently.
>>
> 
> I'm not understanding why, in the scenarios outlined above, more NOHZ idle
> balancing is a good thing.
> 
> Considering only housekeeping CPUs, they're all covered by wakeup, periodic
> and idle balancing (on top of NOHZ idle balancing when relevant). So if
> find_new_ilb() never finds a NOHZ-idle CPU, then that means your HK CPUs
> are either always busy or never stopping the tick when going idle, IOW they
> always have some work to do within a jiffy boundary.
> > Am I missing something?
>

I agree with your description about the housekeeping CPUs. In the worst case,
the system only has one housekeeping CPU and this housekeeping CPU is so busy
that:
1) This housekeeping CPU is unlikely idle;
2) and this housekeeping CPU is unlikely in 'nohz.idle_cpus_mask' because tick
is not stopped.
Therefore find_new_ilb() may very likely return -1. *No* CPU can be selected
to do NOHZ idle load balancing.

This patch tries to fix the imbalance of NOHZ idle CPUs (CPUs in nohz.idle_cpus_mask).
Here is more background:

When running llama on arm64 server, some CPUs *keep* idle while others
are 100% busy. All CPUs are in 'nohz_full=' cpu list, and CONFIG_NO_HZ_FULL
is set.

The problem is caused by two issues:
1) Some idle CPUs cannot be added to 'nohz.idle_cpus_mask',
this bug is fixed by another patch:
https://lore.kernel.org/all/20250815065115.289337-2-adamli@os.amperecomputing.com/

2) Even if the idle CPUs are in 'nohz.idle_cpus_mask', *no* CPU can be selected to
do NOHZ idle load balancing because conditions in find_new_ilb() is too strict.
This patch tries to solve this issue.

Hope this information helps.

Thanks,
-adam