Re: [PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing

[K Prateek Nayak <kprateek.nayak@amd.com>]

Hello Andrea,

On 8/1/2026 12:49 AM, Andrea Righi wrote:
> find_new_ilb() selects the first idle housekeeping CPU without
> considering whether another thread is running on the same physical core.
> On an SMT system, the idle load balancer can therefore activate both
> siblings even when another housekeeping CPU has an entirely idle core.
> 
> On most SMT systems, this is not problematic because the idle load
> balancer is a short-lived activity and the transient wakeup of a sibling
> has negligible performance impact.
> 
> However, this can be particularly costly on NVIDIA Olympus cores used in
> Vera. Briefly activating an otherwise idle sibling can reduce the
> performance available to the other sibling and this effect does not
> necessarily end once the activated sibling becomes idle: after the ILB
> finishes and its CPU enters WFI, full single-thread performance is
> restored only after the sibling has remained idle for a qualification
> interval (10 Ki cycles on the tested Vera system). Repeated short
> sibling wakeups can therefore sustain the interference even with little
> actual overlap.
> 
> Prevent this by preferring an idle housekeeping CPU whose entire SMT
> core is idle. Retain the first idle CPU as a fallback when no fully idle
> core is available, so NOHZ balancing continues to make forward progress.
> Once a partially busy core has been examined, skip its remaining SMT
> siblings to avoid repeating the core-idle check on wide SMT systems.
> 
> Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
> task per SMT core within its CPU affinity mask improved from
> approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
> 
> Note that this preference may wake a fully idle physical core instead of
> using an idle sibling of an active core, potentially increasing ILB
> wakeup latency or energy consumption on some architectures. It may also
> scan additional CPUs before selecting the one to run the ILB. The
> selection falls back to the first idle CPU when no fully idle SMT core
> is available. Non-SMT systems continue to select the first idle
> housekeeping CPU.

I tested this on my end and couldn't really find a workload that shows a
significant difference but I like the general idea of using an idle core
for newidle balance so FWIW, feel free to include:

Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>

[..snip..]

> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 37001c63452e5..574b6b3ee922a 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13965,28 +13965,66 @@ static inline int on_null_domain(struct rq *rq)
>  static inline int find_new_ilb(void)
>  {
>  	int this_cpu = smp_processor_id();
> -	const struct cpumask *hk_mask;
> -	int ilb_cpu;
> +	struct cpumask *ilb_cpus;
> +	int ilb_cpu, fallback = -1;

nit. In case you are re-spinning this, could you rearrange these
two line to follow reverse x-mas tree.

-- 
Thanks and Regards,
Prateek