[PATCH v2] sched/cache: Avoid publishing partial LLC utilization

[PATCH v2] sched/cache: Avoid publishing partial LLC utilization

[Jemmy Wong]

update_sg_lb_stats() sums utilization over the CPUs in both
sched_group_span(group) and env->cpus, but takes capacity from
group->sgc->capacity for the whole group. record_sg_llc_stats()
publishes both.

Rejecting an all-affinity-pinned source runqueue removes that CPU from
env->cpus and retries group selection. The retry can publish the
partial utilization with the full capacity.

can_migrate_llc() then treats a busy preferred LLC as having spare
capacity. More tasks can aggregate into it, migrations out can be
blocked, and two busy LLCs may not fall back to generic balancing.
This increases CPU contention and task latency. Found by inspection.

Skip publication on affinity retries using LBF_ALL_PINNED, retaining
the first-pass snapshots and avoiding repeated cpumask checks. On the
first pass, publish either shared LLC statistic only when env->cpus
covers the whole group. Load balancing still uses the affinity-filtered
statistics.

The same window exists during CPU hotplug. env->cpus drops a CPU at
set_cpu_active(false), but the group span and sgc->capacity still
include it until cpuset_cpu_inactive() rebuilds the domains. Keeping
the previous snapshot avoids the mismatched publish. After the rebuild,
the next periodic balance records both statistics again.

Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Suggested-by: Tim Chen <tim.c.chen@linux.intel.com>
Fixes: f025ef275388 ("sched/cache: Record per LLC utilization to guide cache aware scheduling decisions")
Signed-off-by: Jemmy Wong <jemmywong512@gmail.com>
---

v1: https://lore.kernel.org/r/20261010113425.34829-1-jemmywong512@gmail.com/

Changes since v1:
- Skip publication on affinity retries using LBF_ALL_PINNED, retaining
  first-pass snapshots and avoiding repeated cpumask checks, as suggested
  by Prateek and Tim. Keep cpumask_subset() for partial group coverage
  during CPU hotplug on the first pass.
- Update the comment and commit message to explain the retry behavior.
- Add Fixes and Suggested-by tags.

 kernel/sched/fair.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 57360f5cdde4..dc01fdc0526d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12111,6 +12111,14 @@ static void record_sg_llc_stats(struct lb_env *env,
 	if (env->sd->child != rcu_dereference_all(per_cpu(sd_llc, env->dst_cpu)))
 		return;

+	/*
+	 * A redo repeats stats published on the first pass. Hotplug can
+	 * still leave env->cpus partial, so util would not match capacity.
+	 */
+	if ((env->flags & LBF_ALL_PINNED) ||
+	    !cpumask_subset(sched_group_span(group), env->cpus))
+		return;
+
 	/*
 	 * At this point we know this group spans a LLC domain.
 	 * Record the statistic of this group in its corresponding

--
2.54.0 (Apple Git-157)

Re: [PATCH v2] sched/cache: Avoid publishing partial LLC utilization

[Tim Chen]

On Tue, 2026-10-13 at 16:43 +0800, Jemmy Wong wrote:
> update_sg_lb_stats() sums utilization over the CPUs in both
> sched_group_span(group) and env->cpus, but takes capacity from
> group->sgc->capacity for the whole group. record_sg_llc_stats()
> publishes both.
> 
> Rejecting an all-affinity-pinned source runqueue removes that CPU from
> env->cpus and retries group selection. The retry can publish the
> partial utilization with the full capacity.
> 
> can_migrate_llc() then treats a busy preferred LLC as having spare
> capacity. More tasks can aggregate into it, migrations out can be
> blocked, and two busy LLCs may not fall back to generic balancing.
> This increases CPU contention and task latency. Found by inspection.
> 
> Skip publication on affinity retries using LBF_ALL_PINNED, retaining
> the first-pass snapshots and avoiding repeated cpumask checks. On the
> first pass, publish either shared LLC statistic only when env->cpus
> covers the whole group. Load balancing still uses the affinity-filtered
> statistics.
> 
> The same window exists during CPU hotplug. env->cpus drops a CPU at
> set_cpu_active(false), but the group span and sgc->capacity still
> include it until cpuset_cpu_inactive() rebuilds the domains. Keeping
> the previous snapshot avoids the mismatched publish. After the rebuild,
> the next periodic balance records both statistics again.

Thanks.

Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>

Tim

> 
> Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> Suggested-by: Tim Chen <tim.c.chen@linux.intel.com>
> Fixes: f025ef275388 ("sched/cache: Record per LLC utilization to guide cache aware scheduling decisions")
> Signed-off-by: Jemmy Wong <jemmywong512@gmail.com>
> ---
> 
> v1: https://lore.kernel.org/r/20261010113425.34829-1-jemmywong512@gmail.com/
> 
> Changes since v1:
> - Skip publication on affinity retries using LBF_ALL_PINNED, retaining
>   first-pass snapshots and avoiding repeated cpumask checks, as suggested
>   by Prateek and Tim. Keep cpumask_subset() for partial group coverage
>   during CPU hotplug on the first pass.
> - Update the comment and commit message to explain the retry behavior.
> - Add Fixes and Suggested-by tags.
> 
>  kernel/sched/fair.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 57360f5cdde4..dc01fdc0526d 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -12111,6 +12111,14 @@ static void record_sg_llc_stats(struct lb_env *env,
>  	if (env->sd->child != rcu_dereference_all(per_cpu(sd_llc, env->dst_cpu)))
>  		return;
> 
> +	/*
> +	 * A redo repeats stats published on the first pass. Hotplug can
> +	 * still leave env->cpus partial, so util would not match capacity.
> +	 */
> +	if ((env->flags & LBF_ALL_PINNED) ||
> +	    !cpumask_subset(sched_group_span(group), env->cpus))
> +		return;
> +
>  	/*
>  	 * At this point we know this group spans a LLC domain.
>  	 * Record the statistic of this group in its corresponding
> 
> --
> 2.54.0 (Apple Git-157)

Re: [PATCH v2] sched/cache: Avoid publishing partial LLC utilization

[Tim Chen]

On Tue, 2026-10-13 at 16:43 +0800, Jemmy Wong wrote:
> update_sg_lb_stats() sums utilization over the CPUs in both
> sched_group_span(group) and env->cpus, but takes capacity from
> group->sgc->capacity for the whole group. record_sg_llc_stats()
> publishes both.
> 
> Rejecting an all-affinity-pinned source runqueue removes that CPU from
> env->cpus and retries group selection. The retry can publish the
> partial utilization with the full capacity.
> 
> can_migrate_llc() then treats a busy preferred LLC as having spare
> capacity. More tasks can aggregate into it, migrations out can be
> blocked, and two busy LLCs may not fall back to generic balancing.
> This increases CPU contention and task latency. Found by inspection.
> 
> Skip publication on affinity retries using LBF_ALL_PINNED, retaining
> the first-pass snapshots and avoiding repeated cpumask checks. On the
> first pass, publish either shared LLC statistic only when env->cpus
> covers the whole group. Load balancing still uses the affinity-filtered
> statistics.
> 
> The same window exists during CPU hotplug. env->cpus drops a CPU at
> set_cpu_active(false), but the group span and sgc->capacity still
> include it until cpuset_cpu_inactive() rebuilds the domains. Keeping
> the previous snapshot avoids the mismatched publish. After the rebuild,
> the next periodic balance records both statistics again.
> 
> Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> Suggested-by: Tim Chen <tim.c.chen@linux.intel.com>
> Fixes: f025ef275388 ("sched/cache: Record per LLC utilization to guide cache aware scheduling decisions")
> Signed-off-by: Jemmy Wong <jemmywong512@gmail.com>

BTW, I think this is also a candidate for 7.2.x stable.

Tim

> ---
> 
> v1: https://lore.kernel.org/r/20261010113425.34829-1-jemmywong512@gmail.com/
> 
> Changes since v1:
> - Skip publication on affinity retries using LBF_ALL_PINNED, retaining
>   first-pass snapshots and avoiding repeated cpumask checks, as suggested
>   by Prateek and Tim. Keep cpumask_subset() for partial group coverage
>   during CPU hotplug on the first pass.
> - Update the comment and commit message to explain the retry behavior.
> - Add Fixes and Suggested-by tags.
> 
>  kernel/sched/fair.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 57360f5cdde4..dc01fdc0526d 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -12111,6 +12111,14 @@ static void record_sg_llc_stats(struct lb_env *env,
>  	if (env->sd->child != rcu_dereference_all(per_cpu(sd_llc, env->dst_cpu)))
>  		return;
> 
> +	/*
> +	 * A redo repeats stats published on the first pass. Hotplug can
> +	 * still leave env->cpus partial, so util would not match capacity.
> +	 */
> +	if ((env->flags & LBF_ALL_PINNED) ||
> +	    !cpumask_subset(sched_group_span(group), env->cpus))
> +		return;
> +
>  	/*
>  	 * At this point we know this group spans a LLC domain.
>  	 * Record the statistic of this group in its corresponding
> 
> --
> 2.54.0 (Apple Git-157)
> 

Re: [PATCH v2] sched/cache: Avoid publishing partial LLC utilization

[Chen Yu]

On Tue, Oct 13, 2026 at 04:43:38PM +0800, Jemmy Wong wrote:
> Date: Tue, 13 Oct 2026 16:43:38 +0800
> From: Jemmy Wong <jemmywong512@gmail.com>
> To: mingo@redhat.com, peterz@infradead.org, tim.c.chen@linux.intel.com,
>  kprateek.nayak@amd.com, yu.c.chen@intel.com
> Cc: juri.lelli@redhat.com, vincent.guittot@linaro.org,
>  dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com,
>  mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, Jemmy
>  Wong <jemmywong512@gmail.com>
> Subject: [PATCH v2] sched/cache: Avoid publishing partial LLC utilization
> X-Mailer: git-send-email 2.54.0
> 
> Skip publication on affinity retries using LBF_ALL_PINNED, retaining
> the first-pass snapshots and avoiding repeated cpumask checks. On the

[ ... ]

> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 57360f5cdde4..dc01fdc0526d 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -12111,6 +12111,14 @@ static void record_sg_llc_stats(struct lb_env *env,
>  	if (env->sd->child != rcu_dereference_all(per_cpu(sd_llc, env->dst_cpu)))
>  		return;
> 
> +	/*
> +	 * A redo repeats stats published on the first pass. Hotplug can

It took me a while to understand the check of LBF_ALL_PINNED, to make it clear
how about adding comment below for future reference:

Once we enter record_sg_llc_stats(), LBF_ALL_PINNED being set is a
necessary and sufficient condition for being in a redo. Since
env->cpus may be incomplete when LBF_ALL_PINNED is set, and to avoid
recomputing the statistics of the same group during a redo, we skip
this update. The impact of skipping is small, because the interval
between the first pass and the subsequent redo is short.

Otherwise it looks good to me and

Reviewed-by: Chen Yu <yu.c.chen@intel.com>

thanks,
Chenyu