[PATCH v6 0/6] sched: Fix cluster scheduling in the presence of asymmetric capacity

[PATCH v6 0/6] sched: Fix cluster scheduling in the presence of asymmetric capacity

[Ricardo Neri]

Hi,

This is v6 of this series. The main change is restoring the
SD_PREFER_SIBLING flag to scheduling domains with asymmetric capacity,
not only those with child cluster domains.

I also replaced arch_scale_cpu_capacity() with get_actual_cpu_capacity() to
account for hardware and cpufreq pressure when identifying equal-capacity
clusters, as Vincent suggested.

Cluster scheduling aims to maximize performance by spreading load across
clusters of CPUs that share mid-level resources [1]. It works well on
uniform systems, but it breaks down on topologies with big and small
cores arranged in clusters. As a result, it fails on several generations
of Intel processors already shipped and upcoming.

Consider the topology below of big (B) cores and clusters of small (s)
cores.
         ------   ------
         | B  |   | B  |   -----------------   -----------------
         |    |   |    |   | s | s | s | s |   | s | s | s | s |
         ------   ------   -----------------   -----------------
         | L2 |   | L2 |   |      L2       |   |       L2      |
         -------------------------------------------------------
         |                          L3                         |
         -------------------------------------------------------

On a partially busy system (one with idle CPUs; busy CPUs have one task
each), scheduling for asymmetric capacity ensures that misfit tasks land on
the big CPUs. The remaining tasks, misfit or not, run on the small CPUs.
When CONFIG_SCHED_CLUSTER is enabled, these remaining tasks are supposed to
be evenly spread among the small-CPU clusters. Today, this does not
happen.

Several issues in the load balancer prevent a small CPU in one cluster
from pulling tasks from another:

 a) update_sd_pick_busiest() may select a fully_busy group with higher
    per-CPU capacity as the busiest, preventing a subsequent fully_busy
    group of equal capacity from being correctly selected.
 b) Misfit-load statistics are used to identify tasks that would benefit
    from migrating to bigger CPUs. Accounting misfit load is pointless if
    the destination CPU is equally small, and it also blocks balancing
    between clusters.
 c) Due to b), groups that are truly has_spare or fully_busy get
    misclassified as misfit_task. update_sd_pick_busiest() then skips
    them, since a small destination CPU cannot help with misfit tasks.
 d) Once a busiest group has been identified, sched_balance_find_src_rq()
    will refuse to migrate tasks to CPUs of equal capacity, even when
    doing so is precisely what is required to balance small-CPU clusters.
 e) The SD_PREFER_SIBLING flag is missing from scheduling domains with
    asymmetric capacity, preventing the balancer from equalizing load
    across sibling small-core clusters.

Together, these issues prevent cluster-level balancing on systems with
asymmetric CPU capacity.

This series addresses each problem and restores the intended behavior.
Details, rationale, and code changes are explained in each patch.

I tested these patches on Alder Lake, which has both SMT Pcores and
clusters of Ecores. I tested with SMT both disabled and enabled. I also
tested on Lunar Lake and Panther Lake, which have an Ecore cluster not
connected to the L3 cache. I repeated the same experiment with
CONFIG_SCHED_CLUSTER disabled. The load balancer behaves as expected.

I also tested the series using a patch from Rafael [2] to use
policy->cpuinfo.max_freq as fallback for cpufreq_pressure when
arch_scale_freq_ref() is not defined. Tasks duly spread and consolidate in
the absence/presence of cpufreq pressure.

Christian also tested this patchset on a synthetic arm64 qemu topology and
observed the expected behavior [3].

Andrea tested this patchset on Vera Rubin (arm64) and found no
regressions [4].

Link: https://lore.kernel.org/r/20210924085104.44806-1-21cnbao@gmail.com/ [1]
Link: https://lore.kernel.org/all/5086499.GXAFRqVoOG@rafael.j.wysocki/ [2]
Link: https://lore.kernel.org/all/e08492e0-d9f3-4574-8841-b633db008507@arm.com/ [3]
Link: https://lore.kernel.org/all/akJu2S8SNgp1IaqH@gpd4/[4]

Changes in v6:
 - Patch 6: Restored the SD_PREFER_SIBLING flag to all scheduling domains
   with asymmetric capacity, not only those with child domains with
   clusters. (Vincent)
 - Patch 5: Used get_actual_cpu_capacity() instead of
   arch_scale_cpu_capacity() to identify clusters of equal capacity.
   (Vincent)
 - Patch 5: Renamed a local variable in sched_balance_find_src_rq() for
   improved readability. (Andrea)
 - Added Reviewed-by tags from Vincent. Thanks!
 - Added Tested-by tags from Andrea. Thanks!
 - Link to v5: https://lore.kernel.org/r/20260622-rneri-fix-cas-clusters-v5-0-19968f2d1497@linux.intel.com

Changes in v5:
 - Added Tested-by tags from Christian. Thanks!
 - Patch 1 (pre-work): Optimized logic to identify CPUs with busy SMT
   siblings only when needed. (Prateek, Chen Yu)
 - Patch 5: Optimized logic to check for architectural capacity only when
   needed.
 - Added Reviewed-by tag from Prateek. Thanks!
 - Link to v4: https://lore.kernel.org/r/20260608-rneri-fix-cas-clusters-v4-0-1526711c944c@linux.intel.com

Changes in v4:
 - Patch 1 (pre-work): Fixed a bug that would block load balancing on SMT
   cores with more than one busy sibling.
 - Patch 2 (pre-work): Fixed a bug that would needlessly update
   sg_overloaded.
 - Patch 5: Reworked logic using a local variable for improved
   readability.
 - Added Reviewed-by tags from Chen Yu, Tim, and Vincent. Thanks!
 - Link to v3: https://lore.kernel.org/r/20260514-rneri-fix-cas-clusters-v3-0-0037869554bd@linux.intel.com

Changes in v3:
 - Patch 3: Reverted the inverted runtime capacity check. The inverted
   form resulted in migrations to CPUs of slightly lower capacity. Guarded
   the check for architectural capacity with the sched_cluster_active
   static key.
 - Patch 4: Expanded the patch description to explain the behavior of
   overloaded groups and low-capacity clusters with spare capacity.
 - Added Reviewed-by tags from Christian. Thanks!
 - Link to v2: https://lore.kernel.org/r/20260429-rneri-fix-cas-clusters-v2-0-cd787de35cc6@linux.intel.com

Changes in v2:
 - Patch 1: Rewrote patch description for clarity. Added a note
   clarifying that SD_ASYM_CPUCAPACITY and SMT are mutually
   exclusive. (Tim)
 - Patch 2: Fixed a bug where the capacity check inadvertently broke
   the mutual exclusion of the sched_reduced_capacity() path. Keep
   marking the root domain as overloaded when misfit tasks are present
   to allow bigger CPUs to help via newly idle balance. (sashiko)
   Fixed the description to state that capacity_greater() looks for
   differences of ~5% or more, not 20%. (Christian)
 - Patch 3: Use arch_scale_cpu_capacity() instead of capacity_of() to
   ignore runtime capacity variability. Inverted the capacity check.
   (Christian)
 - Patch 4: Reworded the patch description for clarity.
 - Link to v1: https://lore.kernel.org/r/20260330-rneri-fix-cas-clusters-v1-0-1e465b6fecb2@linux.intel.com/

---
Ricardo Neri (6):
      sched/fair: Do not skip CPUs of similar capacity with busy SMT siblings
      sched/fair: Also gate overloaded status update for SD_ASYM_CPUCAPACITY
      sched/fair: Check CPU capacity before comparing group types during load balance
      sched/fair: Skip misfit load accounting when the destination CPU cannot help
      sched/fair: Allow load balancing between CPUs of identical capacity
      sched/topology: Restore SD_PREFER_SIBLING in domains with asymmetric capacity

 include/linux/sched/sd_flags.h |  3 +-
 kernel/sched/fair.c            | 66 ++++++++++++++++++++++++++++++------------
 kernel/sched/topology.c        |  4 ---
 3 files changed, 49 insertions(+), 24 deletions(-)
---
base-commit: 26b6066f005f1c3cb7c23b1800eb4c3c67ed85e0
change-id: 20250620-rneri-fix-cas-clusters-bb4287d1e152

Best regards,
-- 
Ricardo Neri <ricardo.neri-calderon@linux.intel.com>

[PATCH v6 1/6] sched/fair: Do not skip CPUs of similar capacity with busy SMT siblings

[Ricardo Neri]

When picking a busiest CPU with only one running task, the function
sched_balance_find_src_rq() skips candidate CPUs if the destination CPU has
less than ~5% extra capacity. This condition only holds if all the SMT
siblings of a CPU are idle.

SMT siblings share the computing resources of a physical core and this
results in reduced capacity if more than one sibling is busy.

Skipping a CPU as described would prevent the load balancer from pulling
tasks from a scheduling group previously and correctly identified as
group_smt_balance (i.e., one with more than one task running).

Do not skip a candidate CPU of similar capacity if it has busy SMT
siblings.

Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
Changes in v6:
 * Added Tested-by tag from Andrea. Thanks!
 * Added Reviewed-by tag from Vincent. Thanks!

Changes in v5:
 * Optimized logic to identify CPUs with busy SMT siblings only when
   needed. (Prateek, Chen Yu)
 * Added Reviewed-by tag from Prateek. Thanks!
 * Christian also provided his Reviewed-by tag, but the patch changed
   significantly since then. I did not think it was correct to keep it
   without him reviewing the updated patch first.

Changes in v4:
 * Introduced this patch.

Changes in v3:
 * N/A

Changes in v2:
 * N/A
---
 kernel/sched/fair.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 37001c63452e..15a0e1ce90b7 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13088,9 +13088,17 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 		 * average load.
 		 */
 		if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
-		    !capacity_greater(capacity_of(env->dst_cpu), capacity) &&
-		    nr_running == 1)
-			continue;
+		    nr_running == 1) {
+			bool smt_degraded_cap = sched_smt_active() && !is_core_idle(i);
+
+			/*
+			 * Busy SMT siblings reduce the capacity of CPU @i. Do
+			 * not skip it in this case.
+			 */
+			if (!smt_degraded_cap &&
+			    !capacity_greater(capacity_of(env->dst_cpu), capacity))
+				continue;
+		}
 
 		/*
 		 * Make sure we only pull tasks from a CPU of lower priority

-- 
2.43.0

[PATCH v6 2/6] sched/fair: Also gate overloaded status update for SD_ASYM_CPUCAPACITY

[Ricardo Neri]

The argument sg_overloaded of update_sg_lb_stats() is only consumed when
balancing at the root domain. It only makes sense to update it in such a
case. Commit 3229adbe7875 ("sched/fair: Do not compute overloaded status
unnecessarily during lb") updated the logic accordingly but missed the case
in which the root domain has the SD_ASYM_CPUCAPACITY flag. Fix this.

Fixes: 3229adbe7875 ("sched/fair: Do not compute overloaded status unnecessarily during lb")
Reported-by: Chen Yu <yu.c.chen@intel.com>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
Changes in v6:
 * Added Reviewed-by tag from Vincent. Thanks!
 * Added Tested-by tag from Andrea. Thanks!

Changes in v5:
 * Added Tested-by tag from Christian. Thanks!

Changes in v4:
 * Introduced this patch.

Changes in v3:
 * N/A

Changes in v2:
 * N/A
---
 kernel/sched/fair.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 15a0e1ce90b7..4e82bc9963e6 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11975,7 +11975,9 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 			/* Check for a misfit task on the cpu */
 			if (sgs->group_misfit_task_load < rq->misfit_task_load) {
 				sgs->group_misfit_task_load = rq->misfit_task_load;
-				*sg_overloaded = 1;
+
+				if (balancing_at_rd)
+					*sg_overloaded = 1;
 			}
 		} else if (env->idle && sched_reduced_capacity(rq, env->sd)) {
 			/* Check for a task running on a CPU with reduced capacity */

-- 
2.43.0

[PATCH v6 3/6] sched/fair: Check CPU capacity before comparing group types during load balance

[Ricardo Neri]

update_sd_pick_busiest() may incorrectly select a fully_busy group as the
busiest group when its per-CPU capacity exceeds that of the destination
CPU. This happens because the type of busiest group is initialized to
group_has_spare and allows the fully_busy group to win the type comparison.

update_sd_pick_busiest() should not choose a candidate scheduling group
with at most one runnable task if its per-CPU capacity is greater than that
of the destination CPU. Such a check already exists, but it is done too
late: after the type comparison, preventing a subsequent fully_busy group
of equal per-CPU capacity from being correctly selected.

Move this check to occur before comparing group types.

Fixes: 0b0695f2b34a ("sched/fair: Rework load_balance()")
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
Changes in v6:
 * Add Tested-by tag from Andrea. Thanks!

Changes in v5:
 * Added Tested-by tag from Christian. Thanks!

Changes in v4:
 * Dropped note on SMT not being affected since SMT + asym capacity is
   now supported.
 * Added Reviewed-by tags from Vincent, Tim, and Chen Yu. Thanks!

Changes in v3:
 * Added a Fixes tag. (Christian)
 * Added Reviewed-by tag from Christian. Thanks!

Changes in v2:
 * Added a note clarifying that SMT and SD_ASYM_CPUCAPACITY are mutually
   exclusive. (Tim)
 * Kept parentheses around bitwise operators for clarity.
 * Rewrote patch description for clarity.
---
 kernel/sched/fair.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 4e82bc9963e6..9132fcd621dd 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12056,6 +12056,17 @@ static bool update_sd_pick_busiest(struct lb_env *env,
 	     sds->local_stat.group_type != group_has_spare))
 		return false;
 
+	/*
+	 * Candidate sg has no more than one task per CPU and has higher
+	 * per-CPU capacity. Migrating tasks to less capable CPUs may harm
+	 * throughput. Maximize throughput, power/energy consequences are not
+	 * considered.
+	 */
+	if ((env->sd->flags & SD_ASYM_CPUCAPACITY) &&
+	    (sgs->group_type <= group_fully_busy) &&
+	    (capacity_greater(sg->sgc->min_capacity, capacity_of(env->dst_cpu))))
+		return false;
+
 	if (sgs->group_type > busiest->group_type)
 		return true;
 
@@ -12162,17 +12173,6 @@ static bool update_sd_pick_busiest(struct lb_env *env,
 		break;
 	}
 
-	/*
-	 * Candidate sg has no more than one task per CPU and has higher
-	 * per-CPU capacity. Migrating tasks to less capable CPUs may harm
-	 * throughput. Maximize throughput, power/energy consequences are not
-	 * considered.
-	 */
-	if ((env->sd->flags & SD_ASYM_CPUCAPACITY) &&
-	    (sgs->group_type <= group_fully_busy) &&
-	    (capacity_greater(sg->sgc->min_capacity, capacity_of(env->dst_cpu))))
-		return false;
-
 	return true;
 }
 

-- 
2.43.0

[PATCH v6 4/6] sched/fair: Skip misfit load accounting when the destination CPU cannot help

[Ricardo Neri]

In domains with asymmetric capacity, identifying misfit load in a
scheduling group is not useful when the destination CPU cannot help (i.e.,
its capacity exceeds the group's maximum CPU capacity by less than ~5%). In
such cases, it also prevents load balance among clusters of equal capacity
when CONFIG_SCHED_CLUSTER is enabled. This happens because
update_sd_pick_busiest() skips candidate groups of type misfit_task if the
destination CPU has similar capacity.

Skipping misfit load accounting in this situation allows the group to be
classified as has_spare or fully_busy and lets load balancing proceed. Keep
marking scheduling groups as overloaded when misfit tasks are present. The
sg_overloaded flag propagates to the root domain and allows bigger CPUs in
it to help via newly idle balance.

Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
Changes in v6:
 * Added Tested-by tag from Andrea. Thanks!

Changes in v5:
 * Added Tested-by tag from Christian. Thanks!

Changes in v4:
 * Added Reviewed-by tags from Vincent and Chen Yu. Thanks!

Changes in v3:
 * Added Reviewed-by tag from Christian. Thanks!

Changes in v2:
 * Moved the check of the destination CPU capacity inside the code block
   used for SD_ASYM_CPUCAPACITY. v1 inadvertently broke the mutual
   exclusion of the sched_reduced_capacity() path.
 * Keep marking the root domain as overloaded to allow bigger CPUs to
   help. (sashiko)
 * Fixed patch description to clarify that the capacity_greater() looks
   for differences of 5% or more. (Christian)
 * Reworded the patch description for clarity.
 * I did not include the Reviewed-by tag from Christian since the patch
   changed functionally.
---
 kernel/sched/fair.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 9132fcd621dd..feea47e6abea 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11972,12 +11972,25 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 			continue;
 
 		if (sd_flags & SD_ASYM_CPUCAPACITY) {
-			/* Check for a misfit task on the cpu */
-			if (sgs->group_misfit_task_load < rq->misfit_task_load) {
-				sgs->group_misfit_task_load = rq->misfit_task_load;
-
+			if (rq->misfit_task_load) {
+				/*
+				 * Always mark the root domain overloaded so big
+				 * CPUs can pick up misfit tasks via newly idle
+				 * balance.
+				 */
 				if (balancing_at_rd)
 					*sg_overloaded = 1;
+
+				/*
+				 * Only account misfit load if @dst_cpu can
+				 * help; otherwise, the group may be classified
+				 * as misfit_task and update_sd_pick_busiest()
+				 * will skip it.
+				 */
+				if (capacity_greater(capacity_of(env->dst_cpu),
+						     group->sgc->max_capacity) &&
+				    (sgs->group_misfit_task_load < rq->misfit_task_load))
+					sgs->group_misfit_task_load = rq->misfit_task_load;
 			}
 		} else if (env->idle && sched_reduced_capacity(rq, env->sd)) {
 			/* Check for a task running on a CPU with reduced capacity */

-- 
2.43.0

[PATCH v6 5/6] sched/fair: Allow load balancing between CPUs of identical capacity

[Ricardo Neri]

sched_balance_find_src_rq() avoids selecting a runqueue with a single
running task as busiest if doing so results in migrating the task to a
CPU with less than ~5% of extra capacity. It also unintentionally
prevents migrations between CPUs of identical capacity.

When CONFIG_SCHED_CLUSTER is enabled, load should be balanced across
clusters of CPUs with the same capacity. Allowing migration between CPUs
of identical capacity is necessary to meet this goal.

Use get_actual_cpu_capacity() to reflect architectural capacity as well
as diminished capacity due to hardware or cpufreq pressure. Guard this
check with the sched_cluster_active static key so that systems without
cluster topology are unaffected.

Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
Changes in v6:
 * Switched to use get_actual_cpu_capacity() instead of
   arch_scale_cpu_capacity(). The former considers rq->avg_hw.load_avg and
   cpufreq_pressure and their impact on CPU capacity. (Vincent)
 * Renamed the variable same_arch_cluster as cluster_equal_cap for
   clarity. (Andrea)
 * Added Tested-by tag from Andrea. Thanks!

Changes in v5:
 * Optimized logic to identify same-arch clusters only when needed.
 * Added Tested-by tag from Christian. Thanks!

Changes in v4:
 * Implemented the check for cluster with a local variable for improved
   readability.

Changes in v3:
 * Reverted the inverted capacity check; the inverted form incorrectly
   allows migrations to CPUs of slightly less capacity.
 * Guarded the check for architectural capacity with the
   sched_cluster_active static key.

Changes in v2:
 * Used arch_scale_cpu_capacity() instead of capacity_of() to ignore
   runtime variability.
 * Inverted the check for runtime capacity. (Christian)
 * Reworded patch description for clarity.
---
 kernel/sched/fair.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index feea47e6abea..de4189b562ac 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13104,13 +13104,20 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 		 */
 		if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
 		    nr_running == 1) {
+			bool cluster_equal_cap = static_branch_unlikely(&sched_cluster_active) &&
+						 (get_actual_cpu_capacity(env->dst_cpu) ==
+						  get_actual_cpu_capacity(i));
 			bool smt_degraded_cap = sched_smt_active() && !is_core_idle(i);
 
 			/*
 			 * Busy SMT siblings reduce the capacity of CPU @i. Do
 			 * not skip it in this case.
+			 *
+			 * CONFIG_SCHED_CLUSTER requires balancing load across
+			 * clusters of identical capacity, accounting for
+			 * hardware and cpufreq pressure.
 			 */
-			if (!smt_degraded_cap &&
+			if (!smt_degraded_cap && !cluster_equal_cap &&
 			    !capacity_greater(capacity_of(env->dst_cpu), capacity))
 				continue;
 		}

-- 
2.43.0

[PATCH v6 6/6] sched/topology: Restore SD_PREFER_SIBLING in domains with asymmetric capacity

[Ricardo Neri]

Commit 9c63e84db29b ("sched/core: Disable SD_PREFER_SIBLING on asymmetric
CPU capacity domains") removed the SD_PREFER_SIBLING from the domains with
asymmetric capacity. This was done to avoid spreading tasks to sibling
scheduling groups with less capacity, but this does not happen: checks for
capacity in update_sd_pick_busiest(), sched_balance_find_src_group(), and
sched_balance_find_src_rq() prevent migrations from high- to low-capacity
CPUs if the busiest group is not overloaded.

The cluster topology is a notable example: some systems have scheduling
domains spanning CPUs of asymmetric capacity, grouped into two or more
equal-capacity clusters sharing an L2 cache. When CONFIG_SCHED_CLUSTER is
enabled, SD_PREFER_SIBLING is needed in the domain to spread load across
these clusters.

CPUs with spare capacity, big or small, have always helped overloaded
groups. Once the overloading condition disappears, misfit load will still
be used to move high-utilization tasks to bigger CPUs if they have spare
capacity.

Adding the SD_PREFER_SIBLING flag shifts load balancing in shared-LLC
domains from equalizing the number of idle CPUs to equalizing the number
of running tasks. This enables migrations among clusters from newly-idle
load balance, where the outgoing task is already dequeued but the CPU
has not yet transitioned to idle.

Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
Changes in v6:
 * Extended the patch to keep SD_PREFER_SIBLING in all asymmetric
   topologies. (Vincent)
 * I removed the Reviewed-by tag from Tim, since the updated patch is
   significantly different to what he reviewed. I am happy to re-apply
   the tag on an updated review.
 * Added Tested-by tag from Andrea. Thanks!

Changes in v5:
 * Improved inline comments for accuracy.
 * Added Tested-by tag from Christian. Thanks!

Changes in v4:
 * Added Reviewed-by tag from Tim. Thanks!

Changes in v3:
 * Updated documentation of SD_PREFER_SIBLING.
 * Expanded the patch description to explain the behavior when overloaded
   groups are involved.

Changes in v2:
 * Reworded the patch description for clarity.
 * Kept parentheses around bitwise operators for clarity.
---
 include/linux/sched/sd_flags.h | 3 +--
 kernel/sched/topology.c        | 4 ----
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/include/linux/sched/sd_flags.h b/include/linux/sched/sd_flags.h
index 42839cfa2778..dc3ec2452ee1 100644
--- a/include/linux/sched/sd_flags.h
+++ b/include/linux/sched/sd_flags.h
@@ -146,8 +146,7 @@ SD_FLAG(SD_ASYM_PACKING, SDF_NEEDS_GROUPS)
 /*
  * Prefer to place tasks in a sibling domain
  *
- * Set up until domains start spanning NUMA nodes. Close to being a SHARED_CHILD
- * flag, but cleared below domains with SD_ASYM_CPUCAPACITY.
+ * Set up until domains start spanning NUMA nodes.
  *
  * NEEDS_GROUPS: Load balancing flag.
  */
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 622e2e01974c..21e816ad23ee 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -1995,10 +1995,6 @@ sd_init(struct sched_domain_topology_level *tl,
 	/*
 	 * Convert topological properties into behaviour.
 	 */
-	/* Don't attempt to spread across CPUs of different capacities. */
-	if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child)
-		sd->child->flags &= ~SD_PREFER_SIBLING;
-
 	if (sd->flags & SD_SHARE_CPUCAPACITY) {
 		sd->imbalance_pct = 110;
 

-- 
2.43.0

Re: [PATCH v6 5/6] sched/fair: Allow load balancing between CPUs of identical capacity

[Christian Loehle]

On 7/21/26 03:43, Ricardo Neri wrote:
> sched_balance_find_src_rq() avoids selecting a runqueue with a single
> running task as busiest if doing so results in migrating the task to a
> CPU with less than ~5% of extra capacity. It also unintentionally
> prevents migrations between CPUs of identical capacity.
> 
> When CONFIG_SCHED_CLUSTER is enabled, load should be balanced across
> clusters of CPUs with the same capacity. Allowing migration between CPUs
> of identical capacity is necessary to meet this goal.
> 
> Use get_actual_cpu_capacity() to reflect architectural capacity as well
> as diminished capacity due to hardware or cpufreq pressure. Guard this
> check with the sched_cluster_active static key so that systems without
> cluster topology are unaffected.
> 
> Tested-by: Christian Loehle <christian.loehle@arm.com>
> Tested-by: Andrea Righi <arighi@nvidia.com>
> Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> ---
> Changes in v6:
>  * Switched to use get_actual_cpu_capacity() instead of
>    arch_scale_cpu_capacity(). The former considers rq->avg_hw.load_avg and
>    cpufreq_pressure and their impact on CPU capacity. (Vincent)
>  * Renamed the variable same_arch_cluster as cluster_equal_cap for
>    clarity. (Andrea)
>  * Added Tested-by tag from Andrea. Thanks!
> 
> Changes in v5:
>  * Optimized logic to identify same-arch clusters only when needed.
>  * Added Tested-by tag from Christian. Thanks!
> 
> Changes in v4:
>  * Implemented the check for cluster with a local variable for improved
>    readability.
> 
> Changes in v3:
>  * Reverted the inverted capacity check; the inverted form incorrectly
>    allows migrations to CPUs of slightly less capacity.
>  * Guarded the check for architectural capacity with the
>    sched_cluster_active static key.
> 
> Changes in v2:
>  * Used arch_scale_cpu_capacity() instead of capacity_of() to ignore
>    runtime variability.
>  * Inverted the check for runtime capacity. (Christian)
>  * Reworded patch description for clarity.
> ---
>  kernel/sched/fair.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index feea47e6abea..de4189b562ac 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13104,13 +13104,20 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
>  		 */
>  		if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
>  		    nr_running == 1) {
> +			bool cluster_equal_cap = static_branch_unlikely(&sched_cluster_active) &&
> +						 (get_actual_cpu_capacity(env->dst_cpu) ==
> +						  get_actual_cpu_capacity(i));


I guess it's extremely unlikely, but it _feels_ wrong to have to clusters of different
arch_scale_cpu_capacity() equal to true here because of system/thermal pressure (which is
obviously considered more transient).
Adding && arch_scale_cpu_capacity(env->dst_cpu) == arch_scale_cpu_capacity(i) might even
make the check cheaper because it's better for the branch predictor than get_actual_cpu_capacity().
Vincent, would you be fine with requiring both: equal get_actual_cpu_capacity() and
arch_scale_cpu_capacity()?

Given how unlikely it actually is in any case:
Reviewed-by: Christian Loehle <christian.loehle@arm.com>

>  			bool smt_degraded_cap = sched_smt_active() && !is_core_idle(i);
>  
>  			/*
>  			 * Busy SMT siblings reduce the capacity of CPU @i. Do
>  			 * not skip it in this case.
> +			 *
> +			 * CONFIG_SCHED_CLUSTER requires balancing load across
> +			 * clusters of identical capacity, accounting for
> +			 * hardware and cpufreq pressure.
>  			 */
> -			if (!smt_degraded_cap &&
> +			if (!smt_degraded_cap && !cluster_equal_cap &&
>  			    !capacity_greater(capacity_of(env->dst_cpu), capacity))
>  				continue;
>  		}
> 

Re: [PATCH v6 5/6] sched/fair: Allow load balancing between CPUs of identical capacity

[Vincent Guittot]

On Tue, 21 Jul 2026 at 04:33, Ricardo Neri
<ricardo.neri-calderon@linux.intel.com> wrote:
>
> sched_balance_find_src_rq() avoids selecting a runqueue with a single
> running task as busiest if doing so results in migrating the task to a
> CPU with less than ~5% of extra capacity. It also unintentionally
> prevents migrations between CPUs of identical capacity.
>
> When CONFIG_SCHED_CLUSTER is enabled, load should be balanced across
> clusters of CPUs with the same capacity. Allowing migration between CPUs
> of identical capacity is necessary to meet this goal.
>
> Use get_actual_cpu_capacity() to reflect architectural capacity as well
> as diminished capacity due to hardware or cpufreq pressure. Guard this
> check with the sched_cluster_active static key so that systems without
> cluster topology are unaffected.
>
> Tested-by: Christian Loehle <christian.loehle@arm.com>
> Tested-by: Andrea Righi <arighi@nvidia.com>
> Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>

Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>

> ---
> Changes in v6:
>  * Switched to use get_actual_cpu_capacity() instead of
>    arch_scale_cpu_capacity(). The former considers rq->avg_hw.load_avg and
>    cpufreq_pressure and their impact on CPU capacity. (Vincent)
>  * Renamed the variable same_arch_cluster as cluster_equal_cap for
>    clarity. (Andrea)
>  * Added Tested-by tag from Andrea. Thanks!
>
> Changes in v5:
>  * Optimized logic to identify same-arch clusters only when needed.
>  * Added Tested-by tag from Christian. Thanks!
>
> Changes in v4:
>  * Implemented the check for cluster with a local variable for improved
>    readability.
>
> Changes in v3:
>  * Reverted the inverted capacity check; the inverted form incorrectly
>    allows migrations to CPUs of slightly less capacity.
>  * Guarded the check for architectural capacity with the
>    sched_cluster_active static key.
>
> Changes in v2:
>  * Used arch_scale_cpu_capacity() instead of capacity_of() to ignore
>    runtime variability.
>  * Inverted the check for runtime capacity. (Christian)
>  * Reworded patch description for clarity.
> ---
>  kernel/sched/fair.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index feea47e6abea..de4189b562ac 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13104,13 +13104,20 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
>                  */
>                 if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
>                     nr_running == 1) {
> +                       bool cluster_equal_cap = static_branch_unlikely(&sched_cluster_active) &&
> +                                                (get_actual_cpu_capacity(env->dst_cpu) ==
> +                                                 get_actual_cpu_capacity(i));
>                         bool smt_degraded_cap = sched_smt_active() && !is_core_idle(i);
>
>                         /*
>                          * Busy SMT siblings reduce the capacity of CPU @i. Do
>                          * not skip it in this case.
> +                        *
> +                        * CONFIG_SCHED_CLUSTER requires balancing load across
> +                        * clusters of identical capacity, accounting for
> +                        * hardware and cpufreq pressure.
>                          */
> -                       if (!smt_degraded_cap &&
> +                       if (!smt_degraded_cap && !cluster_equal_cap &&
>                             !capacity_greater(capacity_of(env->dst_cpu), capacity))
>                                 continue;
>                 }
>
> --
> 2.43.0
>

Re: [PATCH v6 5/6] sched/fair: Allow load balancing between CPUs of identical capacity

[Vincent Guittot]

On Thu, 23 Jul 2026 at 09:11, Christian Loehle <christian.loehle@arm.com> wrote:
>
> On 7/21/26 03:43, Ricardo Neri wrote:
> > sched_balance_find_src_rq() avoids selecting a runqueue with a single
> > running task as busiest if doing so results in migrating the task to a
> > CPU with less than ~5% of extra capacity. It also unintentionally
> > prevents migrations between CPUs of identical capacity.
> >
> > When CONFIG_SCHED_CLUSTER is enabled, load should be balanced across
> > clusters of CPUs with the same capacity. Allowing migration between CPUs
> > of identical capacity is necessary to meet this goal.
> >
> > Use get_actual_cpu_capacity() to reflect architectural capacity as well
> > as diminished capacity due to hardware or cpufreq pressure. Guard this
> > check with the sched_cluster_active static key so that systems without
> > cluster topology are unaffected.
> >
> > Tested-by: Christian Loehle <christian.loehle@arm.com>
> > Tested-by: Andrea Righi <arighi@nvidia.com>
> > Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> > ---
> > Changes in v6:
> >  * Switched to use get_actual_cpu_capacity() instead of
> >    arch_scale_cpu_capacity(). The former considers rq->avg_hw.load_avg and
> >    cpufreq_pressure and their impact on CPU capacity. (Vincent)
> >  * Renamed the variable same_arch_cluster as cluster_equal_cap for
> >    clarity. (Andrea)
> >  * Added Tested-by tag from Andrea. Thanks!
> >
> > Changes in v5:
> >  * Optimized logic to identify same-arch clusters only when needed.
> >  * Added Tested-by tag from Christian. Thanks!
> >
> > Changes in v4:
> >  * Implemented the check for cluster with a local variable for improved
> >    readability.
> >
> > Changes in v3:
> >  * Reverted the inverted capacity check; the inverted form incorrectly
> >    allows migrations to CPUs of slightly less capacity.
> >  * Guarded the check for architectural capacity with the
> >    sched_cluster_active static key.
> >
> > Changes in v2:
> >  * Used arch_scale_cpu_capacity() instead of capacity_of() to ignore
> >    runtime variability.
> >  * Inverted the check for runtime capacity. (Christian)
> >  * Reworded patch description for clarity.
> > ---
> >  kernel/sched/fair.c | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index feea47e6abea..de4189b562ac 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -13104,13 +13104,20 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
> >                */
> >               if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
> >                   nr_running == 1) {
> > +                     bool cluster_equal_cap = static_branch_unlikely(&sched_cluster_active) &&
> > +                                              (get_actual_cpu_capacity(env->dst_cpu) ==
> > +                                               get_actual_cpu_capacity(i));
>
>
> I guess it's extremely unlikely, but it _feels_ wrong to have to clusters of different
> arch_scale_cpu_capacity() equal to true here because of system/thermal pressure (which is
> obviously considered more transient).
> Adding && arch_scale_cpu_capacity(env->dst_cpu) == arch_scale_cpu_capacity(i) might even
> make the check cheaper because it's better for the branch predictor than get_actual_cpu_capacity().
> Vincent, would you be fine with requiring both: equal get_actual_cpu_capacity() and
> arch_scale_cpu_capacity()?

TBH, I don't have a strong opinion on this. I have in mind that the
cpufreq pressure can last a long time (i.e., several hundreds of ms)
so it could make sense to spread tasks between clusters even if one
has lower max capacity. The task will migrate back to the cpu with
higher capacity once the cpufreq pressure is removed but this would
need some test results


>
> Given how unlikely it actually is in any case:
> Reviewed-by: Christian Loehle <christian.loehle@arm.com>
>
> >                       bool smt_degraded_cap = sched_smt_active() && !is_core_idle(i);
> >
> >                       /*
> >                        * Busy SMT siblings reduce the capacity of CPU @i. Do
> >                        * not skip it in this case.
> > +                      *
> > +                      * CONFIG_SCHED_CLUSTER requires balancing load across
> > +                      * clusters of identical capacity, accounting for
> > +                      * hardware and cpufreq pressure.
> >                        */
> > -                     if (!smt_degraded_cap &&
> > +                     if (!smt_degraded_cap && !cluster_equal_cap &&
> >                           !capacity_greater(capacity_of(env->dst_cpu), capacity))
> >                               continue;
> >               }
> >
>

Re: [PATCH v6 6/6] sched/topology: Restore SD_PREFER_SIBLING in domains with asymmetric capacity

[Vincent Guittot]

On Tue, 21 Jul 2026 at 04:33, Ricardo Neri
<ricardo.neri-calderon@linux.intel.com> wrote:
>
> Commit 9c63e84db29b ("sched/core: Disable SD_PREFER_SIBLING on asymmetric
> CPU capacity domains") removed the SD_PREFER_SIBLING from the domains with
> asymmetric capacity. This was done to avoid spreading tasks to sibling
> scheduling groups with less capacity, but this does not happen: checks for
> capacity in update_sd_pick_busiest(), sched_balance_find_src_group(), and
> sched_balance_find_src_rq() prevent migrations from high- to low-capacity
> CPUs if the busiest group is not overloaded.
>
> The cluster topology is a notable example: some systems have scheduling
> domains spanning CPUs of asymmetric capacity, grouped into two or more
> equal-capacity clusters sharing an L2 cache. When CONFIG_SCHED_CLUSTER is
> enabled, SD_PREFER_SIBLING is needed in the domain to spread load across
> these clusters.
>
> CPUs with spare capacity, big or small, have always helped overloaded
> groups. Once the overloading condition disappears, misfit load will still
> be used to move high-utilization tasks to bigger CPUs if they have spare
> capacity.
>
> Adding the SD_PREFER_SIBLING flag shifts load balancing in shared-LLC
> domains from equalizing the number of idle CPUs to equalizing the number
> of running tasks. This enables migrations among clusters from newly-idle
> load balance, where the outgoing task is already dequeued but the CPU
> has not yet transitioned to idle.
>
> Tested-by: Christian Loehle <christian.loehle@arm.com>
> Tested-by: Andrea Righi <arighi@nvidia.com>
> Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>

Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>


> ---
> Changes in v6:
>  * Extended the patch to keep SD_PREFER_SIBLING in all asymmetric
>    topologies. (Vincent)
>  * I removed the Reviewed-by tag from Tim, since the updated patch is
>    significantly different to what he reviewed. I am happy to re-apply
>    the tag on an updated review.
>  * Added Tested-by tag from Andrea. Thanks!
>
> Changes in v5:
>  * Improved inline comments for accuracy.
>  * Added Tested-by tag from Christian. Thanks!
>
> Changes in v4:
>  * Added Reviewed-by tag from Tim. Thanks!
>
> Changes in v3:
>  * Updated documentation of SD_PREFER_SIBLING.
>  * Expanded the patch description to explain the behavior when overloaded
>    groups are involved.
>
> Changes in v2:
>  * Reworded the patch description for clarity.
>  * Kept parentheses around bitwise operators for clarity.
> ---
>  include/linux/sched/sd_flags.h | 3 +--
>  kernel/sched/topology.c        | 4 ----
>  2 files changed, 1 insertion(+), 6 deletions(-)
>
> diff --git a/include/linux/sched/sd_flags.h b/include/linux/sched/sd_flags.h
> index 42839cfa2778..dc3ec2452ee1 100644
> --- a/include/linux/sched/sd_flags.h
> +++ b/include/linux/sched/sd_flags.h
> @@ -146,8 +146,7 @@ SD_FLAG(SD_ASYM_PACKING, SDF_NEEDS_GROUPS)
>  /*
>   * Prefer to place tasks in a sibling domain
>   *
> - * Set up until domains start spanning NUMA nodes. Close to being a SHARED_CHILD
> - * flag, but cleared below domains with SD_ASYM_CPUCAPACITY.
> + * Set up until domains start spanning NUMA nodes.
>   *
>   * NEEDS_GROUPS: Load balancing flag.
>   */
> diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
> index 622e2e01974c..21e816ad23ee 100644
> --- a/kernel/sched/topology.c
> +++ b/kernel/sched/topology.c
> @@ -1995,10 +1995,6 @@ sd_init(struct sched_domain_topology_level *tl,
>         /*
>          * Convert topological properties into behaviour.
>          */
> -       /* Don't attempt to spread across CPUs of different capacities. */
> -       if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child)
> -               sd->child->flags &= ~SD_PREFER_SIBLING;
> -
>         if (sd->flags & SD_SHARE_CPUCAPACITY) {
>                 sd->imbalance_pct = 110;
>
>
> --
> 2.43.0
>

Re: [PATCH v6 5/6] sched/fair: Allow load balancing between CPUs of identical capacity

[Ricardo Neri]

On Tue, Aug 04, 2026 at 11:55:52AM +0200, Vincent Guittot wrote:
> On Thu, 23 Jul 2026 at 09:11, Christian Loehle <christian.loehle@arm.com> wrote:
> >
> > On 7/21/26 03:43, Ricardo Neri wrote:
> > > sched_balance_find_src_rq() avoids selecting a runqueue with a single
> > > running task as busiest if doing so results in migrating the task to a
> > > CPU with less than ~5% of extra capacity. It also unintentionally
> > > prevents migrations between CPUs of identical capacity.
> > >
> > > When CONFIG_SCHED_CLUSTER is enabled, load should be balanced across
> > > clusters of CPUs with the same capacity. Allowing migration between CPUs
> > > of identical capacity is necessary to meet this goal.
> > >
> > > Use get_actual_cpu_capacity() to reflect architectural capacity as well
> > > as diminished capacity due to hardware or cpufreq pressure. Guard this
> > > check with the sched_cluster_active static key so that systems without
> > > cluster topology are unaffected.
> > >
> > > Tested-by: Christian Loehle <christian.loehle@arm.com>
> > > Tested-by: Andrea Righi <arighi@nvidia.com>
> > > Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> > > ---
> > > Changes in v6:
> > >  * Switched to use get_actual_cpu_capacity() instead of
> > >    arch_scale_cpu_capacity(). The former considers rq->avg_hw.load_avg and
> > >    cpufreq_pressure and their impact on CPU capacity. (Vincent)
> > >  * Renamed the variable same_arch_cluster as cluster_equal_cap for
> > >    clarity. (Andrea)
> > >  * Added Tested-by tag from Andrea. Thanks!
> > >
> > > Changes in v5:
> > >  * Optimized logic to identify same-arch clusters only when needed.
> > >  * Added Tested-by tag from Christian. Thanks!
> > >
> > > Changes in v4:
> > >  * Implemented the check for cluster with a local variable for improved
> > >    readability.
> > >
> > > Changes in v3:
> > >  * Reverted the inverted capacity check; the inverted form incorrectly
> > >    allows migrations to CPUs of slightly less capacity.
> > >  * Guarded the check for architectural capacity with the
> > >    sched_cluster_active static key.
> > >
> > > Changes in v2:
> > >  * Used arch_scale_cpu_capacity() instead of capacity_of() to ignore
> > >    runtime variability.
> > >  * Inverted the check for runtime capacity. (Christian)
> > >  * Reworded patch description for clarity.
> > > ---
> > >  kernel/sched/fair.c | 9 ++++++++-
> > >  1 file changed, 8 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index feea47e6abea..de4189b562ac 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -13104,13 +13104,20 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
> > >                */
> > >               if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
> > >                   nr_running == 1) {
> > > +                     bool cluster_equal_cap = static_branch_unlikely(&sched_cluster_active) &&
> > > +                                              (get_actual_cpu_capacity(env->dst_cpu) ==
> > > +                                               get_actual_cpu_capacity(i));
> >
> >
> > I guess it's extremely unlikely, but it _feels_ wrong to have to clusters of different
> > arch_scale_cpu_capacity() equal to true here because of system/thermal pressure (which is
> > obviously considered more transient).
> > Adding && arch_scale_cpu_capacity(env->dst_cpu) == arch_scale_cpu_capacity(i) might even
> > make the check cheaper because it's better for the branch predictor than get_actual_cpu_capacity().
> > Vincent, would you be fine with requiring both: equal get_actual_cpu_capacity() and
> > arch_scale_cpu_capacity()?
> 
> TBH, I don't have a strong opinion on this. I have in mind that the
> cpufreq pressure can last a long time (i.e., several hundreds of ms)
> so it could make sense to spread tasks between clusters even if one
> has lower max capacity. The task will migrate back to the cpu with
> higher capacity once the cpufreq pressure is removed but this would
> need some test results

I am with Vincent in this: it would be better to spread tasks among
clusters of equal _actual_ capacity even if they arch capacity is
different. Besides diminished compute capacity, packing tasks in one
cluser would cause L2 cache contention, which is what cluster
scheduling wants to avoid.

I am running some workloads and will report results.

[tip: sched/core] sched/topology: Restore SD_PREFER_SIBLING in domains with asymmetric capacity

[tip-bot2 for Ricardo Neri]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     f2c2ba7219e535afdb2ae7d66e6e3df0332eab70
Gitweb:        https://git.kernel.org/tip/f2c2ba7219e535afdb2ae7d66e6e3df0332eab70
Author:        Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
AuthorDate:    Mon, 20 Jul 2026 19:43:22 -07:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 07 Aug 2026 18:27:11 +02:00

sched/topology: Restore SD_PREFER_SIBLING in domains with asymmetric capacity

Commit 9c63e84db29b ("sched/core: Disable SD_PREFER_SIBLING on asymmetric
CPU capacity domains") removed the SD_PREFER_SIBLING from the domains with
asymmetric capacity. This was done to avoid spreading tasks to sibling
scheduling groups with less capacity, but this does not happen: checks for
capacity in update_sd_pick_busiest(), sched_balance_find_src_group(), and
sched_balance_find_src_rq() prevent migrations from high- to low-capacity
CPUs if the busiest group is not overloaded.

The cluster topology is a notable example: some systems have scheduling
domains spanning CPUs of asymmetric capacity, grouped into two or more
equal-capacity clusters sharing an L2 cache. When CONFIG_SCHED_CLUSTER is
enabled, SD_PREFER_SIBLING is needed in the domain to spread load across
these clusters.

CPUs with spare capacity, big or small, have always helped overloaded
groups. Once the overloading condition disappears, misfit load will still
be used to move high-utilization tasks to bigger CPUs if they have spare
capacity.

Adding the SD_PREFER_SIBLING flag shifts load balancing in shared-LLC
domains from equalizing the number of idle CPUs to equalizing the number
of running tasks. This enables migrations among clusters from newly-idle
load balance, where the outgoing task is already dequeued but the CPU
has not yet transitioned to idle.

Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-6-bb500bf4afd4@linux.intel.com
---
 include/linux/sched/sd_flags.h | 3 +--
 kernel/sched/topology.c        | 4 ----
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/include/linux/sched/sd_flags.h b/include/linux/sched/sd_flags.h
index 42839cf..dc3ec24 100644
--- a/include/linux/sched/sd_flags.h
+++ b/include/linux/sched/sd_flags.h
@@ -146,8 +146,7 @@ SD_FLAG(SD_ASYM_PACKING, SDF_NEEDS_GROUPS)
 /*
  * Prefer to place tasks in a sibling domain
  *
- * Set up until domains start spanning NUMA nodes. Close to being a SHARED_CHILD
- * flag, but cleared below domains with SD_ASYM_CPUCAPACITY.
+ * Set up until domains start spanning NUMA nodes.
  *
  * NEEDS_GROUPS: Load balancing flag.
  */
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 622e2e0..21e816a 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -1995,10 +1995,6 @@ sd_init(struct sched_domain_topology_level *tl,
 	/*
 	 * Convert topological properties into behaviour.
 	 */
-	/* Don't attempt to spread across CPUs of different capacities. */
-	if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child)
-		sd->child->flags &= ~SD_PREFER_SIBLING;
-
 	if (sd->flags & SD_SHARE_CPUCAPACITY) {
 		sd->imbalance_pct = 110;
 

[tip: sched/core] sched/fair: Allow load balancing between CPUs of identical capacity

[tip-bot2 for Ricardo Neri]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     7fd540b1bcaf59289e6e921463037d4eadc1d75b
Gitweb:        https://git.kernel.org/tip/7fd540b1bcaf59289e6e921463037d4eadc1d75b
Author:        Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
AuthorDate:    Mon, 20 Jul 2026 19:43:21 -07:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 07 Aug 2026 18:27:11 +02:00

sched/fair: Allow load balancing between CPUs of identical capacity

sched_balance_find_src_rq() avoids selecting a runqueue with a single
running task as busiest if doing so results in migrating the task to a
CPU with less than ~5% of extra capacity. It also unintentionally
prevents migrations between CPUs of identical capacity.

When CONFIG_SCHED_CLUSTER is enabled, load should be balanced across
clusters of CPUs with the same capacity. Allowing migration between CPUs
of identical capacity is necessary to meet this goal.

Use get_actual_cpu_capacity() to reflect architectural capacity as well
as diminished capacity due to hardware or cpufreq pressure. Guard this
check with the sched_cluster_active static key so that systems without
cluster topology are unaffected.

Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-5-bb500bf4afd4@linux.intel.com
---
 kernel/sched/fair.c |  9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index f1b4db3..dcf860c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13102,13 +13102,20 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 		 */
 		if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
 		    nr_running == 1) {
+			bool cluster_equal_cap = static_branch_unlikely(&sched_cluster_active) &&
+						 (get_actual_cpu_capacity(env->dst_cpu) ==
+						  get_actual_cpu_capacity(i));
 			bool smt_degraded_cap = sched_smt_active() && !is_core_idle(i);
 
 			/*
 			 * Busy SMT siblings reduce the capacity of CPU @i. Do
 			 * not skip it in this case.
+			 *
+			 * CONFIG_SCHED_CLUSTER requires balancing load across
+			 * clusters of identical capacity, accounting for
+			 * hardware and cpufreq pressure.
 			 */
-			if (!smt_degraded_cap &&
+			if (!smt_degraded_cap && !cluster_equal_cap &&
 			    !capacity_greater(capacity_of(env->dst_cpu), capacity))
 				continue;
 		}

[tip: sched/core] sched/fair: Skip misfit load accounting when the destination CPU cannot help

[tip-bot2 for Ricardo Neri]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     0fbd428d078876cefe908efdbee47a4d5adc5f24
Gitweb:        https://git.kernel.org/tip/0fbd428d078876cefe908efdbee47a4d5adc5f24
Author:        Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
AuthorDate:    Mon, 20 Jul 2026 19:43:20 -07:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 07 Aug 2026 18:27:10 +02:00

sched/fair: Skip misfit load accounting when the destination CPU cannot help

In domains with asymmetric capacity, identifying misfit load in a
scheduling group is not useful when the destination CPU cannot help (i.e.,
its capacity exceeds the group's maximum CPU capacity by less than ~5%). In
such cases, it also prevents load balance among clusters of equal capacity
when CONFIG_SCHED_CLUSTER is enabled. This happens because
update_sd_pick_busiest() skips candidate groups of type misfit_task if the
destination CPU has similar capacity.

Skipping misfit load accounting in this situation allows the group to be
classified as has_spare or fully_busy and lets load balancing proceed. Keep
marking scheduling groups as overloaded when misfit tasks are present. The
sg_overloaded flag propagates to the root domain and allows bigger CPUs in
it to help via newly idle balance.

Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-4-bb500bf4afd4@linux.intel.com
---
 kernel/sched/fair.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index f9d0edd..f1b4db3 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11970,12 +11970,25 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 			continue;
 
 		if (sd_flags & SD_ASYM_CPUCAPACITY) {
-			/* Check for a misfit task on the cpu */
-			if (sgs->group_misfit_task_load < rq->misfit_task_load) {
-				sgs->group_misfit_task_load = rq->misfit_task_load;
-
+			if (rq->misfit_task_load) {
+				/*
+				 * Always mark the root domain overloaded so big
+				 * CPUs can pick up misfit tasks via newly idle
+				 * balance.
+				 */
 				if (balancing_at_rd)
 					*sg_overloaded = 1;
+
+				/*
+				 * Only account misfit load if @dst_cpu can
+				 * help; otherwise, the group may be classified
+				 * as misfit_task and update_sd_pick_busiest()
+				 * will skip it.
+				 */
+				if (capacity_greater(capacity_of(env->dst_cpu),
+						     group->sgc->max_capacity) &&
+				    (sgs->group_misfit_task_load < rq->misfit_task_load))
+					sgs->group_misfit_task_load = rq->misfit_task_load;
 			}
 		} else if (env->idle && sched_reduced_capacity(rq, env->sd)) {
 			/* Check for a task running on a CPU with reduced capacity */

[tip: sched/core] sched/fair: Check CPU capacity before comparing group types during load balance

[tip-bot2 for Ricardo Neri]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     50b101f6e586b4417d060a976fd831cd87e86e2b
Gitweb:        https://git.kernel.org/tip/50b101f6e586b4417d060a976fd831cd87e86e2b
Author:        Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
AuthorDate:    Mon, 20 Jul 2026 19:43:19 -07:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 07 Aug 2026 18:27:10 +02:00

sched/fair: Check CPU capacity before comparing group types during load balance

update_sd_pick_busiest() may incorrectly select a fully_busy group as the
busiest group when its per-CPU capacity exceeds that of the destination
CPU. This happens because the type of busiest group is initialized to
group_has_spare and allows the fully_busy group to win the type comparison.

update_sd_pick_busiest() should not choose a candidate scheduling group
with at most one runnable task if its per-CPU capacity is greater than that
of the destination CPU. Such a check already exists, but it is done too
late: after the type comparison, preventing a subsequent fully_busy group
of equal per-CPU capacity from being correctly selected.

Move this check to occur before comparing group types.

Fixes: 0b0695f2b34a ("sched/fair: Rework load_balance()")
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-3-bb500bf4afd4@linux.intel.com
---
 kernel/sched/fair.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 2c5cfec..f9d0edd 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12054,6 +12054,17 @@ static bool update_sd_pick_busiest(struct lb_env *env,
 	     sds->local_stat.group_type != group_has_spare))
 		return false;
 
+	/*
+	 * Candidate sg has no more than one task per CPU and has higher
+	 * per-CPU capacity. Migrating tasks to less capable CPUs may harm
+	 * throughput. Maximize throughput, power/energy consequences are not
+	 * considered.
+	 */
+	if ((env->sd->flags & SD_ASYM_CPUCAPACITY) &&
+	    (sgs->group_type <= group_fully_busy) &&
+	    (capacity_greater(sg->sgc->min_capacity, capacity_of(env->dst_cpu))))
+		return false;
+
 	if (sgs->group_type > busiest->group_type)
 		return true;
 
@@ -12160,17 +12171,6 @@ has_spare:
 		break;
 	}
 
-	/*
-	 * Candidate sg has no more than one task per CPU and has higher
-	 * per-CPU capacity. Migrating tasks to less capable CPUs may harm
-	 * throughput. Maximize throughput, power/energy consequences are not
-	 * considered.
-	 */
-	if ((env->sd->flags & SD_ASYM_CPUCAPACITY) &&
-	    (sgs->group_type <= group_fully_busy) &&
-	    (capacity_greater(sg->sgc->min_capacity, capacity_of(env->dst_cpu))))
-		return false;
-
 	return true;
 }
 

[tip: sched/core] sched/fair: Also gate overloaded status update for SD_ASYM_CPUCAPACITY

[tip-bot2 for Ricardo Neri]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     6060d61d13a10da8c90da4eadf4a421825149883
Gitweb:        https://git.kernel.org/tip/6060d61d13a10da8c90da4eadf4a421825149883
Author:        Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
AuthorDate:    Mon, 20 Jul 2026 19:43:18 -07:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 07 Aug 2026 18:27:10 +02:00

sched/fair: Also gate overloaded status update for SD_ASYM_CPUCAPACITY

The argument sg_overloaded of update_sg_lb_stats() is only consumed when
balancing at the root domain. It only makes sense to update it in such a
case. Commit 3229adbe7875 ("sched/fair: Do not compute overloaded status
unnecessarily during lb") updated the logic accordingly but missed the case
in which the root domain has the SD_ASYM_CPUCAPACITY flag. Fix this.

Fixes: 3229adbe7875 ("sched/fair: Do not compute overloaded status unnecessarily during lb")
Reported-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-2-bb500bf4afd4@linux.intel.com
---
 kernel/sched/fair.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 974e091..2c5cfec 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11973,7 +11973,9 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 			/* Check for a misfit task on the cpu */
 			if (sgs->group_misfit_task_load < rq->misfit_task_load) {
 				sgs->group_misfit_task_load = rq->misfit_task_load;
-				*sg_overloaded = 1;
+
+				if (balancing_at_rd)
+					*sg_overloaded = 1;
 			}
 		} else if (env->idle && sched_reduced_capacity(rq, env->sd)) {
 			/* Check for a task running on a CPU with reduced capacity */

[tip: sched/core] sched/fair: Do not skip CPUs of similar capacity with busy SMT siblings

[tip-bot2 for Ricardo Neri]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     180ff97c186971c2106e68470e4370ace0b8fbbc
Gitweb:        https://git.kernel.org/tip/180ff97c186971c2106e68470e4370ace0b8fbbc
Author:        Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
AuthorDate:    Mon, 20 Jul 2026 19:43:17 -07:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 07 Aug 2026 18:27:09 +02:00

sched/fair: Do not skip CPUs of similar capacity with busy SMT siblings

When picking a busiest CPU with only one running task, the function
sched_balance_find_src_rq() skips candidate CPUs if the destination CPU has
less than ~5% extra capacity. This condition only holds if all the SMT
siblings of a CPU are idle.

SMT siblings share the computing resources of a physical core and this
results in reduced capacity if more than one sibling is busy.

Skipping a CPU as described would prevent the load balancer from pulling
tasks from a scheduling group previously and correctly identified as
group_smt_balance (i.e., one with more than one task running).

Do not skip a candidate CPU of similar capacity if it has busy SMT
siblings.

Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-1-bb500bf4afd4@linux.intel.com
---
 kernel/sched/fair.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index a24dd20..974e091 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13086,9 +13086,17 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 		 * average load.
 		 */
 		if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
-		    !capacity_greater(capacity_of(env->dst_cpu), capacity) &&
-		    nr_running == 1)
-			continue;
+		    nr_running == 1) {
+			bool smt_degraded_cap = sched_smt_active() && !is_core_idle(i);
+
+			/*
+			 * Busy SMT siblings reduce the capacity of CPU @i. Do
+			 * not skip it in this case.
+			 */
+			if (!smt_degraded_cap &&
+			    !capacity_greater(capacity_of(env->dst_cpu), capacity))
+				continue;
+		}
 
 		/*
 		 * Make sure we only pull tasks from a CPU of lower priority