[PATCH RESEND 0/4] sched: Fix cluster scheduling in the presence of asymmetric capacity

[PATCH RESEND 0/4] sched: Fix cluster scheduling in the presence of asymmetric capacity

[Ricardo Neri]

Cluster scheduling balances load among clusters of CPUs sharing a resource
[1]. It was broken on Intel hybrid processors using asymmetric packing of
tasks. Tim fixed that [2]. It is broken again when combined with asymmetric
CPU capacity.

The diagram below shows a processor with big (B) and small (s) CPUs. Also,
small CPUs are grouped in cluster sharing mid-level cache. This topology is
common in Intel hybrid processors.

         ------   ------
         | B  |   | B  |   -----------------   -----------------
         |    |   |    |   | s | s | s | s |   | s | s | s | s |
         ------   ------   -----------------   -----------------
         | L2 |   | L2 |   |      L2       |   |       L2      |
         -------------------------------------------------------
         |                          L3                         |
         -------------------------------------------------------

On a partially busy system (one with idle CPUs; busy CPUs have one task
each), scheduling for asymmetric capacity ensures that misfit tasks are
placed on the big CPUs. The remaining tasks, misfit or not, run on the
small CPUs. If CONFIG_SCHED_CLUSTER is enabled, these remaining tasks
should be evenly spread between the two small-CPU clusters.

This does not happen today because various checks in the load balancer
prevent a small CPU in one cluster from pulling tasks from another:

  * A bug in update_sd_pick_busiest() causes it to not check for capacity
    when preferring a fully_busy big CPU (which it cannot help) vs a has_
    spare small-CPU cluster (which it can).

  * Accounting misfit load in a group is pointless if the destination CPU
    is equally a small CPU. Moreover, update_sd_pick_busiest() will not
    pick such group as busiest anyway.

  * Once a busiest group has been identified, sched_balance_find_src_rq()
    will refuse to migrate tasks to CPUs of equal capacity.

  * The SD_PREFER_SIBLING flag is removed from scheduling domains with
    asymmetric capacity.

I address these issues in this series. Details are in the changelog of each
patch.

I tested these patches on an Alder Lake system with Hyper-Threading
disabled. I also tested with CONFIG_SCHED_CLUSTER=n to ensure that
processors without clusters continue to work.

[1]. https://lore.kernel.org/r/20210924085104.44806-1-21cnbao@gmail.com/
[2]. https://lore.kernel.org/r/cover.1688770494.git.tim.c.chen@linux.intel.com/

---
Ricardo Neri (4):
      sched/fair: Always skip fully_busy higher-capacity groups for load balance
      sched/fair: Ignore misfit load if the destination CPU cannot help
      sched/fair: Allow load balancing between CPUs of equal capacity
      sched/topology: Keep SD_PREFER_SIBLING for domains with clusters

 kernel/sched/fair.c     | 27 +++++++++++++++------------
 kernel/sched/topology.c | 11 +++++++++--
 2 files changed, 24 insertions(+), 14 deletions(-)
---
base-commit: e51a38e71974982abb3f2f16141763a1511f7a3f
change-id: 20250620-rneri-fix-cas-clusters-bb4287d1e152

Best regards,
-- 
Ricardo Neri <ricardo.neri-calderon@linux.intel.com>

[PATCH RESEND 1/4] sched/fair: Always skip fully_busy higher-capacity groups for load balance

[Ricardo Neri]

update_sd_pick_busiest() is supposed to avoid picking as busiest a
candidate scheduling group with no more than one task if its per-CPU
capacity is greater than that of the destination CPU.

update_sd_pick_busiest() selects as busiest a group if its type is greater
than has_spare (the type of the busiest group is initialized as has_spare).
As a result, a fully_busy group with higher per-CPU capacity can still
be selected as busiest.

Relocate the existing comparison of capacities to occur before comparing
the types of the candidate and busiest groups.

Remove unnecessary parentheses while here.

Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
 kernel/sched/fair.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 7e2963efe800..9da5014f8387 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10372,6 +10372,17 @@ static bool update_sd_pick_busiest(struct lb_env *env,
 	     sds->local_stat.group_type != group_has_spare))
 		return false;
 
+	/*
+	 * Candidate sg has no more than one task per CPU and has higher
+	 * per-CPU capacity. Migrating tasks to less capable CPUs may harm
+	 * throughput. Maximize throughput, power/energy consequences are not
+	 * considered.
+	 */
+	if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
+	    sgs->group_type <= group_fully_busy &&
+	    capacity_greater(sg->sgc->min_capacity, capacity_of(env->dst_cpu)))
+		return false;
+
 	if (sgs->group_type > busiest->group_type)
 		return true;
 
@@ -10474,17 +10485,6 @@ static bool update_sd_pick_busiest(struct lb_env *env,
 		break;
 	}
 
-	/*
-	 * Candidate sg has no more than one task per CPU and has higher
-	 * per-CPU capacity. Migrating tasks to less capable CPUs may harm
-	 * throughput. Maximize throughput, power/energy consequences are not
-	 * considered.
-	 */
-	if ((env->sd->flags & SD_ASYM_CPUCAPACITY) &&
-	    (sgs->group_type <= group_fully_busy) &&
-	    (capacity_greater(sg->sgc->min_capacity, capacity_of(env->dst_cpu))))
-		return false;
-
 	return true;
 }
 

-- 
2.43.0

[PATCH RESEND 2/4] sched/fair: Ignore misfit load if the destination CPU cannot help

[Ricardo Neri]

There is no point in identifying scheduling groups with misfit tasks if the
destination CPU cannot help (i.e., it has less than 20% greater capacity
than the most performant CPU in the group).

Since migrating misfit tasks takes precedence over relieving fully_busy
groups, identifying a group with misfit tasks causes a destination CPU of
smaller maximum capacity to back off (see capacity checks in update_sd_
pick_busiest()) even if it can help: it could help a group of equally small
maximum capacity if classified as fully_busy or has_spare.

The described situation can happen if a scheduling domain has groups of
big CPUs alongside two or more clusters of smaller CPUs that share L2
cache. Load should be balanced between these sets of smaller CPUs when
CONFIG_SCHED_CLUSTER is enabled.

Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
 kernel/sched/fair.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 9da5014f8387..3c50ecffa4c7 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10302,7 +10302,9 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 		if (local_group)
 			continue;
 
-		if (sd_flags & SD_ASYM_CPUCAPACITY) {
+		/* Only look for misfit load if dst_cpu can help */
+		if (sd_flags & SD_ASYM_CPUCAPACITY &&
+		    capacity_greater(capacity_of(env->dst_cpu), group->sgc->max_capacity)) {
 			/* Check for a misfit task on the cpu */
 			if (sgs->group_misfit_task_load < rq->misfit_task_load) {
 				sgs->group_misfit_task_load = rq->misfit_task_load;

-- 
2.43.0

[PATCH RESEND 3/4] sched/fair: Allow load balancing between CPUs of equal capacity

[Ricardo Neri]

sched_balance_find_src_rq() is supposed to avoid picking as busiest a
runqueue with a single running task since that would result in the task
migrating to a lower-capacity CPU. It also prevents migrations between CPUs
of equal capacity.

Migrating tasks between CPUs of equal capacity helps when balancing load in
a scheduling domain in which there are CPUs of different capacity and are
grouped in clusters of CPUs of equal capacity that share L2 cache. Load
should be balanced among these clusters when CONFIG_SCHED_CLUSTER is
enabled.

Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
 kernel/sched/fair.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3c50ecffa4c7..a7fd4f1f4348 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11388,6 +11388,7 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 		 * average load.
 		 */
 		if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
+		    capacity_of(env->dst_cpu) != capacity &&
 		    !capacity_greater(capacity_of(env->dst_cpu), capacity) &&
 		    nr_running == 1)
 			continue;

-- 
2.43.0

[PATCH RESEND 4/4] sched/topology: Keep SD_PREFER_SIBLING for domains with clusters

[Ricardo Neri]

There are topologies with scheduling domains that contain CPUs of
asymmetric capacity and grouped into two or more clusters of CPUs of
equal capacity sharing L2 cache. CONFIG_SCHED_CLUSTER requires to
balance load among clusters sharing a resource.

Keep the SD_PREFER_SIBLING in the child domains to indicate to the load
balancer that it should spread load among cluster siblings.

Checks for capacity in the load balancer will prevent migrations from
high- to low-capacity CPUs. Likewise, misfit load will still be used to
move high-load tasks to bigger CPUs.

Remove unnecessary parentheses while here.

Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
 kernel/sched/topology.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 8e06b1d22e91..61786cfdc78f 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -1689,8 +1689,15 @@ sd_init(struct sched_domain_topology_level *tl,
 	/*
 	 * Convert topological properties into behaviour.
 	 */
-	/* Don't attempt to spread across CPUs of different capacities. */
-	if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child)
+	/*
+	 * Don't attempt to spread across CPUs of different capacities. An
+	 * exception to this rule are domains in which there are clusters of
+	 * CPUs sharing a resource. Keep the flag in such case to balance load
+	 * among them. The load balancer will prevent task migrations from
+	 * high- to low-capacity CPUs.
+	 */
+	if (sd->flags & SD_ASYM_CPUCAPACITY && sd->child &&
+	    !(sd->child->flags & SD_CLUSTER))
 		sd->child->flags &= ~SD_PREFER_SIBLING;
 
 	if (sd->flags & SD_SHARE_CPUCAPACITY) {

-- 
2.43.0

Re: [PATCH RESEND 3/4] sched/fair: Allow load balancing between CPUs of equal capacity

[Christian Loehle]

On 3/30/26 23:20, Ricardo Neri wrote:
> sched_balance_find_src_rq() is supposed to avoid picking as busiest a
> runqueue with a single running task since that would result in the task
> migrating to a lower-capacity CPU. It also prevents migrations between CPUs
> of equal capacity.
> 
> Migrating tasks between CPUs of equal capacity helps when balancing load in
> a scheduling domain in which there are CPUs of different capacity and are
> grouped in clusters of CPUs of equal capacity that share L2 cache. Load
> should be balanced among these clusters when CONFIG_SCHED_CLUSTER is
> enabled.
> 
> Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> ---
>  kernel/sched/fair.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 3c50ecffa4c7..a7fd4f1f4348 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -11388,6 +11388,7 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
>  		 * average load.
>  		 */
>  		if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
> +		    capacity_of(env->dst_cpu) != capacity &&
>  		    !capacity_greater(capacity_of(env->dst_cpu), capacity) &&
>  		    nr_running == 1)
>  			continue;
> 

For consistency, doesn't it make sense to just invert capacity_greater() here?
So:

    capacity_greater(capacity, capacity_of(env->dst_cpu))
  		if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
  		    capacity_greater(capacity, capacity_of(env->dst_cpu)) &&
  		    nr_running == 1)
  			continue;

Re: [PATCH RESEND 2/4] sched/fair: Ignore misfit load if the destination CPU cannot help

[Christian Loehle]

On 3/30/26 23:20, Ricardo Neri wrote:
> There is no point in identifying scheduling groups with misfit tasks if the
> destination CPU cannot help (i.e., it has less than 20% greater capacity
> than the most performant CPU in the group).

There's a mismatch here because capacity_greater() is 5%?
You could use 20% fits fits_capacity() I'd say it's too strict here.

> 
> Since migrating misfit tasks takes precedence over relieving fully_busy
> groups, identifying a group with misfit tasks causes a destination CPU of
> smaller maximum capacity to back off (see capacity checks in update_sd_
> pick_busiest()) even if it can help: it could help a group of equally small
> maximum capacity if classified as fully_busy or has_spare.
> 
> The described situation can happen if a scheduling domain has groups of
> big CPUs alongside two or more clusters of smaller CPUs that share L2
> cache. Load should be balanced between these sets of smaller CPUs when
> CONFIG_SCHED_CLUSTER is enabled.
> 
> Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> ---
>  kernel/sched/fair.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 9da5014f8387..3c50ecffa4c7 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -10302,7 +10302,9 @@ static inline void update_sg_lb_stats(struct lb_env *env,
>  		if (local_group)
>  			continue;
>  
> -		if (sd_flags & SD_ASYM_CPUCAPACITY) {
> +		/* Only look for misfit load if dst_cpu can help */
> +		if (sd_flags & SD_ASYM_CPUCAPACITY &&
> +		    capacity_greater(capacity_of(env->dst_cpu), group->sgc->max_capacity)) {
>  			/* Check for a misfit task on the cpu */
>  			if (sgs->group_misfit_task_load < rq->misfit_task_load) {
>  				sgs->group_misfit_task_load = rq->misfit_task_load;
> 

Apart from the above nit:

Reviewed-by: Christian Loehle <christian.loehle@arm.com>

Re: [PATCH RESEND 2/4] sched/fair: Ignore misfit load if the destination CPU cannot help

[Ricardo Neri]

On Wed, Apr 01, 2026 at 10:30:53AM +0100, Christian Loehle wrote:
> On 3/30/26 23:20, Ricardo Neri wrote:
> > There is no point in identifying scheduling groups with misfit tasks if the
> > destination CPU cannot help (i.e., it has less than 20% greater capacity
> > than the most performant CPU in the group).

Thanks for looking at the patches Christian!

> 
> There's a mismatch here because capacity_greater() is 5%?
> You could use 20% fits fits_capacity() I'd say it's too strict here.

My intent was to describe the difference as 5%. I made a mistake here.
I think checking for a 5% mismatch is more appropriate since should skip
misfit load if dst_cpu and the max per-CPU capacities are similar.

> 
> > 
> > Since migrating misfit tasks takes precedence over relieving fully_busy
> > groups, identifying a group with misfit tasks causes a destination CPU of
> > smaller maximum capacity to back off (see capacity checks in update_sd_
> > pick_busiest()) even if it can help: it could help a group of equally small
> > maximum capacity if classified as fully_busy or has_spare.
> > 
> > The described situation can happen if a scheduling domain has groups of
> > big CPUs alongside two or more clusters of smaller CPUs that share L2
> > cache. Load should be balanced between these sets of smaller CPUs when
> > CONFIG_SCHED_CLUSTER is enabled.
> > 
> > Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> > ---
> >  kernel/sched/fair.c | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> > 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 9da5014f8387..3c50ecffa4c7 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -10302,7 +10302,9 @@ static inline void update_sg_lb_stats(struct lb_env *env,
> >  		if (local_group)
> >  			continue;
> >  
> > -		if (sd_flags & SD_ASYM_CPUCAPACITY) {
> > +		/* Only look for misfit load if dst_cpu can help */
> > +		if (sd_flags & SD_ASYM_CPUCAPACITY &&
> > +		    capacity_greater(capacity_of(env->dst_cpu), group->sgc->max_capacity)) {
> >  			/* Check for a misfit task on the cpu */
> >  			if (sgs->group_misfit_task_load < rq->misfit_task_load) {
> >  				sgs->group_misfit_task_load = rq->misfit_task_load;
> > 
> 
> Apart from the above nit:
> 
> Reviewed-by: Christian Loehle <christian.loehle@arm.com>

Thank you!

Re: [PATCH RESEND 3/4] sched/fair: Allow load balancing between CPUs of equal capacity

[Ricardo Neri]

On Wed, Apr 01, 2026 at 09:56:17AM +0100, Christian Loehle wrote:
> On 3/30/26 23:20, Ricardo Neri wrote:
> > sched_balance_find_src_rq() is supposed to avoid picking as busiest a
> > runqueue with a single running task since that would result in the task
> > migrating to a lower-capacity CPU. It also prevents migrations between CPUs
> > of equal capacity.
> > 
> > Migrating tasks between CPUs of equal capacity helps when balancing load in
> > a scheduling domain in which there are CPUs of different capacity and are
> > grouped in clusters of CPUs of equal capacity that share L2 cache. Load
> > should be balanced among these clusters when CONFIG_SCHED_CLUSTER is
> > enabled.
> > 
> > Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> > ---
> >  kernel/sched/fair.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 3c50ecffa4c7..a7fd4f1f4348 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -11388,6 +11388,7 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
> >  		 * average load.
> >  		 */
> >  		if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
> > +		    capacity_of(env->dst_cpu) != capacity &&
> >  		    !capacity_greater(capacity_of(env->dst_cpu), capacity) &&
> >  		    nr_running == 1)
> >  			continue;
> > 
> 
> For consistency, doesn't it make sense to just invert capacity_greater() here?
> So:
> 
>     capacity_greater(capacity, capacity_of(env->dst_cpu))
>   		if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
>   		    capacity_greater(capacity, capacity_of(env->dst_cpu)) &&
>   		    nr_running == 1)
>   			continue;

Sure! It also makes the code more readable.