[PATCH] sched/cache: Check reduced capacity before alb_break_llc()

[PATCH] sched/cache: Check reduced capacity before alb_break_llc()

[Zhan Xusheng]

From: Zhan Xusheng <zhanxusheng@xiaomi.com>

need_active_balance() moves the only CFS task off a CPU whose capacity
is reduced, for example by IRQ load, to an idle CPU with more.
alb_break_llc() runs first and refuses to move a lone task off its
preferred LLC, so if no other CPU in that LLC is idle the task stays on
the reduced CPU.  Like asym packing, which was moved ahead of
alb_break_llc() for the same reason, this is a performance case that
cache locality should not override.

With two LLCs of two CPUs, the other CPU of the task's LLC busy and 60%
of the task's CPU spent in a hardirq, a CPU-bound two-thread process
stayed there for 20 seconds at 38% of its normal throughput.  With this
change it moves to the idle LLC within a second, as it does with cache
aware scheduling disabled.

Fixes: 714059f79ff0 ("sched/cache: Handle moving single tasks to/from their preferred LLC")
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
---
 kernel/sched/fair.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index cfc982205b98..e313c6d212c6 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13585,12 +13585,6 @@ static int need_active_balance(struct lb_env *env)
 	if (asym_active_balance(env))
 		return 1;
 
-	if (alb_break_llc(env))
-		return 0;
-
-	if (imbalanced_active_balance(env))
-		return 1;
-
 	/*
 	 * The dst_cpu is idle and the src_cpu CPU has only 1 CFS task.
 	 * It's worth migrating the task if the src_cpu's capacity is reduced
@@ -13604,6 +13598,12 @@ static int need_active_balance(struct lb_env *env)
 			return 1;
 	}
 
+	if (alb_break_llc(env))
+		return 0;
+
+	if (imbalanced_active_balance(env))
+		return 1;
+
 	if (env->migration_type == migrate_misfit ||
 	    env->migration_type == migrate_llc_task)
 		return 1;

base-commit: 65ba8f34ec886ec72b2c07ddc8cf3d815c32869d
-- 
2.43.0

Re: [PATCH] sched/cache: Check reduced capacity before alb_break_llc()

[Kayra Cizmeci]

On Thu, Oct 29, 2026 at 10:36:50AM +0800, Zhan Xusheng wrote:

Hi Zhan,

> From: Zhan Xusheng <zhanxusheng@xiaomi.com>
> 
> need_active_balance() moves the only CFS task off a CPU whose capacity
> is reduced, for example by IRQ load, to an idle CPU with more.
> alb_break_llc() runs first and refuses to move a lone task off its
> preferred LLC, so if no other CPU in that LLC is idle the task stays on
> the reduced CPU.  Like asym packing, which was moved ahead of
> alb_break_llc() for the same reason, this is a performance case that
> cache locality should not override.
> 
> With two LLCs of two CPUs, the other CPU of the task's LLC busy and 60%
> of the task's CPU spent in a hardirq, a CPU-bound two-thread process
> stayed there for 20 seconds at 38% of its normal throughput.  With this
> change it moves to the idle LLC within a second, as it does with cache
> aware scheduling disabled.

Sweet.

> 
> Fixes: 714059f79ff0 ("sched/cache: Handle moving single tasks to/from their preferred LLC")
> Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
> ---
>  kernel/sched/fair.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index cfc982205b98..e313c6d212c6 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13585,12 +13585,6 @@ static int need_active_balance(struct lb_env *env)
>  	if (asym_active_balance(env))
>  		return 1;
>  
> -	if (alb_break_llc(env))
> -		return 0;
> -
> -	if (imbalanced_active_balance(env))
> -		return 1;
> -
>  	/*
>  	 * The dst_cpu is idle and the src_cpu CPU has only 1 CFS task.
>  	 * It's worth migrating the task if the src_cpu's capacity is reduced
> @@ -13604,6 +13598,12 @@ static int need_active_balance(struct lb_env *env)
>  			return 1;
>  	}
>  
> +	if (alb_break_llc(env))
> +		return 0;
> +
> +	if (imbalanced_active_balance(env))
> +		return 1;
> +
>  	if (env->migration_type == migrate_misfit ||
>  	    env->migration_type == migrate_llc_task)
>  		return 1;
> 
> base-commit: 65ba8f34ec886ec72b2c07ddc8cf3d815c32869d
> -- 
> 2.43.0
> 

Well yeah. alb_break_llc() just returns true in the nr_running <= 1
case. FWIW, on my tip tree this just creates a conflict. On me
alb_break_llc() is first, then asym_active_balance() and then
imbalanced_active_balance(). I think in that case moving all three
functions would be better? (below this other block that's for this case)



Thanks,

Kayra

Re: [PATCH] sched/cache: Check reduced capacity before alb_break_llc()

[Zhan Xusheng]

Hi Kayra,

On Thu, Oct 29, 2026 at 10:13:10AM +0300, Kayra Cizmeci wrote:
> Well yeah. alb_break_llc() just returns true in the nr_running <= 1
> case. FWIW, on my tip tree this just creates a conflict. On me
> alb_break_llc() is first, then asym_active_balance() and then
> imbalanced_active_balance(). I think in that case moving all three
> functions would be better? (below this other block that's for this case)

That's sched/core, which doesn't have ee92b7ef4c1e ("sched/cache: Honor
asym packing over cache aware scheduling on hybrid systems") yet.  The
patch is on top of it and applies to tip/master.

Only alb_break_llc() needs to go below the capacity check, as it is the
only one that returns 0.  The other two only return 1, so where they
sit relative to the capacity check doesn't matter.
imbalanced_active_balance() moves only to stay behind alb_break_llc().

Thanks,
Zhan Xusheng

Re: [PATCH] sched/cache: Check reduced capacity before alb_break_llc()

[Kayra Cizmeci]

On Thu, Oct 29, 2026 at 03:50:43PM +0800, Zhan Xusheng wrote:
> Hi Kayra,
> 
> On Thu, Oct 29, 2026 at 10:13:10AM +0300, Kayra Cizmeci wrote:
> > Well yeah. alb_break_llc() just returns true in the nr_running <= 1
> > case. FWIW, on my tip tree this just creates a conflict. On me
> > alb_break_llc() is first, then asym_active_balance() and then
> > imbalanced_active_balance(). I think in that case moving all three
> > functions would be better? (below this other block that's for this case)
> 
> That's sched/core, which doesn't have ee92b7ef4c1e ("sched/cache: Honor
> asym packing over cache aware scheduling on hybrid systems") yet.  The
> patch is on top of it and applies to tip/master.
> 
> Only alb_break_llc() needs to go below the capacity check, as it is the
> only one that returns 0.  The other two only return 1, so where they
> sit relative to the capacity check doesn't matter.
> imbalanced_active_balance() moves only to stay behind alb_break_llc().

Ah, my bad :-).


Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com>

> 
> Thanks,
> Zhan Xusheng

Thanks,

Kayra

Re: [PATCH] sched/cache: Check reduced capacity before alb_break_llc()

[Chen Yu]

On Thu, Oct 29, 2026 at 10:13:10AM +0300, Kayra Cizmeci wrote:
> On Thu, Oct 29, 2026 at 10:36:50AM +0800, Zhan Xusheng wrote:
> 
> Hi Zhan,
> 
> > From: Zhan Xusheng <zhanxusheng@xiaomi.com>
> > 
> > need_active_balance() moves the only CFS task off a CPU whose capacity
> > is reduced, for example by IRQ load, to an idle CPU with more.
> > alb_break_llc() runs first and refuses to move a lone task off its
> > preferred LLC, so if no other CPU in that LLC is idle the task stays on
> > the reduced CPU.  Like asym packing, which was moved ahead of
> > alb_break_llc() for the same reason, this is a performance case that
> > cache locality should not override.
> > 
> > With two LLCs of two CPUs, the other CPU of the task's LLC busy and 60%
> > of the task's CPU spent in a hardirq, a CPU-bound two-thread process
> > stayed there for 20 seconds at 38% of its normal throughput.  With this
> > change it moves to the idle LLC within a second, as it does with cache
> > aware scheduling disabled.
> 
> Sweet.
> 
> > 
> > Fixes: 714059f79ff0 ("sched/cache: Handle moving single tasks to/from their preferred LLC")
> > Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
> > ---
> >  kernel/sched/fair.c | 12 ++++++------
> >  1 file changed, 6 insertions(+), 6 deletions(-)
> > 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index cfc982205b98..e313c6d212c6 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -13585,12 +13585,6 @@ static int need_active_balance(struct lb_env *env)
> >  	if (asym_active_balance(env))
> >  		return 1;
> >  
> > -	if (alb_break_llc(env))
> > -		return 0;
> > -
> > -	if (imbalanced_active_balance(env))
> > -		return 1;
> > -
> >  	/*
> >  	 * The dst_cpu is idle and the src_cpu CPU has only 1 CFS task.
> >  	 * It's worth migrating the task if the src_cpu's capacity is reduced
> > @@ -13604,6 +13598,12 @@ static int need_active_balance(struct lb_env *env)
> >  			return 1;
> >  	}
> >  
> > +	if (alb_break_llc(env))
> > +		return 0;
> > +
> > +	if (imbalanced_active_balance(env))
> > +		return 1;
> > +
> >  	if (env->migration_type == migrate_misfit ||
> >  	    env->migration_type == migrate_llc_task)
> >  		return 1;
> > 
> > base-commit: 65ba8f34ec886ec72b2c07ddc8cf3d815c32869d
> > -- 
> > 2.43.0
> > 
> 
> Well yeah. alb_break_llc() just returns true in the nr_running <= 1
> case. FWIW, on my tip tree this just creates a conflict. On me
> alb_break_llc() is first, then asym_active_balance() and then
> imbalanced_active_balance(). I think in that case moving all three
> functions would be better? (below this other block that's for this case)
>

The position of asym has been adjusted in
https://lore.kernel.org/lkml/179284198232.4109677.752653731564503349.tip-bot2@tip-bot2/

thanks,
Chenyu

Re: [PATCH] sched/cache: Check reduced capacity before alb_break_llc()

[Chen Yu]

Hi Xusheng,

On Thu, Oct 29, 2026 at 10:36:50AM +0800, Zhan Xusheng wrote:
> From: Zhan Xusheng <zhanxusheng@xiaomi.com>
> 
> need_active_balance() moves the only CFS task off a CPU whose capacity
> is reduced, for example by IRQ load, to an idle CPU with more.
> alb_break_llc() runs first and refuses to move a lone task off its
> preferred LLC, so if no other CPU in that LLC is idle the task stays on
> the reduced CPU.  Like asym packing, which was moved ahead of
> alb_break_llc() for the same reason, this is a performance case that
> cache locality should not override.
> 
> With two LLCs of two CPUs, the other CPU of the task's LLC busy and 60%
> of the task's CPU spent in a hardirq, a CPU-bound two-thread process
> stayed there for 20 seconds at 38% of its normal throughput.  With this
> change it moves to the idle LLC within a second, as it does with cache
> aware scheduling disabled.
> 
> Fixes: 714059f79ff0 ("sched/cache: Handle moving single tasks to/from their preferred LLC")
> Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
> ---
>  kernel/sched/fair.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index cfc982205b98..e313c6d212c6 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13585,12 +13585,6 @@ static int need_active_balance(struct lb_env *env)
>  	if (asym_active_balance(env))
>  		return 1;
>  
> -	if (alb_break_llc(env))
> -		return 0;
> -
> -	if (imbalanced_active_balance(env))
> -		return 1;
> -
>  	/*
>  	 * The dst_cpu is idle and the src_cpu CPU has only 1 CFS task.
>  	 * It's worth migrating the task if the src_cpu's capacity is reduced
> @@ -13604,6 +13598,12 @@ static int need_active_balance(struct lb_env *env)
>  			return 1;
>  	}
>  
> +	if (alb_break_llc(env))
> +		return 0;
> +
> +	if (imbalanced_active_balance(env))
> +		return 1;
> +
>  	if (env->migration_type == migrate_misfit ||
>  	    env->migration_type == migrate_llc_task)
>  		return 1;
>

Moving the capacity check earlier seems to completely ignore the
task's LLC preference. It seems to be a trade-off between choosing
CPU capacity and cache re-population on a remote LLC, and cross-LLC
access latency due to cross-LLC migration. I don't know which one has
more impact, it looks to be workload-specific. May I know which
benchmark/platform you used to find this issue? I'd like to launch
hackbench/tbench/schbenh/netperf(or yours) to evaluate the impact.

May I know if the issue is because of the
if (env->src_rq->nr_running <= 1)
	return true
in alb_break_llc()?

If yes, to mitigate the possible impact, do you think tuning the
alb_break_llc() would be safer?

thanks,
Chenyu

patch1 introduces a helper to consider the capacity between src_cpu
and dst_cpu, no functional change:

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 0f2364479f9d..35e6cc7a0fb4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13573,10 +13573,27 @@ imbalanced_active_balance(struct lb_env *env)
 	return 0;
 }
 
-static int need_active_balance(struct lb_env *env)
+static inline bool
+reduced_cap_alb(struct lb_env *env)
 {
 	struct sched_domain *sd = env->sd;
 
+	/*
+	 * The dst_cpu is idle and the src_cpu CPU has only 1 CFS task.
+	 * It's worth migrating the task if the src_cpu's capacity is reduced
+	 * because of other sched_class or IRQs if more capacity stays
+	 * available on dst_cpu.
+	 */
+	return env->idle &&
+	       (env->src_rq->cfs.h_nr_runnable == 1) &&
+	       check_cpu_capacity(env->src_rq, sd) &&
+	       (capacity_of(env->src_cpu)*sd->imbalance_pct < capacity_of(env->dst_cpu)*100);
+}
+
+static int need_active_balance(struct lb_env *env)
+{
+	bool enough_cap_src = !reduced_cap_alb(env);
+
 	if (alb_break_llc(env))
 		return 0;
 
@@ -13586,18 +13603,8 @@ static int need_active_balance(struct lb_env *env)
 	if (imbalanced_active_balance(env))
 		return 1;
 
-	/*
-	 * The dst_cpu is idle and the src_cpu CPU has only 1 CFS task.
-	 * It's worth migrating the task if the src_cpu's capacity is reduced
-	 * because of other sched_class or IRQs if more capacity stays
-	 * available on dst_cpu.
-	 */
-	if (env->idle &&
-	    (env->src_rq->cfs.h_nr_runnable == 1)) {
-		if ((check_cpu_capacity(env->src_rq, sd)) &&
-		    (capacity_of(env->src_cpu)*sd->imbalance_pct < capacity_of(env->dst_cpu)*100))
-			return 1;
-	}
+	if (!enough_cap_src)
+		return 1;
 
 	if (env->migration_type == migrate_misfit ||
 	    env->migration_type == migrate_llc_task)



patch2 tunes the abl_break_llc():

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 35e6cc7a0fb4..43ce929f8ad1 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10994,7 +10994,7 @@ static enum llc_mig can_migrate_llc_task(struct lb_env *env,
  * migrating the only runnable task away.
  */
 static inline bool
-alb_break_llc(struct lb_env *env)
+alb_break_llc(struct lb_env *env, bool enough_cap_src)
 {
 	if (!sched_cache_enabled())
 		return false;
@@ -11022,7 +11022,7 @@ alb_break_llc(struct lb_env *env)
 		if (env->migration_type == migrate_misfit)
 			return false;
 
-		if (env->src_rq->nr_running <= 1)
+		if (env->src_rq->nr_running <= 1 && enough_cap_src)
 			return true;
 
 		cur = rcu_dereference_all(env->src_rq->curr);
@@ -11098,7 +11098,7 @@ static inline bool get_llc_stats(int cpu, unsigned long *util,
 }
 
 static inline bool
-alb_break_llc(struct lb_env *env)
+alb_break_llc(struct lb_env *env, bool enough_cap_src)
 {
 	return false;
 }
@@ -13594,7 +13594,7 @@ static int need_active_balance(struct lb_env *env)
 {
 	bool enough_cap_src = !reduced_cap_alb(env);
 
-	if (alb_break_llc(env))
+	if (alb_break_llc(env, enough_cap_src))
 		return 0;
 
 	if (asym_active_balance(env))

Re: [PATCH] sched/cache: Check reduced capacity before alb_break_llc()

[Zhan Xusheng]

Hi Chenyu,

On Thu, Oct 29, 2026 at 04:18:32PM +0800, Chen Yu wrote:
> Moving the capacity check earlier seems to completely ignore the
> task's LLC preference. It seems to be a trade-off between choosing
> CPU capacity and cache re-population on a remote LLC, and cross-LLC
> access latency due to cross-LLC migration. I don't know which one has
> more impact, it looks to be workload-specific. May I know which
> benchmark/platform you used to find this issue? I'd like to launch
> hackbench/tbench/schbenh/netperf(or yours) to evaluate the impact.

No benchmark, I found it reading the code and reproduced it in QEMU
with -smp 4,sockets=2,cores=2 and IRQ_TIME_ACCOUNTING.  A module spends
600us of every 1ms in an hrtimer handler on CPU0, CPU1 runs a pinned
busy loop, and the test process has one busy thread and one thread that
only sleeps (cache aware scheduling skips single-threaded processes).
It starts pinned to CPU0 and is then allowed on all CPUs.  I can send
the module and the program.

The LLC preference is only overridden when the capacity check fires,
which is the same condition as your reduced_cap_alb().

> May I know if the issue is because of the
> if (env->src_rq->nr_running <= 1)
> 	return true
> in alb_break_llc()?

Yes.

> If yes, to mitigate the possible impact, do you think tuning the
> alb_break_llc() would be safer?

I tried your two patches on the same setup.  The task still stays on
CPU0 for the whole 20 seconds, at about 1.8e7 iterations/s against
4.6e7 with cache aware scheduling disabled.  Without the early return,
alb_break_llc() falls through to can_migrate_llc(), which still
forbids moving the task off its preferred LLC here.

Thanks,
Zhan Xusheng