[RFC PATCH RESEND 00/10] sched/fair: A series of load balance patches to improve real-time performance of CFS tasks

[RFC PATCH RESEND 00/10] sched/fair: A series of load balance patches to improve real-time performance of CFS tasks

[Xin Zhao]

Embedded platforms commonly use CONFIG_HZ_250, and testing has revealed
that there are numerous instances of unreasonable CPU idle events on such
platforms. Unreasonable CPU idle refers to situations where the CPU enters
an idle state for a duration of time (t), while there are tasks that can
run on the idle CPU and are not limited by cgroup constraints, yet these
tasks remain unscheduled for a duration greater than (t), t > 2.5 ms.

Testing has shown that over 95% of these events last less than 4 ms, but
there are still some instances of longer durations between 4-5ms, even
occasionally between 5-10 ms. For a real-time system, scheduling delays
greater than 4 ms can lead to performance spikes.

Enabling this option can effectively reduce the occurrence of unreasonable
CPU idle events on low HZ systems like CONFIG_HZ_250, and completely
eliminate events exceeding 4 ms. Note that the feature only affects fair
tasks.

Note that enabling this feature will increase sys%, as it uses CPU time
that would have been idle to expedite the scheduling of tasks. There will
also be some CPU overhead involved in searching for suitable tasks.

This feature has been split into several smaller patches, which will be
elaborated on one by one later. Below are some test data:

Test one compares the number of unreasonable CPU idle events and their
distribution when this feature is enabled versus when it is not, under
the same fillback scenario. The test duration was 60 seconds.

      LB_PROMOTE(on/off) 2.5-3ms 3-4ms  4ms+
index 0               on       0     0     0
index 1              off       4    13     1
index 2               on       0     0     0
index 3              off       6     3     0
index 4               on       0     0     0
index 5              off       1     1     0

Test two compares the performance of the system with and without the
feature enabled, based on the same fillback scenario. Each test lasts for
25 minutes, and a total of 15 comparative tests were conducted. The
results include a comparison of the maximum and median values of
end-to-end latency and sys%.

LB_PROMOTE(on/off)                    on        off
end-to-end latency(max)              172        180
end-to-end latency(median of avg)    166     167.68
sys%(max)                           9.68       9.35
sys%(median of avg)                 8.81       8.55

All of the patches are derived from analysis through ftrace and process
logs of load balance code flow corresponding to every unreasonable CPU
idle event captured during testing.

In patch 4, we define a new feature called LB_PROMOTE, which means to
promote load balance. Currently, this feature only optimizes the CFS load
balance part. The first three patches are prerequisite patches; patch 1
addresses a common issue unrelated to the LB_PROMOTE feature, while
patches 2 and 3 are two prerequisite patches that can be applied
independently of LB_PROMOTE feature but are primarily intended to support
patches 6 and 7. Patches 5, 7, 8, and 10 are all based on the LB_PROMOTE
feature introduced in patch 4. Patch 5 implements an improved version of
select_task_rq_fair() for embedded platforms, enhancing performance and
real-time capabilities. Patch 7 implements preemptive active balance,
while patch 6 allows active_load_balance_cpu_stop() to be reused in
preemptive active balance scenarios. Patch 8 removes checks related to
avg_idle. Patch 9 is a minor optimization that does not depend on the
LB_PROMOTE feature, but since it modifies sched_balance_rq() as patch 10
does, it serves as a precursor to patch 10.

Xin Zhao (10):
  sched/fair: Do not set_rd_overloaded() if rd->online != env->cpus
  scbed/fair: Remove duplicate check for busiest_cpu in
    active_load_balance_cpu_stop()
  sched/fair: Clear active_balance at the end of
    active_load_balance_cpu_stop()
  sched/fair: Add LB_PROMOTE feature to enhance real-time performance of
    fair tasks
  sched/fair: Introduce select_task_rq_fair_thin() to select rq when
    LB_PROMOTE
  sched/fair: Modify active_load_balance_cpu_stop() to accommodate more
    scenarios
  sched/fair: Trigger active balance if a CFS task is preempted when
    LB_PROMOTE
  sched/fair: Do not check avg_idle to prematurely exit newly idle when
    LB_PROMOTE
  sched/fair: Not goto more_balance if newly idle and has pending task
    when LBF_NEED_BREAK
  sched/fair: Strive to find a task to migrate if newly idle when
    LB_PROMOTE

 kernel/sched/core.c     |   3 +
 kernel/sched/fair.c     | 220 +++++++++++++++++++++++++++++++++++-----
 kernel/sched/features.h |  24 +++++
 kernel/sched/sched.h    |   2 +
 4 files changed, 224 insertions(+), 25 deletions(-)

-- 
2.34.1

[RFC PATCH RESEND 01/10] sched/fair: Do not set_rd_overloaded() if rd->online != env->cpus

[Xin Zhao]

In update_sg_lb_stats(), it only traverses sched_group that belongs to
env->cpus, but env->cpus may not necessarily equal rd->online. This can
lead to the incorrect clearing of the overloaded flag of rd. For example,
if cpuA belongs to the online CPU mask of the rd but does not belong to
env->cpus, and cpuA consistently maintains nr_running >= 2, while other
CPUs in rd->online keep rq->nr_running <= 1, the overloaded flag of rd
will not be set until next update of update_sd_lb_stats() for that rd.
During this period, sched_balance_newidle() will prematurely return due to
the incorrect assumption that the rd is in a non-overloaded state.

In update_sd_lb_stats(), add a check to verify whether rd->online is equal
to env->cpus before calling set_rd_overloaded() to avoid such incorrect
settings.

Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
---
 kernel/sched/fair.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index dcf860c59a14..13e873b1ef58 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12679,8 +12679,12 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd
 		env->fbq_type = fbq_classify_group(&sds->busiest_stat);
 
 	if (!env->sd->parent) {
-		/* update overload indicator if we are at root domain */
-		set_rd_overloaded(env->dst_rq->rd, sg_overloaded);
+		/*
+		 * Update overload indicator if we are at root domain.
+		 * Note that env->cpus may change during sched_balance_rq().
+		 */
+		if (cpumask_equal(env->dst_rq->rd->online, env->cpus))
+			set_rd_overloaded(env->dst_rq->rd, sg_overloaded);
 
 		/* Update over-utilization (tipping point, U >= 0) indicator */
 		set_rd_overutilized(env->dst_rq->rd, sg_overutilized);
-- 
2.34.1

Re: [RFC PATCH RESEND 01/10] sched/fair: Do not set_rd_overloaded() if rd->online != env->cpus

[K Prateek Nayak]

Hello Xin,

On 9/10/2026 9:59 AM, Xin Zhao wrote:
> In update_sg_lb_stats(), it only traverses sched_group that belongs to
> env->cpus, but env->cpus may not necessarily equal rd->online. This can
> lead to the incorrect clearing of the overloaded flag of rd. For example,
> if cpuA belongs to the online CPU mask of the rd but does not belong to
> env->cpus, and cpuA consistently maintains nr_running >= 2, while other
> CPUs in rd->online keep rq->nr_running <= 1, the overloaded flag of rd
> will not be set until next update of update_sd_lb_stats() for that rd.
> During this period, sched_balance_newidle() will prematurely return due to
> the incorrect assumption that the rd is in a non-overloaded state.

Only two cases manipulate env.cpus:

1. LBF_DST_PINNED: The CPU doing the load balancing clears itself from
   env.cpus since pinned tasks cannot be moved to it and goes to
   "more_balance" but "more_balance" does not recompute stats and never
   reaches update_sd_lb_stats().

2. LBF_ALL_PINNED: CPU with no movable task is cleared from env.cpus.
   How will rd->overload being set for a CPU that cannot be helped make
   newidle balance any more efficient?

  Since LBF_ALL_PINNED is known with busiest's rq_lock held, maybe you
  can set a rq->flag and later consume it in add_nr_running() to
  do set_rd_overloaded() selectively.


Now a combination of (1) -> (2) -> redo can actually leave the original
"dest_cpu" out of the env.cpus which might be problematic.

Vincent, do you know why we clear the original dest_cpu (the CPU doing
load balancing) from "env.cpus" in LBF_DST_PINNED?

We update the destination to env.new_dst_cpu, "busiest" is till the same
and instead of moving load from A -> B, we are moving it from A -> C.

Later, if we do a "redo", B can still be a valid target for a different
busiest CPU with movable tasks right?

> 
> In update_sd_lb_stats(), add a check to verify whether rd->online is equal
> to env->cpus before calling set_rd_overloaded() to avoid such incorrect
> settings.
> 
> Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
> ---
>  kernel/sched/fair.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index dcf860c59a14..13e873b1ef58 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -12679,8 +12679,12 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd
>  		env->fbq_type = fbq_classify_group(&sds->busiest_stat);
>  
>  	if (!env->sd->parent) {
> -		/* update overload indicator if we are at root domain */
> -		set_rd_overloaded(env->dst_rq->rd, sg_overloaded);
> +		/*
> +		 * Update overload indicator if we are at root domain.
> +		 * Note that env->cpus may change during sched_balance_rq().
> +		 */
> +		if (cpumask_equal(env->dst_rq->rd->online, env->cpus))
> +			set_rd_overloaded(env->dst_rq->rd, sg_overloaded);
>  
>  		/* Update over-utilization (tipping point, U >= 0) indicator */
>  		set_rd_overutilized(env->dst_rq->rd, sg_overutilized);

-- 
Thanks and Regards,
Prateek

Re: [RFC PATCH RESEND 01/10] sched/fair: Do not set_rd_overloaded() if rd->online != env->cpus

[Vincent Guittot]

On Thu, 10 Sept 2026 at 10:30, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>
> Hello Xin,
>
> On 9/10/2026 9:59 AM, Xin Zhao wrote:
> > In update_sg_lb_stats(), it only traverses sched_group that belongs to
> > env->cpus, but env->cpus may not necessarily equal rd->online. This can
> > lead to the incorrect clearing of the overloaded flag of rd. For example,
> > if cpuA belongs to the online CPU mask of the rd but does not belong to
> > env->cpus, and cpuA consistently maintains nr_running >= 2, while other
> > CPUs in rd->online keep rq->nr_running <= 1, the overloaded flag of rd
> > will not be set until next update of update_sd_lb_stats() for that rd.
> > During this period, sched_balance_newidle() will prematurely return due to
> > the incorrect assumption that the rd is in a non-overloaded state.
>
> Only two cases manipulate env.cpus:
>
> 1. LBF_DST_PINNED: The CPU doing the load balancing clears itself from
>    env.cpus since pinned tasks cannot be moved to it and goes to
>    "more_balance" but "more_balance" does not recompute stats and never
>    reaches update_sd_lb_stats().
>
> 2. LBF_ALL_PINNED: CPU with no movable task is cleared from env.cpus.
>    How will rd->overload being set for a CPU that cannot be helped make
>    newidle balance any more efficient?
>
>   Since LBF_ALL_PINNED is known with busiest's rq_lock held, maybe you
>   can set a rq->flag and later consume it in add_nr_running() to
>   do set_rd_overloaded() selectively.
>
>
> Now a combination of (1) -> (2) -> redo can actually leave the original
> "dest_cpu" out of the env.cpus which might be problematic.
>
> Vincent, do you know why we clear the original dest_cpu (the CPU doing
> load balancing) from "env.cpus" in LBF_DST_PINNED?

We clear cpu_busiest when all pinned to prevent re-selecting it next
time and as we can't do anything on it, we try to balance without it

For dest_cpu, it's a bit less obvious but IIRC we can have cases where
we ping pong between a few number of dst cpus because the task checked
with can_migrate goes at the end of the list so I suppose that we can
have:
dst_cpu= cpu0 tries to pull task TA  but can't because of cpumask
it sets new_dst_cpu to cpu1,
dst_cpu=cpu1 tries to pull task TB but can't because of cpumask
it sets new_dst_cpu to cpu0,
go back to the beginning

now in case of redo, we mainly want to keep cleared the busiest cpu
but we could probably add back dst_cpu because the busiest_cpu will be
different

>
> We update the destination to env.new_dst_cpu, "busiest" is till the same
> and instead of moving load from A -> B, we are moving it from A -> C.
>
> Later, if we do a "redo", B can still be a valid target for a different
> busiest CPU with movable tasks right?
>
> >
> > In update_sd_lb_stats(), add a check to verify whether rd->online is equal
> > to env->cpus before calling set_rd_overloaded() to avoid such incorrect
> > settings.
> >
> > Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
> > ---
> >  kernel/sched/fair.c | 8 ++++++--
> >  1 file changed, 6 insertions(+), 2 deletions(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index dcf860c59a14..13e873b1ef58 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -12679,8 +12679,12 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd
> >               env->fbq_type = fbq_classify_group(&sds->busiest_stat);
> >
> >       if (!env->sd->parent) {
> > -             /* update overload indicator if we are at root domain */
> > -             set_rd_overloaded(env->dst_rq->rd, sg_overloaded);
> > +             /*
> > +              * Update overload indicator if we are at root domain.
> > +              * Note that env->cpus may change during sched_balance_rq().
> > +              */
> > +             if (cpumask_equal(env->dst_rq->rd->online, env->cpus))
> > +                     set_rd_overloaded(env->dst_rq->rd, sg_overloaded);
> >
> >               /* Update over-utilization (tipping point, U >= 0) indicator */
> >               set_rd_overutilized(env->dst_rq->rd, sg_overutilized);
>
> --
> Thanks and Regards,
> Prateek
>

Re: [RFC PATCH RESEND 01/10] sched/fair: Do not set_rd_overloaded() if rd->online != env->cpus

[Xin Zhao]

On Thu, 10 Sep 2026 14:00:43 +0530 K Prateek Nayak <kprateek.nayak@amd.com> wrote:

> Only two cases manipulate env.cpus:
> 
> 1. LBF_DST_PINNED: The CPU doing the load balancing clears itself from
>    env.cpus since pinned tasks cannot be moved to it and goes to
>    "more_balance" but "more_balance" does not recompute stats and never
>    reaches update_sd_lb_stats().
> 
> 2. LBF_ALL_PINNED: CPU with no movable task is cleared from env.cpus.
>    How will rd->overload being set for a CPU that cannot be helped make
>    newidle balance any more efficient?

The effective range of LBF_ALL_PINNED is specific to a particular src CPU
and a particular dst CPU, whereas rd->overload is indeed a global marker
that affects all CPUs with idle states. Regardless of whether case 1 has
been processed, it seems unreasonable to me that rd->overload could be
incorrectly cleared due to case 2, because the scopes of the LBF_ALL_PINNED
and ->overload flags are not equivalent.


>  Since LBF_ALL_PINNED is known with busiest's rq_lock held, maybe you
>  can set a rq->flag and later consume it in add_nr_running() to
>  do set_rd_overloaded() selectively.

If the global rd->overload is incorrectly cleared due to LBF_ALL_PINNED
from src (CPUA) to dst (CPUB), it is possible that CPUB may experience
no changes in nr_running for a certain period of time. Therefore, I think
modifying it in add_nr_running may not be appropriate. I'm not sure if my
understanding is correct.


--
Xin Zhao

Re: [RFC PATCH RESEND 01/10] sched/fair: Do not set_rd_overloaded() if rd->online != env->cpus

[K Prateek Nayak]

Hello Xin,

On 9/11/2026 6:36 AM, Xin Zhao wrote:
> On Thu, 10 Sep 2026 14:00:43 +0530 K Prateek Nayak <kprateek.nayak@amd.com> wrote:
> 
>> Only two cases manipulate env.cpus:
>>
>> 1. LBF_DST_PINNED: The CPU doing the load balancing clears itself from
>>    env.cpus since pinned tasks cannot be moved to it and goes to
>>    "more_balance" but "more_balance" does not recompute stats and never
>>    reaches update_sd_lb_stats().
>>
>> 2. LBF_ALL_PINNED: CPU with no movable task is cleared from env.cpus.
>>    How will rd->overload being set for a CPU that cannot be helped make
>>    newidle balance any more efficient?
> 
> The effective range of LBF_ALL_PINNED is specific to a particular src CPU
> and a particular dst CPU, whereas rd->overload is indeed a global marker
> that affects all CPUs with idle states. Regardless of whether case 1 has
> been processed, it seems unreasonable to me that rd->overload could be
> incorrectly cleared due to case 2, because the scopes of the LBF_ALL_PINNED
> and ->overload flags are not equivalent.

What is the point of doing load balancing if the CPUs that are overloaded
have all their tasks pinned? Those are just wasted cycles.
>>  Since LBF_ALL_PINNED is known with busiest's rq_lock held, maybe you
>>  can set a rq->flag and later consume it in add_nr_running() to
>>  do set_rd_overloaded() selectively.
> 
> If the global rd->overload is incorrectly cleared due to LBF_ALL_PINNED
> from src (CPUA) to dst (CPUB), it is possible that CPUB may experience
> no changes in nr_running for a certain period of time.

Why? Tasks can still wake up on it no?

All that clearing rq->overloaded does is indicate to newidle balance
that there aren't any CPUs with movable tasks on them and it is futile
to do any load balancing.

Do you have any numbers where Patch 1 specifically improves stuff?

> Therefore, I think
> modifying it in add_nr_running may not be appropriate. I'm not sure if my
> understanding is correct.

I was thinking something along the lines of:

  (Only build tested)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 5de115f67065..e78bbdab637f 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3000,6 +3000,15 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 			complete = true;
 		}
 
+		/*
+		 * At least one task on this rq might be movable again.
+		 * Check if rq->overloaded needs to be changed.
+		 */
+		if (rq->all_pinned && rq->nr_running > 1) {
+			set_rd_overloaded(rq->rd, 1);
+			rq->all_pinned = 0;
+		}
+
 		preempt_disable();
 		task_rq_unlock(rq, p, rf);
 		if (push_task) {
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 66e3b5cd5902..563327eb5ae8 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13614,6 +13614,19 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 		 */
 		cur_ld_moved = detach_tasks(&env);
 
+		/*
+		 * Indicate this rq currently has all its tasks pinned.
+		 * Next enqueue will reset rd->overloaded accordingly
+		 * if it was cleared during load balancing.
+		 *
+		 * XXX: Do this only when update_sd_lb_stats() clears
+		 * sd_overloaded? Can this be used to skip CPUs with
+		 * pinned tasks in sched_balance_find_src_rq()?
+		 */
+		if (!sd_parent &&
+		    ((env.flags & (LBF_DST_PINNED | LBF_ALL_PINNED)) == LBF_ALL_PINNED))
+			busiest->all_pinned = 1;
+
 		/*
 		 * We've detached some tasks from busiest_rq. Every
 		 * task is masked "TASK_ON_RQ_MIGRATING", so we can safely
@@ -13751,6 +13764,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 
 		/* Record that we found at least one task that could run on this_cpu */
 		env.flags &= ~LBF_ALL_PINNED;
+		busiest->all_pinned = 0;
 
 		/*
 		 * ->active_balance synchronizes accesses to
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 5950391b873d..651a6e637e37 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1361,6 +1361,9 @@ struct rq {
 	struct cpuidle_state	*idle_state;
 #endif
 
+	unsigned char		all_pinned;
+				/* hole */
+
 	unsigned int		nr_pinned;
 	unsigned int		push_busy;
 	struct cpu_stop_work	push_work;
@@ -3060,8 +3063,10 @@ static inline void add_nr_running(struct rq *rq, unsigned count)
 		call_trace_sched_update_nr_running(rq, count);
 	}
 
-	if (prev_nr < 2 && rq->nr_running >= 2)
+	if ((prev_nr < 2 || rq->all_pinned) && rq->nr_running >= 2) {
 		set_rd_overloaded(rq->rd, 1);
+		rq->all_pinned = 0;
+	}
 
 	sched_update_tick_dependency(rq);
 }
-- 
Thanks and Regards,
Prateek

Re: [RFC PATCH RESEND 01/10] sched/fair: Do not set_rd_overloaded() if rd->online != env->cpus

[Xin Zhao]

On Fri, 11 Sep 2026 11:51:10 +0530 K Prateek Nayak <kprateek.nayak@amd.com> wrote:

> What is the point of doing load balancing if the CPUs that are overloaded
> have all their tasks pinned? Those are just wasted cycles.

I don't think so. I sometimes make basic mistakes, so I'll express my complete
viewpoint, and you can check if there's anything wrong with what I'm saying below:

The simplified code in can_migrate_task() related LBF_ALL_PINNED is as follows:

The simplified code in can_migrate_task() related LBF_ALL_PINNED is as follows:
static
int can_migrate_task(struct task_struct *p, struct lb_env *env)
{
    ...
    if (!cpumask_test_cpu(env->dst_cpu, p->cpus_ptr)) {
        ...
        if (env->idle == CPU_NEWLY_IDLE ||
		    env->flags & (LBF_DST_PINNED | LBF_ACTIVE_LB))
			return 0;
        /* Prevent to re-select dst_cpu via env's CPUs: */
		cpu = cpumask_first_and_and(env->dst_grpmask, env->cpus, p->cpus_ptr);

		if (cpu < nr_cpu_ids) {
			env->flags |= LBF_DST_PINNED;
			env->new_dst_cpu = cpu;
		}

		return 0;
    }

	/* Record that we found at least one task that could run on dst_cpu */
	env->flags &= ~LBF_ALL_PINNED;
    ...
}

Thus, LBF_ALL_PINNED does not mean that tasks are all pinned to their current CPU;
it merely indicates that tasks on the src CPU cannot be migrated to the dst CPU.

Example:

Assuming a system with two clusters, each containing two physical CPUs, the
structure is as follows:

                          DIE
           USTER0              CLUSTER1
         cpu0       cpu1        cpu2       cpu3
T0       p1,p2      p3           p4         p5

At time T0:

CPU0 has tasks p1 and p2, and both have a CPU mask of CPU0 and CPU2.
CPU1 has only task p3, with a CPU mask of CPU1.
CPU2 has only task `p4 with a CPU mask of CPU2.
CPU3 has only task p5, with a CPU mask of CPU3.

At time T1:

Task p5 on CPU3 goes to sleep, and then a newly load balance (newly lb) is
executed. It first looks for load balancing within CLUSTER1 and finds no tasks
to migrate.
It then goes up to the DIE level scheduling and finds the busiest CPU, which is
CPU0's queue. However, it discovers that none of the tasks can be migrated to
CPU3. Since this is a newly idle load balance, it won't perform the binding
check for whether other CPUs in env->dst_grpmask can serve as a CPU.
Consequently, it removes the busiest CPU (CPU0) from the CPU set, but CPU1
remains in the CPU set. Therefore, the check if 

(!cpumask_subset(cpus, env.dst_grpmask)) {

passes leading to the redo phase.
Then it executes sched_balance_find_src_group(), followed by update_sd_lb_stats().
At this point, aside from CPU0, there is at most one task on the other CPUs,
causing the _overloaded flag in rd to be cleared.

At time T2:

Task p4 on CPU2 goes to sleep, and a newly idle load balance (newly lb) is
executed. However, since thesg_overloadedflag inrdhas been set,sched_balance_newidle()`
returns early.
But in reality, at this point, it is possible to migrate a task p1 to CPU2.


--
Xin Zhao

Re: [RFC PATCH RESEND 01/10] sched/fair: Do not set_rd_overloaded() if rd->online != env->cpus

[Xin Zhao]

Hello Prateek,

I apologize for the confusion. Please disregard the previous email response; this
email is essentially the same, so you only need to refer to this one.


On Fri, 11 Sep 2026 11:51:10 +0530 K Prateek Nayak <kprateek.nayak@amd.com> wrote:

> What is the point of doing load balancing if the CPUs that are overloaded
> have all their tasks pinned? Those are just wasted cycles.

I don't think so. I sometimes make basic mistakes, so I'll express my complete
viewpoint, and you can check if there's anything wrong with what I'm saying below:

The simplified code in can_migrate_task() related LBF_ALL_PINNED is as follows:
static
int can_migrate_task(struct task_struct *p, struct lb_env *env)
{
    ...
    if (!cpumask_test_cpu(env->dst_cpu, p->cpus_ptr)) {
        ...
        if (env->idle == CPU_NEWLY_IDLE ||
		    env->flags & (LBF_DST_PINNED | LBF_ACTIVE_LB))
			return 0;
        /* Prevent to re-select dst_cpu via env's CPUs: */
		cpu = cpumask_first_and_and(env->dst_grpmask, env->cpus, p->cpus_ptr);

		if (cpu < nr_cpu_ids) {
			env->flags |= LBF_DST_PINNED;
			env->new_dst_cpu = cpu;
		}

		return 0;
    }

	/* Record that we found at least one task that could run on dst_cpu */
	env->flags &= ~LBF_ALL_PINNED;
    ...
}

Thus, LBF_ALL_PINNED does not mean that tasks are all pinned to their current CPU;
it merely indicates that tasks on the src CPU cannot be migrated to the dst CPU.

Example:

Assuming a system with two clusters, each containing two physical CPUs, the
structure is as follows:

                          DIE
           CLUSTER0              CLUSTER1
         cpu0       cpu1        cpu2       cpu3
T0       p1,p2      p3           p4         p5

At time T0:

CPU0 has tasks p1 and p2, and both have a CPU mask of CPU0 and CPU2.
CPU1 has only task p3, with a CPU mask of CPU1.
CPU2 has only task `p4 with a CPU mask of CPU2.
CPU3 has only task p5, with a CPU mask of CPU3.


                          DIE
           CLUSTER0              CLUSTER1
         cpu0       cpu1        cpu2       cpu3
T1       p1,p2      p3           p4         newly lb
At time T1:

Task p5 on CPU3 goes to sleep, and then a newly load balance (newly lb) is
executed. It first looks for load balancing within CLUSTER1 and finds no tasks
to migrate.
It then goes up to the DIE level scheduling and finds the busiest CPU, which is
CPU0's queue. However, it discovers that none of the tasks can be migrated to
CPU3. Since this is a newly idle load balance, it won't perform the binding
check for whether other CPUs in env->dst_grpmask can serve as a CPU.
Consequently, it removes the busiest CPU (CPU0) from the CPU set, but CPU1
remains in the CPU set. Therefore, the check if 

(!cpumask_subset(cpus, env.dst_grpmask)) {

passes leading to the redo phase.
Then it executes sched_balance_find_src_group(), followed by update_sd_lb_stats().
At this point, aside from CPU0, there is at most one task on the other CPUs,
causing the _overloaded flag in rd to be cleared.


                          DIE
           CLUSTER0              CLUSTER1
         cpu0       cpu1        cpu2       cpu3
T2       p1,p2      p3           newly lb
At time T2:

Task p4 on CPU2 goes to sleep, and a newly idle load balance (newly lb) is
executed. However, since thesg_overloadedflag inrdhas been set,sched_balance_newidle()`
returns early.
But in reality, at this point, it is possible to migrate a task p1 to CPU2.


--
Xin Zhao

[RFC PATCH RESEND 02/10] scbed/fair: Remove duplicate check for busiest_cpu in active_load_balance_cpu_stop()

[Xin Zhao]

The check for cpu_active(busiest_cpu) already ensures that busiest_cpu has
not gone down. An additional check for busiest_cpu != smp_processor_id()
is redundant. After this modification, busiest_cpu will no longer be bound
to smp_processor_id(), allowing the active_load_balance_cpu_stop function
to accommodate more scenarios, such as preempt active balancing feature
that will be addressed in later patches.

Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
---
 kernel/sched/fair.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 13e873b1ef58..11c104010b2e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13769,9 +13769,7 @@ static int active_load_balance_cpu_stop(void *data)
 	if (!cpu_active(busiest_cpu) || !cpu_active(target_cpu))
 		goto out_unlock;
 
-	/* Make sure the requested CPU hasn't gone down in the meantime: */
-	if (unlikely(busiest_cpu != smp_processor_id() ||
-		     !busiest_rq->active_balance))
+	if (unlikely(!busiest_rq->active_balance))
 		goto out_unlock;
 
 	/* Is there any task to move? */
-- 
2.34.1

Re: [RFC PATCH RESEND 02/10] scbed/fair: Remove duplicate check for busiest_cpu in active_load_balance_cpu_stop()

[Kayra Cizmeci]

Hello Xin,

> The check for cpu_active(busiest_cpu) already ensures that busiest_cpu has
> not gone down. An additional check for busiest_cpu != smp_processor_id()
> is redundant. After this modification, busiest_cpu will no longer be bound
> to smp_processor_id(), allowing the active_load_balance_cpu_stop function
> to accommodate more scenarios, such as preempt active balancing feature
> that will be addressed in later patches.

The code is complicated and I usually not come near here.

But how did you reach the conclusion of smp_processor_id() check is redundant?

The cpu_active(busiest_cpu) just looks at the CPU mask of the given CPU or busiest in our case.
But smp_processor_id() doesn't performs the same checks. And even if it's doing the same checks,
how are you ensuring that it looks to the busiest CPU without even getting any parameters?

For your conclusion to be right, the busiest CPU must be the CPU that smp_processor_id() looks
and the cpu_active(busiest_cpu), smp_processor_id() needs to behave the same.

And please add some details to the commit message why smp_processor_id() is redundant rather than
skipping what smp_processor_id() does in any case.

Please notify me if I'm wrong :_:

Thanks,
Kayra :-)

Re: [RFC PATCH RESEND 02/10] scbed/fair: Remove duplicate check for busiest_cpu in active_load_balance_cpu_stop()

[Xin Zhao]

On Thu, 10 Sep 2026 14:41:09 +0300 Kayra Cizmeci <kayracizmeci@gmail.com> wrote:

> > The check for cpu_active(busiest_cpu) already ensures that busiest_cpu has
> > not gone down. An additional check for busiest_cpu != smp_processor_id()
> > is redundant. After this modification, busiest_cpu will no longer be bound
> > to smp_processor_id(), allowing the active_load_balance_cpu_stop function
> > to accommodate more scenarios, such as preempt active balancing feature
> > that will be addressed in later patches.
> 
> The code is complicated and I usually not come near here.
> 
> But how did you reach the conclusion of smp_processor_id() check is redundant?
> 
> The cpu_active(busiest_cpu) just looks at the CPU mask of the given CPU or busiest in our case.
> But smp_processor_id() doesn't performs the same checks. And even if it's doing the same checks,
> how are you ensuring that it looks to the busiest CPU without even getting any parameters?
> 
> For your conclusion to be right, the busiest CPU must be the CPU that smp_processor_id() looks
> and the cpu_active(busiest_cpu), smp_processor_id() needs to behave the same.
> 
> And please add some details to the commit message why smp_processor_id() is redundant rather than
> skipping what smp_processor_id() does in any case.
> 
> Please notify me if I'm wrong :_:

The active_load_balance_cpu_stop() is executed on the CPU busiest_cpu where the
stop task is dispatched. Therefore, unless it is during a special time when the
CPU is offline, busiest_cpu is always equal to smp_processor_id()`.

--
Xin Zhao

Re: [RFC PATCH RESEND 02/10] scbed/fair: Remove duplicate check for busiest_cpu in active_load_balance_cpu_stop()

[Kayra Cizmeci]

> The active_load_balance_cpu_stop() is executed on the CPU busiest_cpu where the
> stop task is dispatched. Therefore, unless it is during a special time when the
> CPU is offline, busiest_cpu is always equal to smp_processor_id()`.

Okay.

Could you add something to the commit message that describes this on the next version?

Thanks,
Kayra :_:

[RFC PATCH RESEND 03/10] sched/fair: Clear active_balance at the end of active_load_balance_cpu_stop()

[Xin Zhao]

The rq->active_balance flag is used to prevent multiple CPUs from
simultaneously dispatching active balance stop tasks. Since there can only
ever be one consumer of the stop task, it is not strictly necessary to
protect the setting of rq->active_balance to 0 with the rq lock in
active_load_balance_cpu_stop(). Therefore, we can move the action of
clearing rq->active_balance to the end of active_load_balance_cpu_stop().
The benefit of this approach is that the task load of dst_rq will change
due to the execution of attach_one_task(), which helps avoid prematurely
clearing rq->active_balance before attach_one_task(), thus preventing
unnecessary dispatch of duplicate active balance stop tasks.

Active balance stop task is triggered only when rq->active_balance flag
changes from 0 to 1, and there can be at most one consumer of active
balance stop task at any given time. Therefore, we should never see zero
rq->active_balance in active_load_balance_cpu_stop(), use WARN_ON_ONCE
instead.

Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
---
 kernel/sched/fair.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 11c104010b2e..20d03ceed9d7 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13769,8 +13769,7 @@ static int active_load_balance_cpu_stop(void *data)
 	if (!cpu_active(busiest_cpu) || !cpu_active(target_cpu))
 		goto out_unlock;
 
-	if (unlikely(!busiest_rq->active_balance))
-		goto out_unlock;
+	WARN_ON_ONCE(!busiest_rq->active_balance);
 
 	/* Is there any task to move? */
 	if (busiest_rq->nr_running <= 1)
@@ -13815,13 +13814,13 @@ static int active_load_balance_cpu_stop(void *data)
 	}
 	rcu_read_unlock();
 out_unlock:
-	busiest_rq->active_balance = 0;
 	rq_unlock(busiest_rq, &rf);
 
 	if (p)
 		attach_one_task(target_rq, p);
 
 	local_irq_enable();
+	busiest_rq->active_balance = 0;
 
 	return 0;
 }
-- 
2.34.1

Re: [RFC PATCH RESEND 03/10] sched/fair: Clear active_balance at the end of active_load_balance_cpu_stop()

[K Prateek Nayak]

Hello Xin,

On 9/10/2026 9:59 AM, Xin Zhao wrote:
> The rq->active_balance flag is used to prevent multiple CPUs from
> simultaneously dispatching active balance stop tasks. Since there can only
> ever be one consumer of the stop task, it is not strictly necessary to
> protect the setting of rq->active_balance to 0 with the rq lock in
> active_load_balance_cpu_stop(). Therefore, we can move the action of
> clearing rq->active_balance to the end of active_load_balance_cpu_stop().
> The benefit of this approach is that the task load of dst_rq will change
> due to the execution of attach_one_task(), which helps avoid prematurely
> clearing rq->active_balance before attach_one_task(), thus preventing
> unnecessary dispatch of duplicate active balance stop tasks.

Aren't we moving tasks *out* of the busiest CPU where the stopper is
scheduled?

As soon as we do detach_one_task() within the rq_lock, the load is
reflected correctly. There is no need to wait until we attach task to
a remote target_rq. TASK_ON_RQ_MIGRATING will immediately detach its
PELT signal from busiest.

That last statement seems to be inaccurate.

> 
> Active balance stop task is triggered only when rq->active_balance flag
> changes from 0 to 1, and there can be at most one consumer of active
> balance stop task at any given time. Therefore, we should never see zero
> rq->active_balance in active_load_balance_cpu_stop(), use WARN_ON_ONCE
> instead.
> 
> Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
> ---
>  kernel/sched/fair.c | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 11c104010b2e..20d03ceed9d7 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13769,8 +13769,7 @@ static int active_load_balance_cpu_stop(void *data)
>  	if (!cpu_active(busiest_cpu) || !cpu_active(target_cpu))
>  		goto out_unlock;
>  
> -	if (unlikely(!busiest_rq->active_balance))
> -		goto out_unlock;
> +	WARN_ON_ONCE(!busiest_rq->active_balance);
>  
>  	/* Is there any task to move? */
>  	if (busiest_rq->nr_running <= 1)
> @@ -13815,13 +13814,13 @@ static int active_load_balance_cpu_stop(void *data)
>  	}
>  	rcu_read_unlock();
>  out_unlock:
> -	busiest_rq->active_balance = 0;
>  	rq_unlock(busiest_rq, &rf);
>  
>  	if (p)
>  		attach_one_task(target_rq, p);
>  
>  	local_irq_enable();

As soon as we enable IRQs, a timer for a remote tick may go off which
may want to push the task from this CPU again but it sees
->active_balance still set.

At the very least, I think this should be done before IRQs are enabled
but I'm not convinced by the justification for this in the commit
message.

> +	busiest_rq->active_balance = 0;
>  
>  	return 0;
>  }

-- 
Thanks and Regards,
Prateek

Re: [RFC PATCH RESEND 03/10] sched/fair: Clear active_balance at the end of active_load_balance_cpu_stop()

[Xin Zhao]

On Thu, 10 Sep 2026 13:39:10 +0530 K Prateek Nayak <kprateek.nayak@amd.com> wrote:

> > The rq->active_balance flag is used to prevent multiple CPUs from
> > simultaneously dispatching active balance stop tasks. Since there can only
> > ever be one consumer of the stop task, it is not strictly necessary to
> > protect the setting of rq->active_balance to 0 with the rq lock in
> > active_load_balance_cpu_stop(). Therefore, we can move the action of
> > clearing rq->active_balance to the end of active_load_balance_cpu_stop().
> > The benefit of this approach is that the task load of dst_rq will change
> > due to the execution of attach_one_task(), which helps avoid prematurely
> > clearing rq->active_balance before attach_one_task(), thus preventing
> > unnecessary dispatch of duplicate active balance stop tasks.
> 
> Aren't we moving tasks *out* of the busiest CPU where the stopper is
> scheduled?
> 
> As soon as we do detach_one_task() within the rq_lock, the load is
> reflected correctly. There is no need to wait until we attach task to
> a remote target_rq. TASK_ON_RQ_MIGRATING will immediately detach its
> PELT signal from busiest.
> 
> That last statement seems to be inaccurate.

The load on the source CPU is indeed updated after detach_one_task, but the
load on the destination CPU is not.

Suppose that while sched_balance_rq() is being executed on CPU A, an active
balance triggered with the source being CPU B and the destination being CPU A.

Meanwhile, CPU C might also be executing sched_balance_rq() and changing the
destination CPU to CPU A (referring to the code related to

env.dst_rq =_rq(env.new_dst_cpu);). 

If the active_balance flag is cleared to 0 before attach_one_task(), then CPU C
may trigger a duplicate active balance from source CPU B to destination CPU A.
If I'm not mistaken, I may need to add this explanation to the commit log.


> As soon as we enable IRQs, a timer for a remote tick may go off which
> may want to push the task from this CPU again but it sees
> ->active_balance still set.
> 
> At the very least, I think this should be done before IRQs are enabled
> but I'm not convinced by the justification for this in the commit
> message.
> 
> > +	busiest_rq->active_balance = 0;
> >  
> >  	return 0;
> >  }

I understand. Indeed, the action of clearing the active_balance flag after
enabling interrupts is completely unnecessary; it should be placed before
enabling interrupts.

--
Xin Zhao

[RFC PATCH RESEND 04/10] sched/fair: Add LB_PROMOTE feature to enhance real-time performance of fair tasks

[Xin Zhao]

Embedded platforms commonly use CONFIG_HZ_250, and testing has revealed
that there are numerous instances of unreasonable CPU idle events on such
platforms. Unreasonable CPU idle refers to situations where the CPU enters
an idle state for a duration of time (t), while there are tasks that can
run on the idle CPU and are not limited by cgroup constraints, yet these
tasks remain unscheduled for a duration greater than (t), t > 2.5 ms.

Testing has shown that over 95% of these events last less than 4 ms, but
there are still some instances of longer durations between 4-5ms, even
occasionally between 5-10 ms. For a real-time system, scheduling delays
greater than 4 ms can lead to performance spikes.

Enabling this option can effectively reduce the occurrence of unreasonable
CPU idle events on low HZ systems like CONFIG_HZ_250, and completely
eliminate events exceeding 4 ms. Note that the feature only affects fair
tasks.

Note that enabling this feature will increase sys%, as it uses CPU time
that would have been idle to expedite the scheduling of tasks. There will
also be some CPU overhead involved in searching for suitable tasks.

This feature has been split into several smaller patches, which will be
elaborated on one by one later. Below are some test data:

Test one compares the number of unreasonable CPU idle events and their
distribution when this feature is enabled versus when it is not, under
the same fillback scenario. The test duration was 60 seconds.

      LB_PROMOTE(on/off) 2.5-3ms 3-4ms  4ms+
index 0               on       0     0     0
index 1              off       4    13     1
index 2               on       0     0     0
index 3              off       6     3     0
index 4               on       0     0     0
index 5              off       1     1     0

Test two compares the performance of the system with and without the
feature enabled, based on the same fillback scenario. Each test lasts for
25 minutes, and a total of 15 comparative tests were conducted. The
results include a comparison of the maximum and median values of
end-to-end latency and sys%.

LB_PROMOTE(on/off)                    on        off
end-to-end latency(max)              172        180
end-to-end latency(median of avg)    166     167.68
sys%(max)                           9.68       9.35
sys%(median of avg)                 8.81       8.55

Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
---
 kernel/sched/features.h | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index 8f0dee8fc475..4916a4b89ab3 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -142,3 +142,27 @@ SCHED_FEAT(LATENCY_WARN, false)
  */
 SCHED_FEAT(NI_RANDOM, true)
 SCHED_FEAT(NI_RATE, true)
+
+/*
+ * Embedded platforms commonly use CONFIG_HZ_250, and testing has revealed that
+ * there are numerous instances of unreasonable CPU idle events on such
+ * platforms. Unreasonable CPU idle refers to situations where the CPU enters
+ * an idle state for a duration of time (t), while there are tasks that can run
+ * on the idle CPU and are not limited by cgroup constraints, yet these tasks
+ * remain unscheduled for a duration greater than (t), t > 2.5 ms.
+ *
+ * Testing has shown that over 95% of these events last less than 4 ms, but
+ * there are still some instances of longer durations between 4-5ms, even
+ * occasionally between 5-10 ms. For a real-time system, scheduling delays
+ * greater than 4 ms can lead to performance spikes.
+ *
+ * Enabling this option can effectively reduce the occurrence of unreasonable
+ * CPU idle events on low HZ systems like CONFIG_HZ_250, and completely
+ * eliminate events exceeding 4 ms. Note that the feature only affects fair
+ * tasks.
+ *
+ * Note that enabling this feature will increase sys%, as it uses CPU time that
+ * would have been idle to expedite the scheduling of tasks. There will also be
+ * some CPU overhead involved in searching for suitable tasks.
+ */
+SCHED_FEAT(LB_PROMOTE, false)
-- 
2.34.1

Re: [RFC PATCH RESEND 04/10] sched/fair: Add LB_PROMOTE feature to enhance real-time performance of fair tasks

[Vincent Guittot]

On Thu, 10 Sept 2026 at 06:30, Xin Zhao <jackzxcui1989@163.com> wrote:
>
> Embedded platforms commonly use CONFIG_HZ_250, and testing has revealed
> that there are numerous instances of unreasonable CPU idle events on such
> platforms. Unreasonable CPU idle refers to situations where the CPU enters
> an idle state for a duration of time (t), while there are tasks that can
> run on the idle CPU and are not limited by cgroup constraints, yet these
> tasks remain unscheduled for a duration greater than (t), t > 2.5 ms.

Which looks reasonable with a 4ms tick

The ILB fixes this at next tick and we have a similar version for slow
path wake up

Also you might want to look at the newly idle load balance path

>
> Testing has shown that over 95% of these events last less than 4 ms, but
> there are still some instances of longer durations between 4-5ms, even
> occasionally between 5-10 ms. For a real-time system, scheduling delays
> greater than 4 ms can lead to performance spikes.

For a real-time system, use the real-time scheduler but you might want
to refer interactive systems

>
> Enabling this option can effectively reduce the occurrence of unreasonable
> CPU idle events on low HZ systems like CONFIG_HZ_250, and completely
> eliminate events exceeding 4 ms. Note that the feature only affects fair
> tasks.
>
> Note that enabling this feature will increase sys%, as it uses CPU time
> that would have been idle to expedite the scheduling of tasks. There will
> also be some CPU overhead involved in searching for suitable tasks.
>
> This feature has been split into several smaller patches, which will be
> elaborated on one by one later. Below are some test data:
>
> Test one compares the number of unreasonable CPU idle events and their
> distribution when this feature is enabled versus when it is not, under
> the same fillback scenario. The test duration was 60 seconds.
>
>       LB_PROMOTE(on/off) 2.5-3ms 3-4ms  4ms+
> index 0               on       0     0     0
> index 1              off       4    13     1
> index 2               on       0     0     0
> index 3              off       6     3     0
> index 4               on       0     0     0
> index 5              off       1     1     0
>
> Test two compares the performance of the system with and without the
> feature enabled, based on the same fillback scenario. Each test lasts for
> 25 minutes, and a total of 15 comparative tests were conducted. The
> results include a comparison of the maximum and median values of
> end-to-end latency and sys%.
>
> LB_PROMOTE(on/off)                    on        off
> end-to-end latency(max)              172        180
> end-to-end latency(median of avg)    166     167.68
> sys%(max)                           9.68       9.35
> sys%(median of avg)                 8.81       8.55
>
> Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
> ---
>  kernel/sched/features.h | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
>
> diff --git a/kernel/sched/features.h b/kernel/sched/features.h
> index 8f0dee8fc475..4916a4b89ab3 100644
> --- a/kernel/sched/features.h
> +++ b/kernel/sched/features.h
> @@ -142,3 +142,27 @@ SCHED_FEAT(LATENCY_WARN, false)
>   */
>  SCHED_FEAT(NI_RANDOM, true)
>  SCHED_FEAT(NI_RATE, true)
> +
> +/*
> + * Embedded platforms commonly use CONFIG_HZ_250, and testing has revealed that
> + * there are numerous instances of unreasonable CPU idle events on such
> + * platforms. Unreasonable CPU idle refers to situations where the CPU enters
> + * an idle state for a duration of time (t), while there are tasks that can run
> + * on the idle CPU and are not limited by cgroup constraints, yet these tasks
> + * remain unscheduled for a duration greater than (t), t > 2.5 ms.
> + *
> + * Testing has shown that over 95% of these events last less than 4 ms, but
> + * there are still some instances of longer durations between 4-5ms, even
> + * occasionally between 5-10 ms. For a real-time system, scheduling delays
> + * greater than 4 ms can lead to performance spikes.
> + *
> + * Enabling this option can effectively reduce the occurrence of unreasonable
> + * CPU idle events on low HZ systems like CONFIG_HZ_250, and completely
> + * eliminate events exceeding 4 ms. Note that the feature only affects fair
> + * tasks.
> + *
> + * Note that enabling this feature will increase sys%, as it uses CPU time that
> + * would have been idle to expedite the scheduling of tasks. There will also be
> + * some CPU overhead involved in searching for suitable tasks.
> + */
> +SCHED_FEAT(LB_PROMOTE, false)
> --
> 2.34.1
>

Re: [RFC PATCH RESEND 04/10] sched/fair: Add LB_PROMOTE feature to enhance real-time performance of fair tasks

[Xin Zhao]

On Fri, 11 Sep 2026 14:32:25 +0200 Vincent Guittot <vincent.guittot@linaro.org> wrote:

> > Embedded platforms commonly use CONFIG_HZ_250, and testing has revealed
> > that there are numerous instances of unreasonable CPU idle events on such
> > platforms. Unreasonable CPU idle refers to situations where the CPU enters
> > an idle state for a duration of time (t), while there are tasks that can
> > run on the idle CPU and are not limited by cgroup constraints, yet these
> > tasks remain unscheduled for a duration greater than (t), t > 2.5 ms.
> 
> Which looks reasonable with a 4ms tick
> 
> The ILB fixes this at next tick and we have a similar version for slow
> path wake up
> 
> Also you might want to look at the newly idle load balance path

I have already thoroughly reviewed the logic of ILB before working on this series
of patches. Indeed, according to the current implementation, the scheduling delays
in a 250Hz can be explained from the code perspective.

However, when it comes to using Linux for real-time systems in embedded environments,
I wonder why we can't do better in this area. As far as I know, there are an
increasing number of players Linux for real-time systems. Sacrificing 0.3% of average
system consumption in exchange for eliminating scheduling delay cases of 4ms or more,
as well as significantly reducing cases with scheduling delays between 2.5ms and ms,
should benefit everyone using Linux for real-time embedded systems. I believe it is
necessary to consider merging this feature into the mainline for embedded scenarios.

Of course, there may be better implementations. For instance, I initially thought
about replacing some of ILB logic with hrtimer, which could trigger checks earlier
and potentially yield better results. However, the existing logic in load balancing
certainly has many areas to improve real-time performance. I feel that these
improvements are not mutually exclusive; at least we enhance the real-time performance
under the scenario where the LB_PROMOTE feature is enabled.

In addition to modifying the wakeup selection logic, I have also made some changes to
the load balancing logic, as seen in the subsequent patches 7, 8, 10, particularly in
patch 10.


--
Xin Zhao

[RFC PATCH RESEND 05/10] sched/fair: Introduce select_task_rq_fair_thin() to select rq when LB_PROMOTE

[Xin Zhao]

The logic of select_task_rq_fair() is relatively complex, and on typical
embedded systems, the number of CPUs in sd_llc domain is often limited to
a maximum of 4, and sometimes only 2. This makes the complex logic of
select_task_rq_fair() seem less necessary. Additionally, the update logic
for the nr_idle_scan value that select_task_rq_fair() relies on is also
quite time-consuming.

Moreover, embedded systems require better real-time performance, and with
fewer CPUs available, it becomes necessary to bind certain tasks across
the sd_llc range. The default enabled feature, SD_WAKE_AFFINE, causes
select_task_rq_fair() to take the fast path, often overlooking some idle
CPUs across sd_llc domain, leading to increased scheduling latency.

To address this, we introduce the select_task_rq_fair_thin() function,
which serves as a streamlined version of select_task_rq_fair(). It can
quickly perform CPU selection while also considering the real-time
requirements of embedded systems. select_task_rq_fair_thin() retains the
priority selection logic for prev_cpu and recent_used_cpu, and it will
prioritize CPUs within the sd_llc. If there are no idle CPUs in the
sd_llc, it will then look for other available CPUs to run.

When the LB_PROMOTE feature is enabled, select_task_rq_fair_thin() will
replace the original select_task_rq_fair(), and the update logic for
nr_idle_scan that select_task_rq_fair() relies on will no longer need to
be executed.

Testing has shown that in our system with 18 CPUs running at 2.1GHz, where
the first three sd_llc domains each contains 4 CPUs and the last sd_llc
contains 2 CPUs, under same fillback scenario, select_task_rq_fair_thin()
executes 25% faster than the original select_task_rq_fair(). It saves 22ms
over a 10-second period, with this optimization accounting for 0.174% of
total system time. Additionally, we measured the execution time of
update_idle_cpu_scan, which took 0.5ms over the same 10-second period. If
we use select_task_rq_fair_thin() instead, this time can be eliminated,
accounting for 0.04% of total system time. Therefore, the overall
optimization contributes to a reduction of 0.214% of total system time.

Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
---
 kernel/sched/fair.c | 52 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 51 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 20d03ceed9d7..f10e709921fd 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9661,6 +9661,51 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
 	return target;
 }
 
+/*
+ * A streamlined version of select_task_rq_fair().
+ * It runs faster than select_task_rq_fair, especially when there are not
+ * many CPUs. It will prioritize selecting an idle CPU in the following order:
+ * 1. prev_cpu
+ * 2. recent_used_cpu
+ * 3. cpu belongs to intersection of sd_llc and cpus_ptr
+ * 4. cpu belongs to cpus_ptr but not belongs to sd_llc
+ * If there is no idle CPU in cpus_ptr, it will select prev_cpu.
+ */
+static int select_task_rq_fair_thin(struct task_struct *p, int prev_cpu, int wake_flags)
+{
+	int recent_used_cpu, target, cpu, start = nr_cpu_ids;
+	struct sched_domain *sd;
+
+	if (likely(available_idle_cpu(prev_cpu)))
+		return prev_cpu;
+
+	recent_used_cpu = p->recent_used_cpu;
+	p->recent_used_cpu = prev_cpu;
+	if (recent_used_cpu != prev_cpu && available_idle_cpu(recent_used_cpu))
+		return recent_used_cpu;
+
+	target = prev_cpu;
+	rcu_read_lock();
+
+	sd = rcu_dereference(per_cpu(sd_llc, target));
+	if (sd)
+		start = cpumask_first_and(sched_domain_span(sd), p->cpus_ptr);
+	if (start >= nr_cpu_ids)
+		start = cpumask_first(p->cpus_ptr);
+
+	for_each_cpu_wrap(cpu, p->cpus_ptr, start) {
+		if (available_idle_cpu(cpu)) {
+			target = cpu;
+			goto unlock;
+		}
+	}
+
+unlock:
+	rcu_read_unlock();
+
+	return target;
+}
+
 /*
  * select_task_rq_fair: Select target runqueue for the waking task in domains
  * that have the relevant SD flag set. In practice, this is SD_BALANCE_WAKE,
@@ -9682,6 +9727,9 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	/* SD_flags and WF_flags share the first nibble */
 	int sd_flag = wake_flags & 0xF;
 
+	if (sched_feat(LB_PROMOTE))
+		return select_task_rq_fair_thin(p, prev_cpu, wake_flags);
+
 	/*
 	 * required for stable ->cpus_allowed
 	 */
@@ -12564,8 +12612,10 @@ static void update_idle_cpu_scan(struct lb_env *env,
 	 * So the write of this hint only occurs during periodic load
 	 * balancing, rather than CPU_NEWLY_IDLE, because the latter
 	 * can fire way more frequently than the former.
+	 * When LB_PROMOTE is enabled, select_task_rq_fair() is no longer
+	 * used, and there is no need to update nr_idle_scan.
 	 */
-	if (!sched_feat(SIS_UTIL) || env->idle == CPU_NEWLY_IDLE)
+	if (!sched_feat(SIS_UTIL) || env->idle == CPU_NEWLY_IDLE || sched_feat(LB_PROMOTE))
 		return;
 
 	sd_share = sd->shared;
-- 
2.34.1

Re: [RFC PATCH RESEND 05/10] sched/fair: Introduce select_task_rq_fair_thin() to select rq when LB_PROMOTE

[Vincent Guittot]

On Thu, 10 Sept 2026 at 06:30, Xin Zhao <jackzxcui1989@163.com> wrote:
>
> The logic of select_task_rq_fair() is relatively complex, and on typical
> embedded systems, the number of CPUs in sd_llc domain is often limited to
> a maximum of 4, and sometimes only 2. This makes the complex logic of
> select_task_rq_fair() seem less necessary. Additionally, the update logic
> for the nr_idle_scan value that select_task_rq_fair() relies on is also
> quite time-consuming.
>
> Moreover, embedded systems require better real-time performance, and with
> fewer CPUs available, it becomes necessary to bind certain tasks across
> the sd_llc range. The default enabled feature, SD_WAKE_AFFINE, causes
> select_task_rq_fair() to take the fast path, often overlooking some idle
> CPUs across sd_llc domain, leading to increased scheduling latency.
>
> To address this, we introduce the select_task_rq_fair_thin() function,
> which serves as a streamlined version of select_task_rq_fair(). It can
> quickly perform CPU selection while also considering the real-time
> requirements of embedded systems. select_task_rq_fair_thin() retains the
> priority selection logic for prev_cpu and recent_used_cpu, and it will
> prioritize CPUs within the sd_llc. If there are no idle CPUs in the
> sd_llc, it will then look for other available CPUs to run.
>
> When the LB_PROMOTE feature is enabled, select_task_rq_fair_thin() will
> replace the original select_task_rq_fair(), and the update logic for
> nr_idle_scan that select_task_rq_fair() relies on will no longer need to
> be executed.
>
> Testing has shown that in our system with 18 CPUs running at 2.1GHz, where
> the first three sd_llc domains each contains 4 CPUs and the last sd_llc
> contains 2 CPUs, under same fillback scenario, select_task_rq_fair_thin()
> executes 25% faster than the original select_task_rq_fair(). It saves 22ms
> over a 10-second period, with this optimization accounting for 0.174% of
> total system time. Additionally, we measured the execution time of
> update_idle_cpu_scan, which took 0.5ms over the same 10-second period. If
> we use select_task_rq_fair_thin() instead, this time can be eliminated,
> accounting for 0.04% of total system time. Therefore, the overall
> optimization contributes to a reduction of 0.214% of total system time.
>
> Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
> ---
>  kernel/sched/fair.c | 52 ++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 51 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 20d03ceed9d7..f10e709921fd 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9661,6 +9661,51 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
>         return target;
>  }
>
> +/*
> + * A streamlined version of select_task_rq_fair().
> + * It runs faster than select_task_rq_fair, especially when there are not
> + * many CPUs. It will prioritize selecting an idle CPU in the following order:
> + * 1. prev_cpu
> + * 2. recent_used_cpu
> + * 3. cpu belongs to intersection of sd_llc and cpus_ptr
> + * 4. cpu belongs to cpus_ptr but not belongs to sd_llc
> + * If there is no idle CPU in cpus_ptr, it will select prev_cpu.
> + */
> +static int select_task_rq_fair_thin(struct task_struct *p, int prev_cpu, int wake_flags)

We don't want yet another select idle cpu function.
The only reason select_idle_sibling() would not select an idle cpu in
your llc is nr_idle_scan which will abort the loop before checking all
CPUs. You should look at why nr_idle_scan doesn't keep all CPUs in
your small size use cases


> +{
> +       int recent_used_cpu, target, cpu, start = nr_cpu_ids;
> +       struct sched_domain *sd;
> +
> +       if (likely(available_idle_cpu(prev_cpu)))
> +               return prev_cpu;
> +
> +       recent_used_cpu = p->recent_used_cpu;
> +       p->recent_used_cpu = prev_cpu;
> +       if (recent_used_cpu != prev_cpu && available_idle_cpu(recent_used_cpu))
> +               return recent_used_cpu;
> +
> +       target = prev_cpu;
> +       rcu_read_lock();
> +
> +       sd = rcu_dereference(per_cpu(sd_llc, target));
> +       if (sd)
> +               start = cpumask_first_and(sched_domain_span(sd), p->cpus_ptr);
> +       if (start >= nr_cpu_ids)
> +               start = cpumask_first(p->cpus_ptr);
> +
> +       for_each_cpu_wrap(cpu, p->cpus_ptr, start) {
> +               if (available_idle_cpu(cpu)) {
> +                       target = cpu;
> +                       goto unlock;
> +               }
> +       }
> +
> +unlock:
> +       rcu_read_unlock();
> +
> +       return target;
> +}
> +
>  /*
>   * select_task_rq_fair: Select target runqueue for the waking task in domains
>   * that have the relevant SD flag set. In practice, this is SD_BALANCE_WAKE,
> @@ -9682,6 +9727,9 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>         /* SD_flags and WF_flags share the first nibble */
>         int sd_flag = wake_flags & 0xF;
>
> +       if (sched_feat(LB_PROMOTE))
> +               return select_task_rq_fair_thin(p, prev_cpu, wake_flags);
> +
>         /*
>          * required for stable ->cpus_allowed
>          */
> @@ -12564,8 +12612,10 @@ static void update_idle_cpu_scan(struct lb_env *env,
>          * So the write of this hint only occurs during periodic load
>          * balancing, rather than CPU_NEWLY_IDLE, because the latter
>          * can fire way more frequently than the former.
> +        * When LB_PROMOTE is enabled, select_task_rq_fair() is no longer
> +        * used, and there is no need to update nr_idle_scan.
>          */
> -       if (!sched_feat(SIS_UTIL) || env->idle == CPU_NEWLY_IDLE)
> +       if (!sched_feat(SIS_UTIL) || env->idle == CPU_NEWLY_IDLE || sched_feat(LB_PROMOTE))
>                 return;
>
>         sd_share = sd->shared;
> --
> 2.34.1
>

Re: [RFC PATCH RESEND 05/10] sched/fair: Introduce select_task_rq_fair_thin() to select rq when LB_PROMOTE

[Xin Zhao]

On Thu, 10 Sep 2026 10:19:13 +0200 Vincent Guittot <vincent.guittot@linaro.org> wrote:

> > +/*
> > + * A streamlined version of select_task_rq_fair().
> > + * It runs faster than select_task_rq_fair, especially when there are not
> > + * many CPUs. It will prioritize selecting an idle CPU in the following order:
> > + * 1. prev_cpu
> > + * 2. recent_used_cpu
> > + * 3. cpu belongs to intersection of sd_llc and cpus_ptr
> > + * 4. cpu belongs to cpus_ptr but not belongs to sd_llc
> > + * If there is no idle CPU in cpus_ptr, it will select prev_cpu.
> > + */
> > +static int select_task_rq_fair_thin(struct task_struct *p, int prev_cpu, int wake_flags)
> 
> We don't want yet another select idle cpu function.
> The only reason select_idle_sibling() would not select an idle cpu in
> your llc is nr_idle_scan which will abort the loop before checking all
> CPUs. You should look at why nr_idle_scan doesn't keep all CPUs in
> your small size use cases

Adding such a function to the mainline code is indeed not a good idea. What I
actually want is to prioritize selecting idle CPUs within the LLC first, and
then select idle CPUs outside the LLC. However, once the fast path taken, if
I’m not mistaken, it only selects CPUs within the LLC, which is not very
friendly for machines on embedded platforms, as it becomes difficult to bind
tasks without crossing LLC boundaries when the number of CPUs is small.

I’m considering I can modify it as follows:
want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr) && !sched_feat(LB_PROMOTE);
to force the slow path for core selection whenLB_PROMOTE` is enabled.

BTW,
Regarding nr_idle_scan, I have derived some data and the analysis is as follows:

If nr_idle_scan is 2, then: y must be at least 512, tmp must be <=512, and tmp
must be <= 512 * 10000 * 1024. This leads to tmp <= sqrt(512 * 10000 * 1024 / 117 / 117),
which is the square of 382,999.488, resulting in approximately 618.8695 after
taking the square root.

Multiplying this by 4 gives us 2476, so if the estimated utilization of CFS
tasks exceeds 60.54%,nr_idle_scan` must be less than 3 and becomes 2, which
may result in missing idle CPUs within the LLC.


--
Xin Zhao

Re: [RFC PATCH RESEND 05/10] sched/fair: Introduce select_task_rq_fair_thin() to select rq when LB_PROMOTE

[Vincent Guittot]

On Thu, 10 Sept 2026 at 16:39, Xin Zhao <jackzxcui1989@163.com> wrote:
>
> On Thu, 10 Sep 2026 10:19:13 +0200 Vincent Guittot <vincent.guittot@linaro.org> wrote:
>
> > > +/*
> > > + * A streamlined version of select_task_rq_fair().
> > > + * It runs faster than select_task_rq_fair, especially when there are not
> > > + * many CPUs. It will prioritize selecting an idle CPU in the following order:
> > > + * 1. prev_cpu
> > > + * 2. recent_used_cpu
> > > + * 3. cpu belongs to intersection of sd_llc and cpus_ptr
> > > + * 4. cpu belongs to cpus_ptr but not belongs to sd_llc
> > > + * If there is no idle CPU in cpus_ptr, it will select prev_cpu.
> > > + */
> > > +static int select_task_rq_fair_thin(struct task_struct *p, int prev_cpu, int wake_flags)
> >
> > We don't want yet another select idle cpu function.
> > The only reason select_idle_sibling() would not select an idle cpu in
> > your llc is nr_idle_scan which will abort the loop before checking all
> > CPUs. You should look at why nr_idle_scan doesn't keep all CPUs in
> > your small size use cases
>
> Adding such a function to the mainline code is indeed not a good idea. What I
> actually want is to prioritize selecting idle CPUs within the LLC first, and
> then select idle CPUs outside the LLC. However, once the fast path taken, if
> I’m not mistaken, it only selects CPUs within the LLC, which is not very
> friendly for machines on embedded platforms, as it becomes difficult to bind
> tasks without crossing LLC boundaries when the number of CPUs is small.

Ok, I didn't notice that you were looking at all CPUs, not only the
last LLC. Do you have more details about your cpu topology to share
with us?

we have sched_balance_find_dst_cpu() which looks wider but needs
SD_BALANCE_WAKE to be set in your sched domain topology

>
> I’m considering I can modify it as follows:
> want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr) && !sched_feat(LB_PROMOTE);
> to force the slow path for core selection whenLB_PROMOTE` is enabled.
>
> BTW,
> Regarding nr_idle_scan, I have derived some data and the analysis is as follows:
>
> If nr_idle_scan is 2, then: y must be at least 512, tmp must be <=512, and tmp
> must be <= 512 * 10000 * 1024. This leads to tmp <= sqrt(512 * 10000 * 1024 / 117 / 117),
> which is the square of 382,999.488, resulting in approximately 618.8695 after
> taking the square root.
>
> Multiplying this by 4 gives us 2476, so if the estimated utilization of CFS
> tasks exceeds 60.54%,nr_idle_scan` must be less than 3 and becomes 2, which
> may result in missing idle CPUs within the LLC.

nr_idle_scan has been design with large llc in mind where scanning
hundreds of CPU is significant but when you have only few cores, we
could be more relax in the number of cpu to scan

>
>
> --
> Xin Zhao
>

Re: [RFC PATCH RESEND 05/10] sched/fair: Introduce select_task_rq_fair_thin() to select rq when LB_PROMOTE

[Xin Zhao]

On Thu, 10 Sep 2026 17:31:43 +0200 Vincent Guittot <vincent.guittot@linaro.org> wrote:

> > Adding such a function to the mainline code is indeed not a good idea. What I
> > actually want is to prioritize selecting idle CPUs within the LLC first, and
> > then select idle CPUs outside the LLC. However, once the fast path taken, if
> > I’m not mistaken, it only selects CPUs within the LLC, which is not very
> > friendly for machines on embedded platforms, as it becomes difficult to bind
> > tasks without crossing LLC boundaries when the number of CPUs is small.
> 
> Ok, I didn't notice that you were looking at all CPUs, not only the
> last LLC. Do you have more details about your cpu topology to share
> with us?
> 
> we have sched_balance_find_dst_cpu() which looks wider but needs
> SD_BALANCE_WAKE to be set in your sched domain topology

Our system has a two-level scheduling hierarchy with 18 CPUs and 4 clusters:
0-1, 2-5, 6-9, 10-13, and 14-17. Each cluster is LLC (Last Level Cache), and
there are no finer levels of hierarchy.

I did overlook the detail regarding the SD_BALANCE_WAKE flag; our project did
not change the default state of the SD_BALANCE_WAKE flag, which is set to 0
Moreover, as far as I know, most embedded Linux projects have real-time
requirements and generally do not modify this SD_BALANCE_WAKE flag.

Given this situation, is it necessary to create a new branch for such a "_thin"
select_task_rq_fair() variant when enabling LB_PROMOTE?


> > BTW,
> > Regarding nr_idle_scan, I have derived some data and the analysis is as follows:
> >
> > If nr_idle_scan is 2, then: y must be at least 512, tmp must be <=512, and tmp
> > must be <= 512 * 10000 * 1024. This leads to tmp <= sqrt(512 * 10000 * 1024 / 117 / 117),
> > which is the square of 382,999.488, resulting in approximately 618.8695 after
> > taking the square root.
> >
> > Multiplying this by 4 gives us 2476, so if the estimated utilization of CFS
> > tasks exceeds 60.54%,nr_idle_scan` must be less than 3 and becomes 2, which
> > may result in missing idle CPUs within the LLC.
> 
> nr_idle_scan has been design with large llc in mind where scanning
> hundreds of CPU is significant but when you have only few cores, we
> could be more relax in the number of cpu to scan

I can understand that the logic related to nr_idle_scan is very necessary when
there are a large number of CPUs. I realize that many x86 machines, even personal
PCs, have a considerable number of CPUs in their LLC.


--
Xin Zhao

Re: [RFC PATCH RESEND 05/10] sched/fair: Introduce select_task_rq_fair_thin() to select rq when LB_PROMOTE

[Vincent Guittot]

On Thu, 10 Sept 2026 at 17:57, Xin Zhao <jackzxcui1989@163.com> wrote:
>
> On Thu, 10 Sep 2026 17:31:43 +0200 Vincent Guittot <vincent.guittot@linaro.org> wrote:
>
> > > Adding such a function to the mainline code is indeed not a good idea. What I
> > > actually want is to prioritize selecting idle CPUs within the LLC first, and
> > > then select idle CPUs outside the LLC. However, once the fast path taken, if
> > > I’m not mistaken, it only selects CPUs within the LLC, which is not very
> > > friendly for machines on embedded platforms, as it becomes difficult to bind
> > > tasks without crossing LLC boundaries when the number of CPUs is small.
> >
> > Ok, I didn't notice that you were looking at all CPUs, not only the
> > last LLC. Do you have more details about your cpu topology to share
> > with us?
> >
> > we have sched_balance_find_dst_cpu() which looks wider but needs
> > SD_BALANCE_WAKE to be set in your sched domain topology
>
> Our system has a two-level scheduling hierarchy with 18 CPUs and 4 clusters:
> 0-1, 2-5, 6-9, 10-13, and 14-17. Each cluster is LLC (Last Level Cache), and
> there are no finer levels of hierarchy.
>
> I did overlook the detail regarding the SD_BALANCE_WAKE flag; our project did
> not change the default state of the SD_BALANCE_WAKE flag, which is set to 0
> Moreover, as far as I know, most embedded Linux projects have real-time
> requirements and generally do not modify this SD_BALANCE_WAKE flag

But if you have real-time requirements, why don't you use the
real-time scheduler ?

>
> Given this situation, is it necessary to create a new branch for such a "_thin"
> select_task_rq_fair() variant when enabling LB_PROMOTE?

At now, I don't see a need for a new branch nor a LB_PROMOTE. You
failed to describe your problem and came up with thos
platform-specific solution instead of fixing current code. Putting it
behind LB_PROMOTE, doesn't make it better.


>
>
> > > BTW,
> > > Regarding nr_idle_scan, I have derived some data and the analysis is as follows:
> > >
> > > If nr_idle_scan is 2, then: y must be at least 512, tmp must be <=512, and tmp
> > > must be <= 512 * 10000 * 1024. This leads to tmp <= sqrt(512 * 10000 * 1024 / 117 / 117),
> > > which is the square of 382,999.488, resulting in approximately 618.8695 after
> > > taking the square root.
> > >
> > > Multiplying this by 4 gives us 2476, so if the estimated utilization of CFS
> > > tasks exceeds 60.54%,nr_idle_scan` must be less than 3 and becomes 2, which
> > > may result in missing idle CPUs within the LLC.
> >
> > nr_idle_scan has been design with large llc in mind where scanning
> > hundreds of CPU is significant but when you have only few cores, we
> > could be more relax in the number of cpu to scan
>
> I can understand that the logic related to nr_idle_scan is very necessary when
> there are a large number of CPUs. I realize that many x86 machines, even personal
> PCs, have a considerable number of CPUs in their LLC.
>
>
> --
> Xin Zhao
>

Re: [RFC PATCH RESEND 05/10] sched/fair: Introduce select_task_rq_fair_thin() to select rq when LB_PROMOTE

[Xin Zhao]

On Fri, 11 Sep 2026 14:27:04 +0200 Vincent Guittot <vincent.guittot@linaro.org> wrote:

> > I did overlook the detail regarding the SD_BALANCE_WAKE flag; our project did
> > not change the default state of the SD_BALANCE_WAKE flag, which is set to 0
> > Moreover, as far as I know, most embedded Linux projects have real-time
> > requirements and generally do not modify this SD_BALANCE_WAKE flag
> 
> But if you have real-time requirements, why don't you use the
> real-time scheduler ?

Due to fair scheduling, especially in a 250Hz system, we often experience significant
scheduling delays. Therefore, we have already converted many critical tasks to real-time
in our projecct. Currently, there are already a considerable number of real-time tasks
running on our system. The following is the data on our system:

CPU ID | Total Tasks | Real-time Tasks
------------------------------------------
0 | 369          | 197
1 | 293          | 132
2 | 166          | 53
3 | 103          | 27
4 | 89          | 26
5 | 63          | 29
6 | 173          | 61
7 | 215          | 39
8 | 181          | 53
9 | 77          | 21
10 | 90          | 47
11 | 119          | 63
12 | 55          | 23
13 | 50          | 27
14 | 85          | 33
15 | 143          | 50
16 | 156          | 40
17 | 159          | 44


The RT tasks/total tasks percent is already very high. It is no longer feasible to expand
the scope of real-time tasks, including tasks like kworker, ksoftirqd, and kswap, which
are not suitable for real-time scheduling, right? In particular, within kworker and
ksoftirqd, there are still many scenarios where insufficient scheduling timeliness leads
to increased processing delays in the entire pipeline.

The LB_PROM feature and its associated series of patches are specifically designed to
address this issue. One of the most significant core changes is patch 10; you may want to
take a look at it first. The remaining patches (patch 5, this patch, patch7, patch 8, and
patch 9) all contribute to improving fair scheduling delays.


> > Given this situation, is it necessary to create a new branch for such a "_thin"
> > select_task_rq_fair() variant when enabling LB_PROMOTE?
> 
> At now, I don't see a need for a new branch nor a LB_PROMOTE. You
> failed to describe your problem and came up with thos
> platform-specific solution instead of fixing current code. Putting it
> behind LB_PROMOTE, doesn't make it better.

I took a closer look at the implementation of sched_balance_find_dst_cpu and believe that
this function already meets the requirements. I'm wondering if it would be possible to add
the SD_BALANCE_WAKE flag to the scheduling domains (sd) of each CPU when the LB_PROMOTE
feature is enabled, and then revert it back to 0 after the LB_PROMOTE feature is disabled.
Would that be feasible?


--
Xin Zhao

Re: [RFC PATCH RESEND 05/10] sched/fair: Introduce select_task_rq_fair_thin() to select rq when LB_PROMOTE

[Kayra Cizmeci]

Hello Xin,

> Testing has shown that in our system with 18 CPUs running at 2.1GHz, where
> the first three sd_llc domains each contains 4 CPUs and the last sd_llc
> contains 2 CPUs, under same fillback scenario, select_task_rq_fair_thin()
> executes 25% faster than the original select_task_rq_fair(). It saves 22ms
> over a 10-second period, with this optimization accounting for 0.174% of
> total system time. Additionally, we measured the execution time of
> update_idle_cpu_scan, which took 0.5ms over the same 10-second period. If
> we use select_task_rq_fair_thin() instead, this time can be eliminated,
> accounting for 0.04% of total system time. Therefore, the overall
> optimization contributes to a reduction of 0.214% of total system time.

Okay. Can you specify which tests you ran or what you used? If you can?

Thanks,
Kayra

Re: [RFC PATCH RESEND 05/10] sched/fair: Introduce select_task_rq_fair_thin() to select rq when LB_PROMOTE

[Xin Zhao]

On Thu, 10 Sep 2026 15:00:29 +0300 Kayra Cizmeci <kayracizmeci@gmail.com> wrote:

> > Testing has shown that in our system with 18 CPUs running at 2.1GHz, where
> > the first three sd_llc domains each contains 4 CPUs and the last sd_llc
> > contains 2 CPUs, under same fillback scenario, select_task_rq_fair_thin()
> > executes 25% faster than the original select_task_rq_fair(). It saves 22ms
> > over a 10-second period, with this optimization accounting for 0.174% of
> > total system time. Additionally, we measured the execution time of
> > update_idle_cpu_scan, which took 0.5ms over the same 10-second period. If
> > we use select_task_rq_fair_thin() instead, this time can be eliminated,
> > accounting for 0.04% of total system time. Therefore, the overall
> > optimization contributes to a reduction of 0.214% of total system time.
> 
> Okay. Can you specify which tests you ran or what you used? If you can?

I used a relatively crude and straightforward method, which involves
subtracting the time at the beginning and end of the select_task_rq_fair()
function and then accumulating the per-CPU time. I retrieve this accumulated
value a kernel module (ko). The detailed script is as follows:

Below is the test script for comparing the effect of select_task_rq_fair()
with and without the patch:

#!/bin/bash

insmod testselecttask. patch=0 on=1
sleep 10
insmod testselecttask.ko patch=0 on=0
cat testselecttask_patch_0.txt

sleep 1
insmod testselecttask.ko patch=1 on=1sleep 10
insmod testselecttask.ko patch=1 on=0
cat testselecttask_patch_1.txt

Test results:

root@hobot:/map/zhaoxin# ./smalltest.sh
ins: ERROR: could not insert module testselecttask.ko: Invalid parameters
insmod: ERROR: could not insert module testselecttask.ko: Invalid parameters
zhaoxin: enable[0] count[179226] timens[09325] avg[460]
insmod: ERROR: could not insert module testselecttask.ko: Invalid parameters
insmod: ERROR: could not insert module testselecttask.ko: Invalid parameters
zhaoxin: enable[1 count[180836] timens[59444800] avg[328]

From the test results, combined with the following information:

Currently, the machine's overall sys time is about 7%, multiplied by 18
cores, which results approximately 126% for a multi-core CPU. The test
duration is 10 seconds, and within that time, 22 ms is saved, which
translates to 2.2 ms per second, equating to 0.0022 for multi, or 0.22%.
The time taken by this function is generally linked to the total sys time,
and the optimized portion accounts for about 0.174% of the total sys time.

Below are the test results for the execution time ofupdate_idle_cpu_scan`:

root@hobot:/map/zhaoxin# ./smalltest.sh
insmod: ERROR: could not insert module testselecttask.ko: Invalid parameters
insmod: ERROR: could not insert moduleselecttask.ko: Invalid parameters
zhaoxin: enable[0] count[179226] timens[82509325] avg[460]
insmod: ERROR: could not insert module testselecttask.ko: Invalid parameters
mod: ERROR: could not insert module testselecttask.ko: Invalid parameters
zhaoxin: enable[1] count[180836] timens[59444800] avg[328]

From the test results, combined with following information:

Currently, the machine's overall sys time is about 7%, multiplied by 18
cores, which results in approximately 126% for a multi-core CPU. The test
duration is 10 seconds, and within that time, 22 is saved, which translates
to 2.2 ms per second, equating to 0.0022 for multi-core, or 0.22%. The time
taken by this function is generally linked to the total sys time, and the
optimized accounts for about 0.174% of the total sys time.

--
Xin Zhao

[RFC PATCH RESEND 06/10] sched/fair: Modify active_load_balance_cpu_stop() to accommodate more scenarios

[Xin Zhao]

Modify active_load_balance_cpu_stop() to make it more generalized, no
longer limited to the context of finding the busiest rq in the
sched_balance_rq() scenario to migrate to the dst rq. Change the variables
that start with 'busiest' to start with 'src'.

Additionally, adjust the reference to the active_balance variable from
busiest_rq->active_balance to this_rq()->active_balance. With these
changes, the CPU where the stop task is located can serve as either a dst
CPU or a src CPU, laying the groundwork for the upcoming preempt active
balance feature.

Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
---
 kernel/sched/fair.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index f10e709921fd..9ffd01717599 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13796,33 +13796,33 @@ update_next_balance(struct sched_domain *sd, unsigned long *next_balance)
 
 /*
  * active_load_balance_cpu_stop is run by the CPU stopper. It pushes
- * running tasks off the busiest CPU onto idle CPUs. It requires at
+ * one running task off the src CPU onto dst CPU. It requires at
  * least 1 task to be running on each physical CPU where possible, and
  * avoids physical / logical imbalances.
  */
 static int active_load_balance_cpu_stop(void *data)
 {
-	struct rq *busiest_rq = data;
-	int busiest_cpu = cpu_of(busiest_rq);
-	int target_cpu = busiest_rq->push_cpu;
+	struct rq *src_rq = data;
+	int src_cpu = cpu_of(src_rq);
+	int target_cpu = this_rq()->push_cpu;
 	struct rq *target_rq = cpu_rq(target_cpu);
 	struct sched_domain *sd;
 	struct task_struct *p = NULL;
 	struct rq_flags rf;
 
-	rq_lock_irq(busiest_rq, &rf);
+	rq_lock_irq(src_rq, &rf);
 	/*
 	 * Between queueing the stop-work and running it is a hole in which
 	 * CPUs can become inactive. We should not move tasks from or to
 	 * inactive CPUs.
 	 */
-	if (!cpu_active(busiest_cpu) || !cpu_active(target_cpu))
+	if (!cpu_active(src_cpu) || !cpu_active(target_cpu))
 		goto out_unlock;
 
-	WARN_ON_ONCE(!busiest_rq->active_balance);
+	WARN_ON_ONCE(!this_rq()->active_balance);
 
 	/* Is there any task to move? */
-	if (busiest_rq->nr_running <= 1)
+	if (src_rq->nr_running <= 1)
 		goto out_unlock;
 
 	/*
@@ -13830,12 +13830,12 @@ static int active_load_balance_cpu_stop(void *data)
 	 * we need to fix it. Originally reported by
 	 * Bjorn Helgaas on a 128-CPU setup.
 	 */
-	WARN_ON_ONCE(busiest_rq == target_rq);
+	WARN_ON_ONCE(src_rq == target_rq);
 
 	/* Search for an sd spanning us and the target CPU. */
 	rcu_read_lock();
 	for_each_domain(target_cpu, sd) {
-		if (cpumask_test_cpu(busiest_cpu, sched_domain_span(sd)))
+		if (cpumask_test_cpu(src_cpu, sched_domain_span(sd)))
 			break;
 	}
 
@@ -13844,14 +13844,14 @@ static int active_load_balance_cpu_stop(void *data)
 			.sd		= sd,
 			.dst_cpu	= target_cpu,
 			.dst_rq		= target_rq,
-			.src_cpu	= busiest_rq->cpu,
-			.src_rq		= busiest_rq,
+			.src_cpu	= src_rq->cpu,
+			.src_rq		= src_rq,
 			.idle		= CPU_IDLE,
 			.flags		= LBF_ACTIVE_LB,
 		};
 
 		schedstat_inc(sd->alb_count);
-		update_rq_clock(busiest_rq);
+		update_rq_clock(src_rq);
 
 		p = detach_one_task(&env);
 		if (p) {
@@ -13864,13 +13864,13 @@ static int active_load_balance_cpu_stop(void *data)
 	}
 	rcu_read_unlock();
 out_unlock:
-	rq_unlock(busiest_rq, &rf);
+	rq_unlock(src_rq, &rf);
 
 	if (p)
 		attach_one_task(target_rq, p);
 
 	local_irq_enable();
-	busiest_rq->active_balance = 0;
+	this_rq()->active_balance = 0;
 
 	return 0;
 }
-- 
2.34.1

[RFC PATCH RESEND 07/10] sched/fair: Trigger active balance if a CFS task is preempted when LB_PROMOTE

[Xin Zhao]

When LB_PROMOTE is on, in addition to the scenarios modified in the
previous patches regarding .select_task_rq implement, preemption of CFS
tasks also needs to be addressed. Once a CFS task is preempted by a
real-time task, if that task is executing logic in a critical section that
does not support priority inheritance, such as read-write locks or
read-write semaphores, it may lead to performance issues.

By adding this checkpoint for when a CFS task is preempted, we can migrate
the task to an idle CPU, thus alleviating such performance problems.
Additionally, the patch set for the LB_PROMOTE feature does not alter the
logic of task_hot(), so this patch will not significantly increase the
frequency of load balance migration.

The newly introduced function preempt_active_balance() will be triggered
in __schedule() when it detects that the previous task has been preempted
while the LB_PROMOTE feature is enabled. preempt_active_balance() will
first check whether the currently preempted task has other CPUs available
to run on. If there are idle CPUs available, the preempted task will be
migrated to one of those idle CPUs.

The implementation of preempt_active_balance() considers the desire not to
introduce additional "holes" of rq locks, so the migration triggering
action is deferred to the balance_callback which is trigger_preempt_alb().
trigger_preempt_alb() will issue a stop work to allow the idle target CPU
to perform the migration.

After the modifications made by the previous patches, we can reuse
active_load_balance_cpu_stop() to assist in this migration action. In
preempt_active_balance(), we use list_move_tail to move the currently
preempted task to position where detach_one_task() will first traverse,
allowing active_load_balance_cpu_stop() to prioritize finding the task.

Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
---
 kernel/sched/core.c  |  3 ++
 kernel/sched/fair.c  | 80 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h |  2 ++
 3 files changed, 85 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 2e7cde033a31..757ea5ef303d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7227,6 +7227,9 @@ static void __sched notrace __schedule(int sched_mode)
 
 		trace_sched_switch(preempt, prev, next, prev_state);
 
+		if (sched_feat(LB_PROMOTE) && preempt)
+			preempt_active_balance(prev);
+
 		/* Also unlocks the rq: */
 		rq = context_switch(rq, prev, next, &rf);
 	} else {
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 9ffd01717599..d2b538abdf3e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13875,6 +13875,86 @@ static int active_load_balance_cpu_stop(void *data)
 	return 0;
 }
 
+static DEFINE_PER_CPU(struct balance_callback, preempt_push_head);
+static DEFINE_PER_CPU(int, push_cpu);
+
+static void trigger_preempt_alb(struct rq *rq)
+{
+	unsigned long flags;
+	int active_balance = 0;
+	int src_cpu = rq->cpu;
+	int idle_cpu = per_cpu(push_cpu, src_cpu);
+	struct rq *dst_rq = cpu_rq(idle_cpu);
+
+	preempt_disable();
+	raw_spin_rq_unlock(rq);
+	raw_spin_rq_lock_irqsave(dst_rq, flags);
+	if (dst_rq->active_balance)
+		goto unlock_rq;
+	dst_rq->active_balance = 1;
+	dst_rq->push_cpu = idle_cpu;
+	active_balance = 1;
+
+unlock_rq:
+	raw_spin_rq_unlock_irqrestore(dst_rq, flags);
+	if (active_balance) {
+		stop_one_cpu_nowait(idle_cpu, active_load_balance_cpu_stop, rq,
+				    &dst_rq->active_balance_work);
+	}
+	preempt_enable();
+	raw_spin_rq_lock(rq);
+}
+
+static void queue_preempt_alb_callback(struct rq *rq, int dst_cpu)
+{
+	per_cpu(push_cpu, rq->cpu) = dst_cpu;
+	queue_balance_callback(rq, &per_cpu(preempt_push_head, rq->cpu), trigger_preempt_alb);
+}
+
+void preempt_active_balance(struct task_struct *prev)
+{
+	struct rq *this_rq = this_rq();
+	int this_cpu = smp_processor_id(), cpu;
+	int start = nr_cpu_ids, idle_cpu = nr_cpu_ids;
+	struct sched_domain *sd;
+
+	if (unlikely(prev->sched_class != &fair_sched_class))
+		return;
+
+	if (unlikely(prev->migration_disabled))
+		return;
+
+	cpu = prev->recent_used_cpu;
+	if (cpu != this_cpu && available_idle_cpu(cpu) && !cpu_rq(cpu)->active_balance) {
+		idle_cpu = cpu;
+		goto active_balance;
+	}
+
+	rcu_read_lock();
+
+	sd = rcu_dereference(per_cpu(sd_llc, this_cpu));
+	if (sd)
+		start = cpumask_first_and(sched_domain_span(sd), prev->cpus_ptr);
+	if (start >= nr_cpu_ids)
+		start = cpumask_first(prev->cpus_ptr);
+
+	for_each_cpu_wrap(cpu, prev->cpus_ptr, start) {
+		if (cpu != this_cpu && available_idle_cpu(cpu) && !cpu_rq(cpu)->active_balance) {
+			idle_cpu = cpu;
+			goto unlock;
+		}
+	}
+
+unlock:
+	rcu_read_unlock();
+	if (idle_cpu == nr_cpu_ids)
+		return;
+
+active_balance:
+	list_move_tail(&prev->se.group_node, &this_rq->cfs_tasks);
+	queue_preempt_alb_callback(this_rq, idle_cpu);
+}
+
 /*
  * Scale the max sched_balance_rq interval with the number of CPUs in the system.
  * This trades load-balance latency on larger machines for less cross talk.
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 26ae13c86b69..11848708e5ce 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -4193,6 +4193,8 @@ extern struct balance_callback *splice_balance_callbacks(struct rq *rq);
 extern void __balance_callbacks(struct rq *rq, struct rq_flags *rf);
 extern void balance_callbacks(struct rq *rq, struct balance_callback *head);
 
+extern void preempt_active_balance(struct task_struct *prev);
+
 /*
  * The 'sched_change' pattern is the safe, easy and slow way of changing a
  * task's scheduling properties. It dequeues a task, such that the scheduler
-- 
2.34.1

[RFC PATCH RESEND 08/10] sched/fair: Do not check avg_idle to prematurely exit newly idle when LB_PROMOTE

[Xin Zhao]

When LB_PROMOTE is on, there are high real-time requirements, we should
avoid prematurely exiting in sched_balance_newidle() due to a short
avg_idle. This is because after any newly idle state, if no tasks are
found to run, there may be a long-term lack of new incoming tasks to wake
up on that CPU. Meanwhile, there may be tasks waiting to run on other
cores, leading to unreasonable CPU idleness. The definition of
unreasonable CPU idleness can be found in the commit-log of the previous
patch that introduced the LB_PROMOTE feature.

Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
---
 kernel/sched/fair.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d2b538abdf3e..8a1d2763a923 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -14694,7 +14694,7 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)
 		goto out;
 
 	if (!get_rd_overloaded(this_rq->rd) ||
-	    this_rq->avg_idle < sd->max_newidle_lb_cost) {
+	    (!sched_feat(LB_PROMOTE) && this_rq->avg_idle < sd->max_newidle_lb_cost)) {
 
 		update_next_balance(sd, &next_balance);
 		goto out;
@@ -14716,7 +14716,8 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)
 
 		update_next_balance(sd, &next_balance);
 
-		if (this_rq->avg_idle < curr_cost + sd->max_newidle_lb_cost)
+		if (!sched_feat(LB_PROMOTE) &&
+		    this_rq->avg_idle < curr_cost + sd->max_newidle_lb_cost)
 			break;
 
 		if (sd->flags & SD_BALANCE_NEWIDLE) {
-- 
2.34.1

[RFC PATCH RESEND 09/10] sched/fair: Not goto more_balance if newly idle and has pending task when LBF_NEED_BREAK

[Xin Zhao]

When LBF_NEED_BREAK flag is set in env.flags, no longer unconditionally go
to more_balance. Instead, we exclude the case when it is newly idle and
there are pending tasks. This helps avoid unnecessary CPU wastage caused
by repeatedly going to more_balance when the task load is too high during
sched_balance_rq().

In another 'goto more_balance' case when LBF_DST_PINNED flag is set in
env.flags, we do not need to add the check. Because LBF_DST_PINNED flag is
only set within can_migrate_task(). Before setting LBF_DST_PINNED flag in
can_migrate_task(), there is a check to see if it is newly idle.
can_migrate_task() will exit without setting the LBF_DST_PINNED flag if it
is newly idle.

Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
---
 kernel/sched/fair.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8a1d2763a923..1ae351ea5949 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13561,7 +13561,9 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 
 		if (env.flags & LBF_NEED_BREAK) {
 			env.flags &= ~LBF_NEED_BREAK;
-			goto more_balance;
+			if (!(env.idle == CPU_NEWLY_IDLE &&
+			      (env.dst_rq->nr_running > 0 || env.dst_rq->ttwu_pending)))
+				goto more_balance;
 		}
 
 		/*
-- 
2.34.1

[RFC PATCH RESEND 10/10] sched/fair: Strive to find a task to migrate if newly idle when LB_PROMOTE

[Xin Zhao]

This patch is a core modification within this patch set. The simple_find
label in this patch implements a straightforward logic for finding a
migration task. It allows any instances in sched_balance_rq() that are
unable to find a migration task for various reasons to fallback to logic
executing the simple_find label to find one before exiting newly idle
process.

This patch addresses the newly idle scenario by preventing early exit in
cases when !ld_moved && !active_balance, effectively executing the logic
of simple_find label. Testing has shown that the situations listed below,
account for a significant proportion of early exits:
1. Failure in sched_balance_find_src_group
2. Failure in sched_balance_find_src_rq
3. ld_moved is 0 and active_balance has not been triggered

Of course, even with this change, it is still possible that no migration
task can be found. However, it at least ensures that all selectable CPUs
within the sched_domain have been thoroughly traversed.

Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
---
 kernel/sched/fair.c | 38 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 1ae351ea5949..10ec7bb9e18c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13470,6 +13470,8 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 	struct rq *busiest;
 	struct rq_flags rf;
 	struct cpumask *cpus = this_cpu_cpumask_var_ptr(load_balance_mask);
+	int cpu;
+	bool sfind = false;
 	struct lb_env env = {
 		.sd		= sd,
 		.dst_cpu	= this_cpu,
@@ -13504,15 +13506,41 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 	group = sched_balance_find_src_group(&env);
 	if (!group) {
 		schedstat_inc(sd->lb_nobusyg[idle]);
+		if (sched_feat(LB_PROMOTE))
+			goto simple_find;
 		goto out_balanced;
 	}
 
 	busiest = sched_balance_find_src_rq(&env, group);
 	if (!busiest) {
 		schedstat_inc(sd->lb_nobusyq[idle]);
+		if (sched_feat(LB_PROMOTE))
+			goto simple_find;
 		goto out_balanced;
 	}
+	goto begin_balance;
+
+simple_find:
+	if (env.idle != CPU_NEWLY_IDLE ||
+	    (env.dst_rq->nr_running > 0 || env.dst_rq->ttwu_pending))
+		goto out_balanced;
 
+	sfind = true;
+	env.migration_type = migrate_task;
+	env.imbalance = 1;
+
+	for_each_cpu_andnot(cpu, env.cpus, env.dst_grpmask) {
+		busiest = cpu_rq(cpu);
+		if (busiest->nr_running <= 1) {
+			__cpumask_clear_cpu(cpu, cpus);
+			continue;
+		}
+		break;
+	}
+	if (cpu >= nr_cpu_ids)
+		goto out_balanced;
+
+begin_balance:
 	WARN_ON_ONCE(busiest == env.dst_rq);
 
 	update_lb_imbalance_stat(&env, sd, idle);
@@ -13615,6 +13643,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 
 		/* All tasks on this runqueue were pinned by CPU affinity */
 		if (unlikely(env.flags & LBF_ALL_PINNED)) {
+check_redo:
 			__cpumask_clear_cpu(cpu_of(busiest), cpus);
 			/*
 			 * Attempting to continue load balancing at the current
@@ -13627,6 +13656,8 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 			if (!cpumask_subset(cpus, env.dst_grpmask)) {
 				env.loop = 0;
 				env.loop_break = SCHED_NR_MIGRATE_BREAK;
+				if (sfind)
+					goto simple_find;
 				goto redo;
 			}
 			goto out_all_pinned;
@@ -13668,8 +13699,11 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 		 * if the curr task on busiest CPU can't be
 		 * moved to this_cpu:
 		 */
-		if (!cpumask_test_cpu(this_cpu, busiest->curr->cpus_ptr))
+		if (!cpumask_test_cpu(this_cpu, busiest->curr->cpus_ptr)) {
+			if (sched_feat(LB_PROMOTE) && env.idle == CPU_NEWLY_IDLE)
+				goto check_redo;
 			goto out_one_pinned;
+		}
 
 		/* Record that we found at least one task that could run on this_cpu */
 		env.flags &= ~LBF_ALL_PINNED;
@@ -13703,6 +13737,8 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 	preempt_enable();
 
 out_unbalanced:
+	if (sched_feat(LB_PROMOTE) && !active_balance && env.idle == CPU_NEWLY_IDLE)
+		goto check_redo;
 	/* We were unbalanced, so reset the balancing interval */
 	sd->balance_interval = sd->min_interval;
 	goto out;
-- 
2.34.1