[PATCH 0/6 v2] sched/eevdf: Improve scheduling latency of short slice task

[PATCH 0/6 v2] sched/eevdf: Improve scheduling latency of short slice task

[Vincent Guittot]

This series continues to improve the scheduling latency of tasks with
shorter slice duration by mainly canceling, updating or minimizing the
protection of the running tasks when appropriate.

Benchmarks, like hackbench, haven't seen any noticeable performance
differences with this patchset (The default 2.8ms slice has been used for
testing performance regressions)

Several use cases has been used to testing the scheduling latency of short
slice tasks:
- cyclictest with a 3777us period and a 8ms slice alone
- cyclictest with a 3777us period and a 8ms slice. 2xNR_CPUS rt-app
  tasks that run (8177us) and sleep (17777us) with a 16ms slice.
- cyclictest with a 3777us period and a 8ms slice. Hackbench with
  1 group using thread and pipe and a 16ms slice.

NB: periods and run duration have been chosen to minimize alignment
with tick or other periodic activities.

scheduling latency (us) for cyclictest
                   tip/sched/core| this patchset | tip/sched/core
slice                 8ms        |  8ms          |  2.8ms
90th Percentile               80 |    79 (  1 %) |    56 (+30 %)
99th Percentile               93 |    90 (+ 3 %) |    60 (+35 %)
99.9th Percentile            184 |   145 (+21 %) |    67 (+64 %)
Maximum                     7526 |  2649 (+65 %) |  2647 (+65 %)

scheduling latency (us) for cyclictest and rt-app 
                   tip/sched/core| this patchset | tip/sched/core
slice                 8ms / 16ms |  8ms  / 16 ms |  2.8ms / 2.8ms
90th Percentile               61 |    66 (- 8 %) |   440 (-621 %)
99th Percentile             8453 |  2359 (+72 %) |  3167 (+ 63 %)
99.9th Percentile          14053 |  5026 (+64 %) |  5290 (+ 62 %)
Maximum                    20924 |  7851 (+62 %) |  8727 (+ 58 %)

scheduling latency (us) for cyclictest and hackbench 
                   tip/sched/core| this patchset | tip/sched/core
slice                 8ms / 16ms |  8ms  / 16 ms |  2.8ms / 2.8ms
90th Percentile               63 |    63 (  0 %) |  1237 (-1863 %)
99th Percentile               76 |    79 (- 4 %) |  4575 (-5920 %)
99.9th Percentile           2827 |  1021 (+64 %) |  8312 (- 194 %)
Maximum                    15795 |  4766 (+70 %) | 14519 (+   8 %)

Since v1:
- Use the correct min_vruntime() instead of min()

Vincent Guittot (6):
  sched/fair: Set next buddy for preempt short
  sched/eevdf: Take into account current's lag when updating slice
    protection
  sched/eevdf: Update slice protection even when resched is already set
  sched/eevdf: Cancel slice protection if short slice task is eligible
  sched/eevdf: Always update slice protection
  sched/eevdf: Speedup short slice task scheduling

 kernel/sched/fair.c | 43 +++++++++++++++++++++++++------------------
 1 file changed, 25 insertions(+), 18 deletions(-)

-- 
2.43.0

[PATCH 1/6 v2] sched/fair: Set next buddy for preempt short

[Vincent Guittot]

If a shorter slice task can preempt current at wakeup, we make sure that
the decision will not be overwritten in between by setting the task as the
next buddy. This still implies that the waking task remains eligible when
the scheduler will actually pick the next task to run.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1..83bce5a04f3d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9903,7 +9903,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 preempt:
 	if (preempt_action == PREEMPT_WAKEUP_SHORT) {
 		cancel_protect_slice(se);
-		clear_buddies(cfs_rq, se);
+		set_next_buddy(&p->se);
 	}
 
 	resched_curr_lazy(rq);
-- 
2.43.0

[PATCH 2/6 v2] sched/eevdf: Take into account current's lag when updating slice protection

[Vincent Guittot]

Take into account the lag of current task when setting the slice protection
in order to ensure that the absolute value of lags will remain in the
range [0 : slice+tick]
A task that already has a negative lag will see its protection reduced
whereas a task with positive lag will keep a full slice protection.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 kernel/sched/fair.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 83bce5a04f3d..b8d5d9bcc014 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1083,6 +1083,7 @@ struct sched_entity *__pick_first_entity(struct cfs_rq *cfs_rq)
  */
 static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+	u64 vruntime = min_vruntime(se->vruntime, avg_vruntime(cfs_rq));
 	u64 slice = normalized_sysctl_sched_base_slice;
 	u64 vprot = se->deadline;
 
@@ -1090,8 +1091,8 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
 		slice = cfs_rq_min_slice(cfs_rq);
 
 	slice = min(slice, se->slice);
-	if (slice != se->slice)
-		vprot = min_vruntime(vprot, se->vruntime + calc_delta_fair(slice, se));
+	if (vruntime != se->vruntime || slice != se->slice)
+		vprot = min_vruntime(vprot, vruntime + calc_delta_fair(slice, se));
 
 	se->vprot = vprot;
 }
@@ -1099,8 +1100,9 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
 static inline void update_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	u64 slice = cfs_rq_min_slice(cfs_rq);
+	u64 vruntime = min_vruntime(se->vruntime, avg_vruntime(cfs_rq));
 
-	se->vprot = min_vruntime(se->vprot, se->vruntime + calc_delta_fair(slice, se));
+	se->vprot = min_vruntime(se->vprot, vruntime + calc_delta_fair(slice, se));
 }
 
 static inline bool protect_slice(struct sched_entity *se)
-- 
2.43.0

[PATCH 3/6 v2] sched/eevdf: Update slice protection even when resched is already set

[Vincent Guittot]

Even if resched is already set, we might want to update or even cancel
the slice protection and ensure that the newly waking task will be the
next one to run.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b8d5d9bcc014..b98d67650a98 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9805,7 +9805,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	 * prevents us from potentially nominating it as a false LAST_BUDDY
 	 * below.
 	 */
-	if (test_tsk_need_resched(rq->curr))
+	if (!sched_feat(PREEMPT_SHORT) && test_tsk_need_resched(rq->curr))
 		return;
 
 	if (!sched_feat(WAKEUP_PREEMPTION))
-- 
2.43.0

[PATCH 4/6 v2] sched/eevdf: Cancel slice protection if short slice task is eligible

[Vincent Guittot]

If a short slice task will not be the next to be picked but is eligible,
we cancel the slice protection to speedup the time when the short slice
task will be the next to run.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 kernel/sched/fair.c | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b98d67650a98..4923bb28dde3 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9817,18 +9817,13 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	cse_is_idle = se_is_idle(se);
 	pse_is_idle = se_is_idle(pse);
 
+	nse = se;
 	/*
 	 * Preempt an idle entity in favor of a non-idle entity (and don't preempt
 	 * in the inverse case).
 	 */
-	if (cse_is_idle && !pse_is_idle) {
-		/*
-		 * When non-idle entity preempt an idle entity,
-		 * don't give idle entity slice protection.
-		 */
-		preempt_action = PREEMPT_WAKEUP_SHORT;
+	if (cse_is_idle && !pse_is_idle)
 		goto preempt;
-	}
 
 	if (cse_is_idle != pse_is_idle)
 		return;
@@ -9897,16 +9892,23 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	if (!nse && cfs_rq->nr_queued)
 		goto pick;
 
+	/*
+	 * If @p is eligible but not the next task to run then cancel protection
+	 * to prevent large scheduling latency
+	 */
+	if (preempt_action == PREEMPT_WAKEUP_SHORT && entity_eligible(cfs_rq, pse))
+		goto preempt;
+
 	if (sched_feat(RUN_TO_PARITY))
 		update_protect_slice(cfs_rq, se);
 
 	return;
 
 preempt:
-	if (preempt_action == PREEMPT_WAKEUP_SHORT) {
-		cancel_protect_slice(se);
+	cancel_protect_slice(se);
+
+	if (preempt_action == PREEMPT_WAKEUP_SHORT && nse == pse)
 		set_next_buddy(&p->se);
-	}
 
 	resched_curr_lazy(rq);
 }
-- 
2.43.0

[PATCH 5/6 v2] sched/eevdf: Always update slice protection

[Vincent Guittot]

Even if p will not preempt current, it modifies the avg_vruntime and
possibly the min slice. Make sure to update the slice protection with the
updated figures. As an example, Batch and Sched Idle tasks can otherwise
get a larger lag than their slice and finaly delay the scheduling of a
normal task, which deadline will be a later.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 kernel/sched/fair.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 4923bb28dde3..601c67cff185 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9826,13 +9826,13 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 		goto preempt;
 
 	if (cse_is_idle != pse_is_idle)
-		return;
+		goto update;
 
 	/*
 	 * BATCH and IDLE tasks do not preempt others.
 	 */
 	if (unlikely(!normal_policy(p->policy)))
-		return;
+		goto update;
 
 	cfs_rq = cfs_rq_of(se);
 	update_curr(cfs_rq);
@@ -9852,7 +9852,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	 * EEVDF to forcibly queue an ineligible task.
 	 */
 	if ((wake_flags & WF_FORK) || pse->sched_delayed)
-		return;
+		goto update;
 
 	/* Prefer picking wakee soon if appropriate. */
 	if (sched_feat(NEXT_BUDDY) &&
@@ -9898,7 +9898,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	 */
 	if (preempt_action == PREEMPT_WAKEUP_SHORT && entity_eligible(cfs_rq, pse))
 		goto preempt;
-
+update:
 	if (sched_feat(RUN_TO_PARITY))
 		update_protect_slice(cfs_rq, se);
 
-- 
2.43.0

[PATCH 6/6 v2] sched/eevdf: Speedup short slice task scheduling

[Vincent Guittot]

When a task with a shorter slice is enqueued, we protect the running
task which has a longer slice until it becomes ineligible instead of a
full slice in order to speedup the switch to other tasks until the task
with the shortest slice is scheduled. This helps to the task to not wait
too many full slices before running.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 kernel/sched/fair.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 601c67cff185..994fcf3ea702 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1091,7 +1091,10 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
 		slice = cfs_rq_min_slice(cfs_rq);
 
 	slice = min(slice, se->slice);
-	if (vruntime != se->vruntime || slice != se->slice)
+
+	if (sched_feat(PREEMPT_SHORT) && slice < se->slice)
+		vprot = avg_vruntime(cfs_rq);
+	else if ((vruntime != se->vruntime) || (slice != se->slice))
 		vprot = min_vruntime(vprot, vruntime + calc_delta_fair(slice, se));
 
 	se->vprot = vprot;
-- 
2.43.0

Re: [PATCH 2/6 v2] sched/eevdf: Take into account current's lag when updating slice protection

[K Prateek Nayak]

Hello Vincent,

On 6/15/2026 9:54 PM, Vincent Guittot wrote:
> @@ -1083,6 +1083,7 @@ struct sched_entity *__pick_first_entity(struct cfs_rq *cfs_rq)
>   */
>  static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
>  {
> +	u64 vruntime = min_vruntime(se->vruntime, avg_vruntime(cfs_rq));

set_protect_slice() is only called for set_next_entity(... first = true)
which has to come from the pick where the entity has to be eligible and
behind the avg_vruntime(cfs_rq) right?

Is there a case where se->vruntime is ahead of the average here that
I'm missing?

>  	u64 slice = normalized_sysctl_sched_base_slice;
>  	u64 vprot = se->deadline;
>  
> @@ -1090,8 +1091,8 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
>  		slice = cfs_rq_min_slice(cfs_rq);
>  
>  	slice = min(slice, se->slice);
> -	if (slice != se->slice)
> -		vprot = min_vruntime(vprot, se->vruntime + calc_delta_fair(slice, se));
> +	if (vruntime != se->vruntime || slice != se->slice)
> +		vprot = min_vruntime(vprot, vruntime + calc_delta_fair(slice, se));
>  
>  	se->vprot = vprot;
>  }
-- 
Thanks and Regards,
Prateek

Re: [PATCH 4/6 v2] sched/eevdf: Cancel slice protection if short slice task is eligible

[K Prateek Nayak]

Hello Vincent,

On 6/15/2026 9:54 PM, Vincent Guittot wrote:
> @@ -9897,16 +9892,23 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
>  	if (!nse && cfs_rq->nr_queued)
>  		goto pick;
>  
> +	/*
> +	 * If @p is eligible but not the next task to run then cancel protection
> +	 * to prevent large scheduling latency
> +	 */
> +	if (preempt_action == PREEMPT_WAKEUP_SHORT && entity_eligible(cfs_rq, pse))
> +		goto preempt;

Can't we do this independent of the "pick" path if we are going to
resched anyways?

I suppose it is for set_next_buddy() bits but, if nse == pse, we cancel
the slice protection and resched anyways and the pick will naturally
go to pse so does set_next_buddy() even matter?

Instead, can PREEMPT_WAKEUP_SHORT path just do a set_preempt_buddy()
beforehand to ensure the task with the smallest deadline is already
the ->next and pick_next_entity() readily returns that?

> +
>  	if (sched_feat(RUN_TO_PARITY))
>  		update_protect_slice(cfs_rq, se);
>  
>  	return;
>  
>  preempt:
> -	if (preempt_action == PREEMPT_WAKEUP_SHORT) {
> -		cancel_protect_slice(se);
> +	cancel_protect_slice(se);
> +
> +	if (preempt_action == PREEMPT_WAKEUP_SHORT && nse == pse)
>  		set_next_buddy(&p->se);
> -	}
>  
>  	resched_curr_lazy(rq);
>  }

-- 
Thanks and Regards,
Prateek

Re: [PATCH 0/6 v2] sched/eevdf: Improve scheduling latency of short slice task

[K Prateek Nayak]

Hello Vincent,

On 6/15/2026 9:54 PM, Vincent Guittot wrote:
> This series continues to improve the scheduling latency of tasks with
> shorter slice duration by mainly canceling, updating or minimizing the
> protection of the running tasks when appropriate.
> 
> Benchmarks, like hackbench, haven't seen any noticeable performance
> differences with this patchset (The default 2.8ms slice has been used for
> testing performance regressions)

I've left a few comments on the thread but for the vanilla runs, I too
can confirm there aren't any without any slice tuning.

Following are results from a dual socket 4th Generation EPYC system
(2 x 128C/256T) with the series applied on top of
"sched-core-2026-06-14":

  ==================================================================
  Test          : hackbench
  Units         : Normalized time in seconds
  Interpretation: Lower is better
  Statistic     : AMean
  ==================================================================
  Case:           tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
   1-groups     1.00 [ -0.00]( 9.66)     0.86 [ 14.32](14.00)
   2-groups     1.00 [ -0.00]( 9.22)     1.02 [ -1.78](10.45)
   4-groups     1.00 [ -0.00]( 2.14)     0.98 [  2.33]( 1.99)
   8-groups     1.00 [ -0.00]( 2.80)     0.97 [  2.88]( 2.93)
  16-groups     1.00 [ -0.00]( 5.54)     1.00 [  0.49]( 2.58)


  ==================================================================
  Test          : tbench
  Units         : Normalized throughput
  Interpretation: Higher is better
  Statistic     : AMean
  ==================================================================
  Clients:    tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
      1     1.00 [  0.00]( 0.03)     1.00 [  0.36]( 0.20)
      2     1.00 [  0.00]( 0.32)     1.00 [  0.09]( 0.14)
      4     1.00 [  0.00]( 0.34)     1.00 [  0.39]( 0.28)
      8     1.00 [  0.00]( 0.24)     1.00 [  0.01]( 0.24)
     16     1.00 [  0.00]( 0.45)     1.00 [  0.01]( 0.47)
     32     1.00 [  0.00]( 0.58)     1.01 [  0.75]( 0.30)
     64     1.00 [  0.00]( 0.81)     1.02 [  1.62]( 0.62)
    128     1.00 [  0.00]( 0.53)     1.03 [  3.24]( 0.25)
    256     1.00 [  0.00]( 0.30)     1.00 [  0.39]( 0.26)
    512     1.00 [  0.00]( 3.73)     1.01 [  1.47]( 1.13)
   1024     1.00 [  0.00]( 0.23)     1.00 [ -0.10]( 0.37)
   2048     1.00 [  0.00]( 0.14)     1.00 [  0.29]( 0.19)


  ==================================================================
  Test          : stream-10
  Units         : Normalized Bandwidth, MB/s
  Interpretation: Higher is better
  Statistic     : HMean
  ==================================================================
  Test:       tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
   Copy     1.00 [  0.00]( 0.66)     0.99 [ -0.70]( 1.66)
  Scale     1.00 [  0.00]( 0.89)     0.99 [ -0.77]( 1.52)
    Add     1.00 [  0.00]( 0.73)     1.00 [ -0.34]( 1.31)
  Triad     1.00 [  0.00]( 0.70)     0.99 [ -0.52]( 1.24)


  ==================================================================
  Test          : stream-100
  Units         : Normalized Bandwidth, MB/s
  Interpretation: Higher is better
  Statistic     : HMean
  ==================================================================
  Test:       tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
   Copy     1.00 [  0.00]( 0.32)     1.00 [  0.07]( 0.36)
  Scale     1.00 [  0.00]( 0.26)     1.00 [ -0.00]( 0.45)
    Add     1.00 [  0.00]( 0.29)     1.00 [ -0.05]( 0.39)
  Triad     1.00 [  0.00]( 0.27)     1.00 [ -0.05]( 0.37)


  ==================================================================
  Test          : netperf
  Units         : Normalized Througput
  Interpretation: Higher is better
  Statistic     : AMean
  ==================================================================
  Clients:           tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
     1-clients     1.00 [  0.00]( 0.10)     1.00 [  0.02]( 0.19)
     2-clients     1.00 [  0.00]( 0.29)     1.00 [  0.02]( 0.18)
     4-clients     1.00 [  0.00]( 0.36)     1.00 [ -0.01]( 0.23)
     8-clients     1.00 [  0.00]( 0.32)     1.00 [  0.04]( 0.22)
    16-clients     1.00 [  0.00]( 0.24)     1.00 [  0.09]( 0.22)
    32-clients     1.00 [  0.00]( 0.42)     1.00 [  0.30]( 0.33)
    64-clients     1.00 [  0.00]( 0.94)     1.00 [  0.48]( 0.67)
   128-clients     1.00 [  0.00]( 1.10)     1.01 [  0.77]( 1.31)
   256-clients     1.00 [  0.00]( 1.06)     1.02 [  2.02]( 1.18)
   512-clients     1.00 [  0.00]( 4.68)     0.99 [ -1.14]( 5.63)
   768-clients     1.00 [  0.00](34.35)     0.99 [ -1.00](34.84)
  1024-clients     1.00 [  0.00](42.76)     0.99 [ -0.81](45.77)


  ==================================================================
  Test          : schbench
  Units         : Normalized 99th percentile latency in us
  Interpretation: Lower is better
  Statistic     : Median
  ==================================================================
  #workers:  tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
     1     1.00 [ -0.00](18.94)     0.39 [ 61.36]( 5.88)
     2     1.00 [ -0.00]( 1.67)     0.91 [  8.57]( 6.64)
     4     1.00 [ -0.00]( 9.79)     0.84 [ 16.22]( 7.78)
     8     1.00 [ -0.00]( 2.27)     0.89 [ 11.36](10.54)
    16     1.00 [ -0.00]( 0.00)     0.98 [  1.79]( 4.10)
    32     1.00 [ -0.00]( 1.92)     1.01 [ -1.25]( 0.72)
    64     1.00 [ -0.00]( 1.19)     1.01 [ -0.78]( 1.18)
   128     1.00 [ -0.00]( 0.67)     1.01 [ -1.32]( 0.25)
   256     1.00 [ -0.00]( 0.46)     1.03 [ -3.08]( 4.37)
   512     1.00 [ -0.00]( 0.33)     1.01 [ -0.66]( 0.38)
   768     1.00 [ -0.00]( 4.69)     1.02 [ -1.55](10.18)
  1024     1.00 [ -0.00]( 2.71)     1.00 [ -0.00]( 4.43)


  ==================================================================
  Test          : new-schbench-requests-per-second
  Units         : Normalized Requests per second
  Interpretation: Higher is better
  Statistic     : Median
  ==================================================================
  #workers:  tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
     1     1.00 [  0.00]( 0.15)     0.99 [ -0.59]( 0.15)
     2     1.00 [  0.00]( 0.00)     0.99 [ -0.59]( 0.15)
     4     1.00 [  0.00]( 0.00)     0.99 [ -0.88]( 0.31)
     8     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.00)
    16     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.00)
    32     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.15)
    64     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)
   128     1.00 [  0.00](12.53)     0.98 [ -1.77](15.94)
   256     1.00 [  0.00]( 0.15)     0.99 [ -0.85]( 0.39)
   512     1.00 [  0.00]( 0.84)     1.00 [  0.00]( 0.84)
   768     1.00 [  0.00]( 2.05)     0.99 [ -0.94]( 1.94)
  1024     1.00 [  0.00]( 2.90)     1.01 [  1.35]( 2.18)


  ==================================================================
  Test          : new-schbench-wakeup-latency
  Units         : Normalized 99th percentile latency in us
  Interpretation: Lower is better
  Statistic     : Median
  ==================================================================
  #workers:  tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
     1     1.00 [ -0.00](12.99)     1.47 [-46.67](27.72)
     2     1.00 [ -0.00]( 4.08)     0.85 [ 15.38]( 0.00)
     4     1.00 [ -0.00]( 0.00)     0.82 [ 18.18]( 0.00)
     8     1.00 [ -0.00]( 0.00)     1.27 [-27.27]( 3.78)
    16     1.00 [ -0.00]( 4.56)     1.27 [-27.27]( 0.00)
    32     1.00 [ -0.00]( 0.00)     1.00 [ -0.00]( 4.56)
    64     1.00 [ -0.00]( 5.00)     1.00 [ -0.00]( 5.00)
   128     1.00 [ -0.00]( 7.45)     1.25 [-25.00](14.68)
   256     1.00 [ -0.00]( 2.70)     0.96 [  4.48]( 8.12)
   512     1.00 [ -0.00]( 0.00)     1.00 [ -0.00]( 0.00)
   768     1.00 [ -0.00]( 1.66)     1.01 [ -1.47]( 2.52)
  1024     1.00 [ -0.00]( 3.32)     1.01 [ -0.59]( 0.66)

  Note: The absolute numbers are very small until 256 threads (~10-15us)
  which may causes a small variation to appear as a large regression.

  ==================================================================
  Test          : new-schbench-request-latency
  Units         : Normalized 99th percentile latency in us
  Interpretation: Lower is better
  Statistic     : Median
  ==================================================================
  #workers:  tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
     1     1.00 [ -0.00]( 0.14)     1.01 [ -1.06]( 0.27)
     2     1.00 [ -0.00]( 0.14)     1.02 [ -1.60]( 0.23)
     4     1.00 [ -0.00]( 0.00)     1.05 [ -4.53]( 1.73)
     8     1.00 [ -0.00]( 0.14)     1.01 [ -0.53]( 0.14)
    16     1.00 [ -0.00]( 1.49)     1.00 [ -0.26]( 1.23)
    32     1.00 [ -0.00]( 0.89)     1.01 [ -0.79]( 0.00)
    64     1.00 [ -0.00]( 1.43)     1.00 [ -0.26]( 0.98)
   128     1.00 [ -0.00]( 2.78)     1.01 [ -1.18]( 4.09)
   256     1.00 [ -0.00]( 0.13)     1.00 [ -0.25]( 0.26)
   512     1.00 [ -0.00]( 6.72)     1.02 [ -2.20]( 5.45)
   768     1.00 [ -0.00]( 3.42)     1.01 [ -0.52]( 4.21)
  1024     1.00 [ -0.00]( 4.37)     1.01 [ -1.19]( 2.15)

> 
> Several use cases has been used to testing the scheduling latency of short
> slice tasks:
> - cyclictest with a 3777us period and a 8ms slice alone
> - cyclictest with a 3777us period and a 8ms slice. 2xNR_CPUS rt-app
>   tasks that run (8177us) and sleep (17777us) with a 16ms slice.
> - cyclictest with a 3777us period and a 8ms slice. Hackbench with
>   1 group using thread and pipe and a 16ms slice.

I'll go check those configurations next and report back if I find
anything out of the ordinary. Feel free to include:

Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>

-- 
Thanks and Regards,
Prateek

Re: [PATCH 1/6 v2] sched/fair: Set next buddy for preempt short

[Peter Zijlstra]

On Mon, Jun 15, 2026 at 06:24:15PM +0200, Vincent Guittot wrote:
> If a shorter slice task can preempt current at wakeup, we make sure that
> the decision will not be overwritten in between by setting the task as the
> next buddy. This still implies that the waking task remains eligible when
> the scheduler will actually pick the next task to run.
> 
> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> ---
>  kernel/sched/fair.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index d78467ec6ee1..83bce5a04f3d 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9903,7 +9903,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
>  preempt:
>  	if (preempt_action == PREEMPT_WAKEUP_SHORT) {
>  		cancel_protect_slice(se);
> -		clear_buddies(cfs_rq, se);
> +		set_next_buddy(&p->se);
>  	}

Should this not be something along the lines of set_preempt_buddy()?

Re: [PATCH 2/6 v2] sched/eevdf: Take into account current's lag when updating slice protection

[Peter Zijlstra]

On Mon, Jun 15, 2026 at 06:24:16PM +0200, Vincent Guittot wrote:
> Take into account the lag of current task when setting the slice protection
> in order to ensure that the absolute value of lags will remain in the
> range [0 : slice+tick]
> A task that already has a negative lag will see its protection reduced
> whereas a task with positive lag will keep a full slice protection.
> 
> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> ---
>  kernel/sched/fair.c | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 83bce5a04f3d..b8d5d9bcc014 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -1083,6 +1083,7 @@ struct sched_entity *__pick_first_entity(struct cfs_rq *cfs_rq)
>   */
>  static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
>  {
> +	u64 vruntime = min_vruntime(se->vruntime, avg_vruntime(cfs_rq));
>  	u64 slice = normalized_sysctl_sched_base_slice;
>  	u64 vprot = se->deadline;
>  
> @@ -1090,8 +1091,8 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
>  		slice = cfs_rq_min_slice(cfs_rq);
>  
>  	slice = min(slice, se->slice);
> -	if (slice != se->slice)
> -		vprot = min_vruntime(vprot, se->vruntime + calc_delta_fair(slice, se));
> +	if (vruntime != se->vruntime || slice != se->slice)
> +		vprot = min_vruntime(vprot, vruntime + calc_delta_fair(slice, se));
>  
>  	se->vprot = vprot;
>  }

As already noted by Prateek, this doesn't seem to make much sense, since
we just got selected by schedule(), we *must* be left of avg_vruntime(),
otherwise we'd not be eligible and all that.

> @@ -1099,8 +1100,9 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
>  static inline void update_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
>  {
>  	u64 slice = cfs_rq_min_slice(cfs_rq);
> +	u64 vruntime = min_vruntime(se->vruntime, avg_vruntime(cfs_rq));
>  
> -	se->vprot = min_vruntime(se->vprot, se->vruntime + calc_delta_fair(slice, se));
> +	se->vprot = min_vruntime(se->vprot, vruntime + calc_delta_fair(slice, se));
>  }
>  
>  static inline bool protect_slice(struct sched_entity *se)

So:

 - set_protect_slice() is called at: set_next_task(.first = true),
   eg, only when the task gets scheduled().

 - set_protect_slice() takes se->deadline as the baseline, and (when
   RUN_TO_PARITY) computes a shorter vprot [ min_slice vs slice ].

 - update_protect_slice() is called upon (failed) wakeup preemption,
   new tasks have been added and as such the goal is to re-compute the
   min_slice and possibly reduce vprot.

Right?


And while update_protect_slice() would ideally use the original
se->vruntime (as per set_next_task(.first = true) to compute any new
(shorter) vprot, per its use of min_vruntime() it can not in fact end up
with a vprot that is longer than the initial.

Now, your change is to use min(se->vruntime, avg_vruntime()) to increase
the chance of actually computing a shorter vprot. Still very much wrong,
but possibly less wrong.

Rather than taking avg_vruntime(), would it make sense to do something
like:

	slice = cfs_rq_min_slice(cfs_rq);
	slice -= se->sum_exec_runtime - se->prev_sum_exec_runtime;

	vprot = se->vruntime;
	if (slice < 0)
		vprot -= calc_delta_fair(-slice, se);
	else
		vprot += calc_delta_fair(slice, se);

	se->vprot = min_vruntime(se->vprot, vprot);

That is, reduce the slice with the time already ran.

Now, this will go sideways in the unlikely case of renice, and possibly
sched_change pattern (it seems we update prev_sum_exec_runtime for
!first), but overall it might be a better approximation, no?

Re: [PATCH 3/6 v2] sched/eevdf: Update slice protection even when resched is already set

[Peter Zijlstra]

On Mon, Jun 15, 2026 at 06:24:17PM +0200, Vincent Guittot wrote:
> Even if resched is already set, we might want to update or even cancel
> the slice protection and ensure that the newly waking task will be the
> next one to run.
> 
> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> ---
>  kernel/sched/fair.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index b8d5d9bcc014..b98d67650a98 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9805,7 +9805,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
>  	 * prevents us from potentially nominating it as a false LAST_BUDDY
>  	 * below.
>  	 */
> -	if (test_tsk_need_resched(rq->curr))
> +	if (!sched_feat(PREEMPT_SHORT) && test_tsk_need_resched(rq->curr))
>  		return;
>  
>  	if (!sched_feat(WAKEUP_PREEMPTION))

Fair enough.

It did get me thinking about what we would want for SCHED_BATCH tasks. I
think we might want to make that RUN_TO_PARITY check (which selects
min_slice) conditional on normal_policy(). But that's another patch for
another day (and much easier after we land that flat thing).

Re: [PATCH 6/6 v2] sched/eevdf: Speedup short slice task scheduling

[Peter Zijlstra]

On Mon, Jun 15, 2026 at 06:24:20PM +0200, Vincent Guittot wrote:
> When a task with a shorter slice is enqueued, we protect the running
> task which has a longer slice until it becomes ineligible instead of a
> full slice in order to speedup the switch to other tasks until the task
> with the shortest slice is scheduled. This helps to the task to not wait
> too many full slices before running.
> 
> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> ---
>  kernel/sched/fair.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 601c67cff185..994fcf3ea702 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -1091,7 +1091,10 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
>  		slice = cfs_rq_min_slice(cfs_rq);
>  
>  	slice = min(slice, se->slice);
> -	if (vruntime != se->vruntime || slice != se->slice)
> +
> +	if (sched_feat(PREEMPT_SHORT) && slice < se->slice)
> +		vprot = avg_vruntime(cfs_rq);
> +	else if ((vruntime != se->vruntime) || (slice != se->slice))
>  		vprot = min_vruntime(vprot, vruntime + calc_delta_fair(slice, se));
>  
>  	se->vprot = vprot;

I am not entirely sure I understand this one.

avg_vruntime() could be ahead of se->deadline, esp for very short
slices. This would then extend protection beyond the one slice..

Aside from that, there are but two protect_slice() callers that matter:

 - pick_eevdf(): this already has a hard limit on avg_vruntime()

 - update_curr(): this will trigger preemption when reaching either
   ->deadline or ->vprot.


Also, the purpose of vprot is similar to the old min_gran, ensure any
task gets *some* time and avoid the degenerate case of endlessly
scheduling without 'any' real progress.

For EEVDF this happens when tasks get arbitrarily close to
avg_vruntime(). Eg, you have the two tasks A,B with A a virtual ns
before avg (and per necessity the other 1 ns after). You run A until its
just past B, find its not longer eligible, switch to B and do the same.
This then results in max frequency context switches and minimal actual
progress.

The thing that was supposed to stop this is vprot, but if you
consistently set vprot at avg_vruntime, this is effectively disabling
vprot. No?

Now, the conditions for this are such that this only happens for all
tasks not of the minimal slice length in the tree. So in order words,
you get spikes of high frequency scheduling just to burn vtime in order
to achieve eligibility for the earliest min_slice task, right?

So what you really want is not avg_vruntime() but the actual
se->vruntime of this earliest min_slice entity. Then we can simply run
whatever task and not get hit with high frequency scheduling, and still
achieve minimal latency for the waiting task.

Now, we don't actually have a convenient way to get this specific task,
but would something like so work?

  if (sched_feat(PREEMPT_SHORT) && slice != se->slice)
  	vprot = min_vruntime(vprot, __pick_root_entity(cfs_rq)->vruntime);

That is, we protect until the next earliest task becomes eligible.


Or did I go off the rails somewhere?

Re: [PATCH 2/6 v2] sched/eevdf: Take into account current's lag when updating slice protection

[Vincent Guittot]

On Tue, 16 Jun 2026 at 05:52, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>
> Hello Vincent,
>
> On 6/15/2026 9:54 PM, Vincent Guittot wrote:
> > @@ -1083,6 +1083,7 @@ struct sched_entity *__pick_first_entity(struct cfs_rq *cfs_rq)
> >   */
> >  static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >  {
> > +     u64 vruntime = min_vruntime(se->vruntime, avg_vruntime(cfs_rq));
>
> set_protect_slice() is only called for set_next_entity(... first = true)
> which has to come from the pick where the entity has to be eligible and
> behind the avg_vruntime(cfs_rq) right?
>
> Is there a case where se->vruntime is ahead of the average here that
> I'm missing?

The use of avg_vruntime was originaly introduced in patch 6 and while
refactoring the patchset, I wrongly found "better" to introduce its
usage in patch 2 and use same code pattern for both update and
set_protect_slice but obviously the min_vrutime is useless for
set_protect as se->vruntime is earlier than avg_vruntime because it is
eligible




>
> >       u64 slice = normalized_sysctl_sched_base_slice;
> >       u64 vprot = se->deadline;
> >
> > @@ -1090,8 +1091,8 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
> >               slice = cfs_rq_min_slice(cfs_rq);
> >
> >       slice = min(slice, se->slice);
> > -     if (slice != se->slice)
> > -             vprot = min_vruntime(vprot, se->vruntime + calc_delta_fair(slice, se));
> > +     if (vruntime != se->vruntime || slice != se->slice)
> > +             vprot = min_vruntime(vprot, vruntime + calc_delta_fair(slice, se));
> >
> >       se->vprot = vprot;
> >  }
> --
> Thanks and Regards,
> Prateek
>

Re: [PATCH 2/6 v2] sched/eevdf: Take into account current's lag when updating slice protection

[Vincent Guittot]

On Tue, 16 Jun 2026 at 11:30, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Mon, Jun 15, 2026 at 06:24:16PM +0200, Vincent Guittot wrote:
> > Take into account the lag of current task when setting the slice protection
> > in order to ensure that the absolute value of lags will remain in the
> > range [0 : slice+tick]
> > A task that already has a negative lag will see its protection reduced
> > whereas a task with positive lag will keep a full slice protection.
> >
> > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > ---
> >  kernel/sched/fair.c | 8 +++++---
> >  1 file changed, 5 insertions(+), 3 deletions(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 83bce5a04f3d..b8d5d9bcc014 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -1083,6 +1083,7 @@ struct sched_entity *__pick_first_entity(struct cfs_rq *cfs_rq)
> >   */
> >  static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >  {
> > +     u64 vruntime = min_vruntime(se->vruntime, avg_vruntime(cfs_rq));
> >       u64 slice = normalized_sysctl_sched_base_slice;
> >       u64 vprot = se->deadline;
> >
> > @@ -1090,8 +1091,8 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
> >               slice = cfs_rq_min_slice(cfs_rq);
> >
> >       slice = min(slice, se->slice);
> > -     if (slice != se->slice)
> > -             vprot = min_vruntime(vprot, se->vruntime + calc_delta_fair(slice, se));
> > +     if (vruntime != se->vruntime || slice != se->slice)
> > +             vprot = min_vruntime(vprot, vruntime + calc_delta_fair(slice, se));
> >
> >       se->vprot = vprot;
> >  }
>
> As already noted by Prateek, this doesn't seem to make much sense, since
> we just got selected by schedule(), we *must* be left of avg_vruntime(),
> otherwise we'd not be eligible and all that.
>
> > @@ -1099,8 +1100,9 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
> >  static inline void update_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >  {
> >       u64 slice = cfs_rq_min_slice(cfs_rq);
> > +     u64 vruntime = min_vruntime(se->vruntime, avg_vruntime(cfs_rq));
> >
> > -     se->vprot = min_vruntime(se->vprot, se->vruntime + calc_delta_fair(slice, se));
> > +     se->vprot = min_vruntime(se->vprot, vruntime + calc_delta_fair(slice, se));
> >  }
> >
> >  static inline bool protect_slice(struct sched_entity *se)
>
> So:
>
>  - set_protect_slice() is called at: set_next_task(.first = true),
>    eg, only when the task gets scheduled().
>
>  - set_protect_slice() takes se->deadline as the baseline, and (when
>    RUN_TO_PARITY) computes a shorter vprot [ min_slice vs slice ].
>
>  - update_protect_slice() is called upon (failed) wakeup preemption,
>    new tasks have been added and as such the goal is to re-compute the
>    min_slice and possibly reduce vprot.
>
> Right?

Yes

>
>
> And while update_protect_slice() would ideally use the original
> se->vruntime (as per set_next_task(.first = true) to compute any new
> (shorter) vprot, per its use of min_vruntime() it can not in fact end up
> with a vprot that is longer than the initial.
>
> Now, your change is to use min(se->vruntime, avg_vruntime()) to increase
> the chance of actually computing a shorter vprot. Still very much wrong,
> but possibly less wrong.
>
> Rather than taking avg_vruntime(), would it make sense to do something
> like:

avg_vruntime can move back if the newly enqueued task has a positive
lag, accounting for this prevents the lag from increasing further and
possibly exceeding the limit.

As an example,
TA starts to run with a 16ms slice and vprot is set to its deadline
TB wakes up just after with a lag of 15.9ms and a slice of 16ms
TB doesn't preempt TA because of the run to parity
TA will run for 16ms and TB's lag will go above slice + tick
Furthermore, if TC with a 8ms slice wakes up just after TB, it will
not preempt TA because TB has an earlier deadline. Using the time
already consumed by TA would not help further because it would let TA
with almost 8ms of protection.

>
>         slice = cfs_rq_min_slice(cfs_rq);
>         slice -= se->sum_exec_runtime - se->prev_sum_exec_runtime;
>
>         vprot = se->vruntime;
>         if (slice < 0)
>                 vprot -= calc_delta_fair(-slice, se);
>         else
>                 vprot += calc_delta_fair(slice, se);
>
>         se->vprot = min_vruntime(se->vprot, vprot);
>
> That is, reduce the slice with the time already ran.
>
> Now, this will go sideways in the unlikely case of renice, and possibly
> sched_change pattern (it seems we update prev_sum_exec_runtime for
> !first), but overall it might be a better approximation, no?



>

Re: [PATCH 4/6 v2] sched/eevdf: Cancel slice protection if short slice task is eligible

[Vincent Guittot]

On Tue, 16 Jun 2026 at 07:34, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>
> Hello Vincent,
>
> On 6/15/2026 9:54 PM, Vincent Guittot wrote:
> > @@ -9897,16 +9892,23 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
> >       if (!nse && cfs_rq->nr_queued)
> >               goto pick;
> >
> > +     /*
> > +      * If @p is eligible but not the next task to run then cancel protection
> > +      * to prevent large scheduling latency
> > +      */
> > +     if (preempt_action == PREEMPT_WAKEUP_SHORT && entity_eligible(cfs_rq, pse))
> > +             goto preempt;
>
> Can't we do this independent of the "pick" path if we are going to
> resched anyways?

This is for the case where the task is not the next one to be picked
(nse != pse) .i.e. pse is eligible, has a shorter slice than current
but doesn't have the earliest deadline

>
> I suppose it is for set_next_buddy() bits but, if nse == pse, we cancel
> the slice protection and resched anyways and the pick will naturally
> go to pse so does set_next_buddy() even matter?

We will not set_next_buddy in this case but only try to speed up the
scheduling of tasks with earlier deadlines.

>
> Instead, can PREEMPT_WAKEUP_SHORT path just do a set_preempt_buddy()
> beforehand to ensure the task with the smallest deadline is already
> the ->next and pick_next_entity() readily returns that?
>
> > +
> >       if (sched_feat(RUN_TO_PARITY))
> >               update_protect_slice(cfs_rq, se);
> >
> >       return;
> >
> >  preempt:
> > -     if (preempt_action == PREEMPT_WAKEUP_SHORT) {
> > -             cancel_protect_slice(se);
> > +     cancel_protect_slice(se);
> > +
> > +     if (preempt_action == PREEMPT_WAKEUP_SHORT && nse == pse)
> >               set_next_buddy(&p->se);
> > -     }
> >
> >       resched_curr_lazy(rq);
> >  }
>
> --
> Thanks and Regards,
> Prateek
>

Re: [PATCH 1/6 v2] sched/fair: Set next buddy for preempt short

[Vincent Guittot]

On Tue, 16 Jun 2026 at 10:52, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Mon, Jun 15, 2026 at 06:24:15PM +0200, Vincent Guittot wrote:
> > If a shorter slice task can preempt current at wakeup, we make sure that
> > the decision will not be overwritten in between by setting the task as the
> > next buddy. This still implies that the waking task remains eligible when
> > the scheduler will actually pick the next task to run.
> >
> > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > ---
> >  kernel/sched/fair.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index d78467ec6ee1..83bce5a04f3d 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -9903,7 +9903,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
> >  preempt:
> >       if (preempt_action == PREEMPT_WAKEUP_SHORT) {
> >               cancel_protect_slice(se);
> > -             clear_buddies(cfs_rq, se);
> > +             set_next_buddy(&p->se);
> >       }
>
> Should this not be something along the lines of set_preempt_buddy()?

The test with pick_next_entity() ensures that p is the eligible and
before any other task which seems stronger than set_preempt_buddy()
which only look at deadline but not eligibility

Re: [PATCH 3/6 v2] sched/eevdf: Update slice protection even when resched is already set

[Vincent Guittot]

On Tue, 16 Jun 2026 at 11:37, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Mon, Jun 15, 2026 at 06:24:17PM +0200, Vincent Guittot wrote:
> > Even if resched is already set, we might want to update or even cancel
> > the slice protection and ensure that the newly waking task will be the
> > next one to run.
> >
> > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > ---
> >  kernel/sched/fair.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index b8d5d9bcc014..b98d67650a98 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -9805,7 +9805,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
> >        * prevents us from potentially nominating it as a false LAST_BUDDY
> >        * below.
> >        */
> > -     if (test_tsk_need_resched(rq->curr))
> > +     if (!sched_feat(PREEMPT_SHORT) && test_tsk_need_resched(rq->curr))
> >               return;
> >
> >       if (!sched_feat(WAKEUP_PREEMPTION))
>
> Fair enough.
>
> It did get me thinking about what we would want for SCHED_BATCH tasks. I
> think we might want to make that RUN_TO_PARITY check (which selects
> min_slice) conditional on normal_policy(). But that's another patch for
> another day (and much easier after we land that flat thing).

Patch 5 forces the update even for sched idle and batch because even
if they don't preempt current, we don't want them to increase their
lag to an unmanageable level by the time a short slice task wakes up.
Something similar to my example in patch 2
>
>

Re: [PATCH 0/6 v2] sched/eevdf: Improve scheduling latency of short slice task

[Vincent Guittot]

On Tue, 16 Jun 2026 at 09:44, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>
> Hello Vincent,
>
> On 6/15/2026 9:54 PM, Vincent Guittot wrote:
> > This series continues to improve the scheduling latency of tasks with
> > shorter slice duration by mainly canceling, updating or minimizing the
> > protection of the running tasks when appropriate.
> >
> > Benchmarks, like hackbench, haven't seen any noticeable performance
> > differences with this patchset (The default 2.8ms slice has been used for
> > testing performance regressions)
>
> I've left a few comments on the thread but for the vanilla runs, I too
> can confirm there aren't any without any slice tuning.
>
> Following are results from a dual socket 4th Generation EPYC system
> (2 x 128C/256T) with the series applied on top of
> "sched-core-2026-06-14":
>
>   ==================================================================
>   Test          : hackbench
>   Units         : Normalized time in seconds
>   Interpretation: Lower is better
>   Statistic     : AMean
>   ==================================================================
>   Case:           tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
>    1-groups     1.00 [ -0.00]( 9.66)     0.86 [ 14.32](14.00)
>    2-groups     1.00 [ -0.00]( 9.22)     1.02 [ -1.78](10.45)
>    4-groups     1.00 [ -0.00]( 2.14)     0.98 [  2.33]( 1.99)
>    8-groups     1.00 [ -0.00]( 2.80)     0.97 [  2.88]( 2.93)
>   16-groups     1.00 [ -0.00]( 5.54)     1.00 [  0.49]( 2.58)
>
>
>   ==================================================================
>   Test          : tbench
>   Units         : Normalized throughput
>   Interpretation: Higher is better
>   Statistic     : AMean
>   ==================================================================
>   Clients:    tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
>       1     1.00 [  0.00]( 0.03)     1.00 [  0.36]( 0.20)
>       2     1.00 [  0.00]( 0.32)     1.00 [  0.09]( 0.14)
>       4     1.00 [  0.00]( 0.34)     1.00 [  0.39]( 0.28)
>       8     1.00 [  0.00]( 0.24)     1.00 [  0.01]( 0.24)
>      16     1.00 [  0.00]( 0.45)     1.00 [  0.01]( 0.47)
>      32     1.00 [  0.00]( 0.58)     1.01 [  0.75]( 0.30)
>      64     1.00 [  0.00]( 0.81)     1.02 [  1.62]( 0.62)
>     128     1.00 [  0.00]( 0.53)     1.03 [  3.24]( 0.25)
>     256     1.00 [  0.00]( 0.30)     1.00 [  0.39]( 0.26)
>     512     1.00 [  0.00]( 3.73)     1.01 [  1.47]( 1.13)
>    1024     1.00 [  0.00]( 0.23)     1.00 [ -0.10]( 0.37)
>    2048     1.00 [  0.00]( 0.14)     1.00 [  0.29]( 0.19)
>
>
>   ==================================================================
>   Test          : stream-10
>   Units         : Normalized Bandwidth, MB/s
>   Interpretation: Higher is better
>   Statistic     : HMean
>   ==================================================================
>   Test:       tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
>    Copy     1.00 [  0.00]( 0.66)     0.99 [ -0.70]( 1.66)
>   Scale     1.00 [  0.00]( 0.89)     0.99 [ -0.77]( 1.52)
>     Add     1.00 [  0.00]( 0.73)     1.00 [ -0.34]( 1.31)
>   Triad     1.00 [  0.00]( 0.70)     0.99 [ -0.52]( 1.24)
>
>
>   ==================================================================
>   Test          : stream-100
>   Units         : Normalized Bandwidth, MB/s
>   Interpretation: Higher is better
>   Statistic     : HMean
>   ==================================================================
>   Test:       tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
>    Copy     1.00 [  0.00]( 0.32)     1.00 [  0.07]( 0.36)
>   Scale     1.00 [  0.00]( 0.26)     1.00 [ -0.00]( 0.45)
>     Add     1.00 [  0.00]( 0.29)     1.00 [ -0.05]( 0.39)
>   Triad     1.00 [  0.00]( 0.27)     1.00 [ -0.05]( 0.37)
>
>
>   ==================================================================
>   Test          : netperf
>   Units         : Normalized Througput
>   Interpretation: Higher is better
>   Statistic     : AMean
>   ==================================================================
>   Clients:           tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
>      1-clients     1.00 [  0.00]( 0.10)     1.00 [  0.02]( 0.19)
>      2-clients     1.00 [  0.00]( 0.29)     1.00 [  0.02]( 0.18)
>      4-clients     1.00 [  0.00]( 0.36)     1.00 [ -0.01]( 0.23)
>      8-clients     1.00 [  0.00]( 0.32)     1.00 [  0.04]( 0.22)
>     16-clients     1.00 [  0.00]( 0.24)     1.00 [  0.09]( 0.22)
>     32-clients     1.00 [  0.00]( 0.42)     1.00 [  0.30]( 0.33)
>     64-clients     1.00 [  0.00]( 0.94)     1.00 [  0.48]( 0.67)
>    128-clients     1.00 [  0.00]( 1.10)     1.01 [  0.77]( 1.31)
>    256-clients     1.00 [  0.00]( 1.06)     1.02 [  2.02]( 1.18)
>    512-clients     1.00 [  0.00]( 4.68)     0.99 [ -1.14]( 5.63)
>    768-clients     1.00 [  0.00](34.35)     0.99 [ -1.00](34.84)
>   1024-clients     1.00 [  0.00](42.76)     0.99 [ -0.81](45.77)
>
>
>   ==================================================================
>   Test          : schbench
>   Units         : Normalized 99th percentile latency in us
>   Interpretation: Lower is better
>   Statistic     : Median
>   ==================================================================
>   #workers:  tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
>      1     1.00 [ -0.00](18.94)     0.39 [ 61.36]( 5.88)
>      2     1.00 [ -0.00]( 1.67)     0.91 [  8.57]( 6.64)
>      4     1.00 [ -0.00]( 9.79)     0.84 [ 16.22]( 7.78)
>      8     1.00 [ -0.00]( 2.27)     0.89 [ 11.36](10.54)
>     16     1.00 [ -0.00]( 0.00)     0.98 [  1.79]( 4.10)
>     32     1.00 [ -0.00]( 1.92)     1.01 [ -1.25]( 0.72)
>     64     1.00 [ -0.00]( 1.19)     1.01 [ -0.78]( 1.18)
>    128     1.00 [ -0.00]( 0.67)     1.01 [ -1.32]( 0.25)
>    256     1.00 [ -0.00]( 0.46)     1.03 [ -3.08]( 4.37)
>    512     1.00 [ -0.00]( 0.33)     1.01 [ -0.66]( 0.38)
>    768     1.00 [ -0.00]( 4.69)     1.02 [ -1.55](10.18)
>   1024     1.00 [ -0.00]( 2.71)     1.00 [ -0.00]( 4.43)
>
>
>   ==================================================================
>   Test          : new-schbench-requests-per-second
>   Units         : Normalized Requests per second
>   Interpretation: Higher is better
>   Statistic     : Median
>   ==================================================================
>   #workers:  tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
>      1     1.00 [  0.00]( 0.15)     0.99 [ -0.59]( 0.15)
>      2     1.00 [  0.00]( 0.00)     0.99 [ -0.59]( 0.15)
>      4     1.00 [  0.00]( 0.00)     0.99 [ -0.88]( 0.31)
>      8     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.00)
>     16     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.00)
>     32     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.15)
>     64     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)
>    128     1.00 [  0.00](12.53)     0.98 [ -1.77](15.94)
>    256     1.00 [  0.00]( 0.15)     0.99 [ -0.85]( 0.39)
>    512     1.00 [  0.00]( 0.84)     1.00 [  0.00]( 0.84)
>    768     1.00 [  0.00]( 2.05)     0.99 [ -0.94]( 1.94)
>   1024     1.00 [  0.00]( 2.90)     1.01 [  1.35]( 2.18)
>
>
>   ==================================================================
>   Test          : new-schbench-wakeup-latency
>   Units         : Normalized 99th percentile latency in us
>   Interpretation: Lower is better
>   Statistic     : Median
>   ==================================================================
>   #workers:  tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
>      1     1.00 [ -0.00](12.99)     1.47 [-46.67](27.72)
>      2     1.00 [ -0.00]( 4.08)     0.85 [ 15.38]( 0.00)
>      4     1.00 [ -0.00]( 0.00)     0.82 [ 18.18]( 0.00)
>      8     1.00 [ -0.00]( 0.00)     1.27 [-27.27]( 3.78)
>     16     1.00 [ -0.00]( 4.56)     1.27 [-27.27]( 0.00)
>     32     1.00 [ -0.00]( 0.00)     1.00 [ -0.00]( 4.56)
>     64     1.00 [ -0.00]( 5.00)     1.00 [ -0.00]( 5.00)
>    128     1.00 [ -0.00]( 7.45)     1.25 [-25.00](14.68)
>    256     1.00 [ -0.00]( 2.70)     0.96 [  4.48]( 8.12)
>    512     1.00 [ -0.00]( 0.00)     1.00 [ -0.00]( 0.00)
>    768     1.00 [ -0.00]( 1.66)     1.01 [ -1.47]( 2.52)
>   1024     1.00 [ -0.00]( 3.32)     1.01 [ -0.59]( 0.66)
>
>   Note: The absolute numbers are very small until 256 threads (~10-15us)
>   which may causes a small variation to appear as a large regression.
>
>   ==================================================================
>   Test          : new-schbench-request-latency
>   Units         : Normalized 99th percentile latency in us
>   Interpretation: Lower is better
>   Statistic     : Median
>   ==================================================================
>   #workers:  tip[pct imp](CV)    preempt_short_opt[pct imp](CV)
>      1     1.00 [ -0.00]( 0.14)     1.01 [ -1.06]( 0.27)
>      2     1.00 [ -0.00]( 0.14)     1.02 [ -1.60]( 0.23)
>      4     1.00 [ -0.00]( 0.00)     1.05 [ -4.53]( 1.73)
>      8     1.00 [ -0.00]( 0.14)     1.01 [ -0.53]( 0.14)
>     16     1.00 [ -0.00]( 1.49)     1.00 [ -0.26]( 1.23)
>     32     1.00 [ -0.00]( 0.89)     1.01 [ -0.79]( 0.00)
>     64     1.00 [ -0.00]( 1.43)     1.00 [ -0.26]( 0.98)
>    128     1.00 [ -0.00]( 2.78)     1.01 [ -1.18]( 4.09)
>    256     1.00 [ -0.00]( 0.13)     1.00 [ -0.25]( 0.26)
>    512     1.00 [ -0.00]( 6.72)     1.02 [ -2.20]( 5.45)
>    768     1.00 [ -0.00]( 3.42)     1.01 [ -0.52]( 4.21)
>   1024     1.00 [ -0.00]( 4.37)     1.01 [ -1.19]( 2.15)
>
> >
> > Several use cases has been used to testing the scheduling latency of short
> > slice tasks:
> > - cyclictest with a 3777us period and a 8ms slice alone
> > - cyclictest with a 3777us period and a 8ms slice. 2xNR_CPUS rt-app
> >   tasks that run (8177us) and sleep (17777us) with a 16ms slice.
> > - cyclictest with a 3777us period and a 8ms slice. Hackbench with
> >   1 group using thread and pipe and a 16ms slice.
>
> I'll go check those configurations next and report back if I find
> anything out of the ordinary. Feel free to include:
>
> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>

Thanks for the tests

>
> --
> Thanks and Regards,
> Prateek
>

Re: [PATCH 6/6 v2] sched/eevdf: Speedup short slice task scheduling

[Vincent Guittot]

On Tue, 16 Jun 2026 at 12:57, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Mon, Jun 15, 2026 at 06:24:20PM +0200, Vincent Guittot wrote:
> > When a task with a shorter slice is enqueued, we protect the running
> > task which has a longer slice until it becomes ineligible instead of a
> > full slice in order to speedup the switch to other tasks until the task
> > with the shortest slice is scheduled. This helps to the task to not wait
> > too many full slices before running.
> >
> > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > ---
> >  kernel/sched/fair.c | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 601c67cff185..994fcf3ea702 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -1091,7 +1091,10 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
> >               slice = cfs_rq_min_slice(cfs_rq);
> >
> >       slice = min(slice, se->slice);
> > -     if (vruntime != se->vruntime || slice != se->slice)
> > +
> > +     if (sched_feat(PREEMPT_SHORT) && slice < se->slice)
> > +             vprot = avg_vruntime(cfs_rq);
> > +     else if ((vruntime != se->vruntime) || (slice != se->slice))
> >               vprot = min_vruntime(vprot, vruntime + calc_delta_fair(slice, se));
> >
> >       se->vprot = vprot;
>
> I am not entirely sure I understand this one.
>
> avg_vruntime() could be ahead of se->deadline, esp for very short
> slices. This would then extend protection beyond the one slice..

Fair enough, I haven't checked that we were not extending the vprot
(will add it). I don't think this happens that often, particularly
because this only occurs when a task with a shorter slice is enqueued
waiting to run on the cpu and we expect the lag to be shorter than the
slice

>
> Aside from that, there are but two protect_slice() callers that matter:
>
>  - pick_eevdf(): this already has a hard limit on avg_vruntime()
>
>  - update_curr(): this will trigger preemption when reaching either
>    ->deadline or ->vprot.
>
>
> Also, the purpose of vprot is similar to the old min_gran, ensure any
> task gets *some* time and avoid the degenerate case of endlessly
> scheduling without 'any' real progress.
>
> For EEVDF this happens when tasks get arbitrarily close to
> avg_vruntime(). Eg, you have the two tasks A,B with A a virtual ns
> before avg (and per necessity the other 1 ns after). You run A until its
> just past B, find its not longer eligible, switch to B and do the same.
> This then results in max frequency context switches and minimal actual
> progress.
>
> The thing that was supposed to stop this is vprot, but if you
> consistently set vprot at avg_vruntime, this is effectively disabling
> vprot. No?

Yes, that's why it only happens when a shorter slice task is enqueued.
Other tasks that will run before, should have a lag around their slice
when this happens
Note that I'm not using sched hrtick so once picked se will run for a
tick (unless another wakeup happen)

>
> Now, the conditions for this are such that this only happens for all
> tasks not of the minimal slice length in the tree. So in order words,
> you get spikes of high frequency scheduling just to burn vtime in order
> to achieve eligibility for the earliest min_slice task, right?

Not sure what you mean by high frequency scheduling but each task
should run once and just long enough to become ineligible or eligible
but after the short slice task because of deadline update

>
> So what you really want is not avg_vruntime() but the actual
> se->vruntime of this earliest min_slice entity. Then we can simply run
> whatever task and not get hit with high frequency scheduling, and still
> achieve minimal latency for the waiting task.
>
> Now, we don't actually have a convenient way to get this specific task,
> but would something like so work?
>
>   if (sched_feat(PREEMPT_SHORT) && slice != se->slice)
>         vprot = min_vruntime(vprot, __pick_root_entity(cfs_rq)->vruntime);
>
> That is, we protect until the next earliest task becomes eligible.

I probably need to think a bit more about this but if you have several
tasks eligible with very close vruntime, will not this make even
smaller running step because __pick_root_entity(cfs_rq)->vruntime will
be earlier than avg_vruntime().

>
>
> Or did I go off the rails somewhere?

Re: [PATCH 6/6 v2] sched/eevdf: Speedup short slice task scheduling

[Vincent Guittot]

On Tue, 16 Jun 2026 at 17:18, Vincent Guittot
<vincent.guittot@linaro.org> wrote:
>
> On Tue, 16 Jun 2026 at 12:57, Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Mon, Jun 15, 2026 at 06:24:20PM +0200, Vincent Guittot wrote:
> > > When a task with a shorter slice is enqueued, we protect the running
> > > task which has a longer slice until it becomes ineligible instead of a
> > > full slice in order to speedup the switch to other tasks until the task
> > > with the shortest slice is scheduled. This helps to the task to not wait
> > > too many full slices before running.
> > >
> > > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > > ---
> > >  kernel/sched/fair.c | 5 ++++-
> > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index 601c67cff185..994fcf3ea702 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -1091,7 +1091,10 @@ static inline void set_protect_slice(struct cfs_rq *cfs_rq, struct sched_entity
> > >               slice = cfs_rq_min_slice(cfs_rq);
> > >
> > >       slice = min(slice, se->slice);
> > > -     if (vruntime != se->vruntime || slice != se->slice)
> > > +
> > > +     if (sched_feat(PREEMPT_SHORT) && slice < se->slice)
> > > +             vprot = avg_vruntime(cfs_rq);
> > > +     else if ((vruntime != se->vruntime) || (slice != se->slice))
> > >               vprot = min_vruntime(vprot, vruntime + calc_delta_fair(slice, se));
> > >
> > >       se->vprot = vprot;
> >
> > I am not entirely sure I understand this one.
> >
> > avg_vruntime() could be ahead of se->deadline, esp for very short
> > slices. This would then extend protection beyond the one slice..
>
> Fair enough, I haven't checked that we were not extending the vprot
> (will add it). I don't think this happens that often, particularly
> because this only occurs when a task with a shorter slice is enqueued
> waiting to run on the cpu and we expect the lag to be shorter than the
> slice
>
> >
> > Aside from that, there are but two protect_slice() callers that matter:
> >
> >  - pick_eevdf(): this already has a hard limit on avg_vruntime()
> >
> >  - update_curr(): this will trigger preemption when reaching either
> >    ->deadline or ->vprot.
> >
> >
> > Also, the purpose of vprot is similar to the old min_gran, ensure any
> > task gets *some* time and avoid the degenerate case of endlessly
> > scheduling without 'any' real progress.
> >
> > For EEVDF this happens when tasks get arbitrarily close to
> > avg_vruntime(). Eg, you have the two tasks A,B with A a virtual ns
> > before avg (and per necessity the other 1 ns after). You run A until its
> > just past B, find its not longer eligible, switch to B and do the same.
> > This then results in max frequency context switches and minimal actual
> > progress.
> >
> > The thing that was supposed to stop this is vprot, but if you
> > consistently set vprot at avg_vruntime, this is effectively disabling
> > vprot. No?
>
> Yes, that's why it only happens when a shorter slice task is enqueued.
> Other tasks that will run before, should have a lag around their slice
> when this happens
> Note that I'm not using sched hrtick so once picked se will run for a
> tick (unless another wakeup happen)
>
> >
> > Now, the conditions for this are such that this only happens for all
> > tasks not of the minimal slice length in the tree. So in order words,
> > you get spikes of high frequency scheduling just to burn vtime in order
> > to achieve eligibility for the earliest min_slice task, right?
>
> Not sure what you mean by high frequency scheduling but each task
> should run once and just long enough to become ineligible or eligible
> but after the short slice task because of deadline update
>
> >
> > So what you really want is not avg_vruntime() but the actual
> > se->vruntime of this earliest min_slice entity. Then we can simply run
> > whatever task and not get hit with high frequency scheduling, and still
> > achieve minimal latency for the waiting task.
> >
> > Now, we don't actually have a convenient way to get this specific task,
> > but would something like so work?
> >
> >   if (sched_feat(PREEMPT_SHORT) && slice != se->slice)
> >         vprot = min_vruntime(vprot, __pick_root_entity(cfs_rq)->vruntime);
> >
> > That is, we protect until the next earliest task becomes eligible.
>
> I probably need to think a bit more about this but if you have several
> tasks eligible with very close vruntime, will not this make even
> smaller running step because __pick_root_entity(cfs_rq)->vruntime will
> be earlier than avg_vruntime().

When an entity with a shorter slice is enqueued, we want to set vprot
to a value that makes next task to run ineligible so when we
reschedule, we can move to the next entity in the rb_tree until the
entity with the shorter slice runs.
This means we want the average vruntime when next's key is 0 (slightly
above 0), not the current avg_vruntime.
I will try this


>
> >
> >
> > Or did I go off the rails somewhere?