[PATCH v3 0/7] sched: Flatten the pick

[PATCH v3 0/7] sched: Flatten the pick

[Peter Zijlstra]

Hi!

New version, same story [1]. TL;DR:

 - Adds new cgroup_mode knob and implements new policies to address the
   hierarchy level weight mismatch.

 - Builds upon that base to create a flat / single runqueue scheduler where the
   cgroup hierarchy is expressed through dynamic weight management.

I'm hoping to be able to merge these patches early in the next cycle (after
7.2-rc1).

Random benchmark:

Game vs 'for ((i=0; i<8; i++)) do nice ./spin.sh; done':

  Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper)
  Intel Core i7-2600K
  AMD Radeon RX 580

  Shadows Awakening (GOG)

	  default slice(*)

  FPS min   4.0   29.0
      avg  47.5   59.2
      max  83.7   83.7

  FT  min   9.3   10.2
      avg  34.0   17.0
      max 121.2   30.0

  FPS (Frames Per Second)
  FT  (FrameTime)

  [*] Command prefix: 'chrt -o --sched-runtime 100000 0'


Changes since v2:

 - merged debug and prep patches
 - fixed update_entity_lag() on dequeue (Vincent)
 - fixed throttle vs tick (Prateek)
 - fixed wakeup_preempt_fair()
 - rebased on tip/sched/core
 - rewritten cgroup_mode changelogs
 - reworked cgroup_mode concur
 - added cgroup_mode tasks
 - changed default cgroup_mode


[1] - https://lore.kernel.org/r/20260511113104.563854162@infradead.org

Can also be had:

  git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat

 include/linux/cpuset.h |    6 
 include/linux/sched.h  |    1 
 kernel/cgroup/cpuset.c |   15 
 kernel/sched/core.c    |    5 
 kernel/sched/debug.c   |   89 ++++
 kernel/sched/fair.c    |  943 ++++++++++++++++++++++++-------------------------
 kernel/sched/pelt.c    |    6 
 kernel/sched/sched.h   |   30 -
 8 files changed, 607 insertions(+), 488 deletions(-)

[PATCH v3 1/7] sched/fair: Add cgroup_mode switch

[Peter Zijlstra]

The effective task weight (W_t') for a task in cgroup g on CPU n is given by:

                                 W_t
	W_t' = W_g * F_g_n * ----------
                             \Sum W_t_n

Where W_g is the group's weight (cpu.weight), F_g_n is the fraction of the
group weight for CPU n and W_t/W is the relative weight of this task against
all other tasks in the same group on the same CPU.

Furthermore, this makes:

                \Sum W_t_n
	F_g_n = ----------
	         \Sum W_t

The fraction of weight inside the group of CPU n against the whole group.

The problem is with F_g_n, the primary goal of this fraction is to make sure
that the relative weight of tasks, when distributed over CPUs is maintained.
For example, consider 4 (equal weight) tasks and 2 CPUs with a 1:3
distribution, then if F_g_n would simply be 1 (no weight re-distribution) the
effective relative weights (W_t') of the tasks in our group would be:

	CPU0	CPU1
        W_g     W_g/3
	        W_g/3
		W_g/3

IOW, the lucky task on CPU0 would get an equal amount of weight as all 3 tasks
on CPU1 combined. However, with the weight redistribution, this becomes:

	CPU0	CPU1
        W_g/4   W_g/4
	        W_g/4
		W_g/4

All tasks are equal weight (as intended). However, as is already evident from
this example, the more CPUs you add, the smaller F_g_n becomes, which creates a
disparity against tasks not in our group.

Specifically:

	avg(F_g_n) ~ 1/N

This leads to a weight mismatch in the hierarchy. IOW tasks cannot compete
fairly across hierarchy levels.

*Notably*, what is meant by avg(F_g_n) being proportional to 1/N is that when
there are at least N runnable tasks, the average of this fraction tends to 1/N.

For a hierarchy of depth d, this gets even worse, since that gets terms on the
order of:

	avg(F_g_n)^d ~ 1/(N^d)

Given fixed point arithmetic, this also leads to numerical trouble.

However, the meaning of "cpu.weight" is simple and intiutive: the total weight
of the cgroup. But as explored above, there is deception in this simplicity.

Prepare to add a few alternative methods for distributing weight.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/debug.c |   74 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)

--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -633,6 +633,76 @@ static void debugfs_fair_server_init(voi
 	}
 }
 
+#ifdef CONFIG_FAIR_GROUP_SCHED
+static int cgroup_mode = 0;
+
+static const char *cgroup_mode_str[] = {
+	"smp",
+};
+
+static int sched_cgroup_mode(const char *str)
+{
+	for (int i = 0; i < ARRAY_SIZE(cgroup_mode_str); i++) {
+		if (!strcmp(str, cgroup_mode_str[i]))
+			return i;
+	}
+	return -EINVAL;
+}
+
+static ssize_t sched_cgroup_write(struct file *filp, const char __user *ubuf,
+				   size_t cnt, loff_t *ppos)
+{
+	char buf[16];
+	int mode;
+
+	if (cnt > 15)
+		cnt = 15;
+
+	if (copy_from_user(buf, ubuf, cnt))
+		return -EFAULT;
+
+	buf[cnt] = 0;
+	mode = sched_cgroup_mode(strstrip(buf));
+	if (mode < 0)
+		return mode;
+
+	WRITE_ONCE(cgroup_mode, mode);
+
+	*ppos += cnt;
+	return cnt;
+}
+
+static int sched_cgroup_show(struct seq_file *m, void *v)
+{
+	int mode = READ_ONCE(cgroup_mode);
+
+	for (int i = 0; i < ARRAY_SIZE(cgroup_mode_str); i++) {
+		if (mode == i)
+			seq_puts(m, "(");
+		seq_puts(m, cgroup_mode_str[i]);
+		if (mode == i)
+			seq_puts(m, ")");
+
+		seq_puts(m, " ");
+	}
+	seq_puts(m, "\n");
+	return 0;
+}
+
+static int sched_cgroup_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, sched_cgroup_show, NULL);
+}
+
+static const struct file_operations sched_cgroup_fops = {
+	.open		= sched_cgroup_open,
+	.write		= sched_cgroup_write,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+#endif
+
 static __init int sched_init_debug(void)
 {
 	struct dentry __maybe_unused *numa, *llc;
@@ -686,6 +756,10 @@ static __init int sched_init_debug(void)
 
 	debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);
 
+#ifdef CONFIG_FAIR_GROUP_SCHED
+	debugfs_create_file("cgroup_mode", 0644, debugfs_sched, NULL, &sched_cgroup_fops);
+#endif
+
 	debugfs_fair_server_init();
 #ifdef CONFIG_SCHED_CLASS_EXT
 	debugfs_ext_server_init();

[tip: sched/core] sched/fair: Add cgroup_mode switch

[tip-bot2 for Peter Zijlstra]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     4161cb2d867b2210b447afc4a941ab4b96e1bd13
Gitweb:        https://git.kernel.org/tip/4161cb2d867b2210b447afc4a941ab4b96e1bd13
Author:        Peter Zijlstra <peterz@infradead.org>
AuthorDate:    Thu, 12 Mar 2026 14:48:44 +01:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 30 Jun 2026 10:56:51 +02:00

sched/fair: Add cgroup_mode switch

The effective task weight (W_t') for a task in cgroup g on CPU n is given by:

                                 W_t
	W_t' = W_g * F_g_n * ----------
                             \Sum W_t_n

Where W_g is the group's weight (cpu.weight), F_g_n is the fraction of the
group weight for CPU n and W_t/W is the relative weight of this task against
all other tasks in the same group on the same CPU.

Furthermore, this makes:

                \Sum W_t_n
	F_g_n = ----------
	         \Sum W_t

The fraction of weight inside the group of CPU n against the whole group.

The problem is with F_g_n, the primary goal of this fraction is to make sure
that the relative weight of tasks, when distributed over CPUs is maintained.
For example, consider 4 (equal weight) tasks and 2 CPUs with a 1:3
distribution, then if F_g_n would simply be 1 (no weight re-distribution) the
effective relative weights (W_t') of the tasks in our group would be:

	CPU0	CPU1
        W_g     W_g/3
	        W_g/3
		W_g/3

IOW, the lucky task on CPU0 would get an equal amount of weight as all 3 tasks
on CPU1 combined. However, with the weight redistribution, this becomes:

	CPU0	CPU1
        W_g/4   W_g/4
	        W_g/4
		W_g/4

All tasks are equal weight (as intended). However, as is already evident from
this example, the more CPUs you add, the smaller F_g_n becomes, which creates a
disparity against tasks not in our group.

Specifically:

	avg(F_g_n) ~ 1/N

This leads to a weight mismatch in the hierarchy. IOW tasks cannot compete
fairly across hierarchy levels.

*Notably*, what is meant by avg(F_g_n) being proportional to 1/N is that when
there are at least N runnable tasks, the average of this fraction tends to 1/N.

For a hierarchy of depth d, this gets even worse, since that gets terms on the
order of:

	avg(F_g_n)^d ~ 1/(N^d)

Given fixed point arithmetic, this also leads to numerical trouble.

However, the meaning of "cpu.weight" is simple and intiutive: the total weight
of the cgroup. But as explored above, there is deception in this simplicity.

Prepare to add a few alternative methods for distributing weight.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.338602724%40infradead.org
---
 kernel/sched/debug.c | 74 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 74 insertions(+)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 40584b2..507e486 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -633,6 +633,76 @@ static void debugfs_fair_server_init(void)
 	}
 }
 
+#ifdef CONFIG_FAIR_GROUP_SCHED
+static int cgroup_mode = 0;
+
+static const char *cgroup_mode_str[] = {
+	"smp",
+};
+
+static int sched_cgroup_mode(const char *str)
+{
+	for (int i = 0; i < ARRAY_SIZE(cgroup_mode_str); i++) {
+		if (!strcmp(str, cgroup_mode_str[i]))
+			return i;
+	}
+	return -EINVAL;
+}
+
+static ssize_t sched_cgroup_write(struct file *filp, const char __user *ubuf,
+				   size_t cnt, loff_t *ppos)
+{
+	char buf[16];
+	int mode;
+
+	if (cnt > 15)
+		cnt = 15;
+
+	if (copy_from_user(buf, ubuf, cnt))
+		return -EFAULT;
+
+	buf[cnt] = 0;
+	mode = sched_cgroup_mode(strstrip(buf));
+	if (mode < 0)
+		return mode;
+
+	WRITE_ONCE(cgroup_mode, mode);
+
+	*ppos += cnt;
+	return cnt;
+}
+
+static int sched_cgroup_show(struct seq_file *m, void *v)
+{
+	int mode = READ_ONCE(cgroup_mode);
+
+	for (int i = 0; i < ARRAY_SIZE(cgroup_mode_str); i++) {
+		if (mode == i)
+			seq_puts(m, "(");
+		seq_puts(m, cgroup_mode_str[i]);
+		if (mode == i)
+			seq_puts(m, ")");
+
+		seq_puts(m, " ");
+	}
+	seq_puts(m, "\n");
+	return 0;
+}
+
+static int sched_cgroup_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, sched_cgroup_show, NULL);
+}
+
+static const struct file_operations sched_cgroup_fops = {
+	.open		= sched_cgroup_open,
+	.write		= sched_cgroup_write,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+#endif
+
 static __init int sched_init_debug(void)
 {
 	struct dentry __maybe_unused *numa, *llc;
@@ -686,6 +756,10 @@ static __init int sched_init_debug(void)
 
 	debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);
 
+#ifdef CONFIG_FAIR_GROUP_SCHED
+	debugfs_create_file("cgroup_mode", 0644, debugfs_sched, NULL, &sched_cgroup_fops);
+#endif
+
 	debugfs_fair_server_init();
 #ifdef CONFIG_SCHED_CLASS_EXT
 	debugfs_ext_server_init();

[PATCH v3 2/7] sched/fair: Add cgroup_mode: up

[Peter Zijlstra]

Instead of calculating the proportional fraction of the group weight for each
CPU, just give each CPU the full measure, ignoring these pesky SMP problems.

This makes the SMP cgroup fraction (F_g_n) equal to 1, and ensures a single
task in a cgroup competes on equal footing to a task in a level above.

However, as already explored, this is not a very good policy because it gets
the SMP weight distribution wrong. Included for completeness.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/debug.c |    5 ++++-
 kernel/sched/fair.c  |   31 +++++++++++++++++++++++++++++--
 kernel/sched/sched.h |    1 +
 3 files changed, 34 insertions(+), 3 deletions(-)

--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -271,6 +271,7 @@ static ssize_t sched_dynamic_write(struc
 	if (mode < 0)
 		return mode;
 
+	__sched_cgroup_mode_update(mode);
 	sched_dynamic_update(mode);
 
 	*ppos += cnt;
@@ -634,9 +635,11 @@ static void debugfs_fair_server_init(voi
 }
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-static int cgroup_mode = 0;
+static int cgroup_mode = 1;
 
+/* See __sched_cgroup_mode_update(). */
 static const char *cgroup_mode_str[] = {
+	"up",
 	"smp",
 };
 
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -38,6 +38,7 @@
 #include <linux/sched/isolation.h>
 #include <linux/sched/nohz.h>
 #include <linux/sched/prio.h>
+#include <linux/static_call.h>
 
 #include <linux/cpuidle.h>
 #include <linux/interrupt.h>
@@ -4800,7 +4801,7 @@ static inline int throttled_hierarchy(st
  *
  * hence icky!
  */
-static long calc_group_shares(struct cfs_rq *cfs_rq)
+static long calc_smp(struct cfs_rq *cfs_rq)
 {
 	long tg_weight, tg_shares, load, shares;
 	struct task_group *tg = cfs_rq->tg;
@@ -4835,6 +4836,32 @@ static long calc_group_shares(struct cfs
 }
 
 /*
+ * Ignore this pesky SMP stuff, use (4).
+ */
+static long calc_up_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	return READ_ONCE(tg->shares);
+}
+
+DEFINE_STATIC_CALL(calc_group_shares, calc_smp_shares);
+
+void __sched_cgroup_mode_update(int mode)
+{
+	long (*func)(struct cfs_rq *);
+	switch (mode) {
+	case 0:
+		func = &calc_up_shares;
+		break;
+	case 1:
+	default:
+		func = &calc_smp_shares;
+		break;
+	}
+	static_call_update(calc_group_shares, func);
+}
+
+/*
  * Recomputes the group entity based on the current state of its group
  * runqueue.
  */
@@ -4850,7 +4877,7 @@ static void update_cfs_group(struct sche
 	if (!gcfs_rq || !gcfs_rq->load.weight)
 		return;
 
-	shares = calc_group_shares(gcfs_rq);
+	shares = static_call(calc_group_shares)(gcfs_rq);
 	if (unlikely(se->load.weight != shares))
 		reweight_entity(cfs_rq_of(se), se, shares);
 }
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -571,6 +571,7 @@ extern void free_fair_sched_group(struct
 extern int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent);
 extern void online_fair_sched_group(struct task_group *tg);
 extern void unregister_fair_sched_group(struct task_group *tg);
+extern void __sched_cgroup_mode_update(int mode);
 #else /* !CONFIG_FAIR_GROUP_SCHED: */
 static inline void free_fair_sched_group(struct task_group *tg) { }
 static inline int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)

Re: [PATCH v3 2/7] sched/fair: Add cgroup_mode: up

[Peter Zijlstra]

On Fri, Jun 05, 2026 at 02:40:15PM +0200, Peter Zijlstra wrote:
> Instead of calculating the proportional fraction of the group weight for each
> CPU, just give each CPU the full measure, ignoring these pesky SMP problems.
> 
> This makes the SMP cgroup fraction (F_g_n) equal to 1, and ensures a single
> task in a cgroup competes on equal footing to a task in a level above.
> 
> However, as already explored, this is not a very good policy because it gets
> the SMP weight distribution wrong. Included for completeness.
> 
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
>  kernel/sched/debug.c |    5 ++++-
>  kernel/sched/fair.c  |   31 +++++++++++++++++++++++++++++--
>  kernel/sched/sched.h |    1 +
>  3 files changed, 34 insertions(+), 3 deletions(-)
> 
> --- a/kernel/sched/debug.c
> +++ b/kernel/sched/debug.c
> @@ -271,6 +271,7 @@ static ssize_t sched_dynamic_write(struc
>  	if (mode < 0)
>  		return mode;
>  
> +	__sched_cgroup_mode_update(mode);
>  	sched_dynamic_update(mode);
>  
>  	*ppos += cnt;

Yeez, I'm not sute WTF happened here. Let me go fix that up.

[tip: sched/core] sched/fair: Add cgroup_mode: up

[tip-bot2 for Peter Zijlstra]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     80ad6d3338ebe136b0c43c5caa962304a8454e3e
Gitweb:        https://git.kernel.org/tip/80ad6d3338ebe136b0c43c5caa962304a8454e3e
Author:        Peter Zijlstra <peterz@infradead.org>
AuthorDate:    Thu, 12 Mar 2026 14:59:06 +01:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 30 Jun 2026 10:56:52 +02:00

sched/fair: Add cgroup_mode: up

Instead of calculating the proportional fraction of the group weight for each
CPU, just give each CPU the full measure, ignoring these pesky SMP problems.

This makes the SMP cgroup fraction (F_g_n) equal to 1, and ensures a single
task in a cgroup competes on equal footing to a task in a level above.

However, as already explored, this is not a very good policy because it gets
the SMP weight distribution wrong. Included for completeness.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.450303977%40infradead.org
---
 kernel/sched/debug.c |  5 ++++-
 kernel/sched/fair.c  | 31 +++++++++++++++++++++++++++++--
 kernel/sched/sched.h |  1 +
 3 files changed, 34 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 507e486..53b9e82 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -634,9 +634,11 @@ static void debugfs_fair_server_init(void)
 }
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-static int cgroup_mode = 0;
+static int cgroup_mode = 1;
 
+/* See __sched_cgroup_mode_update(). */
 static const char *cgroup_mode_str[] = {
+	"up",
 	"smp",
 };
 
@@ -666,6 +668,7 @@ static ssize_t sched_cgroup_write(struct file *filp, const char __user *ubuf,
 	if (mode < 0)
 		return mode;
 
+	__sched_cgroup_mode_update(mode);
 	WRITE_ONCE(cgroup_mode, mode);
 
 	*ppos += cnt;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e89edbd..3f8a280 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -38,6 +38,7 @@
 #include <linux/sched/isolation.h>
 #include <linux/sched/nohz.h>
 #include <linux/sched/prio.h>
+#include <linux/static_call.h>
 
 #include <linux/cpuidle.h>
 #include <linux/interrupt.h>
@@ -4800,7 +4801,7 @@ static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
  *
  * hence icky!
  */
-static long calc_group_shares(struct cfs_rq *cfs_rq)
+static long calc_smp_shares(struct cfs_rq *cfs_rq)
 {
 	long tg_weight, tg_shares, load, shares;
 	struct task_group *tg = cfs_rq->tg;
@@ -4835,6 +4836,32 @@ static long calc_group_shares(struct cfs_rq *cfs_rq)
 }
 
 /*
+ * Ignore this pesky SMP stuff, use (4).
+ */
+static long calc_up_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	return READ_ONCE(tg->shares);
+}
+
+DEFINE_STATIC_CALL(calc_group_shares, calc_smp_shares);
+
+void __sched_cgroup_mode_update(int mode)
+{
+	long (*func)(struct cfs_rq *);
+	switch (mode) {
+	case 0:
+		func = &calc_up_shares;
+		break;
+	case 1:
+	default:
+		func = &calc_smp_shares;
+		break;
+	}
+	static_call_update(calc_group_shares, func);
+}
+
+/*
  * Recomputes the group entity based on the current state of its group
  * runqueue.
  */
@@ -4850,7 +4877,7 @@ static void update_cfs_group(struct sched_entity *se)
 	if (!gcfs_rq || !gcfs_rq->load.weight)
 		return;
 
-	shares = calc_group_shares(gcfs_rq);
+	shares = static_call(calc_group_shares)(gcfs_rq);
 	if (unlikely(se->load.weight != shares))
 		reweight_entity(cfs_rq_of(se), se, shares);
 }
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 56acf50..922170b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -571,6 +571,7 @@ extern void free_fair_sched_group(struct task_group *tg);
 extern int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent);
 extern void online_fair_sched_group(struct task_group *tg);
 extern void unregister_fair_sched_group(struct task_group *tg);
+extern void __sched_cgroup_mode_update(int mode);
 #else /* !CONFIG_FAIR_GROUP_SCHED: */
 static inline void free_fair_sched_group(struct task_group *tg) { }
 static inline int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)

[PATCH v3 3/7] sched/fair: Add cgroup_mode: max

[Peter Zijlstra]

In order to avoid the average CPU fraction avg(F_g_n) becoming tiny '1/N',
assume each cgroup is maximally concurrent and distrubute 'N*weight', such
that:

	F_g_n' = N * F_g_n

Giving:

	avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1

And while this sounds like it solves things, remember what that ~ meant. There
is the corner case when a cgroup is minimally loaded, eg a single runnable
task, therefore limit the CPU fraction to that of a nice -20 task to avoid
getting too much load.

This last bit is what makes it different from a previous proposal to allow
raising cpu.weight to '100 * N', that would not limit the mininal concurrency
case and results in a very large F_g_n. And just like F_g_n << 1 is
problematic, so is F_g_n >> 1 for the exact same reasons (it would drown the
kthreads, but it also risks overflowing the load values).

So while this might appear to be a better scheme than the current default
scheme, it doesn't really handle less than maximal concurrency nicely -- it
clips and introduces artificially large weights. So where the traditional SMP
mode works well when nr_tasks << nr_cpus, MAX doesn't work well in that regime
and vice-versa.

The meaning of "cpu.weight" would be: weight per allowed CPU.

Included for completeness (and infrastructure).

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 include/linux/cpuset.h |    6 +++++
 kernel/cgroup/cpuset.c |   15 ++++++++++++++
 kernel/sched/debug.c   |    1 
 kernel/sched/fair.c    |   52 ++++++++++++++++++++++++++++++++++++++++++++-----
 4 files changed, 69 insertions(+), 5 deletions(-)

--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_h
 extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
 extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
 extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
+extern int cpuset_num_cpus(struct cgroup *cgroup);
 extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
 #define cpuset_current_mems_allowed (current->mems_allowed)
 void cpuset_init_current_mems_allowed(void);
@@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_f
 	return false;
 }
 
+static inline int cpuset_num_cpus(struct cgroup *cgroup)
+{
+	return num_online_cpus();
+}
+
 static inline nodemask_t cpuset_mems_allowed(struct task_struct *p)
 {
 	return node_possible_map;
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(struct
 	return changed;
 }
 
+int cpuset_num_cpus(struct cgroup *cgrp)
+{
+	int nr = num_online_cpus();
+	struct cpuset *cs;
+
+	if (is_in_v2_mode()) {
+		guard(rcu)();
+		cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys));
+		if (cs)
+			nr = cpumask_weight(cs->effective_cpus);
+	}
+
+	return nr;
+}
+
 void __init cpuset_init_current_mems_allowed(void)
 {
 	nodes_setall(current->mems_allowed);
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -641,6 +641,7 @@ static int cgroup_mode = 1;
 static const char *cgroup_mode_str[] = {
 	"up",
 	"smp",
+	"max",
 };
 
 static int sched_cgroup_mode(const char *str)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4801,12 +4801,10 @@ static inline int throttled_hierarchy(st
  *
  * hence icky!
  */
-static long calc_smp(struct cfs_rq *cfs_rq)
+static long __calc_smp_shares(struct cfs_rq *cfs_rq, long tg_shares, long shares_max)
 {
-	long tg_weight, tg_shares, load, shares;
 	struct task_group *tg = cfs_rq->tg;
-
-	tg_shares = READ_ONCE(tg->shares);
+	long tg_weight, load, shares;
 
 	load = max(scale_load_down(cfs_rq->load.weight), cfs_rq->avg.load_avg);
 
@@ -4832,7 +4830,48 @@ static long calc_smp(struct cfs_rq *cfs_
 	 * case no task is runnable on a CPU MIN_SHARES=2 should be returned
 	 * instead of 0.
 	 */
-	return clamp_t(long, shares, MIN_SHARES, tg_shares);
+	return clamp_t(long, shares, MIN_SHARES, shares_max);
+}
+
+static int tg_cpus(struct task_group *tg)
+{
+	int nr = num_online_cpus();
+
+	if (cpusets_enabled()) {
+		struct cgroup *cgrp = tg->css.cgroup;
+		if (cgrp)
+			nr = cpuset_num_cpus(cgrp);
+	}
+
+	return nr;
+}
+
+/*
+ * Func: min(fraction(nr_cpus * tg->shares), nice -20)
+ *
+ * Scale tg->shares by the maximal number of CPUs; but clip the max shares at
+ * nice -20, otherwise a single spinner on a 512 CPU machine would result in
+ * 512*NICE_0_LOAD, which is also crazy.
+ */
+static long calc_max_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	int nr = tg_cpus(tg);
+	long tg_shares = READ_ONCE(tg->shares);
+	long max_shares = scale_load(sched_prio_to_weight[0]);
+	return __calc_smp_shares(cfs_rq, tg_shares * nr, max_shares);
+}
+
+/*
+ * Func: fraction(tg->shares)
+ *
+ * This infamously results in tiny shares when you have many CPUs.
+ */
+static long calc_smp_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	long tg_shares = READ_ONCE(tg->shares);
+	return __calc_smp_shares(cfs_rq, tg_shares, tg_shares);
 }
 
 /*
@@ -4857,6 +4896,9 @@ void __sched_cgroup_mode_update(int mode
 	default:
 		func = &calc_smp_shares;
 		break;
+	case 2:
+		func = &calc_max_shares;
+		break;
 	}
 	static_call_update(calc_group_shares, func);
 }

Re: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max

[Waiman Long]

On 6/5/26 8:40 AM, Peter Zijlstra wrote:
> In order to avoid the average CPU fraction avg(F_g_n) becoming tiny '1/N',
> assume each cgroup is maximally concurrent and distrubute 'N*weight', such
> that:
>
> 	F_g_n' = N * F_g_n
>
> Giving:
>
> 	avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1
>
> And while this sounds like it solves things, remember what that ~ meant. There
> is the corner case when a cgroup is minimally loaded, eg a single runnable
> task, therefore limit the CPU fraction to that of a nice -20 task to avoid
> getting too much load.
>
> This last bit is what makes it different from a previous proposal to allow
> raising cpu.weight to '100 * N', that would not limit the mininal concurrency
> case and results in a very large F_g_n. And just like F_g_n << 1 is
> problematic, so is F_g_n >> 1 for the exact same reasons (it would drown the
> kthreads, but it also risks overflowing the load values).
>
> So while this might appear to be a better scheme than the current default
> scheme, it doesn't really handle less than maximal concurrency nicely -- it
> clips and introduces artificially large weights. So where the traditional SMP
> mode works well when nr_tasks << nr_cpus, MAX doesn't work well in that regime
> and vice-versa.
>
> The meaning of "cpu.weight" would be: weight per allowed CPU.
>
> Included for completeness (and infrastructure).
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
>   include/linux/cpuset.h |    6 +++++
>   kernel/cgroup/cpuset.c |   15 ++++++++++++++
>   kernel/sched/debug.c   |    1
>   kernel/sched/fair.c    |   52 ++++++++++++++++++++++++++++++++++++++++++++-----
>   4 files changed, 69 insertions(+), 5 deletions(-)
>
> --- a/include/linux/cpuset.h
> +++ b/include/linux/cpuset.h
> @@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_h
>   extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
>   extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
>   extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
> +extern int cpuset_num_cpus(struct cgroup *cgroup);
>   extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
>   #define cpuset_current_mems_allowed (current->mems_allowed)
>   void cpuset_init_current_mems_allowed(void);
> @@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_f
>   	return false;
>   }
>   
> +static inline int cpuset_num_cpus(struct cgroup *cgroup)
> +{
> +	return num_online_cpus();
> +}
> +
>   static inline nodemask_t cpuset_mems_allowed(struct task_struct *p)
>   {
>   	return node_possible_map;
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(struct
>   	return changed;
>   }
>   
> +int cpuset_num_cpus(struct cgroup *cgrp)
> +{
> +	int nr = num_online_cpus();
> +	struct cpuset *cs;
> +
> +	if (is_in_v2_mode()) {
> +		guard(rcu)();
> +		cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys));
> +		if (cs)
> +			nr = cpumask_weight(cs->effective_cpus);
> +	}
> +
> +	return nr;
> +}

I just have a question about cgroup v1 support. I am assuming that 
cgroup v1 without the cpuset_v2_mode mount option is not supported. To 
fully support cgroup v1, you may have to use guarantee_active_cpus() to 
return the actual set of CPUs that the task can run on. Also there is a 
caveat about the arm64 specific task_cpu_possible_mask() for certain 
arm64 CPUs. That is for 32-bit binary running on 64-bit core which are 
allowed only on a selected subset of cores within the CPU.

This is probably not what you want to focus on right now, but it will be 
good to have a comment to list items that are not fully supported here.

Cheers,
Longman

Re: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max

[Waiman Long]

On 6/10/26 11:09 AM, Waiman Long wrote:
> On 6/5/26 8:40 AM, Peter Zijlstra wrote:
>> In order to avoid the average CPU fraction avg(F_g_n) becoming tiny 
>> '1/N',
>> assume each cgroup is maximally concurrent and distrubute 'N*weight', 
>> such
>> that:
>>
>>     F_g_n' = N * F_g_n
>>
>> Giving:
>>
>>     avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1
>>
>> And while this sounds like it solves things, remember what that ~ 
>> meant. There
>> is the corner case when a cgroup is minimally loaded, eg a single 
>> runnable
>> task, therefore limit the CPU fraction to that of a nice -20 task to 
>> avoid
>> getting too much load.
>>
>> This last bit is what makes it different from a previous proposal to 
>> allow
>> raising cpu.weight to '100 * N', that would not limit the mininal 
>> concurrency
>> case and results in a very large F_g_n. And just like F_g_n << 1 is
>> problematic, so is F_g_n >> 1 for the exact same reasons (it would 
>> drown the
>> kthreads, but it also risks overflowing the load values).
>>
>> So while this might appear to be a better scheme than the current 
>> default
>> scheme, it doesn't really handle less than maximal concurrency nicely 
>> -- it
>> clips and introduces artificially large weights. So where the 
>> traditional SMP
>> mode works well when nr_tasks << nr_cpus, MAX doesn't work well in 
>> that regime
>> and vice-versa.
>>
>> The meaning of "cpu.weight" would be: weight per allowed CPU.
>>
>> Included for completeness (and infrastructure).
>>
>> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>> ---
>>   include/linux/cpuset.h |    6 +++++
>>   kernel/cgroup/cpuset.c |   15 ++++++++++++++
>>   kernel/sched/debug.c   |    1
>>   kernel/sched/fair.c    |   52 
>> ++++++++++++++++++++++++++++++++++++++++++++-----
>>   4 files changed, 69 insertions(+), 5 deletions(-)
>>
>> --- a/include/linux/cpuset.h
>> +++ b/include/linux/cpuset.h
>> @@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_h
>>   extern void cpuset_cpus_allowed_locked(struct task_struct *p, 
>> struct cpumask *mask);
>>   extern void cpuset_cpus_allowed(struct task_struct *p, struct 
>> cpumask *mask);
>>   extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
>> +extern int cpuset_num_cpus(struct cgroup *cgroup);
>>   extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
>>   #define cpuset_current_mems_allowed (current->mems_allowed)
>>   void cpuset_init_current_mems_allowed(void);
>> @@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_f
>>       return false;
>>   }
>>   +static inline int cpuset_num_cpus(struct cgroup *cgroup)
>> +{
>> +    return num_online_cpus();
>> +}
>> +
>>   static inline nodemask_t cpuset_mems_allowed(struct task_struct *p)
>>   {
>>       return node_possible_map;
>> --- a/kernel/cgroup/cpuset.c
>> +++ b/kernel/cgroup/cpuset.c
>> @@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(struct
>>       return changed;
>>   }
>>   +int cpuset_num_cpus(struct cgroup *cgrp)
>> +{
>> +    int nr = num_online_cpus();
>> +    struct cpuset *cs;
>> +
>> +    if (is_in_v2_mode()) {
>> +        guard(rcu)();
>> +        cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys));
>> +        if (cs)
>> +            nr = cpumask_weight(cs->effective_cpus);
>> +    }
>> +
>> +    return nr;
>> +}
>
> I just have a question about cgroup v1 support. I am assuming that 
> cgroup v1 without the cpuset_v2_mode mount option is not supported. To 
> fully support cgroup v1, you may have to use guarantee_active_cpus() 
> to return the actual set of CPUs that the task can run on. Also there 
> is a caveat about the arm64 specific task_cpu_possible_mask() for 
> certain arm64 CPUs. That is for 32-bit binary running on 64-bit core 
> which are allowed only on a selected subset of cores within the CPU.
>
> This is probably not what you want to focus on right now, but it will 
> be good to have a comment to list items that are not fully supported 
> here. 

FYI, you may have to take the callback_lock to ensure the stability of 
the effective_cpus mask.

Cheers,
Longman

Re: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max

[Peter Zijlstra]

On Wed, Jun 10, 2026 at 11:42:47AM -0400, Waiman Long wrote:

> > > --- a/kernel/cgroup/cpuset.c
> > > +++ b/kernel/cgroup/cpuset.c
> > > @@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(struct
> > >       return changed;
> > >   }
> > >   +int cpuset_num_cpus(struct cgroup *cgrp)
> > > +{
> > > +    int nr = num_online_cpus();
> > > +    struct cpuset *cs;
> > > +
> > > +    if (is_in_v2_mode()) {
> > > +        guard(rcu)();
> > > +        cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys));
> > > +        if (cs)
> > > +            nr = cpumask_weight(cs->effective_cpus);
> > > +    }
> > > +
> > > +    return nr;
> > > +}

> FYI, you may have to take the callback_lock to ensure the stability of the
> effective_cpus mask.

That seems pointless, the moment we drop that lock, its changeable
again. Either way around nr is but a snapshot.

Re: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max

[Peter Zijlstra]

On Wed, Jun 10, 2026 at 11:09:59AM -0400, Waiman Long wrote:

> > --- a/kernel/cgroup/cpuset.c
> > +++ b/kernel/cgroup/cpuset.c
> > @@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(struct
> >   	return changed;
> >   }
> > +int cpuset_num_cpus(struct cgroup *cgrp)
> > +{
> > +	int nr = num_online_cpus();
> > +	struct cpuset *cs;
> > +
> > +	if (is_in_v2_mode()) {
> > +		guard(rcu)();
> > +		cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys));
> > +		if (cs)
> > +			nr = cpumask_weight(cs->effective_cpus);
> > +	}
> > +
> > +	return nr;
> > +}
> 
> I just have a question about cgroup v1 support. I am assuming that cgroup v1
> without the cpuset_v2_mode mount option is not supported. 

Correct.

> To fully support
> cgroup v1, you may have to use guarantee_active_cpus() to return the actual
> set of CPUs that the task can run on.

Except this is group based, we'd need an iteration of all tasks in the
group and compute a union of guarantee_active_cpus(). Which all seems
far too expensive and not worth the effort.

> Also there is a caveat about the arm64 specific
> task_cpu_possible_mask() for certain arm64 CPUs. That is for 32-bit
> binary running on 64-bit core which are allowed only on a selected
> subset of cores within the CPU.
> 
> This is probably not what you want to focus on right now, but it will be
> good to have a comment to list items that are not fully supported here.

Will add a comment!

Re: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max

[Waiman Long]

On 6/11/26 9:47 AM, Peter Zijlstra wrote:
> On Wed, Jun 10, 2026 at 11:09:59AM -0400, Waiman Long wrote:
>
>>> --- a/kernel/cgroup/cpuset.c
>>> +++ b/kernel/cgroup/cpuset.c
>>> @@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(struct
>>>    	return changed;
>>>    }
>>> +int cpuset_num_cpus(struct cgroup *cgrp)
>>> +{
>>> +	int nr = num_online_cpus();
>>> +	struct cpuset *cs;
>>> +
>>> +	if (is_in_v2_mode()) {
>>> +		guard(rcu)();
>>> +		cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys));
>>> +		if (cs)
>>> +			nr = cpumask_weight(cs->effective_cpus);
>>> +	}
>>> +
>>> +	return nr;
>>> +}
>> I just have a question about cgroup v1 support. I am assuming that cgroup v1
>> without the cpuset_v2_mode mount option is not supported.
> Correct.
>
>> To fully support
>> cgroup v1, you may have to use guarantee_active_cpus() to return the actual
>> set of CPUs that the task can run on.
> Except this is group based, we'd need an iteration of all tasks in the
> group and compute a union of guarantee_active_cpus(). Which all seems
> far too expensive and not worth the effort.
I thought so.
>
>> Also there is a caveat about the arm64 specific
>> task_cpu_possible_mask() for certain arm64 CPUs. That is for 32-bit
>> binary running on 64-bit core which are allowed only on a selected
>> subset of cores within the CPU.
>>
>> This is probably not what you want to focus on right now, but it will be
>> good to have a comment to list items that are not fully supported here.
> Will add a comment!
Thanks,
Longman

[tip: sched/core] sched/fair: Add cgroup_mode: max

[tip-bot2 for Peter Zijlstra]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     90ac22ffef48dbc6e7561434b6e01753a859bb51
Gitweb:        https://git.kernel.org/tip/90ac22ffef48dbc6e7561434b6e01753a859bb51
Author:        Peter Zijlstra <peterz@infradead.org>
AuthorDate:    Thu, 12 Mar 2026 15:42:35 +01:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 30 Jun 2026 10:56:52 +02:00

sched/fair: Add cgroup_mode: max

In order to avoid the average CPU fraction avg(F_g_n) becoming tiny '1/N',
assume each cgroup is maximally concurrent and distrubute 'N*weight', such
that:

	F_g_n' = N * F_g_n

Giving:

	avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1

And while this sounds like it solves things, remember what that ~ meant. There
is the corner case when a cgroup is minimally loaded, eg a single runnable
task, therefore limit the CPU fraction to that of a nice -20 task to avoid
getting too much load.

This last bit is what makes it different from a previous proposal to allow
raising cpu.weight to '100 * N', that would not limit the mininal concurrency
case and results in a very large F_g_n. And just like F_g_n << 1 is
problematic, so is F_g_n >> 1 for the exact same reasons (it would drown the
kthreads, but it also risks overflowing the load values).

So while this might appear to be a better scheme than the current default
scheme, it doesn't really handle less than maximal concurrency nicely -- it
clips and introduces artificially large weights. So where the traditional SMP
mode works well when nr_tasks << nr_cpus, MAX doesn't work well in that regime
and vice-versa.

The meaning of "cpu.weight" would be: weight per allowed CPU.

Included for completeness (and infrastructure).

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.589618504%40infradead.org
---
 include/linux/cpuset.h |  6 +++++-
 kernel/cgroup/cpuset.c | 22 +++++++++++++++++-
 kernel/sched/debug.c   |  1 +-
 kernel/sched/fair.c    | 52 +++++++++++++++++++++++++++++++++++++----
 4 files changed, 76 insertions(+), 5 deletions(-)

diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 65d76a3..9db2d4f 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_held(void);
 extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
 extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
 extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
+extern int cpuset_num_cpus(struct cgroup *cgroup);
 extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
 #define cpuset_current_mems_allowed (current->mems_allowed)
 void cpuset_init_current_mems_allowed(void);
@@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_fallback(struct task_struct *p)
 	return false;
 }
 
+static inline int cpuset_num_cpus(struct cgroup *cgroup)
+{
+	return num_online_cpus();
+}
+
 static inline nodemask_t cpuset_mems_allowed(struct task_struct *p)
 {
 	return node_possible_map;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 591e3aa..e2f3da7 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -4124,6 +4124,28 @@ bool cpuset_cpus_allowed_fallback(struct task_struct *tsk)
 	return changed;
 }
 
+/*
+ * Returns the number of CPUs available for this cgroup.
+ *
+ * This only really works for cgroup-v2 where all the controllers are mounted
+ * in the same hierarchy. If not cgroup-v2 or no cpuset controller is
+ * configured it reverts to num_online_cpus().
+ */
+int cpuset_num_cpus(struct cgroup *cgrp)
+{
+	int nr = num_online_cpus();
+	struct cpuset *cs;
+
+	if (is_in_v2_mode()) {
+		guard(rcu)();
+		cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys));
+		if (cs)
+			nr = cpumask_weight(cs->effective_cpus);
+	}
+
+	return nr;
+}
+
 void __init cpuset_init_current_mems_allowed(void)
 {
 	nodes_setall(current->mems_allowed);
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 53b9e82..84e0ac4 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -640,6 +640,7 @@ static int cgroup_mode = 1;
 static const char *cgroup_mode_str[] = {
 	"up",
 	"smp",
+	"max",
 };
 
 static int sched_cgroup_mode(const char *str)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3f8a280..b556d53 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4801,12 +4801,10 @@ static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
  *
  * hence icky!
  */
-static long calc_smp_shares(struct cfs_rq *cfs_rq)
+static long __calc_smp_shares(struct cfs_rq *cfs_rq, long tg_shares, long shares_max)
 {
-	long tg_weight, tg_shares, load, shares;
 	struct task_group *tg = cfs_rq->tg;
-
-	tg_shares = READ_ONCE(tg->shares);
+	long tg_weight, load, shares;
 
 	load = max(scale_load_down(cfs_rq->load.weight), cfs_rq->avg.load_avg);
 
@@ -4832,7 +4830,48 @@ static long calc_smp_shares(struct cfs_rq *cfs_rq)
 	 * case no task is runnable on a CPU MIN_SHARES=2 should be returned
 	 * instead of 0.
 	 */
-	return clamp_t(long, shares, MIN_SHARES, tg_shares);
+	return clamp_t(long, shares, MIN_SHARES, shares_max);
+}
+
+static int tg_cpus(struct task_group *tg)
+{
+	int nr = num_online_cpus();
+
+	if (cpusets_enabled()) {
+		struct cgroup *cgrp = tg->css.cgroup;
+		if (cgrp)
+			nr = cpuset_num_cpus(cgrp);
+	}
+
+	return nr;
+}
+
+/*
+ * Func: min(fraction(nr_cpus * tg->shares), nice -20)
+ *
+ * Scale tg->shares by the maximal number of CPUs; but clip the max shares at
+ * nice -20, otherwise a single spinner on a 512 CPU machine would result in
+ * 512*NICE_0_LOAD, which is also crazy.
+ */
+static long calc_max_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	int nr = tg_cpus(tg);
+	long tg_shares = READ_ONCE(tg->shares);
+	long max_shares = scale_load(sched_prio_to_weight[0]);
+	return __calc_smp_shares(cfs_rq, tg_shares * nr, max_shares);
+}
+
+/*
+ * Func: fraction(tg->shares)
+ *
+ * This infamously results in tiny shares when you have many CPUs.
+ */
+static long calc_smp_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	long tg_shares = READ_ONCE(tg->shares);
+	return __calc_smp_shares(cfs_rq, tg_shares, tg_shares);
 }
 
 /*
@@ -4857,6 +4896,9 @@ void __sched_cgroup_mode_update(int mode)
 	default:
 		func = &calc_smp_shares;
 		break;
+	case 2:
+		func = &calc_max_shares;
+		break;
 	}
 	static_call_update(calc_group_shares, func);
 }

[PATCH v3 4/7] sched/fair: Add cgroup_mode: concur

[Peter Zijlstra]

Improve upon the previous scheme ("max") by no longer assuming maximal
concurrency. Instead scale by: 'min(nr_tasks, nr_cpus)'. This handles
the low concurrency cases more gracefully:

	F_g_n' = min(M, N) * F_g_n

Notably this is the first mode where:

	avg(F_g_n) = 1

In the single task case it reduces to ("smp") and then it nicely scales up
until it hits N, where it behaves like ("max").

This is no longer clipped at nice -20. Strictly speaking it isn't different
from the normal SMP scenario where all tasks are extremely unbalanced. There
are no unnatural inflations in this scheme.

The meaning of "cpu.weight" would be: weight per active CPU.

NOTE: Compute the group wide number of tasks by extending the tg->load_avg
computation with tg->runnable_avg, since cfs_rq->runnable_avg is based on
cfs_rq->h_nr_running.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/debug.c |    1 +
 kernel/sched/fair.c  |   43 ++++++++++++++++++++++++++++++++++++-------
 kernel/sched/sched.h |    3 +++
 3 files changed, 40 insertions(+), 7 deletions(-)

--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -641,6 +641,7 @@ static int cgroup_mode = 1;
 static const char *cgroup_mode_str[] = {
 	"up",
 	"smp",
+	"concur",
 	"max",
 };
 
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4846,6 +4846,11 @@ static int tg_cpus(struct task_group *tg
 	return nr;
 }
 
+static inline int tg_tasks(struct task_group *tg)
+{
+	return max(1, atomic_long_read(&tg->runnable_avg) >> SCHED_CAPACITY_SHIFT);
+}
+
 /*
  * Func: min(fraction(nr_cpus * tg->shares), nice -20)
  *
@@ -4863,6 +4868,20 @@ static long calc_max_shares(struct cfs_r
 }
 
 /*
+ * Func: fraction(nr * tg->shares); nr = min(nr_tasks, nr_cpus)
+ *
+ * Scales between "smp" and "max" in a natural way. No longer needs clipping
+ * since there are no unnatural inflations like with "max".
+ */
+static long calc_concur_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	int nr = min(tg_tasks(tg), tg_cpus(tg));
+	long tg_shares = READ_ONCE(tg->shares);
+	return __calc_smp_shares(cfs_rq, nr * tg_shares, nr * tg_shares);
+}
+
+/*
  * Func: fraction(tg->shares)
  *
  * This infamously results in tiny shares when you have many CPUs.
@@ -4897,6 +4916,9 @@ void __sched_cgroup_mode_update(int mode
 		func = &calc_smp_shares;
 		break;
 	case 2:
+		func = &calc_concur_shares;
+		break;
+	case 3:
 		func = &calc_max_shares;
 		break;
 	}
@@ -5043,7 +5065,7 @@ static inline bool cfs_rq_is_decayed(str
  */
 static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
 {
-	long delta;
+	long dl, dr;
 	u64 now;
 
 	/*
@@ -5064,17 +5086,21 @@ static inline void update_tg_load_avg(st
 	if (now - cfs_rq->last_update_tg_load_avg < NSEC_PER_MSEC)
 		return;
 
-	delta = cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib;
-	if (abs(delta) > cfs_rq->tg_load_avg_contrib / 64) {
-		atomic_long_add(delta, &cfs_rq->tg->load_avg);
+	dl = cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib;
+	dr = cfs_rq->avg.runnable_avg - cfs_rq->tg_runnable_avg_contrib;
+	if (abs(dl) > cfs_rq->tg_load_avg_contrib / 64 ||
+	    abs(dr) > cfs_rq->tg_runnable_avg_contrib / 64) {
+		atomic_long_add(dl, &cfs_rq->tg->load_avg);
+		atomic_long_add(dr, &cfs_rq->tg->runnable_avg);
 		cfs_rq->tg_load_avg_contrib = cfs_rq->avg.load_avg;
+		cfs_rq->tg_runnable_avg_contrib = cfs_rq->avg.runnable_avg;
 		cfs_rq->last_update_tg_load_avg = now;
 	}
 }
 
 static inline void clear_tg_load_avg(struct cfs_rq *cfs_rq)
 {
-	long delta;
+	long dl, dr;
 	u64 now;
 
 	/*
@@ -5084,9 +5110,12 @@ static inline void clear_tg_load_avg(str
 		return;
 
 	now = rq_clock(rq_of(cfs_rq));
-	delta = 0 - cfs_rq->tg_load_avg_contrib;
-	atomic_long_add(delta, &cfs_rq->tg->load_avg);
+	dl = 0 - cfs_rq->tg_load_avg_contrib;
+	dr = 0 - cfs_rq->tg_runnable_avg_contrib;
+	atomic_long_add(dl, &cfs_rq->tg->load_avg);
+	atomic_long_add(dr, &cfs_rq->tg->runnable_avg);
 	cfs_rq->tg_load_avg_contrib = 0;
+	cfs_rq->tg_runnable_avg_contrib = 0;
 	cfs_rq->last_update_tg_load_avg = now;
 }
 
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -493,6 +493,8 @@ struct task_group {
 	 * will also be accessed at each tick.
 	 */
 	atomic_long_t		load_avg ____cacheline_aligned;
+	atomic_long_t		runnable_avg;
+
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 
 #ifdef CONFIG_RT_GROUP_SCHED
@@ -722,6 +724,7 @@ struct cfs_rq {
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	u64			last_update_tg_load_avg;
 	unsigned long		tg_load_avg_contrib;
+	unsigned long		tg_runnable_avg_contrib;
 	long			propagate;
 	long			prop_runnable_sum;
 

[tip: sched/core] sched/fair: Add cgroup_mode: concur

[tip-bot2 for Peter Zijlstra]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     5c0b58bd1c9f09c4e9363665702c82694c6ee7a0
Gitweb:        https://git.kernel.org/tip/5c0b58bd1c9f09c4e9363665702c82694c6ee7a0
Author:        Peter Zijlstra <peterz@infradead.org>
AuthorDate:    Thu, 12 Mar 2026 16:04:18 +01:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 30 Jun 2026 10:56:52 +02:00

sched/fair: Add cgroup_mode: concur

Improve upon the previous scheme ("max") by no longer assuming maximal
concurrency. Instead scale by: 'min(nr_tasks, nr_cpus)'. This handles
the low concurrency cases more gracefully:

	F_g_n' = min(M, N) * F_g_n

Notably this is the first mode where:

	avg(F_g_n) = 1

In the single task case it reduces to ("smp") and then it nicely scales up
until it hits N, where it behaves like ("max").

This is no longer clipped at nice -20. Strictly speaking it isn't different
from the normal SMP scenario where all tasks are extremely unbalanced. There
are no unnatural inflations in this scheme.

The meaning of "cpu.weight" would be: weight per active CPU.

NOTE: Compute the group wide number of tasks by extending the tg->load_avg
computation with tg->runnable_avg, since cfs_rq->runnable_avg is based on
cfs_rq->h_nr_running.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.740585993%40infradead.org
---
 kernel/sched/debug.c |  1 +-
 kernel/sched/fair.c  | 43 ++++++++++++++++++++++++++++++++++++-------
 kernel/sched/sched.h |  3 +++-
 3 files changed, 40 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 84e0ac4..a5e3f8a 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -640,6 +640,7 @@ static int cgroup_mode = 1;
 static const char *cgroup_mode_str[] = {
 	"up",
 	"smp",
+	"concur",
 	"max",
 };
 
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b556d53..64181e4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4846,6 +4846,11 @@ static int tg_cpus(struct task_group *tg)
 	return nr;
 }
 
+static inline int tg_tasks(struct task_group *tg)
+{
+	return max(1, atomic_long_read(&tg->runnable_avg) >> SCHED_CAPACITY_SHIFT);
+}
+
 /*
  * Func: min(fraction(nr_cpus * tg->shares), nice -20)
  *
@@ -4863,6 +4868,20 @@ static long calc_max_shares(struct cfs_rq *cfs_rq)
 }
 
 /*
+ * Func: fraction(nr * tg->shares); nr = min(nr_tasks, nr_cpus)
+ *
+ * Scales between "smp" and "max" in a natural way. No longer needs clipping
+ * since there are no unnatural inflations like with "max".
+ */
+static long calc_concur_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	int nr = min(tg_tasks(tg), tg_cpus(tg));
+	long tg_shares = READ_ONCE(tg->shares);
+	return __calc_smp_shares(cfs_rq, nr * tg_shares, nr * tg_shares);
+}
+
+/*
  * Func: fraction(tg->shares)
  *
  * This infamously results in tiny shares when you have many CPUs.
@@ -4897,6 +4916,9 @@ void __sched_cgroup_mode_update(int mode)
 		func = &calc_smp_shares;
 		break;
 	case 2:
+		func = &calc_concur_shares;
+		break;
+	case 3:
 		func = &calc_max_shares;
 		break;
 	}
@@ -5043,7 +5065,7 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
  */
 static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
 {
-	long delta;
+	long dl, dr;
 	u64 now;
 
 	/*
@@ -5064,17 +5086,21 @@ static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
 	if (now - cfs_rq->last_update_tg_load_avg < NSEC_PER_MSEC)
 		return;
 
-	delta = cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib;
-	if (abs(delta) > cfs_rq->tg_load_avg_contrib / 64) {
-		atomic_long_add(delta, &cfs_rq->tg->load_avg);
+	dl = cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib;
+	dr = cfs_rq->avg.runnable_avg - cfs_rq->tg_runnable_avg_contrib;
+	if (abs(dl) > cfs_rq->tg_load_avg_contrib / 64 ||
+	    abs(dr) > cfs_rq->tg_runnable_avg_contrib / 64) {
+		atomic_long_add(dl, &cfs_rq->tg->load_avg);
+		atomic_long_add(dr, &cfs_rq->tg->runnable_avg);
 		cfs_rq->tg_load_avg_contrib = cfs_rq->avg.load_avg;
+		cfs_rq->tg_runnable_avg_contrib = cfs_rq->avg.runnable_avg;
 		cfs_rq->last_update_tg_load_avg = now;
 	}
 }
 
 static inline void clear_tg_load_avg(struct cfs_rq *cfs_rq)
 {
-	long delta;
+	long dl, dr;
 	u64 now;
 
 	/*
@@ -5084,9 +5110,12 @@ static inline void clear_tg_load_avg(struct cfs_rq *cfs_rq)
 		return;
 
 	now = rq_clock(rq_of(cfs_rq));
-	delta = 0 - cfs_rq->tg_load_avg_contrib;
-	atomic_long_add(delta, &cfs_rq->tg->load_avg);
+	dl = 0 - cfs_rq->tg_load_avg_contrib;
+	dr = 0 - cfs_rq->tg_runnable_avg_contrib;
+	atomic_long_add(dl, &cfs_rq->tg->load_avg);
+	atomic_long_add(dr, &cfs_rq->tg->runnable_avg);
 	cfs_rq->tg_load_avg_contrib = 0;
+	cfs_rq->tg_runnable_avg_contrib = 0;
 	cfs_rq->last_update_tg_load_avg = now;
 }
 
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 922170b..fd267bf 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -493,6 +493,8 @@ struct task_group {
 	 * will also be accessed at each tick.
 	 */
 	atomic_long_t		load_avg ____cacheline_aligned;
+	atomic_long_t		runnable_avg;
+
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 
 #ifdef CONFIG_RT_GROUP_SCHED
@@ -722,6 +724,7 @@ struct cfs_rq {
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	u64			last_update_tg_load_avg;
 	unsigned long		tg_load_avg_contrib;
+	unsigned long		tg_runnable_avg_contrib;
 	long			propagate;
 	long			prop_runnable_sum;
 

[PATCH v3 5/7] sched/fair: Add cgroup_mode: tasks

[Peter Zijlstra]

Since we are exploring this space; include a scheme that scales by total number
of runnable tasks. This results in:

	F_g_n' = M * F_g_n

This will obviously have: avg(F_g_n') > 1, (it will be ~M/N in fact).

And while that sounds odd, it actually has a fairly straight foward meaning for
"cpu.weight": average weight per member task.

This is an entirely valid and workable option, it is however wildly different
from the traditional meaning.

Included for completeness (and curiosity).

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/debug.c |    1 +
 kernel/sched/fair.c  |   16 ++++++++++++++++
 2 files changed, 17 insertions(+)

--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -643,6 +643,7 @@ static const char *cgroup_mode_str[] = {
 	"smp",
 	"concur",
 	"max",
+	"tasks",
 };
 
 static int sched_cgroup_mode(const char *str)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4852,6 +4852,19 @@ static inline int tg_tasks(struct task_g
 }
 
 /*
+ * Func: fraction(nr_tasks * tg->shares)
+ *
+ * Scale tg->shares by the number of tasks.
+ */
+static long calc_tasks_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	int nr = tg_tasks(tg);
+	long tg_shares = READ_ONCE(tg->shares);
+	return __calc_smp_shares(cfs_rq, nr * tg_shares, nr * tg_shares);
+}
+
+/*
  * Func: min(fraction(nr_cpus * tg->shares), nice -20)
  *
  * Scale tg->shares by the maximal number of CPUs; but clip the max shares at
@@ -4921,6 +4934,9 @@ void __sched_cgroup_mode_update(int mode
 	case 3:
 		func = &calc_max_shares;
 		break;
+	case 4:
+		func = &calc_tasks_shares;
+		break;
 	}
 	static_call_update(calc_group_shares, func);
 }

[tip: sched/core] sched/fair: Add cgroup_mode: tasks

[tip-bot2 for Peter Zijlstra]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     507f910a4e1f140a245b693441b42bcbc8128938
Gitweb:        https://git.kernel.org/tip/507f910a4e1f140a245b693441b42bcbc8128938
Author:        Peter Zijlstra <peterz@infradead.org>
AuthorDate:    Fri, 05 Jun 2026 10:35:01 +02:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 30 Jun 2026 10:56:53 +02:00

sched/fair: Add cgroup_mode: tasks

Since we are exploring this space; include a scheme that scales by total number
of runnable tasks. This results in:

	F_g_n' = M * F_g_n

This will obviously have: avg(F_g_n') > 1, (it will be ~M/N in fact).

And while that sounds odd, it actually has a fairly straight foward meaning for
"cpu.weight": average weight per member task.

This is an entirely valid and workable option, it is however wildly different
from the traditional meaning.

Included for completeness (and curiosity).

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.921991975%40infradead.org
---
 kernel/sched/debug.c |  1 +
 kernel/sched/fair.c  | 16 ++++++++++++++++
 2 files changed, 17 insertions(+)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index a5e3f8a..464d0e4 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -642,6 +642,7 @@ static const char *cgroup_mode_str[] = {
 	"smp",
 	"concur",
 	"max",
+	"tasks",
 };
 
 static int sched_cgroup_mode(const char *str)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 64181e4..4a204eb 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4852,6 +4852,19 @@ static inline int tg_tasks(struct task_group *tg)
 }
 
 /*
+ * Func: fraction(nr_tasks * tg->shares)
+ *
+ * Scale tg->shares by the number of tasks.
+ */
+static long calc_tasks_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	int nr = tg_tasks(tg);
+	long tg_shares = READ_ONCE(tg->shares);
+	return __calc_smp_shares(cfs_rq, nr * tg_shares, nr * tg_shares);
+}
+
+/*
  * Func: min(fraction(nr_cpus * tg->shares), nice -20)
  *
  * Scale tg->shares by the maximal number of CPUs; but clip the max shares at
@@ -4921,6 +4934,9 @@ void __sched_cgroup_mode_update(int mode)
 	case 3:
 		func = &calc_max_shares;
 		break;
+	case 4:
+		func = &calc_tasks_shares;
+		break;
 	}
 	static_call_update(calc_group_shares, func);
 }

[PATCH v3 6/7] sched/fair: Change the default cgroup_mode to concur

[Peter Zijlstra]

For all the reasons described in the preceding patches, the way cgroup
weight is computed is problematic. However, changing it is bound to
also lead to trouble. Esp. since people might have taken to inflating
the weight value where they can.

Since things are configurable, change the default and hope this serves
more people than it hurts, esp. in the longer run.

Specifically, this prepares for a flattened runqueue, where the hierarchical
weight becomes far more important (F_g^d terms), so getting rid of small F_g is
imperative.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/debug.c |    2 +-
 kernel/sched/fair.c  |    4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -635,7 +635,7 @@ static void debugfs_fair_server_init(voi
 }
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-static int cgroup_mode = 1;
+static int cgroup_mode = 2;
 
 /* See __sched_cgroup_mode_update(). */
 static const char *cgroup_mode_str[] = {
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4921,7 +4921,7 @@ static long calc_up_shares(struct cfs_rq
 	return READ_ONCE(tg->shares);
 }
 
-DEFINE_STATIC_CALL(calc_group_shares, calc_smp_shares);
+DEFINE_STATIC_CALL(calc_group_shares, calc_concur_shares);
 
 void __sched_cgroup_mode_update(int mode)
 {
@@ -4931,10 +4931,10 @@ void __sched_cgroup_mode_update(int mode
 		func = &calc_up_shares;
 		break;
 	case 1:
-	default:
 		func = &calc_smp_shares;
 		break;
 	case 2:
+	default:
 		func = &calc_concur_shares;
 		break;
 	case 3:

[tip: sched/core] sched/fair: Change the default cgroup_mode to concur

[tip-bot2 for Peter Zijlstra]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     fb1050ac8e40aaa014291b5b53612d27890c2b09
Gitweb:        https://git.kernel.org/tip/fb1050ac8e40aaa014291b5b53612d27890c2b09
Author:        Peter Zijlstra <peterz@infradead.org>
AuthorDate:    Thu, 04 Jun 2026 15:15:51 +02:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 30 Jun 2026 10:56:53 +02:00

sched/fair: Change the default cgroup_mode to concur

For all the reasons described in the preceding patches, the way cgroup
weight is computed is problematic. However, changing it is bound to
also lead to trouble. Esp. since people might have taken to inflating
the weight value where they can.

Since things are configurable, change the default and hope this serves
more people than it hurts, esp. in the longer run.

Specifically, this prepares for a flattened runqueue, where the hierarchical
weight becomes far more important (F_g^d terms), so getting rid of small F_g is
imperative.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124052.080482755%40infradead.org
---
 kernel/sched/debug.c | 2 +-
 kernel/sched/fair.c  | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 464d0e4..f54e732 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -634,7 +634,7 @@ static void debugfs_fair_server_init(void)
 }
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-static int cgroup_mode = 1;
+static int cgroup_mode = 2;
 
 /* See __sched_cgroup_mode_update(). */
 static const char *cgroup_mode_str[] = {
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 4a204eb..1608c01 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4915,7 +4915,7 @@ static long calc_up_shares(struct cfs_rq *cfs_rq)
 	return READ_ONCE(tg->shares);
 }
 
-DEFINE_STATIC_CALL(calc_group_shares, calc_smp_shares);
+DEFINE_STATIC_CALL(calc_group_shares, calc_concur_shares);
 
 void __sched_cgroup_mode_update(int mode)
 {
@@ -4925,10 +4925,10 @@ void __sched_cgroup_mode_update(int mode)
 		func = &calc_up_shares;
 		break;
 	case 1:
-	default:
 		func = &calc_smp_shares;
 		break;
 	case 2:
+	default:
 		func = &calc_concur_shares;
 		break;
 	case 3:

[PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Peter Zijlstra]

Change fair/cgroup to a single runqueue.

Infamously fair/cgroup isn't working for a number of people; typically
the complaint is latencies and/or overhead. The latency issue is due
to the intermediate entries that represent a combination of tasks and
thereby obfuscate the runnability of tasks.

The approach here is to leave the cgroup hierarchy as is; including
the intermediate enqueue/dequeue but move the actual EEVDF runqueue
outside. This means things like the shares_weight approximation are
fully preserved.

That is, given a hierarchy like:

          R
          |
          se--G1
              / \
        G2--se   se--G3
       / \           |
  T1--se se--T2      se--T3

This is fully maintained for load tracking, however the EEVDF parts of
cfs_rq/se go unused for the intermediates and are instead connected
like:

     _R_
    / | \
   T1 T2 T3

Since the effective weight of the entities is determined by the
hierarchy, this gets recomputed on enqueue,set_next_task and tick.

Notably, the effective weight (se->h_load) is computed from the
hierarchical fraction: se->load / cfs_rq->load.

Since EEVDF is now exclusively operating on rq->cfs, it needs to
consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly,
only tasks can get delayed, simplifying some of the cgroup cleanup.

One place where additional information was required was
set_next_task() / put_prev_task(), where we need to track 'current'
both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense
(cfs_rq->curr).

As a result of only having a single level to pick from, much of the
complications in pick_next_task() and preemption go away.

Since many of the hierarchical operations are still there, this won't
immediately fix the performance issues, but hopefully it will fix some
of the latency issues.

TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 include/linux/sched.h |    1 
 kernel/sched/core.c   |    5 
 kernel/sched/debug.c  |    9 
 kernel/sched/fair.c   |  803 +++++++++++++++++++++-----------------------------
 kernel/sched/pelt.c   |    6 
 kernel/sched/sched.h  |   26 -
 6 files changed, 375 insertions(+), 475 deletions(-)

--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -574,6 +574,7 @@ struct sched_statistics {
 struct sched_entity {
 	/* For load-balancing: */
 	struct load_weight		load;
+	struct load_weight		h_load;
 	struct rb_node			run_node;
 	u64				deadline;
 	u64				min_vruntime;
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5652,11 +5652,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);
  */
 static inline void prefetch_curr_exec_start(struct task_struct *p)
 {
-#ifdef CONFIG_FAIR_GROUP_SCHED
-	struct sched_entity *curr = p->se.cfs_rq->curr;
-#else
 	struct sched_entity *curr = task_rq(p)->cfs.curr;
-#endif
+
 	prefetch(curr);
 	prefetch(&curr->exec_start);
 }
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -975,10 +975,11 @@ print_task(struct seq_file *m, struct rq
 	else
 		SEQ_printf(m, " %c", task_state_to_char(p));
 
-	SEQ_printf(m, " %15s %5d %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
+	SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
 		p->comm, task_pid_nr(p),
+		p->se.h_load.weight,
 		SPLIT_NS(p->se.vruntime),
-		entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
+		entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N',
 		SPLIT_NS(p->se.deadline),
 		p->se.custom_slice ? 'S' : ' ',
 		SPLIT_NS(p->se.slice),
@@ -1007,7 +1008,7 @@ static void print_rq(struct seq_file *m,
 
 	SEQ_printf(m, "\n");
 	SEQ_printf(m, "runnable tasks:\n");
-	SEQ_printf(m, " S            task   PID       vruntime   eligible    "
+	SEQ_printf(m, " S            task   PID     weight       vruntime   eligible    "
 		   "deadline             slice          sum-exec      switches  "
 		   "prio         wait-time        sum-sleep       sum-block"
 #ifdef CONFIG_NUMA_BALANCING
@@ -1115,6 +1116,8 @@ void print_cfs_rq(struct seq_file *m, in
 			cfs_rq->tg_load_avg_contrib);
 	SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_avg",
 			atomic_long_read(&cfs_rq->tg->load_avg));
+	SEQ_printf(m, "  .%-30s: %lu\n", "h_load",
+			cfs_rq->h_load);
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 #ifdef CONFIG_CFS_BANDWIDTH
 	SEQ_printf(m, "  .%-30s: %d\n", "throttled",
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -297,8 +297,8 @@ static u64 __calc_delta(u64 delta_exec,
  */
 static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)
 {
-	if (unlikely(se->load.weight != NICE_0_LOAD))
-		delta = __calc_delta(delta, NICE_0_LOAD, &se->load);
+	if (se->h_load.weight != NICE_0_LOAD)
+		delta = __calc_delta(delta, NICE_0_LOAD, &se->h_load);
 
 	return delta;
 }
@@ -428,38 +428,6 @@ static inline struct sched_entity *paren
 	return se->parent;
 }
 
-static void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
-	int se_depth, pse_depth;
-
-	/*
-	 * preemption test can be made between sibling entities who are in the
-	 * same cfs_rq i.e who have a common parent. Walk up the hierarchy of
-	 * both tasks until we find their ancestors who are siblings of common
-	 * parent.
-	 */
-
-	/* First walk up until both entities are at same depth */
-	se_depth = (*se)->depth;
-	pse_depth = (*pse)->depth;
-
-	while (se_depth > pse_depth) {
-		se_depth--;
-		*se = parent_entity(*se);
-	}
-
-	while (pse_depth > se_depth) {
-		pse_depth--;
-		*pse = parent_entity(*pse);
-	}
-
-	while (!is_same_group(*se, *pse)) {
-		*se = parent_entity(*se);
-		*pse = parent_entity(*pse);
-	}
-}
-
 static int tg_is_idle(struct task_group *tg)
 {
 	return tg->idle > 0;
@@ -503,11 +471,6 @@ static inline struct sched_entity *paren
 	return NULL;
 }
 
-static inline void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
-}
-
 static inline int tg_is_idle(struct task_group *tg)
 {
 	return 0;
@@ -686,7 +649,7 @@ static inline unsigned long avg_vruntime
 static inline void
 __sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+	unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
 	s64 w_vruntime, key = entity_key(cfs_rq, se);
 
 	w_vruntime = key * weight;
@@ -703,7 +666,7 @@ sum_w_vruntime_add_paranoid(struct cfs_r
 	s64 key, tmp;
 
 again:
-	weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+	weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
 	key = entity_key(cfs_rq, se);
 
 	if (check_mul_overflow(key, weight, &key))
@@ -749,7 +712,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rq
 static void
 sum_w_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+	unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
 	s64 key = entity_key(cfs_rq, se);
 
 	cfs_rq->sum_w_vruntime -= key * weight;
@@ -791,7 +754,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)
 		s64 runtime = cfs_rq->sum_w_vruntime;
 
 		if (curr) {
-			unsigned long w = avg_vruntime_weight(cfs_rq, curr->load.weight);
+			unsigned long w = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
 			runtime += entity_key(cfs_rq, curr) * w;
 			weight += w;
@@ -862,8 +825,6 @@ bool update_entity_lag(struct cfs_rq *cf
 	u64 avruntime = avg_vruntime(cfs_rq);
 	s64 vlag = entity_lag(cfs_rq, se, avruntime);
 
-	WARN_ON_ONCE(!se->on_rq);
-
 	if (se->sched_delayed) {
 		/* previous vlag < 0 otherwise se would not be delayed */
 		vlag = max(vlag, se->vlag);
@@ -899,7 +860,7 @@ static int vruntime_eligible(struct cfs_
 	long load = cfs_rq->sum_weight;
 
 	if (curr && curr->on_rq) {
-		unsigned long weight = avg_vruntime_weight(cfs_rq, curr->load.weight);
+		unsigned long weight = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
 		avg += entity_key(cfs_rq, curr) * weight;
 		load += weight;
@@ -1040,6 +1001,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim
  */
 static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+	WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+	WARN_ON_ONCE(!entity_is_task(se));
+
 	sum_w_vruntime_add(cfs_rq, se);
 	se->min_vruntime = se->vruntime;
 	se->min_slice = se->slice;
@@ -1049,6 +1013,9 @@ static void __enqueue_entity(struct cfs_
 
 static void __dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+	WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+	WARN_ON_ONCE(!entity_is_task(se));
+
 	rb_erase_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline,
 				  &min_vruntime_cb);
 	sum_w_vruntime_sub(cfs_rq, se);
@@ -1145,7 +1112,7 @@ static struct sched_entity *pick_eevdf(s
 	 * We can safely skip eligibility check if there is only one entity
 	 * in this cfs_rq, saving some cycles.
 	 */
-	if (cfs_rq->nr_queued == 1)
+	if (cfs_rq->h_nr_queued == 1)
 		return curr && curr->on_rq ? curr : se;
 
 	/*
@@ -1395,8 +1362,6 @@ static s64 update_se(struct rq *rq, stru
 	return delta_exec;
 }
 
-static void set_next_buddy(struct sched_entity *se);
-
 #ifdef CONFIG_SCHED_CACHE
 
 /*
@@ -1991,7 +1956,7 @@ static void update_curr(struct cfs_rq *c
 	 * not necessarily be the actual task running
 	 * (rq->curr.se). This is easy to confuse!
 	 */
-	struct sched_entity *curr = cfs_rq->curr;
+	struct sched_entity *curr = cfs_rq->h_curr;
 	struct rq *rq = rq_of(cfs_rq);
 	s64 delta_exec;
 	bool resched;
@@ -2003,26 +1968,29 @@ static void update_curr(struct cfs_rq *c
 	if (unlikely(delta_exec <= 0))
 		return;
 
+	account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+	if (!entity_is_task(curr))
+		return;
+
+	cfs_rq = &rq->cfs;
+
 	curr->vruntime += calc_delta_fair(delta_exec, curr);
 	resched = update_deadline(cfs_rq, curr);
 
-	if (entity_is_task(curr)) {
-		/*
-		 * If the fair_server is active, we need to account for the
-		 * fair_server time whether or not the task is running on
-		 * behalf of fair_server or not:
-		 *  - If the task is running on behalf of fair_server, we need
-		 *    to limit its time based on the assigned runtime.
-		 *  - Fair task that runs outside of fair_server should account
-		 *    against fair_server such that it can account for this time
-		 *    and possibly avoid running this period.
-		 */
-		dl_server_update(&rq->fair_server, delta_exec);
-	}
-
-	account_cfs_rq_runtime(cfs_rq, delta_exec);
+	/*
+	 * If the fair_server is active, we need to account for the
+	 * fair_server time whether or not the task is running on
+	 * behalf of fair_server or not:
+	 *  - If the task is running on behalf of fair_server, we need
+	 *    to limit its time based on the assigned runtime.
+	 *  - Fair task that runs outside of fair_server should account
+	 *    against fair_server such that it can account for this time
+	 *    and possibly avoid running this period.
+	 */
+	dl_server_update(&rq->fair_server, delta_exec);
 
-	if (cfs_rq->nr_queued == 1)
+	if (cfs_rq->h_nr_queued == 1)
 		return;
 
 	if (resched || !protect_slice(curr)) {
@@ -2033,7 +2001,10 @@ static void update_curr(struct cfs_rq *c
 
 static void update_curr_fair(struct rq *rq)
 {
-	update_curr(cfs_rq_of(&rq->donor->se));
+	struct sched_entity *se = &rq->donor->se;
+
+	for_each_sched_entity(se)
+		update_curr(cfs_rq_of(se));
 }
 
 static inline void
@@ -2109,7 +2080,7 @@ update_stats_enqueue_fair(struct cfs_rq
 	 * Are we enqueueing a waiting task? (for current tasks
 	 * a dequeue/enqueue event is a NOP)
 	 */
-	if (se != cfs_rq->curr)
+	if (se != cfs_rq->h_curr)
 		update_stats_wait_start_fair(cfs_rq, se);
 
 	if (flags & ENQUEUE_WAKEUP)
@@ -2127,7 +2098,7 @@ update_stats_dequeue_fair(struct cfs_rq
 	 * Mark the end of the wait period if dequeueing a
 	 * waiting task:
 	 */
-	if (se != cfs_rq->curr)
+	if (se != cfs_rq->h_curr)
 		update_stats_wait_end_fair(cfs_rq, se);
 
 	if ((flags & DEQUEUE_SLEEP) && entity_is_task(se)) {
@@ -4468,6 +4439,7 @@ static inline void update_scan_period(st
 static void
 account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+	WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
 	update_load_add(&cfs_rq->load, se->load.weight);
 	if (entity_is_task(se)) {
 		struct task_struct *p = task_of(se);
@@ -4483,6 +4455,7 @@ account_entity_enqueue(struct cfs_rq *cf
 static void
 account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+	WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
 	update_load_sub(&cfs_rq->load, se->load.weight);
 	if (entity_is_task(se)) {
 		struct task_struct *p = task_of(se);
@@ -4564,7 +4537,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq,
 static void
 rescale_entity(struct sched_entity *se, unsigned long weight, bool rel_vprot)
 {
-	unsigned long old_weight = se->load.weight;
+	long old_weight = se->h_load.weight;
 
 	/*
 	 * VRUNTIME
@@ -4664,16 +4637,17 @@ rescale_entity(struct sched_entity *se,
 		se->vprot = div64_long(se->vprot * old_weight, weight);
 }
 
-static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
-			    unsigned long weight)
+static void reweight_eevdf(struct cfs_rq *cfs_rq, struct sched_entity *se,
+			   unsigned long weight, bool on_rq)
 {
 	bool curr = cfs_rq->curr == se;
 	bool rel_vprot = false;
 	u64 avruntime = 0;
 
-	if (se->on_rq) {
-		/* commit outstanding execution time */
-		update_curr(cfs_rq);
+	if (se->h_load.weight == weight)
+		return;
+
+	if (on_rq) {
 		avruntime = avg_vruntime(cfs_rq);
 		se->vlag = entity_lag(cfs_rq, se, avruntime);
 		se->deadline -= avruntime;
@@ -4683,46 +4657,90 @@ static void reweight_entity(struct cfs_r
 			rel_vprot = true;
 		}
 
-		cfs_rq->nr_queued--;
+		cfs_rq->h_nr_queued--;
 		if (!curr)
 			__dequeue_entity(cfs_rq, se);
-		update_load_sub(&cfs_rq->load, se->load.weight);
 	}
-	dequeue_load_avg(cfs_rq, se);
 
 	rescale_entity(se, weight, rel_vprot);
 
-	update_load_set(&se->load, weight);
+	update_load_set(&se->h_load, weight);
 
-	do {
-		u32 divider = get_pelt_divider(&se->avg);
-		se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
-	} while (0);
-
-	enqueue_load_avg(cfs_rq, se);
-	if (se->on_rq) {
+	if (on_rq) {
 		if (rel_vprot)
 			se->vprot += avruntime;
 		se->deadline += avruntime;
 		se->rel_deadline = 0;
 		se->vruntime = avruntime - se->vlag;
 
-		update_load_add(&cfs_rq->load, se->load.weight);
 		if (!curr)
 			__enqueue_entity(cfs_rq, se);
-		cfs_rq->nr_queued++;
+		cfs_rq->h_nr_queued++;
 	}
 }
 
+static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
+			    unsigned long weight)
+{
+	if (se->load.weight == weight)
+		return;
+
+	if (se->on_rq) {
+		WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
+		update_load_sub(&cfs_rq->load, se->load.weight);
+	}
+	dequeue_load_avg(cfs_rq, se);
+
+	update_load_set(&se->load, weight);
+
+	do {
+		u32 divider = get_pelt_divider(&se->avg);
+		se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
+	} while (0);
+
+	enqueue_load_avg(cfs_rq, se);
+
+	if (se->on_rq)
+		update_load_add(&cfs_rq->load, se->load.weight);
+}
+
+/*
+ * weight = NICE_0_LOAD;
+ * for_each_entity_se(se)
+ *   weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
+ */
+static __always_inline
+unsigned long __calc_prop_weight(struct cfs_rq *cfs_rq, struct sched_entity *se,
+				 unsigned long weight)
+{
+	weight *= se->load.weight;
+	if (parent_entity(se))
+		weight /= cfs_rq->load.weight;
+	else
+		weight /= NICE_0_LOAD;
+
+	return max(weight, MIN_SHARES);
+}
+
 static void reweight_task_fair(struct rq *rq, struct task_struct *p,
 			       const struct load_weight *lw)
 {
 	struct sched_entity *se = &p->se;
-	struct cfs_rq *cfs_rq = cfs_rq_of(se);
-	struct load_weight *load = &se->load;
+	unsigned long weight = NICE_0_LOAD;
 
-	reweight_entity(cfs_rq, se, lw->weight);
-	load->inv_weight = lw->inv_weight;
+	if (se->on_rq)
+		update_curr_fair(rq);
+
+	reweight_entity(cfs_rq_of(se), se, lw->weight);
+	se->load.inv_weight = lw->inv_weight;
+
+	if (!se->on_rq)
+		return;
+
+	for_each_sched_entity(se)
+		weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
+
+	reweight_eevdf(&rq->cfs, &p->se, weight, p->se.on_rq);
 }
 
 static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
@@ -4958,8 +4976,7 @@ static void update_cfs_group(struct sche
 		return;
 
 	shares = static_call(calc_group_shares)(gcfs_rq);
-	if (unlikely(se->load.weight != shares))
-		reweight_entity(cfs_rq_of(se), se, shares);
+	reweight_entity(cfs_rq_of(se), se, shares);
 }
 
 #else /* !CONFIG_FAIR_GROUP_SCHED: */
@@ -5077,7 +5094,7 @@ static inline bool cfs_rq_is_decayed(str
  * differential update where we store the last value we propagated. This in
  * turn allows skipping updates if the differential is 'small'.
  *
- * Updating tg's load_avg is necessary before update_cfs_share().
+ * Updating tg's load_avg is necessary before update_cfs_group().
  */
 static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
 {
@@ -5544,7 +5561,7 @@ static void migrate_se_pelt_lag(struct s
  * The cfs_rq avg is the direct sum of all its entities (blocked and runnable)
  * avg. The immediate corollary is that all (fair) tasks must be attached.
  *
- * cfs_rq->avg is used for task_h_load() and update_cfs_share() for example.
+ * cfs_rq->avg is used for task_h_load() and update_cfs_group() for example.
  *
  * Return: true if the load decayed or we removed load.
  *
@@ -6082,6 +6099,7 @@ static void
 place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
 	u64 vslice, vruntime = avg_vruntime(cfs_rq);
+	unsigned int nr_queued = cfs_rq->h_nr_queued;
 	bool update_zero = false;
 	s64 lag = 0;
 
@@ -6089,6 +6107,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
 		se->slice = sysctl_sched_base_slice;
 	vslice = calc_delta_fair(se->slice, se);
 
+	if (flags & ENQUEUE_QUEUED)
+		nr_queued -= 1;
+
 	/*
 	 * Due to how V is constructed as the weighted average of entities,
 	 * adding tasks with positive lag, or removing tasks with negative lag
@@ -6097,7 +6118,7 @@ place_entity(struct cfs_rq *cfs_rq, stru
 	 *
 	 * EEVDF: placement strategy #1 / #2
 	 */
-	if (sched_feat(PLACE_LAG) && cfs_rq->nr_queued && se->vlag) {
+	if (sched_feat(PLACE_LAG) && nr_queued && se->vlag) {
 		struct sched_entity *curr = cfs_rq->curr;
 		long load, weight;
 
@@ -6157,9 +6178,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
 		 */
 		load = cfs_rq->sum_weight;
 		if (curr && curr->on_rq)
-			load += avg_vruntime_weight(cfs_rq, curr->load.weight);
+			load += avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
-		weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+		weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
 		lag *= load + weight;
 		if (WARN_ON_ONCE(!load))
 			load = 1;
@@ -6218,22 +6239,8 @@ static void check_enqueue_throttle(struc
 static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
 
 static void
-requeue_delayed_entity(struct sched_entity *se);
-
-static void
 enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
-	bool curr = cfs_rq->curr == se;
-
-	/*
-	 * If we're the current task, we must renormalise before calling
-	 * update_curr().
-	 */
-	if (curr)
-		place_entity(cfs_rq, se, flags);
-
-	update_curr(cfs_rq);
-
 	/*
 	 * When enqueuing a sched_entity, we must:
 	 *   - Update loads to have both entity and cfs_rq synced with now.
@@ -6252,13 +6259,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
 	 */
 	update_cfs_group(se);
 
-	/*
-	 * XXX now that the entity has been re-weighted, and it's lag adjusted,
-	 * we can place the entity.
-	 */
-	if (!curr)
-		place_entity(cfs_rq, se, flags);
-
 	account_entity_enqueue(cfs_rq, se);
 
 	/* Entity has migrated, no longer consider this task hot */
@@ -6267,8 +6267,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
 
 	check_schedstat_required();
 	update_stats_enqueue_fair(cfs_rq, se, flags);
-	if (!curr)
-		__enqueue_entity(cfs_rq, se);
 	se->on_rq = 1;
 
 	if (cfs_rq->nr_queued == 1) {
@@ -6286,21 +6284,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
 	}
 }
 
-static void __clear_buddies_next(struct sched_entity *se)
+static void set_next_buddy(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	for_each_sched_entity(se) {
-		struct cfs_rq *cfs_rq = cfs_rq_of(se);
-		if (cfs_rq->next != se)
-			break;
-
-		cfs_rq->next = NULL;
-	}
+	if (WARN_ON_ONCE(!se->on_rq || se->sched_delayed))
+		return;
+	if (se_is_idle(se))
+		return;
+	cfs_rq->next = se;
 }
 
 static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if (cfs_rq->next == se)
-		__clear_buddies_next(se);
+		cfs_rq->next = NULL;
 }
 
 static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
@@ -6311,7 +6307,7 @@ static void set_delayed(struct sched_ent
 
 	/*
 	 * Delayed se of cfs_rq have no tasks queued on them.
-	 * Do not adjust h_nr_runnable since dequeue_entities()
+	 * Do not adjust h_nr_runnable since __dequeue_task()
 	 * will account it for blocked tasks.
 	 */
 	if (!entity_is_task(se))
@@ -6344,45 +6340,16 @@ static void clear_delayed(struct sched_e
 	}
 }
 
-static bool
+static void
 dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
-	bool sleep = flags & DEQUEUE_SLEEP;
-	int action = 0;
-
-	update_curr(cfs_rq);
-	clear_buddies(cfs_rq, se);
-
-	if (flags & DEQUEUE_DELAYED) {
-		WARN_ON_ONCE(!se->sched_delayed);
-	} else {
-		bool delay = sleep;
-		/*
-		 * DELAY_DEQUEUE relies on spurious wakeups, special task
-		 * states must not suffer spurious wakeups, excempt them.
-		 */
-		if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
-			delay = false;
-
-		WARN_ON_ONCE(delay && se->sched_delayed);
+	int action = UPDATE_TG;
 
-		if (sched_feat(DELAY_DEQUEUE) && delay &&
-		    !entity_eligible(cfs_rq, se)) {
-			if (entity_is_task(se))
-				action |= UPDATE_UTIL_EST;
-			update_load_avg(cfs_rq, se, action);
-			update_entity_lag(cfs_rq, se);
-			set_delayed(se);
-			return false;
-		}
-	}
-
-	action = UPDATE_TG;
 	if (entity_is_task(se)) {
 		if (task_on_rq_migrating(task_of(se)))
 			action |= DO_DETACH;
 
-		if (sleep && !(flags & DEQUEUE_DELAYED))
+		if ((flags & DEQUEUE_SLEEP) && !(flags & DEQUEUE_DELAYED))
 			action |= UPDATE_UTIL_EST;
 	}
 
@@ -6400,14 +6367,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 
 	update_stats_dequeue_fair(cfs_rq, se, flags);
 
-	update_entity_lag(cfs_rq, se);
-	if (sched_feat(PLACE_REL_DEADLINE) && !sleep) {
-		se->deadline -= se->vruntime;
-		se->rel_deadline = 1;
-	}
-
-	if (se != cfs_rq->curr)
-		__dequeue_entity(cfs_rq, se);
 	se->on_rq = 0;
 	account_entity_dequeue(cfs_rq, se);
 
@@ -6416,9 +6375,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 
 	update_cfs_group(se);
 
-	if (flags & DEQUEUE_DELAYED)
-		clear_delayed(se);
-
 	if (cfs_rq->nr_queued == 0) {
 		update_idle_cfs_rq_clock_pelt(cfs_rq);
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -6431,15 +6387,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 		}
 #endif
 	}
-
-	return true;
 }
 
 static void
-set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first)
+set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	clear_buddies(cfs_rq, se);
-
 	/* 'current' is not kept within the tree. */
 	if (se->on_rq) {
 		/*
@@ -6448,16 +6400,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s
 		 * runqueue.
 		 */
 		update_stats_wait_end_fair(cfs_rq, se);
-		__dequeue_entity(cfs_rq, se);
 		update_load_avg(cfs_rq, se, UPDATE_TG);
-
-		if (first)
-			set_protect_slice(cfs_rq, se);
 	}
 
 	update_stats_curr_start(cfs_rq, se);
-	WARN_ON_ONCE(cfs_rq->curr);
-	cfs_rq->curr = se;
+	WARN_ON_ONCE(cfs_rq->h_curr);
+	cfs_rq->h_curr = se;
 
 	/*
 	 * Track our maximum slice length, if the CPU's load is at
@@ -6477,23 +6425,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s
 	se->prev_sum_exec_runtime = se->sum_exec_runtime;
 }
 
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags);
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags);
 
-/*
- * Pick the next process, keeping these things in mind, in this order:
- * 1) keep things fair between processes/task groups
- * 2) pick the "next" process, since someone really wants that to run
- * 3) pick the "last" process, for cache locality
- * 4) do not run the "skip" process, if something else is available
- */
 static struct sched_entity *
-pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
+pick_next_entity(struct rq *rq, bool protect)
 {
+	struct cfs_rq *cfs_rq = &rq->cfs;
 	struct sched_entity *se;
 
 	se = pick_eevdf(cfs_rq, protect);
 	if (se->sched_delayed) {
-		dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+		__dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 		/*
 		 * Must not reference @se again, see __block_task().
 		 */
@@ -6513,13 +6455,11 @@ static void put_prev_entity(struct cfs_r
 
 	if (prev->on_rq) {
 		update_stats_wait_start_fair(cfs_rq, prev);
-		/* Put 'current' back into the tree. */
-		__enqueue_entity(cfs_rq, prev);
 		/* in !on_rq case, update occurred at dequeue */
 		update_load_avg(cfs_rq, prev, 0);
 	}
-	WARN_ON_ONCE(cfs_rq->curr != prev);
-	cfs_rq->curr = NULL;
+	WARN_ON_ONCE(cfs_rq->h_curr != prev);
+	cfs_rq->h_curr = NULL;
 }
 
 static void
@@ -7074,7 +7014,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cf
 	assert_list_leaf_cfs_rq(rq);
 
 	/* Determine whether we need to wake up potentially idle CPU: */
-	if (rq->curr == rq->idle && rq->cfs.nr_queued)
+	if (rq->curr == rq->idle && rq->cfs.h_nr_queued)
 		resched_curr(rq);
 }
 
@@ -7409,7 +7349,7 @@ static void check_enqueue_throttle(struc
 		return;
 
 	/* an active group must be handled by the update_curr() path */
-	if (!cfs_rq->runtime_enabled || cfs_rq->curr)
+	if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)
 		return;
 
 	/* ensure the group is not already throttled */
@@ -7781,7 +7721,7 @@ static void hrtick_start_fair(struct rq
 			resched_curr(rq);
 		return;
 	}
-	delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+	delta = (se->h_load.weight * vdelta) / NICE_0_LOAD;
 
 	/*
 	 * Correct for instantaneous load of other classes.
@@ -7881,10 +7821,8 @@ static int choose_idle_cpu(int cpu, stru
 }
 
 static void
-requeue_delayed_entity(struct sched_entity *se)
+requeue_delayed_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	struct cfs_rq *cfs_rq = cfs_rq_of(se);
-
 	/*
 	 * se->sched_delayed should imply: se->on_rq == 1.
 	 * Because a delayed entity is one that is still on
@@ -7894,19 +7832,58 @@ requeue_delayed_entity(struct sched_enti
 	WARN_ON_ONCE(!se->on_rq);
 
 	if (update_entity_lag(cfs_rq, se)) {
-		cfs_rq->nr_queued--;
+		cfs_rq->h_nr_queued--;
 		if (se != cfs_rq->curr)
 			__dequeue_entity(cfs_rq, se);
 		place_entity(cfs_rq, se, 0);
 		if (se != cfs_rq->curr)
 			__enqueue_entity(cfs_rq, se);
-		cfs_rq->nr_queued++;
+		cfs_rq->h_nr_queued++;
 	}
 
 	update_load_avg(cfs_rq, se, 0);
 	clear_delayed(se);
 }
 
+static unsigned long enqueue_hierarchy(struct task_struct *p, int flags)
+{
+	unsigned long weight = NICE_0_LOAD;
+	int task_new = !(flags & ENQUEUE_WAKEUP);
+	struct sched_entity *se = &p->se;
+	int h_nr_idle = task_has_idle_policy(p);
+	int h_nr_runnable = 1;
+
+	if (task_new && se->sched_delayed)
+		h_nr_runnable = 0;
+
+	for_each_sched_entity(se) {
+		struct cfs_rq *cfs_rq = cfs_rq_of(se);
+
+		update_curr(cfs_rq);
+
+		if (!se->on_rq) {
+			enqueue_entity(cfs_rq, se, flags);
+		} else {
+			update_load_avg(cfs_rq, se, UPDATE_TG);
+			se_update_runnable(se);
+			update_cfs_group(se);
+		}
+
+		cfs_rq->h_nr_runnable += h_nr_runnable;
+		cfs_rq->h_nr_queued++;
+		cfs_rq->h_nr_idle += h_nr_idle;
+
+		if (cfs_rq_is_idle(cfs_rq))
+			h_nr_idle = 1;
+
+		weight = __calc_prop_weight(cfs_rq, se, weight);
+
+		flags = ENQUEUE_WAKEUP;
+	}
+
+	return weight;
+}
+
 /*
  * The enqueue_task method is called before nr_running is
  * increased. Here we update the fair scheduling stats and
@@ -7915,13 +7892,12 @@ requeue_delayed_entity(struct sched_enti
 static void
 enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 {
-	struct cfs_rq *cfs_rq;
-	struct sched_entity *se = &p->se;
-	int h_nr_idle = task_has_idle_policy(p);
-	int h_nr_runnable = 1;
-	int task_new = !(flags & ENQUEUE_WAKEUP);
 	int rq_h_nr_queued = rq->cfs.h_nr_queued;
-	u64 slice = 0;
+	int task_new = !(flags & ENQUEUE_WAKEUP);
+	struct sched_entity *se = &p->se;
+	struct cfs_rq *cfs_rq = &rq->cfs;
+	unsigned long weight;
+	bool curr;
 
 	if (task_is_throttled(p) && enqueue_throttled_task(p))
 		return;
@@ -7933,10 +7909,10 @@ enqueue_task_fair(struct rq *rq, struct
 	 * estimated utilization, before we update schedutil.
 	 */
 	if (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))
-		util_est_enqueue(&rq->cfs, p);
+		util_est_enqueue(cfs_rq, p);
 
 	if (flags & ENQUEUE_DELAYED) {
-		requeue_delayed_entity(se);
+		requeue_delayed_entity(cfs_rq, se);
 		return;
 	}
 
@@ -7948,57 +7924,22 @@ enqueue_task_fair(struct rq *rq, struct
 	if (p->in_iowait)
 		cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT);
 
-	if (task_new && se->sched_delayed)
-		h_nr_runnable = 0;
-
-	for_each_sched_entity(se) {
-		if (se->on_rq) {
-			if (se->sched_delayed)
-				requeue_delayed_entity(se);
-			break;
-		}
-		cfs_rq = cfs_rq_of(se);
-
-		/*
-		 * Basically set the slice of group entries to the min_slice of
-		 * their respective cfs_rq. This ensures the group can service
-		 * its entities in the desired time-frame.
-		 */
-		if (slice) {
-			se->slice = slice;
-			se->custom_slice = 1;
-		}
-		enqueue_entity(cfs_rq, se, flags);
-		slice = cfs_rq_min_slice(cfs_rq);
-
-		cfs_rq->h_nr_runnable += h_nr_runnable;
-		cfs_rq->h_nr_queued++;
-		cfs_rq->h_nr_idle += h_nr_idle;
-
-		if (cfs_rq_is_idle(cfs_rq))
-			h_nr_idle = 1;
-
-		flags = ENQUEUE_WAKEUP;
-	}
-
-	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
+	/*
+	 * XXX comment on the curr thing
+	 */
+	curr = (cfs_rq->curr == se);
+	if (curr)
+		place_entity(cfs_rq, se, flags);
 
-		update_load_avg(cfs_rq, se, UPDATE_TG);
-		se_update_runnable(se);
-		update_cfs_group(se);
+	if (se->on_rq && se->sched_delayed)
+		requeue_delayed_entity(cfs_rq, se);
 
-		se->slice = slice;
-		if (se != cfs_rq->curr)
-			min_vruntime_cb_propagate(&se->run_node, NULL);
-		slice = cfs_rq_min_slice(cfs_rq);
+	weight = enqueue_hierarchy(p, flags);
 
-		cfs_rq->h_nr_runnable += h_nr_runnable;
-		cfs_rq->h_nr_queued++;
-		cfs_rq->h_nr_idle += h_nr_idle;
-
-		if (cfs_rq_is_idle(cfs_rq))
-			h_nr_idle = 1;
+	if (!curr) {
+		reweight_eevdf(cfs_rq, se, weight, false);
+		place_entity(cfs_rq, se, flags | ENQUEUE_QUEUED);
+		__enqueue_entity(cfs_rq, se);
 	}
 
 	if (!rq_h_nr_queued && rq->cfs.h_nr_queued)
@@ -8029,105 +7970,109 @@ enqueue_task_fair(struct rq *rq, struct
 	hrtick_update(rq);
 }
 
-/*
- * Basically dequeue_task_fair(), except it can deal with dequeue_entity()
- * failing half-way through and resume the dequeue later.
- *
- * Returns:
- * -1 - dequeue delayed
- *  0 - dequeue throttled
- *  1 - dequeue complete
- */
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
+static void dequeue_hierarchy(struct task_struct *p, int flags)
 {
-	bool was_sched_idle = sched_idle_rq(rq);
+	struct sched_entity *se = &p->se;
 	bool task_sleep = flags & DEQUEUE_SLEEP;
 	bool task_delayed = flags & DEQUEUE_DELAYED;
 	bool task_throttled = flags & DEQUEUE_THROTTLE;
-	struct task_struct *p = NULL;
-	int h_nr_idle = 0;
-	int h_nr_queued = 0;
 	int h_nr_runnable = 0;
-	struct cfs_rq *cfs_rq;
-	u64 slice = 0;
+	int h_nr_idle = task_has_idle_policy(p);
+	bool dequeue = true;
 
-	if (entity_is_task(se)) {
-		p = task_of(se);
-		h_nr_queued = 1;
-		h_nr_idle = task_has_idle_policy(p);
-		if (task_sleep || task_delayed || !se->sched_delayed)
-			h_nr_runnable = 1;
-	}
+	if (task_sleep || task_delayed || !se->sched_delayed)
+		h_nr_runnable = 1;
 
 	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
+		struct cfs_rq *cfs_rq = cfs_rq_of(se);
 
-		if (!dequeue_entity(cfs_rq, se, flags)) {
-			if (p && &p->se == se)
-				return -1;
+		update_curr(cfs_rq);
 
-			slice = cfs_rq_min_slice(cfs_rq);
-			break;
+		if (dequeue) {
+			dequeue_entity(cfs_rq, se, flags);
+			/* Don't dequeue parent if it has other entities besides us */
+			if (cfs_rq->load.weight)
+				dequeue = false;
+		} else {
+			update_load_avg(cfs_rq, se, UPDATE_TG);
+			se_update_runnable(se);
+			update_cfs_group(se);
 		}
 
 		cfs_rq->h_nr_runnable -= h_nr_runnable;
-		cfs_rq->h_nr_queued -= h_nr_queued;
+		cfs_rq->h_nr_queued--;
 		cfs_rq->h_nr_idle -= h_nr_idle;
 
 		if (cfs_rq_is_idle(cfs_rq))
-			h_nr_idle = h_nr_queued;
+			h_nr_idle = 1;
 
 		if (throttled_hierarchy(cfs_rq) && task_throttled)
 			record_throttle_clock(cfs_rq);
 
-		/* Don't dequeue parent if it has other entities besides us */
-		if (cfs_rq->load.weight) {
-			slice = cfs_rq_min_slice(cfs_rq);
-
-			/* Avoid re-evaluating load for this entity: */
-			se = parent_entity(se);
-			/*
-			 * Bias pick_next to pick a task from this cfs_rq, as
-			 * p is sleeping when it is within its sched_slice.
-			 */
-			if (task_sleep && se)
-				set_next_buddy(se);
-			break;
-		}
 		flags |= DEQUEUE_SLEEP;
 		flags &= ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL);
 	}
+}
 
-	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
+/*
+ * The part of dequeue_task_fair() that is needed to dequeue delayed tasks.
+ *
+ * Returns:
+ *   true  - dequeued
+ *   false - delayed
+ */
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
+{
+	struct sched_entity *se = &p->se;
+	struct cfs_rq *cfs_rq = &rq->cfs;
+	bool was_sched_idle = sched_idle_rq(rq);
+	bool task_sleep = flags & DEQUEUE_SLEEP;
+	bool task_delayed = flags & DEQUEUE_DELAYED;
 
-		update_load_avg(cfs_rq, se, UPDATE_TG);
-		se_update_runnable(se);
-		update_cfs_group(se);
+	clear_buddies(cfs_rq, se);
 
-		se->slice = slice;
-		if (se != cfs_rq->curr)
-			min_vruntime_cb_propagate(&se->run_node, NULL);
-		slice = cfs_rq_min_slice(cfs_rq);
+	update_curr(cfs_rq_of(se));
+	update_entity_lag(cfs_rq, se);
 
-		cfs_rq->h_nr_runnable -= h_nr_runnable;
-		cfs_rq->h_nr_queued -= h_nr_queued;
-		cfs_rq->h_nr_idle -= h_nr_idle;
+	if (flags & DEQUEUE_DELAYED) {
+		WARN_ON_ONCE(!se->sched_delayed);
+	} else {
+		bool delay = task_sleep;
+		/*
+		 * DELAY_DEQUEUE relies on spurious wakeups, special task
+		 * states must not suffer spurious wakeups, excempt them.
+		 */
+		if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
+			delay = false;
 
-		if (cfs_rq_is_idle(cfs_rq))
-			h_nr_idle = h_nr_queued;
+		WARN_ON_ONCE(delay && se->sched_delayed);
 
-		if (throttled_hierarchy(cfs_rq) && task_throttled)
-			record_throttle_clock(cfs_rq);
+		if (sched_feat(DELAY_DEQUEUE) && delay &&
+		    !entity_eligible(cfs_rq, se)) {
+			update_load_avg(cfs_rq_of(se), se, UPDATE_UTIL_EST);
+			set_delayed(se);
+			return false;
+		}
 	}
 
-	sub_nr_running(rq, h_nr_queued);
+	dequeue_hierarchy(p, flags);
+
+	if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
+		se->deadline -= se->vruntime;
+		se->rel_deadline = 1;
+	}
+	if (se != cfs_rq->curr)
+		__dequeue_entity(cfs_rq, se);
+
+	sub_nr_running(rq, 1);
 
 	/* balance early to pull high priority tasks */
 	if (unlikely(!was_sched_idle && sched_idle_rq(rq)))
 		rq->next_balance = jiffies;
 
-	if (p && task_delayed) {
+	if (task_delayed) {
+		clear_delayed(se);
+
 		WARN_ON_ONCE(!task_sleep);
 		WARN_ON_ONCE(p->on_rq != 1);
 
@@ -8139,7 +8084,7 @@ static int dequeue_entities(struct rq *r
 		__block_task(rq, p);
 	}
 
-	return 1;
+	return true;
 }
 
 /*
@@ -8157,11 +8102,11 @@ static bool dequeue_task_fair(struct rq
 	if (!p->se.sched_delayed)
 		util_est_dequeue(&rq->cfs, p);
 
-	if (dequeue_entities(rq, &p->se, flags) < 0)
+	if (!__dequeue_task(rq, p, flags))
 		return false;
 
 	/*
-	 * Must not reference @p after dequeue_entities(DEQUEUE_DELAYED).
+	 * Must not reference @p after __dequeue_task(DEQUEUE_DELAYED).
 	 */
 	return true;
 }
@@ -9749,19 +9694,6 @@ static void migrate_task_rq_fair(struct
 static void task_dead_fair(struct task_struct *p)
 {
 	struct sched_entity *se = &p->se;
-
-	if (se->sched_delayed) {
-		struct rq_flags rf;
-		struct rq *rq;
-
-		rq = task_rq_lock(p, &rf);
-		if (se->sched_delayed) {
-			update_rq_clock(rq);
-			dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-		}
-		task_rq_unlock(rq, p, &rf);
-	}
-
 	remove_entity_load_avg(se);
 }
 
@@ -9795,21 +9727,10 @@ static void set_cpus_allowed_fair(struct
 	set_task_max_allowed_capacity(p);
 }
 
-static void set_next_buddy(struct sched_entity *se)
-{
-	for_each_sched_entity(se) {
-		if (WARN_ON_ONCE(!se->on_rq))
-			return;
-		if (se_is_idle(se))
-			return;
-		cfs_rq_of(se)->next = se;
-	}
-}
-
 enum preempt_wakeup_action {
 	PREEMPT_WAKEUP_NONE,	/* No preemption. */
 	PREEMPT_WAKEUP_SHORT,	/* Ignore slice protection. */
-	PREEMPT_WAKEUP_PICK,	/* Let __pick_eevdf() decide. */
+	PREEMPT_WAKEUP_PICK,	/* Let pick_eevdf() decide. */
 	PREEMPT_WAKEUP_RESCHED,	/* Force reschedule. */
 };
 
@@ -9826,7 +9747,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq,
 	if (cfs_rq->next && entity_before(cfs_rq->next, pse))
 		return false;
 
-	set_next_buddy(pse);
+	set_next_buddy(cfs_rq, pse);
 	return true;
 }
 
@@ -9879,7 +9800,7 @@ static void wakeup_preempt_fair(struct r
 	enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
 	struct task_struct *donor = rq->donor;
 	struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
-	struct cfs_rq *cfs_rq = task_cfs_rq(donor);
+	struct cfs_rq *cfs_rq = &rq->cfs;
 	int cse_is_idle, pse_is_idle;
 
 	/*
@@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r
 	if (!sched_feat(WAKEUP_PREEMPTION))
 		return;
 
-	find_matching_se(&se, &pse);
 	WARN_ON_ONCE(!pse);
 
 	cse_is_idle = se_is_idle(se);
@@ -9944,8 +9864,7 @@ static void wakeup_preempt_fair(struct r
 	if (unlikely(!normal_policy(p->policy)))
 		return;
 
-	cfs_rq = cfs_rq_of(se);
-	update_curr(cfs_rq);
+	update_curr_fair(rq);
 	/*
 	 * If @p has a shorter slice than current and @p is eligible, override
 	 * current's slice protection in order to allow preemption.
@@ -9989,18 +9908,15 @@ static void wakeup_preempt_fair(struct r
 	}
 
 pick:
-	nse = pick_next_entity(rq, cfs_rq, preempt_action != PREEMPT_WAKEUP_SHORT);
-	/* If @p has become the most eligible task, force preemption */
-	if (nse == pse)
-		goto preempt;
-
-	/*
-	 * Because p is enqueued, nse being null can only mean that we
-	 * dequeued a delayed task. If there are still entities queued in
-	 * cfs, check if the next one will be p.
-	 */
-	if (!nse && cfs_rq->nr_queued)
-		goto pick;
+	if (cfs_rq->h_nr_queued) {
+		nse = pick_next_entity(rq, preempt_action != PREEMPT_WAKEUP_SHORT);
+		if (unlikely(!nse))
+			goto pick;
+
+		/* If @p has become the most eligible task, force preemption */
+		if (nse == pse)
+			goto preempt;
+	}
 
 	if (sched_feat(RUN_TO_PARITY))
 		update_protect_slice(cfs_rq, se);
@@ -10019,33 +9935,24 @@ static void wakeup_preempt_fair(struct r
 struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
 	__must_hold(__rq_lockp(rq))
 {
+	struct cfs_rq *cfs_rq = &rq->cfs;
 	struct sched_entity *se;
-	struct cfs_rq *cfs_rq;
 	struct task_struct *p;
-	bool throttled;
 	int new_tasks;
 
 again:
-	cfs_rq = &rq->cfs;
-	if (!cfs_rq->nr_queued)
+	if (!cfs_rq->h_nr_queued)
 		goto idle;
 
-	throttled = false;
-
-	do {
-		/* Might not have done put_prev_entity() */
-		if (cfs_rq->curr && cfs_rq->curr->on_rq)
-			update_curr(cfs_rq);
+	/* Might not have done put_prev_entity() */
+	if (cfs_rq->curr && cfs_rq->curr->on_rq)
+		update_curr(cfs_rq);
 
-		se = pick_next_entity(rq, cfs_rq, true);
-		if (!se)
-			goto again;
-		cfs_rq = group_cfs_rq(se);
-	} while (cfs_rq);
+	se = pick_next_entity(rq, true);
+	if (!se)
+		goto again;
 
 	p = task_of(se);
-	if (unlikely(throttled))
-		task_throttle_setup_work(p);
 	return p;
 
 idle:
@@ -10079,7 +9986,7 @@ void fair_server_init(struct rq *rq)
 static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct task_struct *next)
 {
 	struct sched_entity *se = &prev->se;
-	struct cfs_rq *cfs_rq;
+	struct cfs_rq *cfs_rq = &rq->cfs;
 	struct sched_entity *nse = NULL;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
@@ -10089,7 +9996,7 @@ static void put_prev_task_fair(struct rq
 
 	while (se) {
 		cfs_rq = cfs_rq_of(se);
-		if (!nse || cfs_rq->curr)
+		if (!nse || cfs_rq->h_curr)
 			put_prev_entity(cfs_rq, se);
 #ifdef CONFIG_FAIR_GROUP_SCHED
 		if (nse) {
@@ -10108,6 +10015,14 @@ static void put_prev_task_fair(struct rq
 #endif
 		se = parent_entity(se);
 	}
+
+	/* Put 'current' back into the tree. */
+	cfs_rq = &rq->cfs;
+	se = &prev->se;
+	WARN_ON_ONCE(cfs_rq->curr != se);
+	cfs_rq->curr = NULL;
+	if (se->on_rq)
+		__enqueue_entity(cfs_rq, se);
 }
 
 /*
@@ -10116,8 +10031,8 @@ static void put_prev_task_fair(struct rq
 static void yield_task_fair(struct rq *rq)
 {
 	struct task_struct *curr = rq->donor;
-	struct cfs_rq *cfs_rq = task_cfs_rq(curr);
 	struct sched_entity *se = &curr->se;
+	struct cfs_rq *cfs_rq = &rq->cfs;
 
 	/*
 	 * Are we the only task in the tree?
@@ -10158,11 +10073,11 @@ static bool yield_to_task_fair(struct rq
 	struct sched_entity *se = &p->se;
 
 	/* !se->on_rq also covers throttled task */
-	if (!se->on_rq)
+	if (!se->on_rq || se->sched_delayed)
 		return false;
 
 	/* Tell the scheduler that we'd really like se to run next. */
-	set_next_buddy(se);
+	set_next_buddy(&task_rq(p)->cfs, se);
 
 	yield_task_fair(rq);
 
@@ -10501,15 +10416,10 @@ static inline long migrate_degrades_loca
  */
 static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_cpu)
 {
-	struct cfs_rq *dst_cfs_rq;
+	struct cfs_rq *dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
 
-#ifdef CONFIG_FAIR_GROUP_SCHED
-	dst_cfs_rq = tg_cfs_rq(task_group(p), dest_cpu);
-#else
-	dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
-#endif
-	if (sched_feat(PLACE_LAG) && dst_cfs_rq->nr_queued &&
-	    !entity_eligible(task_cfs_rq(p), &p->se))
+	if (sched_feat(PLACE_LAG) && dst_cfs_rq->h_nr_queued &&
+	    !entity_eligible(&task_rq(p)->cfs, &p->se))
 		return 1;
 
 	return 0;
@@ -11292,7 +11202,7 @@ static void update_cfs_rq_h_load(struct
 	while ((se = READ_ONCE(cfs_rq->h_load_next)) != NULL) {
 		load = cfs_rq->h_load;
 		load = div64_ul(load * se->avg.load_avg,
-			cfs_rq_load_avg(cfs_rq) + 1);
+				cfs_rq_load_avg(cfs_rq) + 1);
 		cfs_rq = group_cfs_rq(se);
 		cfs_rq->h_load = load;
 		cfs_rq->last_h_load_update = now;
@@ -14684,7 +14594,7 @@ static inline void task_tick_core(struct
 	 * MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check
 	 * if we need to give up the CPU.
 	 */
-	if (rq->core->core_forceidle_count && rq->cfs.nr_queued == 1 &&
+	if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued == 1 &&
 	    __entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))
 		resched_curr(rq);
 }
@@ -14893,30 +14803,8 @@ bool cfs_prio_less(const struct task_str
 
 	WARN_ON_ONCE(task_rq(b)->core != rq->core);
 
-#ifdef CONFIG_FAIR_GROUP_SCHED
-	/*
-	 * Find an se in the hierarchy for tasks a and b, such that the se's
-	 * are immediate siblings.
-	 */
-	while (sea->cfs_rq->tg != seb->cfs_rq->tg) {
-		int sea_depth = sea->depth;
-		int seb_depth = seb->depth;
-
-		if (sea_depth >= seb_depth)
-			sea = parent_entity(sea);
-		if (sea_depth <= seb_depth)
-			seb = parent_entity(seb);
-	}
-
-	se_fi_update(sea, rq->core->core_forceidle_seq, in_fi);
-	se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);
-
-	cfs_rqa = sea->cfs_rq;
-	cfs_rqb = seb->cfs_rq;
-#else /* !CONFIG_FAIR_GROUP_SCHED: */
 	cfs_rqa = &task_rq(a)->cfs;
 	cfs_rqb = &task_rq(b)->cfs;
-#endif /* !CONFIG_FAIR_GROUP_SCHED */
 
 	/*
 	 * Find delta after normalizing se's vruntime with its cfs_rq's
@@ -14955,11 +14843,20 @@ static inline void task_tick_core(struct
 static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 {
 	struct sched_entity *se = &curr->se;
-	struct cfs_rq *cfs_rq;
 
-	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
-		entity_tick(cfs_rq, se, queued);
+	if (se->on_rq) {
+		unsigned long weight = NICE_0_LOAD;
+		struct cfs_rq *cfs_rq;
+
+		for_each_sched_entity(se) {
+			cfs_rq = cfs_rq_of(se);
+			entity_tick(cfs_rq, se, queued);
+
+			weight = __calc_prop_weight(cfs_rq, se, weight);
+		}
+
+		se = &curr->se;
+		reweight_eevdf(cfs_rq, se, weight, se->on_rq);
 	}
 
 	if (queued)
@@ -14999,7 +14896,7 @@ prio_changed_fair(struct rq *rq, struct
 	if (p->prio == oldprio)
 		return;
 
-	if (rq->cfs.nr_queued == 1)
+	if (rq->cfs.h_nr_queued == 1)
 		return;
 
 	/*
@@ -15128,33 +15025,44 @@ static void switched_to_fair(struct rq *
 	}
 }
 
-/*
- * Account for a task changing its policy or group.
- *
- * This routine is mostly called to set cfs_rq->curr field when a task
- * migrates between groups/classes.
- */
 static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 {
 	struct sched_entity *se = &p->se;
 	bool throttled = false;
+	struct cfs_rq *cfs_rq = &rq->cfs;
+	unsigned long weight = NICE_0_LOAD;
+	bool on_rq = se->on_rq;
+
+	clear_buddies(cfs_rq, se);
+
+	if (on_rq)
+		__dequeue_entity(cfs_rq, se);
 
 	for_each_sched_entity(se) {
-		struct cfs_rq *cfs_rq = cfs_rq_of(se);
+		cfs_rq = cfs_rq_of(se);
 
-		if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
-		    first && cfs_rq->curr)
-			break;
+		if (!IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) ||
+		    !first || !cfs_rq->h_curr)
+			set_next_entity(cfs_rq, se);
 
-		set_next_entity(cfs_rq, se, first);
 		/* ensure bandwidth has been allocated on our new cfs_rq */
 		throttled |= account_cfs_rq_runtime(cfs_rq, 0);
+
+		if (on_rq)
+			weight = __calc_prop_weight(cfs_rq, se, weight);
 	}
 
 	if (throttled)
 		task_throttle_setup_work(p);
 
 	se = &p->se;
+	cfs_rq->curr = se;
+
+	if (on_rq) {
+		reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+		if (first)
+			set_protect_slice(cfs_rq, se);
+	}
 
 	if (task_on_rq_queued(p)) {
 		/*
@@ -15267,17 +15175,8 @@ void unregister_fair_sched_group(struct
 		struct sched_entity *se = tg_se(tg, cpu);
 		struct rq *rq = cpu_rq(cpu);
 
-		if (se) {
-			if (se->sched_delayed) {
-				guard(rq_lock_irqsave)(rq);
-				if (se->sched_delayed) {
-					update_rq_clock(rq);
-					dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-				}
-				list_del_leaf_cfs_rq(cfs_rq);
-			}
+		if (se)
 			remove_entity_load_avg(se);
-		}
 
 		/*
 		 * Only empty task groups can be destroyed; so we can speculatively
--- a/kernel/sched/pelt.c
+++ b/kernel/sched/pelt.c
@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched
 	/*
 	 * running is a subset of runnable (weight) so running can't be set if
 	 * runnable is clear. But there are some corner cases where the current
-	 * se has been already dequeued but cfs_rq->curr still points to it.
+	 * se has been already dequeued but cfs_rq->h_curr still points to it.
 	 * This means that weight will be 0 but not running for a sched_entity
 	 * but also for a cfs_rq if the latter becomes idle. As an example,
 	 * this happens during sched_balance_newidle() which calls
@@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now
 int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se),
-				cfs_rq->curr == se)) {
+				cfs_rq->h_curr == se)) {
 
 		___update_load_avg(&se->avg, se_weight(se));
 		cfs_se_util_change(&se->avg);
@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, st
 	if (___update_load_sum(now, &cfs_rq->avg,
 				scale_load_down(cfs_rq->load.weight),
 				cfs_rq->h_nr_runnable,
-				cfs_rq->curr != NULL)) {
+				cfs_rq->h_curr != NULL)) {
 
 		___update_load_avg(&cfs_rq->avg, 1);
 		trace_pelt_cfs_tp(cfs_rq);
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -530,21 +530,8 @@ struct task_group {
 
 };
 
-#ifdef CONFIG_GROUP_SCHED_WEIGHT
 #define ROOT_TASK_GROUP_LOAD	NICE_0_LOAD
 
-/*
- * A weight of 0 or 1 can cause arithmetics problems.
- * A weight of a cfs_rq is the sum of weights of which entities
- * are queued on this cfs_rq, so a weight of a entity should not be
- * too large, so as the shares value of a task group.
- * (The default weight is 1024 - so there's no practical
- *  limitation from this.)
- */
-#define MIN_SHARES		(1UL <<  1)
-#define MAX_SHARES		(1UL << 18)
-#endif
-
 typedef int (*tg_visitor)(struct task_group *, void *);
 
 extern int walk_tg_tree_from(struct task_group *from,
@@ -631,6 +618,17 @@ static inline bool cfs_task_bw_constrain
 
 #endif /* !CONFIG_CGROUP_SCHED */
 
+/*
+ * A weight of 0 or 1 can cause arithmetics problems.
+ * A weight of a cfs_rq is the sum of weights of which entities
+ * are queued on this cfs_rq, so a weight of a entity should not be
+ * too large, so as the shares value of a task group.
+ * (The default weight is 1024 - so there's no practical
+ *  limitation from this.)
+ */
+#define MIN_SHARES		(1UL <<  1)
+#define MAX_SHARES		(1UL << 18)
+
 extern void unregister_rt_sched_group(struct task_group *tg);
 extern void free_rt_sched_group(struct task_group *tg);
 extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent);
@@ -709,6 +707,7 @@ struct cfs_rq {
 	/*
 	 * CFS load tracking
 	 */
+	struct sched_entity	*h_curr;
 	struct sched_avg	avg;
 #ifndef CONFIG_64BIT
 	u64			last_update_time_copy;
@@ -2575,6 +2574,7 @@ extern const u32		sched_prio_to_wmult[40
 #define ENQUEUE_MIGRATED	0x00040000
 #define ENQUEUE_INITIAL		0x00080000
 #define ENQUEUE_RQ_SELECTED	0x00100000
+#define ENQUEUE_QUEUED		0x00200000
 
 #define RETRY_TASK		((void *)-1UL)
 

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Chen, Yu C]

On 6/5/2026 8:40 PM, Peter Zijlstra wrote:
> Change fair/cgroup to a single runqueue.
> 
> Infamously fair/cgroup isn't working for a number of people; typically
> the complaint is latencies and/or overhead. The latency issue is due
> to the intermediate entries that represent a combination of tasks and
> thereby obfuscate the runnability of tasks.
> 
> The approach here is to leave the cgroup hierarchy as is; including
> the intermediate enqueue/dequeue but move the actual EEVDF runqueue
> outside. This means things like the shares_weight approximation are
> fully preserved.
> 
> That is, given a hierarchy like:
> 
>            R
>            |
>            se--G1
>                / \
>          G2--se   se--G3
>         / \           |
>    T1--se se--T2      se--T3
> 
> This is fully maintained for load tracking, however the EEVDF parts of
> cfs_rq/se go unused for the intermediates and are instead connected
> like:
> 
>       _R_
>      / | \
>     T1 T2 T3
> 
> Since the effective weight of the entities is determined by the
> hierarchy, this gets recomputed on enqueue,set_next_task and tick.
> 
> Notably, the effective weight (se->h_load) is computed from the
> hierarchical fraction: se->load / cfs_rq->load.
> 
> Since EEVDF is now exclusively operating on rq->cfs, it needs to
> consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly,
> only tasks can get delayed, simplifying some of the cgroup cleanup.
> 
> One place where additional information was required was
> set_next_task() / put_prev_task(), where we need to track 'current'
> both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense
> (cfs_rq->curr).
> 
> As a result of only having a single level to pick from, much of the
> complications in pick_next_task() and preemption go away.
> 
> Since many of the hierarchical operations are still there, this won't
> immediately fix the performance issues, but hopefully it will fix some
> of the latency issues.
> 
> TODO: split struct cfs_rq / struct sched_entity
> TODO: try and get rid of h_curr
> 
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>

A divide-by-zero crash is observed when running hackbench:

   [14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not 
tainted 7.1.0-rc2+
   [14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0
   [14697.506799]  <TASK>
   [14697.507411]  __dequeue_task+0x2b4/0xc70
   [14697.508677]  dequeue_task_fair+0x36/0x370
   [14697.509047]  dequeue_task+0x101/0x2f0
   [14697.509426]  __schedule+0x1b1/0x1a00
   [14697.510868]  anon_pipe_read+0x3da/0x450
   [14697.511400]  vfs_read+0x361/0x390
   [14697.512053]  __x64_sys_read+0x19/0x30

The divide-by-zero happens here:

if (scale_load_down(gcfs_rq->load.weight)) {
         load_sum = div_u64(gcfs_rq->avg.load_sum,
                 scale_load_down(gcfs_rq->load.weight));
}

gcfs_rq->load.weight is an insane large value and is truncated
to the lower 32 bits by div_u64, which happen to be 0.

Using AI for investigation, the cause is a u32 overflow in
update_tg_cfs_runnable(), and flat pickup became a victim when using
tg_tasks():

   u32 new_sum, divider;
   ...
   new_sum = se->avg.runnable_avg * divider; <-- boom

The following sequence shows how this triggers the crash:

   propagate_entity_load_avg()
     update_tg_cfs_runnable()     # u32 overflow corrupts runnable_sum

   __update_load_avg_cfs_rq()
     ___update_load_avg()         # computes insane runnable_avg
   update_tg_load_avg()           # propagates to tg->runnable_avg

   update_cfs_group()
     calc_concur_shares()
       tg_tasks()                 # long-to-int truncation, negative nr
     reweight_entity()            # corrupted se->load.weight
       update_load_add()          # corrupted cfs_rq->load.weight

   propagate_entity_load_avg()
     update_tg_cfs_load()
       div_u64()                  # divide-by-zero

Fix by widening new_sum from u32 to u64(no need to force tg_tasks()
to return unsigned long after this fix)
Assisted-by: Claude:claude-opus-4.6
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
---
  kernel/sched/fair.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d991ea85873a..99ea51448981 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5305,7 +5305,8 @@ static inline void
  update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, 
struct cfs_rq *gcfs_rq)
  {
  	long delta_sum, delta_avg = gcfs_rq->avg.runnable_avg - 
se->avg.runnable_avg;
-	u32 new_sum, divider;
+	u64 new_sum;
+	u32 divider;

  	/* Nothing to update */
  	if (!delta_avg)
@@ -5319,7 +5320,7 @@ update_tg_cfs_runnable(struct cfs_rq *cfs_rq, 
struct sched_entity *se, struct cf

  	/* Set new sched_entity's runnable */
  	se->avg.runnable_avg = gcfs_rq->avg.runnable_avg;
-	new_sum = se->avg.runnable_avg * divider;
+	new_sum = (u64)se->avg.runnable_avg * divider;
  	delta_sum = (long)new_sum - (long)se->avg.runnable_sum;
  	se->avg.runnable_sum = new_sum;

-- 
2.45.2

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Peter Zijlstra]

On Sat, Jun 20, 2026 at 11:54:22AM +0800, Chen, Yu C wrote:

> A divide-by-zero crash is observed when running hackbench:
> 
>   [14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not tainted
> 7.1.0-rc2+
>   [14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0
>   [14697.506799]  <TASK>
>   [14697.507411]  __dequeue_task+0x2b4/0xc70
>   [14697.508677]  dequeue_task_fair+0x36/0x370
>   [14697.509047]  dequeue_task+0x101/0x2f0
>   [14697.509426]  __schedule+0x1b1/0x1a00
>   [14697.510868]  anon_pipe_read+0x3da/0x450
>   [14697.511400]  vfs_read+0x361/0x390
>   [14697.512053]  __x64_sys_read+0x19/0x30
> 
> The divide-by-zero happens here:
> 
> if (scale_load_down(gcfs_rq->load.weight)) {
>         load_sum = div_u64(gcfs_rq->avg.load_sum,
>                 scale_load_down(gcfs_rq->load.weight));
> }
> 
> gcfs_rq->load.weight is an insane large value and is truncated
> to the lower 32 bits by div_u64, which happen to be 0.
> 
> Using AI for investigation, the cause is a u32 overflow in
> update_tg_cfs_runnable(), and flat pickup became a victim when using
> tg_tasks():
> 
>   u32 new_sum, divider;
>   ...
>   new_sum = se->avg.runnable_avg * divider; <-- boom
> 
> The following sequence shows how this triggers the crash:
> 
>   propagate_entity_load_avg()
>     update_tg_cfs_runnable()     # u32 overflow corrupts runnable_sum
> 
>   __update_load_avg_cfs_rq()
>     ___update_load_avg()         # computes insane runnable_avg
>   update_tg_load_avg()           # propagates to tg->runnable_avg
> 
>   update_cfs_group()
>     calc_concur_shares()
>       tg_tasks()                 # long-to-int truncation, negative nr
>     reweight_entity()            # corrupted se->load.weight
>       update_load_add()          # corrupted cfs_rq->load.weight
> 
>   propagate_entity_load_avg()
>     update_tg_cfs_load()
>       div_u64()                  # divide-by-zero
> 
> Fix by widening new_sum from u32 to u64(no need to force tg_tasks()
> to return unsigned long after this fix)
> Assisted-by: Claude:claude-opus-4.6
> Signed-off-by: Chen Yu <yu.c.chen@intel.com>
> ---
>  kernel/sched/fair.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index d991ea85873a..99ea51448981 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -5305,7 +5305,8 @@ static inline void
>  update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cfs_rq *gcfs_rq)
>  {
>  	long delta_sum, delta_avg = gcfs_rq->avg.runnable_avg - se->avg.runnable_avg;
> -	u32 new_sum, divider;
> +	u64 new_sum;
> +	u32 divider;
> 
>  	/* Nothing to update */
>  	if (!delta_avg)
> @@ -5319,7 +5320,7 @@ update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cf
> 
>  	/* Set new sched_entity's runnable */
>  	se->avg.runnable_avg = gcfs_rq->avg.runnable_avg;
> -	new_sum = se->avg.runnable_avg * divider;
> +	new_sum = (u64)se->avg.runnable_avg * divider;
>  	delta_sum = (long)new_sum - (long)se->avg.runnable_sum;
>  	se->avg.runnable_sum = new_sum;

Hmm, nice one. This makes sense because sched_avg::runnable_sum is a u64
itself, so having the delta be one is only sensible.

I do wonder though, this doesn't actually look to be specific to flat,
it just managed to trip it somehow.

I'll stick this on as a separate fix.

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Vincent Guittot]

On Fri, 26 Jun 2026 at 13:40, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Sat, Jun 20, 2026 at 11:54:22AM +0800, Chen, Yu C wrote:
>
> > A divide-by-zero crash is observed when running hackbench:
> >
> >   [14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not tainted
> > 7.1.0-rc2+
> >   [14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0
> >   [14697.506799]  <TASK>
> >   [14697.507411]  __dequeue_task+0x2b4/0xc70
> >   [14697.508677]  dequeue_task_fair+0x36/0x370
> >   [14697.509047]  dequeue_task+0x101/0x2f0
> >   [14697.509426]  __schedule+0x1b1/0x1a00
> >   [14697.510868]  anon_pipe_read+0x3da/0x450
> >   [14697.511400]  vfs_read+0x361/0x390
> >   [14697.512053]  __x64_sys_read+0x19/0x30
> >
> > The divide-by-zero happens here:
> >
> > if (scale_load_down(gcfs_rq->load.weight)) {
> >         load_sum = div_u64(gcfs_rq->avg.load_sum,
> >                 scale_load_down(gcfs_rq->load.weight));
> > }
> >
> > gcfs_rq->load.weight is an insane large value and is truncated
> > to the lower 32 bits by div_u64, which happen to be 0.
> >
> > Using AI for investigation, the cause is a u32 overflow in
> > update_tg_cfs_runnable(), and flat pickup became a victim when using
> > tg_tasks():
> >
> >   u32 new_sum, divider;
> >   ...
> >   new_sum = se->avg.runnable_avg * divider; <-- boom
> >
> > The following sequence shows how this triggers the crash:
> >
> >   propagate_entity_load_avg()
> >     update_tg_cfs_runnable()     # u32 overflow corrupts runnable_sum
> >
> >   __update_load_avg_cfs_rq()
> >     ___update_load_avg()         # computes insane runnable_avg
> >   update_tg_load_avg()           # propagates to tg->runnable_avg
> >
> >   update_cfs_group()
> >     calc_concur_shares()
> >       tg_tasks()                 # long-to-int truncation, negative nr
> >     reweight_entity()            # corrupted se->load.weight
> >       update_load_add()          # corrupted cfs_rq->load.weight
> >
> >   propagate_entity_load_avg()
> >     update_tg_cfs_load()
> >       div_u64()                  # divide-by-zero
> >
> > Fix by widening new_sum from u32 to u64(no need to force tg_tasks()
> > to return unsigned long after this fix)
> > Assisted-by: Claude:claude-opus-4.6
> > Signed-off-by: Chen Yu <yu.c.chen@intel.com>
> > ---
> >  kernel/sched/fair.c | 5 +++--
> >  1 file changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index d991ea85873a..99ea51448981 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -5305,7 +5305,8 @@ static inline void
> >  update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cfs_rq *gcfs_rq)
> >  {
> >       long delta_sum, delta_avg = gcfs_rq->avg.runnable_avg - se->avg.runnable_avg;
> > -     u32 new_sum, divider;
> > +     u64 new_sum;
> > +     u32 divider;
> >
> >       /* Nothing to update */
> >       if (!delta_avg)
> > @@ -5319,7 +5320,7 @@ update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cf
> >
> >       /* Set new sched_entity's runnable */
> >       se->avg.runnable_avg = gcfs_rq->avg.runnable_avg;
> > -     new_sum = se->avg.runnable_avg * divider;
> > +     new_sum = (u64)se->avg.runnable_avg * divider;
> >       delta_sum = (long)new_sum - (long)se->avg.runnable_sum;
> >       se->avg.runnable_sum = new_sum;
>
> Hmm, nice one. This makes sense because sched_avg::runnable_sum is a u64
> itself, so having the delta be one is only sensible.
>
> I do wonder though, this doesn't actually look to be specific to flat,
> it just managed to trip it somehow.

I think, it's specific to the new share policies introduced with flat.
Before flat we were using:
return clamp_t(long, shares, MIN_SHARES, tg_shares); where tg_shares
was clamp(shares, scale_load(MIN_SHARES), scale_load(MAX_SHARES));

With flat, we are using:
return clamp_t(long, shares, MIN_SHARES, shares_max); where shares_max can be:
    max(1, atomic_long_read(&tg->runnable_avg) >>
SCHED_CAPACITY_SHIFT) * clamp(shares, scale_load(MIN_SHARES),
scale_load(MAX_SHARES))





>
> I'll stick this on as a separate fix.

[tip: sched/core] sched/fair: Fix overflow in update_tg_cfs_runnable()

[tip-bot2 for Chen, Yu C]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     4f166adb5cb0525d9e32d45729fd8f28c80acbee
Gitweb:        https://git.kernel.org/tip/4f166adb5cb0525d9e32d45729fd8f28c80acbee
Author:        Chen, Yu C <yu.c.chen@intel.com>
AuthorDate:    Sat, 20 Jun 2026 11:54:22 +08:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 30 Jun 2026 10:56:51 +02:00

sched/fair: Fix overflow in update_tg_cfs_runnable()

A divide-by-zero crash is observed when running hackbench:

   [14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not tainted 7.1.0-rc2+
   [14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0
   [14697.506799]  <TASK>
   [14697.507411]  __dequeue_task+0x2b4/0xc70
   [14697.508677]  dequeue_task_fair+0x36/0x370
   [14697.509047]  dequeue_task+0x101/0x2f0
   [14697.509426]  __schedule+0x1b1/0x1a00
   [14697.510868]  anon_pipe_read+0x3da/0x450
   [14697.511400]  vfs_read+0x361/0x390
   [14697.512053]  __x64_sys_read+0x19/0x30

The divide-by-zero happens here:

  if (scale_load_down(gcfs_rq->load.weight)) {
           load_sum = div_u64(gcfs_rq->avg.load_sum,
                              scale_load_down(gcfs_rq->load.weight));
  }

gcfs_rq->load.weight is an insane large value and is truncated
to the lower 32 bits by div_u64, which happen to be 0.

Using AI for investigation, the cause is a u32 overflow in
update_tg_cfs_runnable(), and flat pickup became a victim when using
tg_tasks():

   u32 new_sum, divider;
   ...
   new_sum = se->avg.runnable_avg * divider; <-- boom

The following sequence shows how this triggers the crash:

   propagate_entity_load_avg()
     update_tg_cfs_runnable()     # u32 overflow corrupts runnable_sum

   __update_load_avg_cfs_rq()
     ___update_load_avg()         # computes insane runnable_avg
   update_tg_load_avg()           # propagates to tg->runnable_avg

   update_cfs_group()
     calc_concur_shares()
       tg_tasks()                 # long-to-int truncation, negative nr
     reweight_entity()            # corrupted se->load.weight
       update_load_add()          # corrupted cfs_rq->load.weight

   propagate_entity_load_avg()
     update_tg_cfs_load()
       div_u64()                  # divide-by-zero

Fix by widening new_sum from u32 to u64 (no need to force tg_tasks()
to return unsigned long after this fix)

Fixes: 95246d1ec80b ("sched/pelt: Relax the sync of runnable_sum with runnable_avg")
Assisted-by: Claude:claude-opus-4.6
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/a22eea2b-4c4a-4623-9a44-d7b18c0c91c8@intel.com
---
 kernel/sched/fair.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index a384f16..e89edbd 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5174,7 +5174,8 @@ static inline void
 update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cfs_rq *gcfs_rq)
 {
 	long delta_sum, delta_avg = gcfs_rq->avg.runnable_avg - se->avg.runnable_avg;
-	u32 new_sum, divider;
+	u64 new_sum;
+	u32 divider;
 
 	/* Nothing to update */
 	if (!delta_avg)
@@ -5188,7 +5189,7 @@ update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cf
 
 	/* Set new sched_entity's runnable */
 	se->avg.runnable_avg = gcfs_rq->avg.runnable_avg;
-	new_sum = se->avg.runnable_avg * divider;
+	new_sum = (u64)se->avg.runnable_avg * divider;
 	delta_sum = (long)new_sum - (long)se->avg.runnable_sum;
 	se->avg.runnable_sum = new_sum;
 

[tip: sched/core] sched/eevdf: Move to a single runqueue

[tip-bot2 for Peter Zijlstra (Intel)]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     85570f10a4c61372c1d437365b9d8dbad512ec6f
Gitweb:        https://git.kernel.org/tip/85570f10a4c61372c1d437365b9d8dbad512ec6f
Author:        Peter Zijlstra (Intel) <peterz@infradead.org>
AuthorDate:    Sat, 06 Dec 2025 10:08:58 +01:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 30 Jun 2026 10:56:53 +02:00

sched/eevdf: Move to a single runqueue

Change fair/cgroup to a single runqueue.

Infamously fair/cgroup isn't working for a number of people; typically
the complaint is latencies and/or overhead. The latency issue is due
to the intermediate entries that represent a combination of tasks and
thereby obfuscate the runnability of tasks.

The approach here is to leave the cgroup hierarchy as is; including
the intermediate enqueue/dequeue but move the actual EEVDF runqueue
outside. This means things like the shares_weight approximation are
fully preserved.

That is, given a hierarchy like:

          R
          |
          se--G1
              / \
        G2--se   se--G3
       / \           |
  T1--se se--T2      se--T3

This is fully maintained for load tracking, however the EEVDF parts of
cfs_rq/se go unused for the intermediates and are instead connected
like:

     _R_
    / | \
   T1 T2 T3

Since the effective weight of the entities is determined by the
hierarchy, this gets recomputed on enqueue,set_next_task and tick.

Notably, the effective weight (se->h_load) is computed from the
hierarchical fraction: se->load / cfs_rq->load.

Since EEVDF is now exclusively operating on rq->cfs, it needs to
consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly,
only tasks can get delayed, simplifying some of the cgroup cleanup.

One place where additional information was required was
set_next_task() / put_prev_task(), where we need to track 'current'
both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense
(cfs_rq->curr).

As a result of only having a single level to pick from, much of the
complications in pick_next_task() and preemption go away.

Since many of the hierarchical operations are still there, this won't
immediately fix the performance issues, but hopefully it will fix some
of the latency issues.

TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124052.227463677%40infradead.org
---
 include/linux/sched.h |   1 +-
 kernel/sched/core.c   |   5 +-
 kernel/sched/debug.c  |   9 +-
 kernel/sched/fair.c   | 801 +++++++++++++++++------------------------
 kernel/sched/pelt.c   |   6 +-
 kernel/sched/sched.h  |  26 +-
 6 files changed, 374 insertions(+), 474 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 373bcc0..12f6335 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -575,6 +575,7 @@ struct sched_statistics {
 struct sched_entity {
 	/* For load-balancing: */
 	struct load_weight		load;
+	struct load_weight		h_load;
 	struct rb_node			run_node;
 	u64				deadline;
 	u64				min_vruntime;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 9622670..2e7cde0 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5657,11 +5657,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);
  */
 static inline void prefetch_curr_exec_start(struct task_struct *p)
 {
-#ifdef CONFIG_FAIR_GROUP_SCHED
-	struct sched_entity *curr = p->se.cfs_rq->curr;
-#else
 	struct sched_entity *curr = task_rq(p)->cfs.curr;
-#endif
+
 	prefetch(curr);
 	prefetch(&curr->exec_start);
 }
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index f54e732..c15291c 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -975,10 +975,11 @@ print_task(struct seq_file *m, struct rq *rq, struct task_struct *p)
 	else
 		SEQ_printf(m, " %c", task_state_to_char(p));
 
-	SEQ_printf(m, " %15s %5d %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
+	SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
 		p->comm, task_pid_nr(p),
+		p->se.h_load.weight,
 		SPLIT_NS(p->se.vruntime),
-		entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
+		entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N',
 		SPLIT_NS(p->se.deadline),
 		p->se.custom_slice ? 'S' : ' ',
 		SPLIT_NS(p->se.slice),
@@ -1007,7 +1008,7 @@ static void print_rq(struct seq_file *m, struct rq *rq, int rq_cpu)
 
 	SEQ_printf(m, "\n");
 	SEQ_printf(m, "runnable tasks:\n");
-	SEQ_printf(m, " S            task   PID       vruntime   eligible    "
+	SEQ_printf(m, " S            task   PID     weight       vruntime   eligible    "
 		   "deadline             slice          sum-exec      switches  "
 		   "prio         wait-time        sum-sleep       sum-block"
 #ifdef CONFIG_NUMA_BALANCING
@@ -1115,6 +1116,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 			cfs_rq->tg_load_avg_contrib);
 	SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_avg",
 			atomic_long_read(&cfs_rq->tg->load_avg));
+	SEQ_printf(m, "  .%-30s: %lu\n", "h_load",
+			cfs_rq->h_load);
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 #ifdef CONFIG_CFS_BANDWIDTH
 	SEQ_printf(m, "  .%-30s: %d\n", "throttled",
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 1608c01..cfaed2c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -297,8 +297,8 @@ static u64 __calc_delta(u64 delta_exec, unsigned long weight, struct load_weight
  */
 static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)
 {
-	if (unlikely(se->load.weight != NICE_0_LOAD))
-		delta = __calc_delta(delta, NICE_0_LOAD, &se->load);
+	if (se->h_load.weight != NICE_0_LOAD)
+		delta = __calc_delta(delta, NICE_0_LOAD, &se->h_load);
 
 	return delta;
 }
@@ -428,38 +428,6 @@ static inline struct sched_entity *parent_entity(const struct sched_entity *se)
 	return se->parent;
 }
 
-static void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
-	int se_depth, pse_depth;
-
-	/*
-	 * preemption test can be made between sibling entities who are in the
-	 * same cfs_rq i.e who have a common parent. Walk up the hierarchy of
-	 * both tasks until we find their ancestors who are siblings of common
-	 * parent.
-	 */
-
-	/* First walk up until both entities are at same depth */
-	se_depth = (*se)->depth;
-	pse_depth = (*pse)->depth;
-
-	while (se_depth > pse_depth) {
-		se_depth--;
-		*se = parent_entity(*se);
-	}
-
-	while (pse_depth > se_depth) {
-		pse_depth--;
-		*pse = parent_entity(*pse);
-	}
-
-	while (!is_same_group(*se, *pse)) {
-		*se = parent_entity(*se);
-		*pse = parent_entity(*pse);
-	}
-}
-
 static int tg_is_idle(struct task_group *tg)
 {
 	return tg->idle > 0;
@@ -503,11 +471,6 @@ static inline struct sched_entity *parent_entity(struct sched_entity *se)
 	return NULL;
 }
 
-static inline void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
-}
-
 static inline int tg_is_idle(struct task_group *tg)
 {
 	return 0;
@@ -686,7 +649,7 @@ static inline unsigned long avg_vruntime_weight(struct cfs_rq *cfs_rq, unsigned 
 static inline void
 __sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+	unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
 	s64 w_vruntime, key = entity_key(cfs_rq, se);
 
 	w_vruntime = key * weight;
@@ -703,7 +666,7 @@ sum_w_vruntime_add_paranoid(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	s64 key, tmp;
 
 again:
-	weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+	weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
 	key = entity_key(cfs_rq, se);
 
 	if (check_mul_overflow(key, weight, &key))
@@ -749,7 +712,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se)
 static void
 sum_w_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+	unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
 	s64 key = entity_key(cfs_rq, se);
 
 	cfs_rq->sum_w_vruntime -= key * weight;
@@ -791,7 +754,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)
 		s64 runtime = cfs_rq->sum_w_vruntime;
 
 		if (curr) {
-			unsigned long w = avg_vruntime_weight(cfs_rq, curr->load.weight);
+			unsigned long w = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
 			runtime += entity_key(cfs_rq, curr) * w;
 			weight += w;
@@ -862,8 +825,6 @@ bool update_entity_lag(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	u64 avruntime = avg_vruntime(cfs_rq);
 	s64 vlag = entity_lag(cfs_rq, se, avruntime);
 
-	WARN_ON_ONCE(!se->on_rq);
-
 	if (se->sched_delayed) {
 		/* previous vlag < 0 otherwise se would not be delayed */
 		vlag = max(vlag, se->vlag);
@@ -899,7 +860,7 @@ static int vruntime_eligible(struct cfs_rq *cfs_rq, u64 vruntime)
 	long load = cfs_rq->sum_weight;
 
 	if (curr && curr->on_rq) {
-		unsigned long weight = avg_vruntime_weight(cfs_rq, curr->load.weight);
+		unsigned long weight = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
 		avg += entity_key(cfs_rq, curr) * weight;
 		load += weight;
@@ -1040,6 +1001,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntime_cb, struct sched_entity,
  */
 static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+	WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+	WARN_ON_ONCE(!entity_is_task(se));
+
 	sum_w_vruntime_add(cfs_rq, se);
 	se->min_vruntime = se->vruntime;
 	se->min_slice = se->slice;
@@ -1049,6 +1013,9 @@ static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 
 static void __dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+	WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+	WARN_ON_ONCE(!entity_is_task(se));
+
 	rb_erase_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline,
 				  &min_vruntime_cb);
 	sum_w_vruntime_sub(cfs_rq, se);
@@ -1145,7 +1112,7 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq, bool protect)
 	 * We can safely skip eligibility check if there is only one entity
 	 * in this cfs_rq, saving some cycles.
 	 */
-	if (cfs_rq->nr_queued == 1)
+	if (cfs_rq->h_nr_queued == 1)
 		return curr && curr->on_rq ? curr : se;
 
 	/*
@@ -1395,8 +1362,6 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)
 	return delta_exec;
 }
 
-static void set_next_buddy(struct sched_entity *se);
-
 #ifdef CONFIG_SCHED_CACHE
 
 /*
@@ -1991,7 +1956,7 @@ static void update_curr(struct cfs_rq *cfs_rq)
 	 * not necessarily be the actual task running
 	 * (rq->curr.se). This is easy to confuse!
 	 */
-	struct sched_entity *curr = cfs_rq->curr;
+	struct sched_entity *curr = cfs_rq->h_curr;
 	struct rq *rq = rq_of(cfs_rq);
 	s64 delta_exec;
 	bool resched;
@@ -2003,26 +1968,29 @@ static void update_curr(struct cfs_rq *cfs_rq)
 	if (unlikely(delta_exec <= 0))
 		return;
 
+	account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+	if (!entity_is_task(curr))
+		return;
+
+	cfs_rq = &rq->cfs;
+
 	curr->vruntime += calc_delta_fair(delta_exec, curr);
 	resched = update_deadline(cfs_rq, curr);
 
-	if (entity_is_task(curr)) {
-		/*
-		 * If the fair_server is active, we need to account for the
-		 * fair_server time whether or not the task is running on
-		 * behalf of fair_server or not:
-		 *  - If the task is running on behalf of fair_server, we need
-		 *    to limit its time based on the assigned runtime.
-		 *  - Fair task that runs outside of fair_server should account
-		 *    against fair_server such that it can account for this time
-		 *    and possibly avoid running this period.
-		 */
-		dl_server_update(&rq->fair_server, delta_exec);
-	}
-
-	account_cfs_rq_runtime(cfs_rq, delta_exec);
+	/*
+	 * If the fair_server is active, we need to account for the
+	 * fair_server time whether or not the task is running on
+	 * behalf of fair_server or not:
+	 *  - If the task is running on behalf of fair_server, we need
+	 *    to limit its time based on the assigned runtime.
+	 *  - Fair task that runs outside of fair_server should account
+	 *    against fair_server such that it can account for this time
+	 *    and possibly avoid running this period.
+	 */
+	dl_server_update(&rq->fair_server, delta_exec);
 
-	if (cfs_rq->nr_queued == 1)
+	if (cfs_rq->h_nr_queued == 1)
 		return;
 
 	if (resched || !protect_slice(curr)) {
@@ -2033,7 +2001,10 @@ static void update_curr(struct cfs_rq *cfs_rq)
 
 static void update_curr_fair(struct rq *rq)
 {
-	update_curr(cfs_rq_of(&rq->donor->se));
+	struct sched_entity *se = &rq->donor->se;
+
+	for_each_sched_entity(se)
+		update_curr(cfs_rq_of(se));
 }
 
 static inline void
@@ -2109,7 +2080,7 @@ update_stats_enqueue_fair(struct cfs_rq *cfs_rq, struct sched_entity *se, int fl
 	 * Are we enqueueing a waiting task? (for current tasks
 	 * a dequeue/enqueue event is a NOP)
 	 */
-	if (se != cfs_rq->curr)
+	if (se != cfs_rq->h_curr)
 		update_stats_wait_start_fair(cfs_rq, se);
 
 	if (flags & ENQUEUE_WAKEUP)
@@ -2127,7 +2098,7 @@ update_stats_dequeue_fair(struct cfs_rq *cfs_rq, struct sched_entity *se, int fl
 	 * Mark the end of the wait period if dequeueing a
 	 * waiting task:
 	 */
-	if (se != cfs_rq->curr)
+	if (se != cfs_rq->h_curr)
 		update_stats_wait_end_fair(cfs_rq, se);
 
 	if ((flags & DEQUEUE_SLEEP) && entity_is_task(se)) {
@@ -4468,6 +4439,7 @@ static inline void update_scan_period(struct task_struct *p, int new_cpu)
 static void
 account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+	WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
 	update_load_add(&cfs_rq->load, se->load.weight);
 	if (entity_is_task(se)) {
 		struct task_struct *p = task_of(se);
@@ -4483,6 +4455,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
 static void
 account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+	WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
 	update_load_sub(&cfs_rq->load, se->load.weight);
 	if (entity_is_task(se)) {
 		struct task_struct *p = task_of(se);
@@ -4564,7 +4537,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se)
 static void
 rescale_entity(struct sched_entity *se, unsigned long weight, bool rel_vprot)
 {
-	unsigned long old_weight = se->load.weight;
+	long old_weight = se->h_load.weight;
 
 	/*
 	 * VRUNTIME
@@ -4664,16 +4637,17 @@ rescale_entity(struct sched_entity *se, unsigned long weight, bool rel_vprot)
 		se->vprot = div64_long(se->vprot * old_weight, weight);
 }
 
-static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
-			    unsigned long weight)
+static void reweight_eevdf(struct cfs_rq *cfs_rq, struct sched_entity *se,
+			   unsigned long weight, bool on_rq)
 {
 	bool curr = cfs_rq->curr == se;
 	bool rel_vprot = false;
 	u64 avruntime = 0;
 
-	if (se->on_rq) {
-		/* commit outstanding execution time */
-		update_curr(cfs_rq);
+	if (se->h_load.weight == weight)
+		return;
+
+	if (on_rq) {
 		avruntime = avg_vruntime(cfs_rq);
 		se->vlag = entity_lag(cfs_rq, se, avruntime);
 		se->deadline -= avruntime;
@@ -4683,46 +4657,90 @@ static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
 			rel_vprot = true;
 		}
 
-		cfs_rq->nr_queued--;
+		cfs_rq->h_nr_queued--;
 		if (!curr)
 			__dequeue_entity(cfs_rq, se);
-		update_load_sub(&cfs_rq->load, se->load.weight);
 	}
-	dequeue_load_avg(cfs_rq, se);
 
 	rescale_entity(se, weight, rel_vprot);
 
-	update_load_set(&se->load, weight);
-
-	do {
-		u32 divider = get_pelt_divider(&se->avg);
-		se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
-	} while (0);
+	update_load_set(&se->h_load, weight);
 
-	enqueue_load_avg(cfs_rq, se);
-	if (se->on_rq) {
+	if (on_rq) {
 		if (rel_vprot)
 			se->vprot += avruntime;
 		se->deadline += avruntime;
 		se->rel_deadline = 0;
 		se->vruntime = avruntime - se->vlag;
 
-		update_load_add(&cfs_rq->load, se->load.weight);
 		if (!curr)
 			__enqueue_entity(cfs_rq, se);
-		cfs_rq->nr_queued++;
+		cfs_rq->h_nr_queued++;
 	}
 }
 
+static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
+			    unsigned long weight)
+{
+	if (se->load.weight == weight)
+		return;
+
+	if (se->on_rq) {
+		WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
+		update_load_sub(&cfs_rq->load, se->load.weight);
+	}
+	dequeue_load_avg(cfs_rq, se);
+
+	update_load_set(&se->load, weight);
+
+	do {
+		u32 divider = get_pelt_divider(&se->avg);
+		se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
+	} while (0);
+
+	enqueue_load_avg(cfs_rq, se);
+
+	if (se->on_rq)
+		update_load_add(&cfs_rq->load, se->load.weight);
+}
+
+/*
+ * weight = NICE_0_LOAD;
+ * for_each_entity_se(se)
+ *   weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
+ */
+static __always_inline
+unsigned long __calc_prop_weight(struct cfs_rq *cfs_rq, struct sched_entity *se,
+				 unsigned long weight)
+{
+	weight *= se->load.weight;
+	if (parent_entity(se))
+		weight /= cfs_rq->load.weight;
+	else
+		weight /= NICE_0_LOAD;
+
+	return max(weight, MIN_SHARES);
+}
+
 static void reweight_task_fair(struct rq *rq, struct task_struct *p,
 			       const struct load_weight *lw)
 {
 	struct sched_entity *se = &p->se;
-	struct cfs_rq *cfs_rq = cfs_rq_of(se);
-	struct load_weight *load = &se->load;
+	unsigned long weight = NICE_0_LOAD;
+
+	if (se->on_rq)
+		update_curr_fair(rq);
 
-	reweight_entity(cfs_rq, se, lw->weight);
-	load->inv_weight = lw->inv_weight;
+	reweight_entity(cfs_rq_of(se), se, lw->weight);
+	se->load.inv_weight = lw->inv_weight;
+
+	if (!se->on_rq)
+		return;
+
+	for_each_sched_entity(se)
+		weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
+
+	reweight_eevdf(&rq->cfs, &p->se, weight, p->se.on_rq);
 }
 
 static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
@@ -4958,8 +4976,7 @@ static void update_cfs_group(struct sched_entity *se)
 		return;
 
 	shares = static_call(calc_group_shares)(gcfs_rq);
-	if (unlikely(se->load.weight != shares))
-		reweight_entity(cfs_rq_of(se), se, shares);
+	reweight_entity(cfs_rq_of(se), se, shares);
 }
 
 #else /* !CONFIG_FAIR_GROUP_SCHED: */
@@ -5077,7 +5094,7 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
  * differential update where we store the last value we propagated. This in
  * turn allows skipping updates if the differential is 'small'.
  *
- * Updating tg's load_avg is necessary before update_cfs_share().
+ * Updating tg's load_avg is necessary before update_cfs_group().
  */
 static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
 {
@@ -5545,7 +5562,7 @@ static void migrate_se_pelt_lag(struct sched_entity *se) {}
  * The cfs_rq avg is the direct sum of all its entities (blocked and runnable)
  * avg. The immediate corollary is that all (fair) tasks must be attached.
  *
- * cfs_rq->avg is used for task_h_load() and update_cfs_share() for example.
+ * cfs_rq->avg is used for task_h_load() and update_cfs_group() for example.
  *
  * Return: true if the load decayed or we removed load.
  *
@@ -6083,6 +6100,7 @@ static void
 place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
 	u64 vslice, vruntime = avg_vruntime(cfs_rq);
+	unsigned int nr_queued = cfs_rq->h_nr_queued;
 	bool update_zero = false;
 	s64 lag = 0;
 
@@ -6090,6 +6108,9 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 		se->slice = sysctl_sched_base_slice;
 	vslice = calc_delta_fair(se->slice, se);
 
+	if (flags & ENQUEUE_QUEUED)
+		nr_queued -= 1;
+
 	/*
 	 * Due to how V is constructed as the weighted average of entities,
 	 * adding tasks with positive lag, or removing tasks with negative lag
@@ -6098,7 +6119,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	 *
 	 * EEVDF: placement strategy #1 / #2
 	 */
-	if (sched_feat(PLACE_LAG) && cfs_rq->nr_queued && se->vlag) {
+	if (sched_feat(PLACE_LAG) && nr_queued && se->vlag) {
 		struct sched_entity *curr = cfs_rq->curr;
 		long load, weight;
 
@@ -6158,9 +6179,9 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 		 */
 		load = cfs_rq->sum_weight;
 		if (curr && curr->on_rq)
-			load += avg_vruntime_weight(cfs_rq, curr->load.weight);
+			load += avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
-		weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+		weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
 		lag *= load + weight;
 		if (WARN_ON_ONCE(!load))
 			load = 1;
@@ -6219,22 +6240,8 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq);
 static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
 
 static void
-requeue_delayed_entity(struct sched_entity *se);
-
-static void
 enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
-	bool curr = cfs_rq->curr == se;
-
-	/*
-	 * If we're the current task, we must renormalise before calling
-	 * update_curr().
-	 */
-	if (curr)
-		place_entity(cfs_rq, se, flags);
-
-	update_curr(cfs_rq);
-
 	/*
 	 * When enqueuing a sched_entity, we must:
 	 *   - Update loads to have both entity and cfs_rq synced with now.
@@ -6253,13 +6260,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	 */
 	update_cfs_group(se);
 
-	/*
-	 * XXX now that the entity has been re-weighted, and it's lag adjusted,
-	 * we can place the entity.
-	 */
-	if (!curr)
-		place_entity(cfs_rq, se, flags);
-
 	account_entity_enqueue(cfs_rq, se);
 
 	/* Entity has migrated, no longer consider this task hot */
@@ -6268,8 +6268,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
 	check_schedstat_required();
 	update_stats_enqueue_fair(cfs_rq, se, flags);
-	if (!curr)
-		__enqueue_entity(cfs_rq, se);
 	se->on_rq = 1;
 
 	if (cfs_rq->nr_queued == 1) {
@@ -6287,21 +6285,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	}
 }
 
-static void __clear_buddies_next(struct sched_entity *se)
+static void set_next_buddy(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	for_each_sched_entity(se) {
-		struct cfs_rq *cfs_rq = cfs_rq_of(se);
-		if (cfs_rq->next != se)
-			break;
-
-		cfs_rq->next = NULL;
-	}
+	if (WARN_ON_ONCE(!se->on_rq || se->sched_delayed))
+		return;
+	if (se_is_idle(se))
+		return;
+	cfs_rq->next = se;
 }
 
 static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if (cfs_rq->next == se)
-		__clear_buddies_next(se);
+		cfs_rq->next = NULL;
 }
 
 static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
@@ -6312,7 +6308,7 @@ static void set_delayed(struct sched_entity *se)
 
 	/*
 	 * Delayed se of cfs_rq have no tasks queued on them.
-	 * Do not adjust h_nr_runnable since dequeue_entities()
+	 * Do not adjust h_nr_runnable since __dequeue_task()
 	 * will account it for blocked tasks.
 	 */
 	if (!entity_is_task(se))
@@ -6345,45 +6341,16 @@ static void clear_delayed(struct sched_entity *se)
 	}
 }
 
-static bool
+static void
 dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
-	bool sleep = flags & DEQUEUE_SLEEP;
-	int action = 0;
+	int action = UPDATE_TG;
 
-	update_curr(cfs_rq);
-	clear_buddies(cfs_rq, se);
-
-	if (flags & DEQUEUE_DELAYED) {
-		WARN_ON_ONCE(!se->sched_delayed);
-	} else {
-		bool delay = sleep;
-		/*
-		 * DELAY_DEQUEUE relies on spurious wakeups, special task
-		 * states must not suffer spurious wakeups, excempt them.
-		 */
-		if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
-			delay = false;
-
-		WARN_ON_ONCE(delay && se->sched_delayed);
-
-		if (sched_feat(DELAY_DEQUEUE) && delay &&
-		    !entity_eligible(cfs_rq, se)) {
-			if (entity_is_task(se))
-				action |= UPDATE_UTIL_EST;
-			update_load_avg(cfs_rq, se, action);
-			update_entity_lag(cfs_rq, se);
-			set_delayed(se);
-			return false;
-		}
-	}
-
-	action = UPDATE_TG;
 	if (entity_is_task(se)) {
 		if (task_on_rq_migrating(task_of(se)))
 			action |= DO_DETACH;
 
-		if (sleep && !(flags & DEQUEUE_DELAYED))
+		if ((flags & DEQUEUE_SLEEP) && !(flags & DEQUEUE_DELAYED))
 			action |= UPDATE_UTIL_EST;
 	}
 
@@ -6401,14 +6368,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
 	update_stats_dequeue_fair(cfs_rq, se, flags);
 
-	update_entity_lag(cfs_rq, se);
-	if (sched_feat(PLACE_REL_DEADLINE) && !sleep) {
-		se->deadline -= se->vruntime;
-		se->rel_deadline = 1;
-	}
-
-	if (se != cfs_rq->curr)
-		__dequeue_entity(cfs_rq, se);
 	se->on_rq = 0;
 	account_entity_dequeue(cfs_rq, se);
 
@@ -6417,9 +6376,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
 	update_cfs_group(se);
 
-	if (flags & DEQUEUE_DELAYED)
-		clear_delayed(se);
-
 	if (cfs_rq->nr_queued == 0) {
 		update_idle_cfs_rq_clock_pelt(cfs_rq);
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -6432,15 +6388,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 		}
 #endif
 	}
-
-	return true;
 }
 
 static void
-set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first)
+set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	clear_buddies(cfs_rq, se);
-
 	/* 'current' is not kept within the tree. */
 	if (se->on_rq) {
 		/*
@@ -6449,16 +6401,12 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first)
 		 * runqueue.
 		 */
 		update_stats_wait_end_fair(cfs_rq, se);
-		__dequeue_entity(cfs_rq, se);
 		update_load_avg(cfs_rq, se, UPDATE_TG);
-
-		if (first)
-			set_protect_slice(cfs_rq, se);
 	}
 
 	update_stats_curr_start(cfs_rq, se);
-	WARN_ON_ONCE(cfs_rq->curr);
-	cfs_rq->curr = se;
+	WARN_ON_ONCE(cfs_rq->h_curr);
+	cfs_rq->h_curr = se;
 
 	/*
 	 * Track our maximum slice length, if the CPU's load is at
@@ -6478,23 +6426,17 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first)
 	se->prev_sum_exec_runtime = se->sum_exec_runtime;
 }
 
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags);
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags);
 
-/*
- * Pick the next process, keeping these things in mind, in this order:
- * 1) keep things fair between processes/task groups
- * 2) pick the "next" process, since someone really wants that to run
- * 3) pick the "last" process, for cache locality
- * 4) do not run the "skip" process, if something else is available
- */
 static struct sched_entity *
-pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
+pick_next_entity(struct rq *rq, bool protect)
 {
+	struct cfs_rq *cfs_rq = &rq->cfs;
 	struct sched_entity *se;
 
 	se = pick_eevdf(cfs_rq, protect);
 	if (se->sched_delayed) {
-		dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+		__dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 		/*
 		 * Must not reference @se again, see __block_task().
 		 */
@@ -6514,13 +6456,11 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
 
 	if (prev->on_rq) {
 		update_stats_wait_start_fair(cfs_rq, prev);
-		/* Put 'current' back into the tree. */
-		__enqueue_entity(cfs_rq, prev);
 		/* in !on_rq case, update occurred at dequeue */
 		update_load_avg(cfs_rq, prev, 0);
 	}
-	WARN_ON_ONCE(cfs_rq->curr != prev);
-	cfs_rq->curr = NULL;
+	WARN_ON_ONCE(cfs_rq->h_curr != prev);
+	cfs_rq->h_curr = NULL;
 }
 
 static void
@@ -7075,7 +7015,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
 	assert_list_leaf_cfs_rq(rq);
 
 	/* Determine whether we need to wake up potentially idle CPU: */
-	if (rq->curr == rq->idle && rq->cfs.nr_queued)
+	if (rq->curr == rq->idle && rq->cfs.h_nr_queued)
 		resched_curr(rq);
 }
 
@@ -7410,7 +7350,7 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq)
 		return;
 
 	/* an active group must be handled by the update_curr() path */
-	if (!cfs_rq->runtime_enabled || cfs_rq->curr)
+	if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)
 		return;
 
 	/* ensure the group is not already throttled */
@@ -7782,7 +7722,7 @@ static void hrtick_start_fair(struct rq *rq, struct task_struct *p)
 			resched_curr(rq);
 		return;
 	}
-	delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+	delta = (se->h_load.weight * vdelta) / NICE_0_LOAD;
 
 	/*
 	 * Correct for instantaneous load of other classes.
@@ -7882,10 +7822,8 @@ static int choose_idle_cpu(int cpu, struct task_struct *p)
 }
 
 static void
-requeue_delayed_entity(struct sched_entity *se)
+requeue_delayed_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	struct cfs_rq *cfs_rq = cfs_rq_of(se);
-
 	/*
 	 * se->sched_delayed should imply: se->on_rq == 1.
 	 * Because a delayed entity is one that is still on
@@ -7895,19 +7833,58 @@ requeue_delayed_entity(struct sched_entity *se)
 	WARN_ON_ONCE(!se->on_rq);
 
 	if (update_entity_lag(cfs_rq, se)) {
-		cfs_rq->nr_queued--;
+		cfs_rq->h_nr_queued--;
 		if (se != cfs_rq->curr)
 			__dequeue_entity(cfs_rq, se);
 		place_entity(cfs_rq, se, 0);
 		if (se != cfs_rq->curr)
 			__enqueue_entity(cfs_rq, se);
-		cfs_rq->nr_queued++;
+		cfs_rq->h_nr_queued++;
 	}
 
 	update_load_avg(cfs_rq, se, 0);
 	clear_delayed(se);
 }
 
+static unsigned long enqueue_hierarchy(struct task_struct *p, int flags)
+{
+	unsigned long weight = NICE_0_LOAD;
+	int task_new = !(flags & ENQUEUE_WAKEUP);
+	struct sched_entity *se = &p->se;
+	int h_nr_idle = task_has_idle_policy(p);
+	int h_nr_runnable = 1;
+
+	if (task_new && se->sched_delayed)
+		h_nr_runnable = 0;
+
+	for_each_sched_entity(se) {
+		struct cfs_rq *cfs_rq = cfs_rq_of(se);
+
+		update_curr(cfs_rq);
+
+		if (!se->on_rq) {
+			enqueue_entity(cfs_rq, se, flags);
+		} else {
+			update_load_avg(cfs_rq, se, UPDATE_TG);
+			se_update_runnable(se);
+			update_cfs_group(se);
+		}
+
+		cfs_rq->h_nr_runnable += h_nr_runnable;
+		cfs_rq->h_nr_queued++;
+		cfs_rq->h_nr_idle += h_nr_idle;
+
+		if (cfs_rq_is_idle(cfs_rq))
+			h_nr_idle = 1;
+
+		weight = __calc_prop_weight(cfs_rq, se, weight);
+
+		flags = ENQUEUE_WAKEUP;
+	}
+
+	return weight;
+}
+
 /*
  * The enqueue_task method is called before nr_running is
  * increased. Here we update the fair scheduling stats and
@@ -7916,13 +7893,12 @@ requeue_delayed_entity(struct sched_entity *se)
 static void
 enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 {
-	struct cfs_rq *cfs_rq;
-	struct sched_entity *se = &p->se;
-	int h_nr_idle = task_has_idle_policy(p);
-	int h_nr_runnable = 1;
-	int task_new = !(flags & ENQUEUE_WAKEUP);
 	int rq_h_nr_queued = rq->cfs.h_nr_queued;
-	u64 slice = 0;
+	int task_new = !(flags & ENQUEUE_WAKEUP);
+	struct sched_entity *se = &p->se;
+	struct cfs_rq *cfs_rq = &rq->cfs;
+	unsigned long weight;
+	bool curr;
 
 	if (task_is_throttled(p) && enqueue_throttled_task(p))
 		return;
@@ -7934,10 +7910,10 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 	 * estimated utilization, before we update schedutil.
 	 */
 	if (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))
-		util_est_enqueue(&rq->cfs, p);
+		util_est_enqueue(cfs_rq, p);
 
 	if (flags & ENQUEUE_DELAYED) {
-		requeue_delayed_entity(se);
+		requeue_delayed_entity(cfs_rq, se);
 		return;
 	}
 
@@ -7949,57 +7925,22 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 	if (p->in_iowait)
 		cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT);
 
-	if (task_new && se->sched_delayed)
-		h_nr_runnable = 0;
-
-	for_each_sched_entity(se) {
-		if (se->on_rq) {
-			if (se->sched_delayed)
-				requeue_delayed_entity(se);
-			break;
-		}
-		cfs_rq = cfs_rq_of(se);
-
-		/*
-		 * Basically set the slice of group entries to the min_slice of
-		 * their respective cfs_rq. This ensures the group can service
-		 * its entities in the desired time-frame.
-		 */
-		if (slice) {
-			se->slice = slice;
-			se->custom_slice = 1;
-		}
-		enqueue_entity(cfs_rq, se, flags);
-		slice = cfs_rq_min_slice(cfs_rq);
-
-		cfs_rq->h_nr_runnable += h_nr_runnable;
-		cfs_rq->h_nr_queued++;
-		cfs_rq->h_nr_idle += h_nr_idle;
-
-		if (cfs_rq_is_idle(cfs_rq))
-			h_nr_idle = 1;
-
-		flags = ENQUEUE_WAKEUP;
-	}
-
-	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
+	/*
+	 * XXX comment on the curr thing
+	 */
+	curr = (cfs_rq->curr == se);
+	if (curr)
+		place_entity(cfs_rq, se, flags);
 
-		update_load_avg(cfs_rq, se, UPDATE_TG);
-		se_update_runnable(se);
-		update_cfs_group(se);
+	if (se->on_rq && se->sched_delayed)
+		requeue_delayed_entity(cfs_rq, se);
 
-		se->slice = slice;
-		if (se != cfs_rq->curr)
-			min_vruntime_cb_propagate(&se->run_node, NULL);
-		slice = cfs_rq_min_slice(cfs_rq);
+	weight = enqueue_hierarchy(p, flags);
 
-		cfs_rq->h_nr_runnable += h_nr_runnable;
-		cfs_rq->h_nr_queued++;
-		cfs_rq->h_nr_idle += h_nr_idle;
-
-		if (cfs_rq_is_idle(cfs_rq))
-			h_nr_idle = 1;
+	if (!curr) {
+		reweight_eevdf(cfs_rq, se, weight, false);
+		place_entity(cfs_rq, se, flags | ENQUEUE_QUEUED);
+		__enqueue_entity(cfs_rq, se);
 	}
 
 	if (!rq_h_nr_queued && rq->cfs.h_nr_queued)
@@ -8030,105 +7971,109 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 	hrtick_update(rq);
 }
 
-/*
- * Basically dequeue_task_fair(), except it can deal with dequeue_entity()
- * failing half-way through and resume the dequeue later.
- *
- * Returns:
- * -1 - dequeue delayed
- *  0 - dequeue throttled
- *  1 - dequeue complete
- */
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
+static void dequeue_hierarchy(struct task_struct *p, int flags)
 {
-	bool was_sched_idle = sched_idle_rq(rq);
+	struct sched_entity *se = &p->se;
 	bool task_sleep = flags & DEQUEUE_SLEEP;
 	bool task_delayed = flags & DEQUEUE_DELAYED;
 	bool task_throttled = flags & DEQUEUE_THROTTLE;
-	struct task_struct *p = NULL;
-	int h_nr_idle = 0;
-	int h_nr_queued = 0;
 	int h_nr_runnable = 0;
-	struct cfs_rq *cfs_rq;
-	u64 slice = 0;
+	int h_nr_idle = task_has_idle_policy(p);
+	bool dequeue = true;
 
-	if (entity_is_task(se)) {
-		p = task_of(se);
-		h_nr_queued = 1;
-		h_nr_idle = task_has_idle_policy(p);
-		if (task_sleep || task_delayed || !se->sched_delayed)
-			h_nr_runnable = 1;
-	}
+	if (task_sleep || task_delayed || !se->sched_delayed)
+		h_nr_runnable = 1;
 
 	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
+		struct cfs_rq *cfs_rq = cfs_rq_of(se);
 
-		if (!dequeue_entity(cfs_rq, se, flags)) {
-			if (p && &p->se == se)
-				return -1;
+		update_curr(cfs_rq);
 
-			slice = cfs_rq_min_slice(cfs_rq);
-			break;
+		if (dequeue) {
+			dequeue_entity(cfs_rq, se, flags);
+			/* Don't dequeue parent if it has other entities besides us */
+			if (cfs_rq->load.weight)
+				dequeue = false;
+		} else {
+			update_load_avg(cfs_rq, se, UPDATE_TG);
+			se_update_runnable(se);
+			update_cfs_group(se);
 		}
 
 		cfs_rq->h_nr_runnable -= h_nr_runnable;
-		cfs_rq->h_nr_queued -= h_nr_queued;
+		cfs_rq->h_nr_queued--;
 		cfs_rq->h_nr_idle -= h_nr_idle;
 
 		if (cfs_rq_is_idle(cfs_rq))
-			h_nr_idle = h_nr_queued;
+			h_nr_idle = 1;
 
 		if (throttled_hierarchy(cfs_rq) && task_throttled)
 			record_throttle_clock(cfs_rq);
 
-		/* Don't dequeue parent if it has other entities besides us */
-		if (cfs_rq->load.weight) {
-			slice = cfs_rq_min_slice(cfs_rq);
-
-			/* Avoid re-evaluating load for this entity: */
-			se = parent_entity(se);
-			/*
-			 * Bias pick_next to pick a task from this cfs_rq, as
-			 * p is sleeping when it is within its sched_slice.
-			 */
-			if (task_sleep && se)
-				set_next_buddy(se);
-			break;
-		}
 		flags |= DEQUEUE_SLEEP;
 		flags &= ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL);
 	}
+}
 
-	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
+/*
+ * The part of dequeue_task_fair() that is needed to dequeue delayed tasks.
+ *
+ * Returns:
+ *   true  - dequeued
+ *   false - delayed
+ */
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
+{
+	struct sched_entity *se = &p->se;
+	struct cfs_rq *cfs_rq = &rq->cfs;
+	bool was_sched_idle = sched_idle_rq(rq);
+	bool task_sleep = flags & DEQUEUE_SLEEP;
+	bool task_delayed = flags & DEQUEUE_DELAYED;
 
-		update_load_avg(cfs_rq, se, UPDATE_TG);
-		se_update_runnable(se);
-		update_cfs_group(se);
+	clear_buddies(cfs_rq, se);
 
-		se->slice = slice;
-		if (se != cfs_rq->curr)
-			min_vruntime_cb_propagate(&se->run_node, NULL);
-		slice = cfs_rq_min_slice(cfs_rq);
+	update_curr(cfs_rq_of(se));
+	update_entity_lag(cfs_rq, se);
 
-		cfs_rq->h_nr_runnable -= h_nr_runnable;
-		cfs_rq->h_nr_queued -= h_nr_queued;
-		cfs_rq->h_nr_idle -= h_nr_idle;
+	if (flags & DEQUEUE_DELAYED) {
+		WARN_ON_ONCE(!se->sched_delayed);
+	} else {
+		bool delay = task_sleep;
+		/*
+		 * DELAY_DEQUEUE relies on spurious wakeups, special task
+		 * states must not suffer spurious wakeups, excempt them.
+		 */
+		if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
+			delay = false;
 
-		if (cfs_rq_is_idle(cfs_rq))
-			h_nr_idle = h_nr_queued;
+		WARN_ON_ONCE(delay && se->sched_delayed);
 
-		if (throttled_hierarchy(cfs_rq) && task_throttled)
-			record_throttle_clock(cfs_rq);
+		if (sched_feat(DELAY_DEQUEUE) && delay &&
+		    !entity_eligible(cfs_rq, se)) {
+			update_load_avg(cfs_rq_of(se), se, UPDATE_UTIL_EST);
+			set_delayed(se);
+			return false;
+		}
 	}
 
-	sub_nr_running(rq, h_nr_queued);
+	dequeue_hierarchy(p, flags);
+
+	if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
+		se->deadline -= se->vruntime;
+		se->rel_deadline = 1;
+	}
+	if (se != cfs_rq->curr)
+		__dequeue_entity(cfs_rq, se);
+
+	sub_nr_running(rq, 1);
 
 	/* balance early to pull high priority tasks */
 	if (unlikely(!was_sched_idle && sched_idle_rq(rq)))
 		rq->next_balance = jiffies;
 
-	if (p && task_delayed) {
+	if (task_delayed) {
+		clear_delayed(se);
+
 		WARN_ON_ONCE(!task_sleep);
 		WARN_ON_ONCE(p->on_rq != 1);
 
@@ -8140,7 +8085,7 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
 		__block_task(rq, p);
 	}
 
-	return 1;
+	return true;
 }
 
 /*
@@ -8158,11 +8103,11 @@ static bool dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 	if (!p->se.sched_delayed)
 		util_est_dequeue(&rq->cfs, p);
 
-	if (dequeue_entities(rq, &p->se, flags) < 0)
+	if (!__dequeue_task(rq, p, flags))
 		return false;
 
 	/*
-	 * Must not reference @p after dequeue_entities(DEQUEUE_DELAYED).
+	 * Must not reference @p after __dequeue_task(DEQUEUE_DELAYED).
 	 */
 	return true;
 }
@@ -9757,19 +9702,6 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)
 static void task_dead_fair(struct task_struct *p)
 {
 	struct sched_entity *se = &p->se;
-
-	if (se->sched_delayed) {
-		struct rq_flags rf;
-		struct rq *rq;
-
-		rq = task_rq_lock(p, &rf);
-		if (se->sched_delayed) {
-			update_rq_clock(rq);
-			dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-		}
-		task_rq_unlock(rq, p, &rf);
-	}
-
 	remove_entity_load_avg(se);
 }
 
@@ -9803,21 +9735,10 @@ static void set_cpus_allowed_fair(struct task_struct *p, struct affinity_context
 	set_task_max_allowed_capacity(p);
 }
 
-static void set_next_buddy(struct sched_entity *se)
-{
-	for_each_sched_entity(se) {
-		if (WARN_ON_ONCE(!se->on_rq))
-			return;
-		if (se_is_idle(se))
-			return;
-		cfs_rq_of(se)->next = se;
-	}
-}
-
 enum preempt_wakeup_action {
 	PREEMPT_WAKEUP_NONE,	/* No preemption. */
 	PREEMPT_WAKEUP_SHORT,	/* Ignore slice protection. */
-	PREEMPT_WAKEUP_PICK,	/* Let __pick_eevdf() decide. */
+	PREEMPT_WAKEUP_PICK,	/* Let pick_eevdf() decide. */
 	PREEMPT_WAKEUP_RESCHED,	/* Force reschedule. */
 };
 
@@ -9834,7 +9755,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq, int wake_flags,
 	if (cfs_rq->next && entity_before(cfs_rq->next, pse))
 		return false;
 
-	set_next_buddy(pse);
+	set_next_buddy(cfs_rq, pse);
 	return true;
 }
 
@@ -9887,7 +9808,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
 	struct task_struct *donor = rq->donor;
 	struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
-	struct cfs_rq *cfs_rq = task_cfs_rq(donor);
+	struct cfs_rq *cfs_rq = &rq->cfs;
 	int cse_is_idle, pse_is_idle;
 
 	/*
@@ -9925,7 +9846,6 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	if (!sched_feat(WAKEUP_PREEMPTION))
 		return;
 
-	find_matching_se(&se, &pse);
 	WARN_ON_ONCE(!pse);
 
 	cse_is_idle = se_is_idle(se);
@@ -9953,8 +9873,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	if (unlikely(!normal_policy(p->policy)))
 		return;
 
-	cfs_rq = cfs_rq_of(se);
-	update_curr(cfs_rq);
+	update_curr_fair(rq);
 	/*
 	 * If @p has a shorter slice than current and @p is eligible, override
 	 * current's slice protection in order to allow preemption.
@@ -9998,18 +9917,15 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	}
 
 pick:
-	nse = pick_next_entity(rq, cfs_rq, preempt_action != PREEMPT_WAKEUP_SHORT);
-	/* If @p has become the most eligible task, force preemption */
-	if (nse == pse)
-		goto preempt;
+	if (cfs_rq->h_nr_queued) {
+		nse = pick_next_entity(rq, preempt_action != PREEMPT_WAKEUP_SHORT);
+		if (unlikely(!nse))
+			goto pick;
 
-	/*
-	 * Because p is enqueued, nse being null can only mean that we
-	 * dequeued a delayed task. If there are still entities queued in
-	 * cfs, check if the next one will be p.
-	 */
-	if (!nse && cfs_rq->nr_queued)
-		goto pick;
+		/* If @p has become the most eligible task, force preemption */
+		if (nse == pse)
+			goto preempt;
+	}
 
 	if (sched_feat(RUN_TO_PARITY))
 		update_protect_slice(cfs_rq, se);
@@ -10028,33 +9944,24 @@ preempt:
 struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
 	__must_hold(__rq_lockp(rq))
 {
+	struct cfs_rq *cfs_rq = &rq->cfs;
 	struct sched_entity *se;
-	struct cfs_rq *cfs_rq;
 	struct task_struct *p;
-	bool throttled;
 	int new_tasks;
 
 again:
-	cfs_rq = &rq->cfs;
-	if (!cfs_rq->nr_queued)
+	if (!cfs_rq->h_nr_queued)
 		goto idle;
 
-	throttled = false;
-
-	do {
-		/* Might not have done put_prev_entity() */
-		if (cfs_rq->curr && cfs_rq->curr->on_rq)
-			update_curr(cfs_rq);
+	/* Might not have done put_prev_entity() */
+	if (cfs_rq->curr && cfs_rq->curr->on_rq)
+		update_curr(cfs_rq);
 
-		se = pick_next_entity(rq, cfs_rq, true);
-		if (!se)
-			goto again;
-		cfs_rq = group_cfs_rq(se);
-	} while (cfs_rq);
+	se = pick_next_entity(rq, true);
+	if (!se)
+		goto again;
 
 	p = task_of(se);
-	if (unlikely(throttled))
-		task_throttle_setup_work(p);
 	return p;
 
 idle:
@@ -10091,7 +9998,7 @@ void fair_server_init(struct rq *rq)
 static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct task_struct *next)
 {
 	struct sched_entity *se = &prev->se;
-	struct cfs_rq *cfs_rq;
+	struct cfs_rq *cfs_rq = &rq->cfs;
 	struct sched_entity *nse = NULL;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
@@ -10101,7 +10008,7 @@ static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t
 
 	while (se) {
 		cfs_rq = cfs_rq_of(se);
-		if (!nse || cfs_rq->curr)
+		if (!nse || cfs_rq->h_curr)
 			put_prev_entity(cfs_rq, se);
 #ifdef CONFIG_FAIR_GROUP_SCHED
 		if (nse) {
@@ -10120,6 +10027,14 @@ static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t
 #endif
 		se = parent_entity(se);
 	}
+
+	/* Put 'current' back into the tree. */
+	cfs_rq = &rq->cfs;
+	se = &prev->se;
+	WARN_ON_ONCE(cfs_rq->curr != se);
+	cfs_rq->curr = NULL;
+	if (se->on_rq)
+		__enqueue_entity(cfs_rq, se);
 }
 
 /*
@@ -10128,8 +10043,8 @@ static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t
 static void yield_task_fair(struct rq *rq)
 {
 	struct task_struct *curr = rq->donor;
-	struct cfs_rq *cfs_rq = task_cfs_rq(curr);
 	struct sched_entity *se = &curr->se;
+	struct cfs_rq *cfs_rq = &rq->cfs;
 
 	/*
 	 * Are we the only task in the tree?
@@ -10170,11 +10085,11 @@ static bool yield_to_task_fair(struct rq *rq, struct task_struct *p)
 	struct sched_entity *se = &p->se;
 
 	/* !se->on_rq also covers throttled task */
-	if (!se->on_rq)
+	if (!se->on_rq || se->sched_delayed)
 		return false;
 
 	/* Tell the scheduler that we'd really like se to run next. */
-	set_next_buddy(se);
+	set_next_buddy(&task_rq(p)->cfs, se);
 
 	yield_task_fair(rq);
 
@@ -10513,15 +10428,10 @@ static inline long migrate_degrades_locality(struct task_struct *p,
  */
 static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_cpu)
 {
-	struct cfs_rq *dst_cfs_rq;
+	struct cfs_rq *dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
 
-#ifdef CONFIG_FAIR_GROUP_SCHED
-	dst_cfs_rq = tg_cfs_rq(task_group(p), dest_cpu);
-#else
-	dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
-#endif
-	if (sched_feat(PLACE_LAG) && dst_cfs_rq->nr_queued &&
-	    !entity_eligible(task_cfs_rq(p), &p->se))
+	if (sched_feat(PLACE_LAG) && dst_cfs_rq->h_nr_queued &&
+	    !entity_eligible(&task_rq(p)->cfs, &p->se))
 		return 1;
 
 	return 0;
@@ -11304,7 +11214,7 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
 	while ((se = READ_ONCE(cfs_rq->h_load_next)) != NULL) {
 		load = cfs_rq->h_load;
 		load = div64_ul(load * se->avg.load_avg,
-			cfs_rq_load_avg(cfs_rq) + 1);
+				cfs_rq_load_avg(cfs_rq) + 1);
 		cfs_rq = group_cfs_rq(se);
 		cfs_rq->h_load = load;
 		cfs_rq->last_h_load_update = now;
@@ -14696,7 +14606,7 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr)
 	 * MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check
 	 * if we need to give up the CPU.
 	 */
-	if (rq->core->core_forceidle_count && rq->cfs.nr_queued == 1 &&
+	if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued == 1 &&
 	    __entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))
 		resched_curr(rq);
 }
@@ -14905,30 +14815,8 @@ bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b,
 
 	WARN_ON_ONCE(task_rq(b)->core != rq->core);
 
-#ifdef CONFIG_FAIR_GROUP_SCHED
-	/*
-	 * Find an se in the hierarchy for tasks a and b, such that the se's
-	 * are immediate siblings.
-	 */
-	while (sea->cfs_rq->tg != seb->cfs_rq->tg) {
-		int sea_depth = sea->depth;
-		int seb_depth = seb->depth;
-
-		if (sea_depth >= seb_depth)
-			sea = parent_entity(sea);
-		if (sea_depth <= seb_depth)
-			seb = parent_entity(seb);
-	}
-
-	se_fi_update(sea, rq->core->core_forceidle_seq, in_fi);
-	se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);
-
-	cfs_rqa = sea->cfs_rq;
-	cfs_rqb = seb->cfs_rq;
-#else /* !CONFIG_FAIR_GROUP_SCHED: */
 	cfs_rqa = &task_rq(a)->cfs;
 	cfs_rqb = &task_rq(b)->cfs;
-#endif /* !CONFIG_FAIR_GROUP_SCHED */
 
 	/*
 	 * Find delta after normalizing se's vruntime with its cfs_rq's
@@ -14967,11 +14855,20 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {}
 static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 {
 	struct sched_entity *se = &curr->se;
-	struct cfs_rq *cfs_rq;
 
-	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
-		entity_tick(cfs_rq, se, queued);
+	if (se->on_rq) {
+		unsigned long weight = NICE_0_LOAD;
+		struct cfs_rq *cfs_rq;
+
+		for_each_sched_entity(se) {
+			cfs_rq = cfs_rq_of(se);
+			entity_tick(cfs_rq, se, queued);
+
+			weight = __calc_prop_weight(cfs_rq, se, weight);
+		}
+
+		se = &curr->se;
+		reweight_eevdf(cfs_rq, se, weight, se->on_rq);
 	}
 
 	if (queued)
@@ -15011,7 +14908,7 @@ prio_changed_fair(struct rq *rq, struct task_struct *p, u64 oldprio)
 	if (p->prio == oldprio)
 		return;
 
-	if (rq->cfs.nr_queued == 1)
+	if (rq->cfs.h_nr_queued == 1)
 		return;
 
 	/*
@@ -15140,33 +15037,44 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p)
 	}
 }
 
-/*
- * Account for a task changing its policy or group.
- *
- * This routine is mostly called to set cfs_rq->curr field when a task
- * migrates between groups/classes.
- */
 static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 {
 	struct sched_entity *se = &p->se;
 	bool throttled = false;
+	struct cfs_rq *cfs_rq = &rq->cfs;
+	unsigned long weight = NICE_0_LOAD;
+	bool on_rq = se->on_rq;
+
+	clear_buddies(cfs_rq, se);
+
+	if (on_rq)
+		__dequeue_entity(cfs_rq, se);
 
 	for_each_sched_entity(se) {
-		struct cfs_rq *cfs_rq = cfs_rq_of(se);
+		cfs_rq = cfs_rq_of(se);
 
-		if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
-		    first && cfs_rq->curr)
-			break;
+		if (!IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) ||
+		    !first || !cfs_rq->h_curr)
+			set_next_entity(cfs_rq, se);
 
-		set_next_entity(cfs_rq, se, first);
 		/* ensure bandwidth has been allocated on our new cfs_rq */
 		throttled |= account_cfs_rq_runtime(cfs_rq, 0);
+
+		if (on_rq)
+			weight = __calc_prop_weight(cfs_rq, se, weight);
 	}
 
 	if (throttled)
 		task_throttle_setup_work(p);
 
 	se = &p->se;
+	cfs_rq->curr = se;
+
+	if (on_rq) {
+		reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+		if (first)
+			set_protect_slice(cfs_rq, se);
+	}
 
 	if (task_on_rq_queued(p)) {
 		/*
@@ -15279,17 +15187,8 @@ void unregister_fair_sched_group(struct task_group *tg)
 		struct sched_entity *se = tg_se(tg, cpu);
 		struct rq *rq = cpu_rq(cpu);
 
-		if (se) {
-			if (se->sched_delayed) {
-				guard(rq_lock_irqsave)(rq);
-				if (se->sched_delayed) {
-					update_rq_clock(rq);
-					dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-				}
-				list_del_leaf_cfs_rq(cfs_rq);
-			}
+		if (se)
 			remove_entity_load_avg(se);
-		}
 
 		/*
 		 * Only empty task groups can be destroyed; so we can speculatively
diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c
index 8977908..779eb58 100644
--- a/kernel/sched/pelt.c
+++ b/kernel/sched/pelt.c
@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched_avg *sa,
 	/*
 	 * running is a subset of runnable (weight) so running can't be set if
 	 * runnable is clear. But there are some corner cases where the current
-	 * se has been already dequeued but cfs_rq->curr still points to it.
+	 * se has been already dequeued but cfs_rq->h_curr still points to it.
 	 * This means that weight will be 0 but not running for a sched_entity
 	 * but also for a cfs_rq if the latter becomes idle. As an example,
 	 * this happens during sched_balance_newidle() which calls
@@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now, struct sched_entity *se)
 int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se),
-				cfs_rq->curr == se)) {
+				cfs_rq->h_curr == se)) {
 
 		___update_load_avg(&se->avg, se_weight(se));
 		cfs_se_util_change(&se->avg);
@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq)
 	if (___update_load_sum(now, &cfs_rq->avg,
 				scale_load_down(cfs_rq->load.weight),
 				cfs_rq->h_nr_runnable,
-				cfs_rq->curr != NULL)) {
+				cfs_rq->h_curr != NULL)) {
 
 		___update_load_avg(&cfs_rq->avg, 1);
 		trace_pelt_cfs_tp(cfs_rq);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index fd267bf..e20823f 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -530,21 +530,8 @@ struct task_group {
 
 };
 
-#ifdef CONFIG_GROUP_SCHED_WEIGHT
 #define ROOT_TASK_GROUP_LOAD	NICE_0_LOAD
 
-/*
- * A weight of 0 or 1 can cause arithmetics problems.
- * A weight of a cfs_rq is the sum of weights of which entities
- * are queued on this cfs_rq, so a weight of a entity should not be
- * too large, so as the shares value of a task group.
- * (The default weight is 1024 - so there's no practical
- *  limitation from this.)
- */
-#define MIN_SHARES		(1UL <<  1)
-#define MAX_SHARES		(1UL << 18)
-#endif
-
 typedef int (*tg_visitor)(struct task_group *, void *);
 
 extern int walk_tg_tree_from(struct task_group *from,
@@ -631,6 +618,17 @@ static inline bool cfs_task_bw_constrained(struct task_struct *p) { return false
 
 #endif /* !CONFIG_CGROUP_SCHED */
 
+/*
+ * A weight of 0 or 1 can cause arithmetics problems.
+ * A weight of a cfs_rq is the sum of weights of which entities
+ * are queued on this cfs_rq, so a weight of a entity should not be
+ * too large, so as the shares value of a task group.
+ * (The default weight is 1024 - so there's no practical
+ *  limitation from this.)
+ */
+#define MIN_SHARES		(1UL <<  1)
+#define MAX_SHARES		(1UL << 18)
+
 extern void unregister_rt_sched_group(struct task_group *tg);
 extern void free_rt_sched_group(struct task_group *tg);
 extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent);
@@ -709,6 +707,7 @@ struct cfs_rq {
 	/*
 	 * CFS load tracking
 	 */
+	struct sched_entity	*h_curr;
 	struct sched_avg	avg;
 #ifndef CONFIG_64BIT
 	u64			last_update_time_copy;
@@ -2575,6 +2574,7 @@ extern const u32		sched_prio_to_wmult[40];
 #define ENQUEUE_MIGRATED	0x00040000
 #define ENQUEUE_INITIAL		0x00080000
 #define ENQUEUE_RQ_SELECTED	0x00100000
+#define ENQUEUE_QUEUED		0x00200000
 
 #define RETRY_TASK		((void *)-1UL)
 

[REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Aishwarya Rambhadran]

Hi Peter,

We have observed some schbench p99 request latency regressions while
comparing v7.2 against v7.3 cycle on an AWS Graviton3 (m7g.metal)
using Fastpath (Linux kernel performance benchmarking tool).

The schbench suite as a whole does not show a significant regression,
and the results appear to be a mix of improvements and regressions
depending on the thread configuration. However, a few configurations
show fairly large p99 request latency regressions. For example,
comparing v7.2 against v7.3-rc4:

message-thread:16, worker-thread:16
v7.2 = 662869 us
v7.3-rc4 = 969557 us
change% = -31.63%

message-thread:64, worker-thread:4
v7.2 = 730453 us
v7.3-rc4 = 942251 us
change% = -22.48%

message-thread:32, worker-thread:4
v7.2 = 74251 us
v7.3-rc4 = 56725 us
change% = +30.90%


So this does not appear to be a general latency degradation; some
configs improve while others regress.

I also ran a Fastpath automated git bisection for the m 16, t 16 p99
request latency regression between v7.2 and v7.3-rc1, which identified
this commit as the first bad commit. The original regression was ~30.9%,
& the bisection reproducibility check showed ~27.8% regression with 20
repeats across 2 boot sessions.

This commit looks relevant since it changes EEVDF runqueue organization
and is intended, in part, to address latency issues with hierarchical fair/
cgroup scheduling. I also noticed that there have been fixes on top of this
change, including 68e37487810a ("sched/fair: Fix flat hierarchy"), which
fixes vruntime handling and has a Fixes: 85570f10a4c6 tag.

Given that the overall schbench result is mostly neutral while some p99
configurations move considerably in both directions, I wanted to check
whether this could be an expected trade-off from the single-runqueue
change, or whether the p99 regressions are worth investigating further.

I can run additional tests or provide the full results/bisection data if
useful.


Thank you.
Aishwarya Rambhadran

On 05/06/26 6:10 PM, Peter Zijlstra wrote:
> Change fair/cgroup to a single runqueue.
>
> Infamously fair/cgroup isn't working for a number of people; typically
> the complaint is latencies and/or overhead. The latency issue is due
> to the intermediate entries that represent a combination of tasks and
> thereby obfuscate the runnability of tasks.
>
> The approach here is to leave the cgroup hierarchy as is; including
> the intermediate enqueue/dequeue but move the actual EEVDF runqueue
> outside. This means things like the shares_weight approximation are
> fully preserved.
>
> That is, given a hierarchy like:
>
>            R
>            |
>            se--G1
>                / \
>          G2--se   se--G3
>         / \           |
>    T1--se se--T2      se--T3
>
> This is fully maintained for load tracking, however the EEVDF parts of
> cfs_rq/se go unused for the intermediates and are instead connected
> like:
>
>       _R_
>      / | \
>     T1 T2 T3
>
> Since the effective weight of the entities is determined by the
> hierarchy, this gets recomputed on enqueue,set_next_task and tick.
>
> Notably, the effective weight (se->h_load) is computed from the
> hierarchical fraction: se->load / cfs_rq->load.
>
> Since EEVDF is now exclusively operating on rq->cfs, it needs to
> consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly,
> only tasks can get delayed, simplifying some of the cgroup cleanup.
>
> One place where additional information was required was
> set_next_task() / put_prev_task(), where we need to track 'current'
> both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense
> (cfs_rq->curr).
>
> As a result of only having a single level to pick from, much of the
> complications in pick_next_task() and preemption go away.
>
> Since many of the hierarchical operations are still there, this won't
> immediately fix the performance issues, but hopefully it will fix some
> of the latency issues.
>
> TODO: split struct cfs_rq / struct sched_entity
> TODO: try and get rid of h_curr
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
>   include/linux/sched.h |    1
>   kernel/sched/core.c   |    5
>   kernel/sched/debug.c  |    9
>   kernel/sched/fair.c   |  803 +++++++++++++++++++++-----------------------------
>   kernel/sched/pelt.c   |    6
>   kernel/sched/sched.h  |   26 -
>   6 files changed, 375 insertions(+), 475 deletions(-)
>
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -574,6 +574,7 @@ struct sched_statistics {
>   struct sched_entity {
>   	/* For load-balancing: */
>   	struct load_weight		load;
> +	struct load_weight		h_load;
>   	struct rb_node			run_node;
>   	u64				deadline;
>   	u64				min_vruntime;
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -5652,11 +5652,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);
>    */
>   static inline void prefetch_curr_exec_start(struct task_struct *p)
>   {
> -#ifdef CONFIG_FAIR_GROUP_SCHED
> -	struct sched_entity *curr = p->se.cfs_rq->curr;
> -#else
>   	struct sched_entity *curr = task_rq(p)->cfs.curr;
> -#endif
> +
>   	prefetch(curr);
>   	prefetch(&curr->exec_start);
>   }
> --- a/kernel/sched/debug.c
> +++ b/kernel/sched/debug.c
> @@ -975,10 +975,11 @@ print_task(struct seq_file *m, struct rq
>   	else
>   		SEQ_printf(m, " %c", task_state_to_char(p));
>   
> -	SEQ_printf(m, " %15s %5d %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
> +	SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
>   		p->comm, task_pid_nr(p),
> +		p->se.h_load.weight,
>   		SPLIT_NS(p->se.vruntime),
> -		entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
> +		entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N',
>   		SPLIT_NS(p->se.deadline),
>   		p->se.custom_slice ? 'S' : ' ',
>   		SPLIT_NS(p->se.slice),
> @@ -1007,7 +1008,7 @@ static void print_rq(struct seq_file *m,
>   
>   	SEQ_printf(m, "\n");
>   	SEQ_printf(m, "runnable tasks:\n");
> -	SEQ_printf(m, " S            task   PID       vruntime   eligible    "
> +	SEQ_printf(m, " S            task   PID     weight       vruntime   eligible    "
>   		   "deadline             slice          sum-exec      switches  "
>   		   "prio         wait-time        sum-sleep       sum-block"
>   #ifdef CONFIG_NUMA_BALANCING
> @@ -1115,6 +1116,8 @@ void print_cfs_rq(struct seq_file *m, in
>   			cfs_rq->tg_load_avg_contrib);
>   	SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_avg",
>   			atomic_long_read(&cfs_rq->tg->load_avg));
> +	SEQ_printf(m, "  .%-30s: %lu\n", "h_load",
> +			cfs_rq->h_load);
>   #endif /* CONFIG_FAIR_GROUP_SCHED */
>   #ifdef CONFIG_CFS_BANDWIDTH
>   	SEQ_printf(m, "  .%-30s: %d\n", "throttled",
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -297,8 +297,8 @@ static u64 __calc_delta(u64 delta_exec,
>    */
>   static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)
>   {
> -	if (unlikely(se->load.weight != NICE_0_LOAD))
> -		delta = __calc_delta(delta, NICE_0_LOAD, &se->load);
> +	if (se->h_load.weight != NICE_0_LOAD)
> +		delta = __calc_delta(delta, NICE_0_LOAD, &se->h_load);
>   
>   	return delta;
>   }
> @@ -428,38 +428,6 @@ static inline struct sched_entity *paren
>   	return se->parent;
>   }
>   
> -static void
> -find_matching_se(struct sched_entity **se, struct sched_entity **pse)
> -{
> -	int se_depth, pse_depth;
> -
> -	/*
> -	 * preemption test can be made between sibling entities who are in the
> -	 * same cfs_rq i.e who have a common parent. Walk up the hierarchy of
> -	 * both tasks until we find their ancestors who are siblings of common
> -	 * parent.
> -	 */
> -
> -	/* First walk up until both entities are at same depth */
> -	se_depth = (*se)->depth;
> -	pse_depth = (*pse)->depth;
> -
> -	while (se_depth > pse_depth) {
> -		se_depth--;
> -		*se = parent_entity(*se);
> -	}
> -
> -	while (pse_depth > se_depth) {
> -		pse_depth--;
> -		*pse = parent_entity(*pse);
> -	}
> -
> -	while (!is_same_group(*se, *pse)) {
> -		*se = parent_entity(*se);
> -		*pse = parent_entity(*pse);
> -	}
> -}
> -
>   static int tg_is_idle(struct task_group *tg)
>   {
>   	return tg->idle > 0;
> @@ -503,11 +471,6 @@ static inline struct sched_entity *paren
>   	return NULL;
>   }
>   
> -static inline void
> -find_matching_se(struct sched_entity **se, struct sched_entity **pse)
> -{
> -}
> -
>   static inline int tg_is_idle(struct task_group *tg)
>   {
>   	return 0;
> @@ -686,7 +649,7 @@ static inline unsigned long avg_vruntime
>   static inline void
>   __sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
> -	unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
> +	unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
>   	s64 w_vruntime, key = entity_key(cfs_rq, se);
>   
>   	w_vruntime = key * weight;
> @@ -703,7 +666,7 @@ sum_w_vruntime_add_paranoid(struct cfs_r
>   	s64 key, tmp;
>   
>   again:
> -	weight = avg_vruntime_weight(cfs_rq, se->load.weight);
> +	weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
>   	key = entity_key(cfs_rq, se);
>   
>   	if (check_mul_overflow(key, weight, &key))
> @@ -749,7 +712,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rq
>   static void
>   sum_w_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
> -	unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
> +	unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
>   	s64 key = entity_key(cfs_rq, se);
>   
>   	cfs_rq->sum_w_vruntime -= key * weight;
> @@ -791,7 +754,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)
>   		s64 runtime = cfs_rq->sum_w_vruntime;
>   
>   		if (curr) {
> -			unsigned long w = avg_vruntime_weight(cfs_rq, curr->load.weight);
> +			unsigned long w = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
>   
>   			runtime += entity_key(cfs_rq, curr) * w;
>   			weight += w;
> @@ -862,8 +825,6 @@ bool update_entity_lag(struct cfs_rq *cf
>   	u64 avruntime = avg_vruntime(cfs_rq);
>   	s64 vlag = entity_lag(cfs_rq, se, avruntime);
>   
> -	WARN_ON_ONCE(!se->on_rq);
> -
>   	if (se->sched_delayed) {
>   		/* previous vlag < 0 otherwise se would not be delayed */
>   		vlag = max(vlag, se->vlag);
> @@ -899,7 +860,7 @@ static int vruntime_eligible(struct cfs_
>   	long load = cfs_rq->sum_weight;
>   
>   	if (curr && curr->on_rq) {
> -		unsigned long weight = avg_vruntime_weight(cfs_rq, curr->load.weight);
> +		unsigned long weight = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
>   
>   		avg += entity_key(cfs_rq, curr) * weight;
>   		load += weight;
> @@ -1040,6 +1001,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim
>    */
>   static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
> +	WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
> +	WARN_ON_ONCE(!entity_is_task(se));
> +
>   	sum_w_vruntime_add(cfs_rq, se);
>   	se->min_vruntime = se->vruntime;
>   	se->min_slice = se->slice;
> @@ -1049,6 +1013,9 @@ static void __enqueue_entity(struct cfs_
>   
>   static void __dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
> +	WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
> +	WARN_ON_ONCE(!entity_is_task(se));
> +
>   	rb_erase_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline,
>   				  &min_vruntime_cb);
>   	sum_w_vruntime_sub(cfs_rq, se);
> @@ -1145,7 +1112,7 @@ static struct sched_entity *pick_eevdf(s
>   	 * We can safely skip eligibility check if there is only one entity
>   	 * in this cfs_rq, saving some cycles.
>   	 */
> -	if (cfs_rq->nr_queued == 1)
> +	if (cfs_rq->h_nr_queued == 1)
>   		return curr && curr->on_rq ? curr : se;
>   
>   	/*
> @@ -1395,8 +1362,6 @@ static s64 update_se(struct rq *rq, stru
>   	return delta_exec;
>   }
>   
> -static void set_next_buddy(struct sched_entity *se);
> -
>   #ifdef CONFIG_SCHED_CACHE
>   
>   /*
> @@ -1991,7 +1956,7 @@ static void update_curr(struct cfs_rq *c
>   	 * not necessarily be the actual task running
>   	 * (rq->curr.se). This is easy to confuse!
>   	 */
> -	struct sched_entity *curr = cfs_rq->curr;
> +	struct sched_entity *curr = cfs_rq->h_curr;
>   	struct rq *rq = rq_of(cfs_rq);
>   	s64 delta_exec;
>   	bool resched;
> @@ -2003,26 +1968,29 @@ static void update_curr(struct cfs_rq *c
>   	if (unlikely(delta_exec <= 0))
>   		return;
>   
> +	account_cfs_rq_runtime(cfs_rq, delta_exec);
> +
> +	if (!entity_is_task(curr))
> +		return;
> +
> +	cfs_rq = &rq->cfs;
> +
>   	curr->vruntime += calc_delta_fair(delta_exec, curr);
>   	resched = update_deadline(cfs_rq, curr);
>   
> -	if (entity_is_task(curr)) {
> -		/*
> -		 * If the fair_server is active, we need to account for the
> -		 * fair_server time whether or not the task is running on
> -		 * behalf of fair_server or not:
> -		 *  - If the task is running on behalf of fair_server, we need
> -		 *    to limit its time based on the assigned runtime.
> -		 *  - Fair task that runs outside of fair_server should account
> -		 *    against fair_server such that it can account for this time
> -		 *    and possibly avoid running this period.
> -		 */
> -		dl_server_update(&rq->fair_server, delta_exec);
> -	}
> -
> -	account_cfs_rq_runtime(cfs_rq, delta_exec);
> +	/*
> +	 * If the fair_server is active, we need to account for the
> +	 * fair_server time whether or not the task is running on
> +	 * behalf of fair_server or not:
> +	 *  - If the task is running on behalf of fair_server, we need
> +	 *    to limit its time based on the assigned runtime.
> +	 *  - Fair task that runs outside of fair_server should account
> +	 *    against fair_server such that it can account for this time
> +	 *    and possibly avoid running this period.
> +	 */
> +	dl_server_update(&rq->fair_server, delta_exec);
>   
> -	if (cfs_rq->nr_queued == 1)
> +	if (cfs_rq->h_nr_queued == 1)
>   		return;
>   
>   	if (resched || !protect_slice(curr)) {
> @@ -2033,7 +2001,10 @@ static void update_curr(struct cfs_rq *c
>   
>   static void update_curr_fair(struct rq *rq)
>   {
> -	update_curr(cfs_rq_of(&rq->donor->se));
> +	struct sched_entity *se = &rq->donor->se;
> +
> +	for_each_sched_entity(se)
> +		update_curr(cfs_rq_of(se));
>   }
>   
>   static inline void
> @@ -2109,7 +2080,7 @@ update_stats_enqueue_fair(struct cfs_rq
>   	 * Are we enqueueing a waiting task? (for current tasks
>   	 * a dequeue/enqueue event is a NOP)
>   	 */
> -	if (se != cfs_rq->curr)
> +	if (se != cfs_rq->h_curr)
>   		update_stats_wait_start_fair(cfs_rq, se);
>   
>   	if (flags & ENQUEUE_WAKEUP)
> @@ -2127,7 +2098,7 @@ update_stats_dequeue_fair(struct cfs_rq
>   	 * Mark the end of the wait period if dequeueing a
>   	 * waiting task:
>   	 */
> -	if (se != cfs_rq->curr)
> +	if (se != cfs_rq->h_curr)
>   		update_stats_wait_end_fair(cfs_rq, se);
>   
>   	if ((flags & DEQUEUE_SLEEP) && entity_is_task(se)) {
> @@ -4468,6 +4439,7 @@ static inline void update_scan_period(st
>   static void
>   account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
> +	WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
>   	update_load_add(&cfs_rq->load, se->load.weight);
>   	if (entity_is_task(se)) {
>   		struct task_struct *p = task_of(se);
> @@ -4483,6 +4455,7 @@ account_entity_enqueue(struct cfs_rq *cf
>   static void
>   account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
> +	WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
>   	update_load_sub(&cfs_rq->load, se->load.weight);
>   	if (entity_is_task(se)) {
>   		struct task_struct *p = task_of(se);
> @@ -4564,7 +4537,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq,
>   static void
>   rescale_entity(struct sched_entity *se, unsigned long weight, bool rel_vprot)
>   {
> -	unsigned long old_weight = se->load.weight;
> +	long old_weight = se->h_load.weight;
>   
>   	/*
>   	 * VRUNTIME
> @@ -4664,16 +4637,17 @@ rescale_entity(struct sched_entity *se,
>   		se->vprot = div64_long(se->vprot * old_weight, weight);
>   }
>   
> -static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
> -			    unsigned long weight)
> +static void reweight_eevdf(struct cfs_rq *cfs_rq, struct sched_entity *se,
> +			   unsigned long weight, bool on_rq)
>   {
>   	bool curr = cfs_rq->curr == se;
>   	bool rel_vprot = false;
>   	u64 avruntime = 0;
>   
> -	if (se->on_rq) {
> -		/* commit outstanding execution time */
> -		update_curr(cfs_rq);
> +	if (se->h_load.weight == weight)
> +		return;
> +
> +	if (on_rq) {
>   		avruntime = avg_vruntime(cfs_rq);
>   		se->vlag = entity_lag(cfs_rq, se, avruntime);
>   		se->deadline -= avruntime;
> @@ -4683,46 +4657,90 @@ static void reweight_entity(struct cfs_r
>   			rel_vprot = true;
>   		}
>   
> -		cfs_rq->nr_queued--;
> +		cfs_rq->h_nr_queued--;
>   		if (!curr)
>   			__dequeue_entity(cfs_rq, se);
> -		update_load_sub(&cfs_rq->load, se->load.weight);
>   	}
> -	dequeue_load_avg(cfs_rq, se);
>   
>   	rescale_entity(se, weight, rel_vprot);
>   
> -	update_load_set(&se->load, weight);
> +	update_load_set(&se->h_load, weight);
>   
> -	do {
> -		u32 divider = get_pelt_divider(&se->avg);
> -		se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
> -	} while (0);
> -
> -	enqueue_load_avg(cfs_rq, se);
> -	if (se->on_rq) {
> +	if (on_rq) {
>   		if (rel_vprot)
>   			se->vprot += avruntime;
>   		se->deadline += avruntime;
>   		se->rel_deadline = 0;
>   		se->vruntime = avruntime - se->vlag;
>   
> -		update_load_add(&cfs_rq->load, se->load.weight);
>   		if (!curr)
>   			__enqueue_entity(cfs_rq, se);
> -		cfs_rq->nr_queued++;
> +		cfs_rq->h_nr_queued++;
>   	}
>   }
>   
> +static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
> +			    unsigned long weight)
> +{
> +	if (se->load.weight == weight)
> +		return;
> +
> +	if (se->on_rq) {
> +		WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
> +		update_load_sub(&cfs_rq->load, se->load.weight);
> +	}
> +	dequeue_load_avg(cfs_rq, se);
> +
> +	update_load_set(&se->load, weight);
> +
> +	do {
> +		u32 divider = get_pelt_divider(&se->avg);
> +		se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
> +	} while (0);
> +
> +	enqueue_load_avg(cfs_rq, se);
> +
> +	if (se->on_rq)
> +		update_load_add(&cfs_rq->load, se->load.weight);
> +}
> +
> +/*
> + * weight = NICE_0_LOAD;
> + * for_each_entity_se(se)
> + *   weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
> + */
> +static __always_inline
> +unsigned long __calc_prop_weight(struct cfs_rq *cfs_rq, struct sched_entity *se,
> +				 unsigned long weight)
> +{
> +	weight *= se->load.weight;
> +	if (parent_entity(se))
> +		weight /= cfs_rq->load.weight;
> +	else
> +		weight /= NICE_0_LOAD;
> +
> +	return max(weight, MIN_SHARES);
> +}
> +
>   static void reweight_task_fair(struct rq *rq, struct task_struct *p,
>   			       const struct load_weight *lw)
>   {
>   	struct sched_entity *se = &p->se;
> -	struct cfs_rq *cfs_rq = cfs_rq_of(se);
> -	struct load_weight *load = &se->load;
> +	unsigned long weight = NICE_0_LOAD;
>   
> -	reweight_entity(cfs_rq, se, lw->weight);
> -	load->inv_weight = lw->inv_weight;
> +	if (se->on_rq)
> +		update_curr_fair(rq);
> +
> +	reweight_entity(cfs_rq_of(se), se, lw->weight);
> +	se->load.inv_weight = lw->inv_weight;
> +
> +	if (!se->on_rq)
> +		return;
> +
> +	for_each_sched_entity(se)
> +		weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
> +
> +	reweight_eevdf(&rq->cfs, &p->se, weight, p->se.on_rq);
>   }
>   
>   static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
> @@ -4958,8 +4976,7 @@ static void update_cfs_group(struct sche
>   		return;
>   
>   	shares = static_call(calc_group_shares)(gcfs_rq);
> -	if (unlikely(se->load.weight != shares))
> -		reweight_entity(cfs_rq_of(se), se, shares);
> +	reweight_entity(cfs_rq_of(se), se, shares);
>   }
>   
>   #else /* !CONFIG_FAIR_GROUP_SCHED: */
> @@ -5077,7 +5094,7 @@ static inline bool cfs_rq_is_decayed(str
>    * differential update where we store the last value we propagated. This in
>    * turn allows skipping updates if the differential is 'small'.
>    *
> - * Updating tg's load_avg is necessary before update_cfs_share().
> + * Updating tg's load_avg is necessary before update_cfs_group().
>    */
>   static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
>   {
> @@ -5544,7 +5561,7 @@ static void migrate_se_pelt_lag(struct s
>    * The cfs_rq avg is the direct sum of all its entities (blocked and runnable)
>    * avg. The immediate corollary is that all (fair) tasks must be attached.
>    *
> - * cfs_rq->avg is used for task_h_load() and update_cfs_share() for example.
> + * cfs_rq->avg is used for task_h_load() and update_cfs_group() for example.
>    *
>    * Return: true if the load decayed or we removed load.
>    *
> @@ -6082,6 +6099,7 @@ static void
>   place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>   {
>   	u64 vslice, vruntime = avg_vruntime(cfs_rq);
> +	unsigned int nr_queued = cfs_rq->h_nr_queued;
>   	bool update_zero = false;
>   	s64 lag = 0;
>   
> @@ -6089,6 +6107,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
>   		se->slice = sysctl_sched_base_slice;
>   	vslice = calc_delta_fair(se->slice, se);
>   
> +	if (flags & ENQUEUE_QUEUED)
> +		nr_queued -= 1;
> +
>   	/*
>   	 * Due to how V is constructed as the weighted average of entities,
>   	 * adding tasks with positive lag, or removing tasks with negative lag
> @@ -6097,7 +6118,7 @@ place_entity(struct cfs_rq *cfs_rq, stru
>   	 *
>   	 * EEVDF: placement strategy #1 / #2
>   	 */
> -	if (sched_feat(PLACE_LAG) && cfs_rq->nr_queued && se->vlag) {
> +	if (sched_feat(PLACE_LAG) && nr_queued && se->vlag) {
>   		struct sched_entity *curr = cfs_rq->curr;
>   		long load, weight;
>   
> @@ -6157,9 +6178,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
>   		 */
>   		load = cfs_rq->sum_weight;
>   		if (curr && curr->on_rq)
> -			load += avg_vruntime_weight(cfs_rq, curr->load.weight);
> +			load += avg_vruntime_weight(cfs_rq, curr->h_load.weight);
>   
> -		weight = avg_vruntime_weight(cfs_rq, se->load.weight);
> +		weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
>   		lag *= load + weight;
>   		if (WARN_ON_ONCE(!load))
>   			load = 1;
> @@ -6218,22 +6239,8 @@ static void check_enqueue_throttle(struc
>   static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
>   
>   static void
> -requeue_delayed_entity(struct sched_entity *se);
> -
> -static void
>   enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>   {
> -	bool curr = cfs_rq->curr == se;
> -
> -	/*
> -	 * If we're the current task, we must renormalise before calling
> -	 * update_curr().
> -	 */
> -	if (curr)
> -		place_entity(cfs_rq, se, flags);
> -
> -	update_curr(cfs_rq);
> -
>   	/*
>   	 * When enqueuing a sched_entity, we must:
>   	 *   - Update loads to have both entity and cfs_rq synced with now.
> @@ -6252,13 +6259,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
>   	 */
>   	update_cfs_group(se);
>   
> -	/*
> -	 * XXX now that the entity has been re-weighted, and it's lag adjusted,
> -	 * we can place the entity.
> -	 */
> -	if (!curr)
> -		place_entity(cfs_rq, se, flags);
> -
>   	account_entity_enqueue(cfs_rq, se);
>   
>   	/* Entity has migrated, no longer consider this task hot */
> @@ -6267,8 +6267,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
>   
>   	check_schedstat_required();
>   	update_stats_enqueue_fair(cfs_rq, se, flags);
> -	if (!curr)
> -		__enqueue_entity(cfs_rq, se);
>   	se->on_rq = 1;
>   
>   	if (cfs_rq->nr_queued == 1) {
> @@ -6286,21 +6284,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
>   	}
>   }
>   
> -static void __clear_buddies_next(struct sched_entity *se)
> +static void set_next_buddy(struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
> -	for_each_sched_entity(se) {
> -		struct cfs_rq *cfs_rq = cfs_rq_of(se);
> -		if (cfs_rq->next != se)
> -			break;
> -
> -		cfs_rq->next = NULL;
> -	}
> +	if (WARN_ON_ONCE(!se->on_rq || se->sched_delayed))
> +		return;
> +	if (se_is_idle(se))
> +		return;
> +	cfs_rq->next = se;
>   }
>   
>   static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
>   	if (cfs_rq->next == se)
> -		__clear_buddies_next(se);
> +		cfs_rq->next = NULL;
>   }
>   
>   static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
> @@ -6311,7 +6307,7 @@ static void set_delayed(struct sched_ent
>   
>   	/*
>   	 * Delayed se of cfs_rq have no tasks queued on them.
> -	 * Do not adjust h_nr_runnable since dequeue_entities()
> +	 * Do not adjust h_nr_runnable since __dequeue_task()
>   	 * will account it for blocked tasks.
>   	 */
>   	if (!entity_is_task(se))
> @@ -6344,45 +6340,16 @@ static void clear_delayed(struct sched_e
>   	}
>   }
>   
> -static bool
> +static void
>   dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>   {
> -	bool sleep = flags & DEQUEUE_SLEEP;
> -	int action = 0;
> -
> -	update_curr(cfs_rq);
> -	clear_buddies(cfs_rq, se);
> -
> -	if (flags & DEQUEUE_DELAYED) {
> -		WARN_ON_ONCE(!se->sched_delayed);
> -	} else {
> -		bool delay = sleep;
> -		/*
> -		 * DELAY_DEQUEUE relies on spurious wakeups, special task
> -		 * states must not suffer spurious wakeups, excempt them.
> -		 */
> -		if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
> -			delay = false;
> -
> -		WARN_ON_ONCE(delay && se->sched_delayed);
> +	int action = UPDATE_TG;
>   
> -		if (sched_feat(DELAY_DEQUEUE) && delay &&
> -		    !entity_eligible(cfs_rq, se)) {
> -			if (entity_is_task(se))
> -				action |= UPDATE_UTIL_EST;
> -			update_load_avg(cfs_rq, se, action);
> -			update_entity_lag(cfs_rq, se);
> -			set_delayed(se);
> -			return false;
> -		}
> -	}
> -
> -	action = UPDATE_TG;
>   	if (entity_is_task(se)) {
>   		if (task_on_rq_migrating(task_of(se)))
>   			action |= DO_DETACH;
>   
> -		if (sleep && !(flags & DEQUEUE_DELAYED))
> +		if ((flags & DEQUEUE_SLEEP) && !(flags & DEQUEUE_DELAYED))
>   			action |= UPDATE_UTIL_EST;
>   	}
>   
> @@ -6400,14 +6367,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
>   
>   	update_stats_dequeue_fair(cfs_rq, se, flags);
>   
> -	update_entity_lag(cfs_rq, se);
> -	if (sched_feat(PLACE_REL_DEADLINE) && !sleep) {
> -		se->deadline -= se->vruntime;
> -		se->rel_deadline = 1;
> -	}
> -
> -	if (se != cfs_rq->curr)
> -		__dequeue_entity(cfs_rq, se);
>   	se->on_rq = 0;
>   	account_entity_dequeue(cfs_rq, se);
>   
> @@ -6416,9 +6375,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
>   
>   	update_cfs_group(se);
>   
> -	if (flags & DEQUEUE_DELAYED)
> -		clear_delayed(se);
> -
>   	if (cfs_rq->nr_queued == 0) {
>   		update_idle_cfs_rq_clock_pelt(cfs_rq);
>   #ifdef CONFIG_CFS_BANDWIDTH
> @@ -6431,15 +6387,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
>   		}
>   #endif
>   	}
> -
> -	return true;
>   }
>   
>   static void
> -set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first)
> +set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
> -	clear_buddies(cfs_rq, se);
> -
>   	/* 'current' is not kept within the tree. */
>   	if (se->on_rq) {
>   		/*
> @@ -6448,16 +6400,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s
>   		 * runqueue.
>   		 */
>   		update_stats_wait_end_fair(cfs_rq, se);
> -		__dequeue_entity(cfs_rq, se);
>   		update_load_avg(cfs_rq, se, UPDATE_TG);
> -
> -		if (first)
> -			set_protect_slice(cfs_rq, se);
>   	}
>   
>   	update_stats_curr_start(cfs_rq, se);
> -	WARN_ON_ONCE(cfs_rq->curr);
> -	cfs_rq->curr = se;
> +	WARN_ON_ONCE(cfs_rq->h_curr);
> +	cfs_rq->h_curr = se;
>   
>   	/*
>   	 * Track our maximum slice length, if the CPU's load is at
> @@ -6477,23 +6425,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s
>   	se->prev_sum_exec_runtime = se->sum_exec_runtime;
>   }
>   
> -static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags);
> +static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags);
>   
> -/*
> - * Pick the next process, keeping these things in mind, in this order:
> - * 1) keep things fair between processes/task groups
> - * 2) pick the "next" process, since someone really wants that to run
> - * 3) pick the "last" process, for cache locality
> - * 4) do not run the "skip" process, if something else is available
> - */
>   static struct sched_entity *
> -pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
> +pick_next_entity(struct rq *rq, bool protect)
>   {
> +	struct cfs_rq *cfs_rq = &rq->cfs;
>   	struct sched_entity *se;
>   
>   	se = pick_eevdf(cfs_rq, protect);
>   	if (se->sched_delayed) {
> -		dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> +		__dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_DELAYED);
>   		/*
>   		 * Must not reference @se again, see __block_task().
>   		 */
> @@ -6513,13 +6455,11 @@ static void put_prev_entity(struct cfs_r
>   
>   	if (prev->on_rq) {
>   		update_stats_wait_start_fair(cfs_rq, prev);
> -		/* Put 'current' back into the tree. */
> -		__enqueue_entity(cfs_rq, prev);
>   		/* in !on_rq case, update occurred at dequeue */
>   		update_load_avg(cfs_rq, prev, 0);
>   	}
> -	WARN_ON_ONCE(cfs_rq->curr != prev);
> -	cfs_rq->curr = NULL;
> +	WARN_ON_ONCE(cfs_rq->h_curr != prev);
> +	cfs_rq->h_curr = NULL;
>   }
>   
>   static void
> @@ -7074,7 +7014,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cf
>   	assert_list_leaf_cfs_rq(rq);
>   
>   	/* Determine whether we need to wake up potentially idle CPU: */
> -	if (rq->curr == rq->idle && rq->cfs.nr_queued)
> +	if (rq->curr == rq->idle && rq->cfs.h_nr_queued)
>   		resched_curr(rq);
>   }
>   
> @@ -7409,7 +7349,7 @@ static void check_enqueue_throttle(struc
>   		return;
>   
>   	/* an active group must be handled by the update_curr() path */
> -	if (!cfs_rq->runtime_enabled || cfs_rq->curr)
> +	if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)
>   		return;
>   
>   	/* ensure the group is not already throttled */
> @@ -7781,7 +7721,7 @@ static void hrtick_start_fair(struct rq
>   			resched_curr(rq);
>   		return;
>   	}
> -	delta = (se->load.weight * vdelta) / NICE_0_LOAD;
> +	delta = (se->h_load.weight * vdelta) / NICE_0_LOAD;
>   
>   	/*
>   	 * Correct for instantaneous load of other classes.
> @@ -7881,10 +7821,8 @@ static int choose_idle_cpu(int cpu, stru
>   }
>   
>   static void
> -requeue_delayed_entity(struct sched_entity *se)
> +requeue_delayed_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
> -	struct cfs_rq *cfs_rq = cfs_rq_of(se);
> -
>   	/*
>   	 * se->sched_delayed should imply: se->on_rq == 1.
>   	 * Because a delayed entity is one that is still on
> @@ -7894,19 +7832,58 @@ requeue_delayed_entity(struct sched_enti
>   	WARN_ON_ONCE(!se->on_rq);
>   
>   	if (update_entity_lag(cfs_rq, se)) {
> -		cfs_rq->nr_queued--;
> +		cfs_rq->h_nr_queued--;
>   		if (se != cfs_rq->curr)
>   			__dequeue_entity(cfs_rq, se);
>   		place_entity(cfs_rq, se, 0);
>   		if (se != cfs_rq->curr)
>   			__enqueue_entity(cfs_rq, se);
> -		cfs_rq->nr_queued++;
> +		cfs_rq->h_nr_queued++;
>   	}
>   
>   	update_load_avg(cfs_rq, se, 0);
>   	clear_delayed(se);
>   }
>   
> +static unsigned long enqueue_hierarchy(struct task_struct *p, int flags)
> +{
> +	unsigned long weight = NICE_0_LOAD;
> +	int task_new = !(flags & ENQUEUE_WAKEUP);
> +	struct sched_entity *se = &p->se;
> +	int h_nr_idle = task_has_idle_policy(p);
> +	int h_nr_runnable = 1;
> +
> +	if (task_new && se->sched_delayed)
> +		h_nr_runnable = 0;
> +
> +	for_each_sched_entity(se) {
> +		struct cfs_rq *cfs_rq = cfs_rq_of(se);
> +
> +		update_curr(cfs_rq);
> +
> +		if (!se->on_rq) {
> +			enqueue_entity(cfs_rq, se, flags);
> +		} else {
> +			update_load_avg(cfs_rq, se, UPDATE_TG);
> +			se_update_runnable(se);
> +			update_cfs_group(se);
> +		}
> +
> +		cfs_rq->h_nr_runnable += h_nr_runnable;
> +		cfs_rq->h_nr_queued++;
> +		cfs_rq->h_nr_idle += h_nr_idle;
> +
> +		if (cfs_rq_is_idle(cfs_rq))
> +			h_nr_idle = 1;
> +
> +		weight = __calc_prop_weight(cfs_rq, se, weight);
> +
> +		flags = ENQUEUE_WAKEUP;
> +	}
> +
> +	return weight;
> +}
> +
>   /*
>    * The enqueue_task method is called before nr_running is
>    * increased. Here we update the fair scheduling stats and
> @@ -7915,13 +7892,12 @@ requeue_delayed_entity(struct sched_enti
>   static void
>   enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
>   {
> -	struct cfs_rq *cfs_rq;
> -	struct sched_entity *se = &p->se;
> -	int h_nr_idle = task_has_idle_policy(p);
> -	int h_nr_runnable = 1;
> -	int task_new = !(flags & ENQUEUE_WAKEUP);
>   	int rq_h_nr_queued = rq->cfs.h_nr_queued;
> -	u64 slice = 0;
> +	int task_new = !(flags & ENQUEUE_WAKEUP);
> +	struct sched_entity *se = &p->se;
> +	struct cfs_rq *cfs_rq = &rq->cfs;
> +	unsigned long weight;
> +	bool curr;
>   
>   	if (task_is_throttled(p) && enqueue_throttled_task(p))
>   		return;
> @@ -7933,10 +7909,10 @@ enqueue_task_fair(struct rq *rq, struct
>   	 * estimated utilization, before we update schedutil.
>   	 */
>   	if (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))
> -		util_est_enqueue(&rq->cfs, p);
> +		util_est_enqueue(cfs_rq, p);
>   
>   	if (flags & ENQUEUE_DELAYED) {
> -		requeue_delayed_entity(se);
> +		requeue_delayed_entity(cfs_rq, se);
>   		return;
>   	}
>   
> @@ -7948,57 +7924,22 @@ enqueue_task_fair(struct rq *rq, struct
>   	if (p->in_iowait)
>   		cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT);
>   
> -	if (task_new && se->sched_delayed)
> -		h_nr_runnable = 0;
> -
> -	for_each_sched_entity(se) {
> -		if (se->on_rq) {
> -			if (se->sched_delayed)
> -				requeue_delayed_entity(se);
> -			break;
> -		}
> -		cfs_rq = cfs_rq_of(se);
> -
> -		/*
> -		 * Basically set the slice of group entries to the min_slice of
> -		 * their respective cfs_rq. This ensures the group can service
> -		 * its entities in the desired time-frame.
> -		 */
> -		if (slice) {
> -			se->slice = slice;
> -			se->custom_slice = 1;
> -		}
> -		enqueue_entity(cfs_rq, se, flags);
> -		slice = cfs_rq_min_slice(cfs_rq);
> -
> -		cfs_rq->h_nr_runnable += h_nr_runnable;
> -		cfs_rq->h_nr_queued++;
> -		cfs_rq->h_nr_idle += h_nr_idle;
> -
> -		if (cfs_rq_is_idle(cfs_rq))
> -			h_nr_idle = 1;
> -
> -		flags = ENQUEUE_WAKEUP;
> -	}
> -
> -	for_each_sched_entity(se) {
> -		cfs_rq = cfs_rq_of(se);
> +	/*
> +	 * XXX comment on the curr thing
> +	 */
> +	curr = (cfs_rq->curr == se);
> +	if (curr)
> +		place_entity(cfs_rq, se, flags);
>   
> -		update_load_avg(cfs_rq, se, UPDATE_TG);
> -		se_update_runnable(se);
> -		update_cfs_group(se);
> +	if (se->on_rq && se->sched_delayed)
> +		requeue_delayed_entity(cfs_rq, se);
>   
> -		se->slice = slice;
> -		if (se != cfs_rq->curr)
> -			min_vruntime_cb_propagate(&se->run_node, NULL);
> -		slice = cfs_rq_min_slice(cfs_rq);
> +	weight = enqueue_hierarchy(p, flags);
>   
> -		cfs_rq->h_nr_runnable += h_nr_runnable;
> -		cfs_rq->h_nr_queued++;
> -		cfs_rq->h_nr_idle += h_nr_idle;
> -
> -		if (cfs_rq_is_idle(cfs_rq))
> -			h_nr_idle = 1;
> +	if (!curr) {
> +		reweight_eevdf(cfs_rq, se, weight, false);
> +		place_entity(cfs_rq, se, flags | ENQUEUE_QUEUED);
> +		__enqueue_entity(cfs_rq, se);
>   	}
>   
>   	if (!rq_h_nr_queued && rq->cfs.h_nr_queued)
> @@ -8029,105 +7970,109 @@ enqueue_task_fair(struct rq *rq, struct
>   	hrtick_update(rq);
>   }
>   
> -/*
> - * Basically dequeue_task_fair(), except it can deal with dequeue_entity()
> - * failing half-way through and resume the dequeue later.
> - *
> - * Returns:
> - * -1 - dequeue delayed
> - *  0 - dequeue throttled
> - *  1 - dequeue complete
> - */
> -static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
> +static void dequeue_hierarchy(struct task_struct *p, int flags)
>   {
> -	bool was_sched_idle = sched_idle_rq(rq);
> +	struct sched_entity *se = &p->se;
>   	bool task_sleep = flags & DEQUEUE_SLEEP;
>   	bool task_delayed = flags & DEQUEUE_DELAYED;
>   	bool task_throttled = flags & DEQUEUE_THROTTLE;
> -	struct task_struct *p = NULL;
> -	int h_nr_idle = 0;
> -	int h_nr_queued = 0;
>   	int h_nr_runnable = 0;
> -	struct cfs_rq *cfs_rq;
> -	u64 slice = 0;
> +	int h_nr_idle = task_has_idle_policy(p);
> +	bool dequeue = true;
>   
> -	if (entity_is_task(se)) {
> -		p = task_of(se);
> -		h_nr_queued = 1;
> -		h_nr_idle = task_has_idle_policy(p);
> -		if (task_sleep || task_delayed || !se->sched_delayed)
> -			h_nr_runnable = 1;
> -	}
> +	if (task_sleep || task_delayed || !se->sched_delayed)
> +		h_nr_runnable = 1;
>   
>   	for_each_sched_entity(se) {
> -		cfs_rq = cfs_rq_of(se);
> +		struct cfs_rq *cfs_rq = cfs_rq_of(se);
>   
> -		if (!dequeue_entity(cfs_rq, se, flags)) {
> -			if (p && &p->se == se)
> -				return -1;
> +		update_curr(cfs_rq);
>   
> -			slice = cfs_rq_min_slice(cfs_rq);
> -			break;
> +		if (dequeue) {
> +			dequeue_entity(cfs_rq, se, flags);
> +			/* Don't dequeue parent if it has other entities besides us */
> +			if (cfs_rq->load.weight)
> +				dequeue = false;
> +		} else {
> +			update_load_avg(cfs_rq, se, UPDATE_TG);
> +			se_update_runnable(se);
> +			update_cfs_group(se);
>   		}
>   
>   		cfs_rq->h_nr_runnable -= h_nr_runnable;
> -		cfs_rq->h_nr_queued -= h_nr_queued;
> +		cfs_rq->h_nr_queued--;
>   		cfs_rq->h_nr_idle -= h_nr_idle;
>   
>   		if (cfs_rq_is_idle(cfs_rq))
> -			h_nr_idle = h_nr_queued;
> +			h_nr_idle = 1;
>   
>   		if (throttled_hierarchy(cfs_rq) && task_throttled)
>   			record_throttle_clock(cfs_rq);
>   
> -		/* Don't dequeue parent if it has other entities besides us */
> -		if (cfs_rq->load.weight) {
> -			slice = cfs_rq_min_slice(cfs_rq);
> -
> -			/* Avoid re-evaluating load for this entity: */
> -			se = parent_entity(se);
> -			/*
> -			 * Bias pick_next to pick a task from this cfs_rq, as
> -			 * p is sleeping when it is within its sched_slice.
> -			 */
> -			if (task_sleep && se)
> -				set_next_buddy(se);
> -			break;
> -		}
>   		flags |= DEQUEUE_SLEEP;
>   		flags &= ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL);
>   	}
> +}
>   
> -	for_each_sched_entity(se) {
> -		cfs_rq = cfs_rq_of(se);
> +/*
> + * The part of dequeue_task_fair() that is needed to dequeue delayed tasks.
> + *
> + * Returns:
> + *   true  - dequeued
> + *   false - delayed
> + */
> +static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
> +{
> +	struct sched_entity *se = &p->se;
> +	struct cfs_rq *cfs_rq = &rq->cfs;
> +	bool was_sched_idle = sched_idle_rq(rq);
> +	bool task_sleep = flags & DEQUEUE_SLEEP;
> +	bool task_delayed = flags & DEQUEUE_DELAYED;
>   
> -		update_load_avg(cfs_rq, se, UPDATE_TG);
> -		se_update_runnable(se);
> -		update_cfs_group(se);
> +	clear_buddies(cfs_rq, se);
>   
> -		se->slice = slice;
> -		if (se != cfs_rq->curr)
> -			min_vruntime_cb_propagate(&se->run_node, NULL);
> -		slice = cfs_rq_min_slice(cfs_rq);
> +	update_curr(cfs_rq_of(se));
> +	update_entity_lag(cfs_rq, se);
>   
> -		cfs_rq->h_nr_runnable -= h_nr_runnable;
> -		cfs_rq->h_nr_queued -= h_nr_queued;
> -		cfs_rq->h_nr_idle -= h_nr_idle;
> +	if (flags & DEQUEUE_DELAYED) {
> +		WARN_ON_ONCE(!se->sched_delayed);
> +	} else {
> +		bool delay = task_sleep;
> +		/*
> +		 * DELAY_DEQUEUE relies on spurious wakeups, special task
> +		 * states must not suffer spurious wakeups, excempt them.
> +		 */
> +		if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
> +			delay = false;
>   
> -		if (cfs_rq_is_idle(cfs_rq))
> -			h_nr_idle = h_nr_queued;
> +		WARN_ON_ONCE(delay && se->sched_delayed);
>   
> -		if (throttled_hierarchy(cfs_rq) && task_throttled)
> -			record_throttle_clock(cfs_rq);
> +		if (sched_feat(DELAY_DEQUEUE) && delay &&
> +		    !entity_eligible(cfs_rq, se)) {
> +			update_load_avg(cfs_rq_of(se), se, UPDATE_UTIL_EST);
> +			set_delayed(se);
> +			return false;
> +		}
>   	}
>   
> -	sub_nr_running(rq, h_nr_queued);
> +	dequeue_hierarchy(p, flags);
> +
> +	if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
> +		se->deadline -= se->vruntime;
> +		se->rel_deadline = 1;
> +	}
> +	if (se != cfs_rq->curr)
> +		__dequeue_entity(cfs_rq, se);
> +
> +	sub_nr_running(rq, 1);
>   
>   	/* balance early to pull high priority tasks */
>   	if (unlikely(!was_sched_idle && sched_idle_rq(rq)))
>   		rq->next_balance = jiffies;
>   
> -	if (p && task_delayed) {
> +	if (task_delayed) {
> +		clear_delayed(se);
> +
>   		WARN_ON_ONCE(!task_sleep);
>   		WARN_ON_ONCE(p->on_rq != 1);
>   
> @@ -8139,7 +8084,7 @@ static int dequeue_entities(struct rq *r
>   		__block_task(rq, p);
>   	}
>   
> -	return 1;
> +	return true;
>   }
>   
>   /*
> @@ -8157,11 +8102,11 @@ static bool dequeue_task_fair(struct rq
>   	if (!p->se.sched_delayed)
>   		util_est_dequeue(&rq->cfs, p);
>   
> -	if (dequeue_entities(rq, &p->se, flags) < 0)
> +	if (!__dequeue_task(rq, p, flags))
>   		return false;
>   
>   	/*
> -	 * Must not reference @p after dequeue_entities(DEQUEUE_DELAYED).
> +	 * Must not reference @p after __dequeue_task(DEQUEUE_DELAYED).
>   	 */
>   	return true;
>   }
> @@ -9749,19 +9694,6 @@ static void migrate_task_rq_fair(struct
>   static void task_dead_fair(struct task_struct *p)
>   {
>   	struct sched_entity *se = &p->se;
> -
> -	if (se->sched_delayed) {
> -		struct rq_flags rf;
> -		struct rq *rq;
> -
> -		rq = task_rq_lock(p, &rf);
> -		if (se->sched_delayed) {
> -			update_rq_clock(rq);
> -			dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> -		}
> -		task_rq_unlock(rq, p, &rf);
> -	}
> -
>   	remove_entity_load_avg(se);
>   }
>   
> @@ -9795,21 +9727,10 @@ static void set_cpus_allowed_fair(struct
>   	set_task_max_allowed_capacity(p);
>   }
>   
> -static void set_next_buddy(struct sched_entity *se)
> -{
> -	for_each_sched_entity(se) {
> -		if (WARN_ON_ONCE(!se->on_rq))
> -			return;
> -		if (se_is_idle(se))
> -			return;
> -		cfs_rq_of(se)->next = se;
> -	}
> -}
> -
>   enum preempt_wakeup_action {
>   	PREEMPT_WAKEUP_NONE,	/* No preemption. */
>   	PREEMPT_WAKEUP_SHORT,	/* Ignore slice protection. */
> -	PREEMPT_WAKEUP_PICK,	/* Let __pick_eevdf() decide. */
> +	PREEMPT_WAKEUP_PICK,	/* Let pick_eevdf() decide. */
>   	PREEMPT_WAKEUP_RESCHED,	/* Force reschedule. */
>   };
>   
> @@ -9826,7 +9747,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq,
>   	if (cfs_rq->next && entity_before(cfs_rq->next, pse))
>   		return false;
>   
> -	set_next_buddy(pse);
> +	set_next_buddy(cfs_rq, pse);
>   	return true;
>   }
>   
> @@ -9879,7 +9800,7 @@ static void wakeup_preempt_fair(struct r
>   	enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
>   	struct task_struct *donor = rq->donor;
>   	struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
> -	struct cfs_rq *cfs_rq = task_cfs_rq(donor);
> +	struct cfs_rq *cfs_rq = &rq->cfs;
>   	int cse_is_idle, pse_is_idle;
>   
>   	/*
> @@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r
>   	if (!sched_feat(WAKEUP_PREEMPTION))
>   		return;
>   
> -	find_matching_se(&se, &pse);
>   	WARN_ON_ONCE(!pse);
>   
>   	cse_is_idle = se_is_idle(se);
> @@ -9944,8 +9864,7 @@ static void wakeup_preempt_fair(struct r
>   	if (unlikely(!normal_policy(p->policy)))
>   		return;
>   
> -	cfs_rq = cfs_rq_of(se);
> -	update_curr(cfs_rq);
> +	update_curr_fair(rq);
>   	/*
>   	 * If @p has a shorter slice than current and @p is eligible, override
>   	 * current's slice protection in order to allow preemption.
> @@ -9989,18 +9908,15 @@ static void wakeup_preempt_fair(struct r
>   	}
>   
>   pick:
> -	nse = pick_next_entity(rq, cfs_rq, preempt_action != PREEMPT_WAKEUP_SHORT);
> -	/* If @p has become the most eligible task, force preemption */
> -	if (nse == pse)
> -		goto preempt;
> -
> -	/*
> -	 * Because p is enqueued, nse being null can only mean that we
> -	 * dequeued a delayed task. If there are still entities queued in
> -	 * cfs, check if the next one will be p.
> -	 */
> -	if (!nse && cfs_rq->nr_queued)
> -		goto pick;
> +	if (cfs_rq->h_nr_queued) {
> +		nse = pick_next_entity(rq, preempt_action != PREEMPT_WAKEUP_SHORT);
> +		if (unlikely(!nse))
> +			goto pick;
> +
> +		/* If @p has become the most eligible task, force preemption */
> +		if (nse == pse)
> +			goto preempt;
> +	}
>   
>   	if (sched_feat(RUN_TO_PARITY))
>   		update_protect_slice(cfs_rq, se);
> @@ -10019,33 +9935,24 @@ static void wakeup_preempt_fair(struct r
>   struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
>   	__must_hold(__rq_lockp(rq))
>   {
> +	struct cfs_rq *cfs_rq = &rq->cfs;
>   	struct sched_entity *se;
> -	struct cfs_rq *cfs_rq;
>   	struct task_struct *p;
> -	bool throttled;
>   	int new_tasks;
>   
>   again:
> -	cfs_rq = &rq->cfs;
> -	if (!cfs_rq->nr_queued)
> +	if (!cfs_rq->h_nr_queued)
>   		goto idle;
>   
> -	throttled = false;
> -
> -	do {
> -		/* Might not have done put_prev_entity() */
> -		if (cfs_rq->curr && cfs_rq->curr->on_rq)
> -			update_curr(cfs_rq);
> +	/* Might not have done put_prev_entity() */
> +	if (cfs_rq->curr && cfs_rq->curr->on_rq)
> +		update_curr(cfs_rq);
>   
> -		se = pick_next_entity(rq, cfs_rq, true);
> -		if (!se)
> -			goto again;
> -		cfs_rq = group_cfs_rq(se);
> -	} while (cfs_rq);
> +	se = pick_next_entity(rq, true);
> +	if (!se)
> +		goto again;
>   
>   	p = task_of(se);
> -	if (unlikely(throttled))
> -		task_throttle_setup_work(p);
>   	return p;
>   
>   idle:
> @@ -10079,7 +9986,7 @@ void fair_server_init(struct rq *rq)
>   static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct task_struct *next)
>   {
>   	struct sched_entity *se = &prev->se;
> -	struct cfs_rq *cfs_rq;
> +	struct cfs_rq *cfs_rq = &rq->cfs;
>   	struct sched_entity *nse = NULL;
>   
>   #ifdef CONFIG_FAIR_GROUP_SCHED
> @@ -10089,7 +9996,7 @@ static void put_prev_task_fair(struct rq
>   
>   	while (se) {
>   		cfs_rq = cfs_rq_of(se);
> -		if (!nse || cfs_rq->curr)
> +		if (!nse || cfs_rq->h_curr)
>   			put_prev_entity(cfs_rq, se);
>   #ifdef CONFIG_FAIR_GROUP_SCHED
>   		if (nse) {
> @@ -10108,6 +10015,14 @@ static void put_prev_task_fair(struct rq
>   #endif
>   		se = parent_entity(se);
>   	}
> +
> +	/* Put 'current' back into the tree. */
> +	cfs_rq = &rq->cfs;
> +	se = &prev->se;
> +	WARN_ON_ONCE(cfs_rq->curr != se);
> +	cfs_rq->curr = NULL;
> +	if (se->on_rq)
> +		__enqueue_entity(cfs_rq, se);
>   }
>   
>   /*
> @@ -10116,8 +10031,8 @@ static void put_prev_task_fair(struct rq
>   static void yield_task_fair(struct rq *rq)
>   {
>   	struct task_struct *curr = rq->donor;
> -	struct cfs_rq *cfs_rq = task_cfs_rq(curr);
>   	struct sched_entity *se = &curr->se;
> +	struct cfs_rq *cfs_rq = &rq->cfs;
>   
>   	/*
>   	 * Are we the only task in the tree?
> @@ -10158,11 +10073,11 @@ static bool yield_to_task_fair(struct rq
>   	struct sched_entity *se = &p->se;
>   
>   	/* !se->on_rq also covers throttled task */
> -	if (!se->on_rq)
> +	if (!se->on_rq || se->sched_delayed)
>   		return false;
>   
>   	/* Tell the scheduler that we'd really like se to run next. */
> -	set_next_buddy(se);
> +	set_next_buddy(&task_rq(p)->cfs, se);
>   
>   	yield_task_fair(rq);
>   
> @@ -10501,15 +10416,10 @@ static inline long migrate_degrades_loca
>    */
>   static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_cpu)
>   {
> -	struct cfs_rq *dst_cfs_rq;
> +	struct cfs_rq *dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
>   
> -#ifdef CONFIG_FAIR_GROUP_SCHED
> -	dst_cfs_rq = tg_cfs_rq(task_group(p), dest_cpu);
> -#else
> -	dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
> -#endif
> -	if (sched_feat(PLACE_LAG) && dst_cfs_rq->nr_queued &&
> -	    !entity_eligible(task_cfs_rq(p), &p->se))
> +	if (sched_feat(PLACE_LAG) && dst_cfs_rq->h_nr_queued &&
> +	    !entity_eligible(&task_rq(p)->cfs, &p->se))
>   		return 1;
>   
>   	return 0;
> @@ -11292,7 +11202,7 @@ static void update_cfs_rq_h_load(struct
>   	while ((se = READ_ONCE(cfs_rq->h_load_next)) != NULL) {
>   		load = cfs_rq->h_load;
>   		load = div64_ul(load * se->avg.load_avg,
> -			cfs_rq_load_avg(cfs_rq) + 1);
> +				cfs_rq_load_avg(cfs_rq) + 1);
>   		cfs_rq = group_cfs_rq(se);
>   		cfs_rq->h_load = load;
>   		cfs_rq->last_h_load_update = now;
> @@ -14684,7 +14594,7 @@ static inline void task_tick_core(struct
>   	 * MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check
>   	 * if we need to give up the CPU.
>   	 */
> -	if (rq->core->core_forceidle_count && rq->cfs.nr_queued == 1 &&
> +	if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued == 1 &&
>   	    __entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))
>   		resched_curr(rq);
>   }
> @@ -14893,30 +14803,8 @@ bool cfs_prio_less(const struct task_str
>   
>   	WARN_ON_ONCE(task_rq(b)->core != rq->core);
>   
> -#ifdef CONFIG_FAIR_GROUP_SCHED
> -	/*
> -	 * Find an se in the hierarchy for tasks a and b, such that the se's
> -	 * are immediate siblings.
> -	 */
> -	while (sea->cfs_rq->tg != seb->cfs_rq->tg) {
> -		int sea_depth = sea->depth;
> -		int seb_depth = seb->depth;
> -
> -		if (sea_depth >= seb_depth)
> -			sea = parent_entity(sea);
> -		if (sea_depth <= seb_depth)
> -			seb = parent_entity(seb);
> -	}
> -
> -	se_fi_update(sea, rq->core->core_forceidle_seq, in_fi);
> -	se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);
> -
> -	cfs_rqa = sea->cfs_rq;
> -	cfs_rqb = seb->cfs_rq;
> -#else /* !CONFIG_FAIR_GROUP_SCHED: */
>   	cfs_rqa = &task_rq(a)->cfs;
>   	cfs_rqb = &task_rq(b)->cfs;
> -#endif /* !CONFIG_FAIR_GROUP_SCHED */
>   
>   	/*
>   	 * Find delta after normalizing se's vruntime with its cfs_rq's
> @@ -14955,11 +14843,20 @@ static inline void task_tick_core(struct
>   static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
>   {
>   	struct sched_entity *se = &curr->se;
> -	struct cfs_rq *cfs_rq;
>   
> -	for_each_sched_entity(se) {
> -		cfs_rq = cfs_rq_of(se);
> -		entity_tick(cfs_rq, se, queued);
> +	if (se->on_rq) {
> +		unsigned long weight = NICE_0_LOAD;
> +		struct cfs_rq *cfs_rq;
> +
> +		for_each_sched_entity(se) {
> +			cfs_rq = cfs_rq_of(se);
> +			entity_tick(cfs_rq, se, queued);
> +
> +			weight = __calc_prop_weight(cfs_rq, se, weight);
> +		}
> +
> +		se = &curr->se;
> +		reweight_eevdf(cfs_rq, se, weight, se->on_rq);
>   	}
>   
>   	if (queued)
> @@ -14999,7 +14896,7 @@ prio_changed_fair(struct rq *rq, struct
>   	if (p->prio == oldprio)
>   		return;
>   
> -	if (rq->cfs.nr_queued == 1)
> +	if (rq->cfs.h_nr_queued == 1)
>   		return;
>   
>   	/*
> @@ -15128,33 +15025,44 @@ static void switched_to_fair(struct rq *
>   	}
>   }
>   
> -/*
> - * Account for a task changing its policy or group.
> - *
> - * This routine is mostly called to set cfs_rq->curr field when a task
> - * migrates between groups/classes.
> - */
>   static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
>   {
>   	struct sched_entity *se = &p->se;
>   	bool throttled = false;
> +	struct cfs_rq *cfs_rq = &rq->cfs;
> +	unsigned long weight = NICE_0_LOAD;
> +	bool on_rq = se->on_rq;
> +
> +	clear_buddies(cfs_rq, se);
> +
> +	if (on_rq)
> +		__dequeue_entity(cfs_rq, se);
>   
>   	for_each_sched_entity(se) {
> -		struct cfs_rq *cfs_rq = cfs_rq_of(se);
> +		cfs_rq = cfs_rq_of(se);
>   
> -		if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
> -		    first && cfs_rq->curr)
> -			break;
> +		if (!IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) ||
> +		    !first || !cfs_rq->h_curr)
> +			set_next_entity(cfs_rq, se);
>   
> -		set_next_entity(cfs_rq, se, first);
>   		/* ensure bandwidth has been allocated on our new cfs_rq */
>   		throttled |= account_cfs_rq_runtime(cfs_rq, 0);
> +
> +		if (on_rq)
> +			weight = __calc_prop_weight(cfs_rq, se, weight);
>   	}
>   
>   	if (throttled)
>   		task_throttle_setup_work(p);
>   
>   	se = &p->se;
> +	cfs_rq->curr = se;
> +
> +	if (on_rq) {
> +		reweight_eevdf(cfs_rq, se, weight, se->on_rq);
> +		if (first)
> +			set_protect_slice(cfs_rq, se);
> +	}
>   
>   	if (task_on_rq_queued(p)) {
>   		/*
> @@ -15267,17 +15175,8 @@ void unregister_fair_sched_group(struct
>   		struct sched_entity *se = tg_se(tg, cpu);
>   		struct rq *rq = cpu_rq(cpu);
>   
> -		if (se) {
> -			if (se->sched_delayed) {
> -				guard(rq_lock_irqsave)(rq);
> -				if (se->sched_delayed) {
> -					update_rq_clock(rq);
> -					dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> -				}
> -				list_del_leaf_cfs_rq(cfs_rq);
> -			}
> +		if (se)
>   			remove_entity_load_avg(se);
> -		}
>   
>   		/*
>   		 * Only empty task groups can be destroyed; so we can speculatively
> --- a/kernel/sched/pelt.c
> +++ b/kernel/sched/pelt.c
> @@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched
>   	/*
>   	 * running is a subset of runnable (weight) so running can't be set if
>   	 * runnable is clear. But there are some corner cases where the current
> -	 * se has been already dequeued but cfs_rq->curr still points to it.
> +	 * se has been already dequeued but cfs_rq->h_curr still points to it.
>   	 * This means that weight will be 0 but not running for a sched_entity
>   	 * but also for a cfs_rq if the latter becomes idle. As an example,
>   	 * this happens during sched_balance_newidle() which calls
> @@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now
>   int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
>   {
>   	if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se),
> -				cfs_rq->curr == se)) {
> +				cfs_rq->h_curr == se)) {
>   
>   		___update_load_avg(&se->avg, se_weight(se));
>   		cfs_se_util_change(&se->avg);
> @@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, st
>   	if (___update_load_sum(now, &cfs_rq->avg,
>   				scale_load_down(cfs_rq->load.weight),
>   				cfs_rq->h_nr_runnable,
> -				cfs_rq->curr != NULL)) {
> +				cfs_rq->h_curr != NULL)) {
>   
>   		___update_load_avg(&cfs_rq->avg, 1);
>   		trace_pelt_cfs_tp(cfs_rq);
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -530,21 +530,8 @@ struct task_group {
>   
>   };
>   
> -#ifdef CONFIG_GROUP_SCHED_WEIGHT
>   #define ROOT_TASK_GROUP_LOAD	NICE_0_LOAD
>   
> -/*
> - * A weight of 0 or 1 can cause arithmetics problems.
> - * A weight of a cfs_rq is the sum of weights of which entities
> - * are queued on this cfs_rq, so a weight of a entity should not be
> - * too large, so as the shares value of a task group.
> - * (The default weight is 1024 - so there's no practical
> - *  limitation from this.)
> - */
> -#define MIN_SHARES		(1UL <<  1)
> -#define MAX_SHARES		(1UL << 18)
> -#endif
> -
>   typedef int (*tg_visitor)(struct task_group *, void *);
>   
>   extern int walk_tg_tree_from(struct task_group *from,
> @@ -631,6 +618,17 @@ static inline bool cfs_task_bw_constrain
>   
>   #endif /* !CONFIG_CGROUP_SCHED */
>   
> +/*
> + * A weight of 0 or 1 can cause arithmetics problems.
> + * A weight of a cfs_rq is the sum of weights of which entities
> + * are queued on this cfs_rq, so a weight of a entity should not be
> + * too large, so as the shares value of a task group.
> + * (The default weight is 1024 - so there's no practical
> + *  limitation from this.)
> + */
> +#define MIN_SHARES		(1UL <<  1)
> +#define MAX_SHARES		(1UL << 18)
> +
>   extern void unregister_rt_sched_group(struct task_group *tg);
>   extern void free_rt_sched_group(struct task_group *tg);
>   extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent);
> @@ -709,6 +707,7 @@ struct cfs_rq {
>   	/*
>   	 * CFS load tracking
>   	 */
> +	struct sched_entity	*h_curr;
>   	struct sched_avg	avg;
>   #ifndef CONFIG_64BIT
>   	u64			last_update_time_copy;
> @@ -2575,6 +2574,7 @@ extern const u32		sched_prio_to_wmult[40
>   #define ENQUEUE_MIGRATED	0x00040000
>   #define ENQUEUE_INITIAL		0x00080000
>   #define ENQUEUE_RQ_SELECTED	0x00100000
> +#define ENQUEUE_QUEUED		0x00200000
>   
>   #define RETRY_TASK		((void *)-1UL)

Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Chen Yu]

On Wed, Sep 23, 2026 at 07:51:35PM +0530, Aishwarya Rambhadran wrote:
> Hi Peter,
> 
> We have observed some schbench p99 request latency regressions while
> comparing v7.2 against v7.3 cycle on an AWS Graviton3 (m7g.metal)
> using Fastpath (Linux kernel performance benchmarking tool).
> 
> The schbench suite as a whole does not show a significant regression,
> and the results appear to be a mix of improvements and regressions
> depending on the thread configuration. However, a few configurations
> show fairly large p99 request latency regressions. For example,
> comparing v7.2 against v7.3-rc4:
> 
> message-thread:16, worker-thread:16
> v7.2 = 662869 us
> v7.3-rc4 = 969557 us
> change% = -31.63%
> 
> message-thread:64, worker-thread:4
> v7.2 = 730453 us
> v7.3-rc4 = 942251 us
> change% = -22.48%
> 
> message-thread:32, worker-thread:4
> v7.2 = 74251 us
> v7.3-rc4 = 56725 us
> change% = +30.90%
> 
> 
> So this does not appear to be a general latency degradation; some
> configs improve while others regress.
> 
> I also ran a Fastpath automated git bisection for the m 16, t 16 p99
> request latency regression between v7.2 and v7.3-rc1, which identified
> this commit as the first bad commit. The original regression was ~30.9%,
> & the bisection reproducibility check showed ~27.8% regression with 20
> repeats across 2 boot sessions.
> 
> This commit looks relevant since it changes EEVDF runqueue organization
> and is intended, in part, to address latency issues with hierarchical fair/
> cgroup scheduling. I also noticed that there have been fixes on top of this
> change, including 68e37487810a ("sched/fair: Fix flat hierarchy"), which
> fixes vruntime handling and has a Fixes: 85570f10a4c6 tag.
> 
> Given that the overall schbench result is mostly neutral while some p99
> configurations move considerably in both directions, I wanted to check
> whether this could be an expected trade-off from the single-runqueue
> change, or whether the p99 regressions are worth investigating further.
> 
> I can run additional tests or provide the full results/bisection data if
> useful.
> 
>

Would disabling WA_WEIGHT help? I happened to find that this workaround works
for me to restore netperf performance (though it is a throughput score rather
than a latency one), it seems that task_h_load() becomes much
larger(also on sched/task_h_load):
echo NO_WA_WEIGHT > /sys/kernel/debug/sched/features
https://lore.kernel.org/lkml/aoxah90s0bQ4tcUW@three-body/ 

thanks,
Chenyu

Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[K Prateek Nayak]

On 9/23/2026 10:18 PM, Chen Yu wrote:
> On Wed, Sep 23, 2026 at 07:51:35PM +0530, Aishwarya Rambhadran wrote:
>> Hi Peter,
>>
>> We have observed some schbench p99 request latency regressions while
>> comparing v7.2 against v7.3 cycle on an AWS Graviton3 (m7g.metal)
>> using Fastpath (Linux kernel performance benchmarking tool).
>>
>> The schbench suite as a whole does not show a significant regression,
>> and the results appear to be a mix of improvements and regressions
>> depending on the thread configuration. However, a few configurations
>> show fairly large p99 request latency regressions. For example,
>> comparing v7.2 against v7.3-rc4:
>>
>> message-thread:16, worker-thread:16
>> v7.2 = 662869 us
>> v7.3-rc4 = 969557 us
>> change% = -31.63%
>>
>> message-thread:64, worker-thread:4
>> v7.2 = 730453 us
>> v7.3-rc4 = 942251 us
>> change% = -22.48%
>>
>> message-thread:32, worker-thread:4
>> v7.2 = 74251 us
>> v7.3-rc4 = 56725 us
>> change% = +30.90%
>>
>>
>> So this does not appear to be a general latency degradation; some
>> configs improve while others regress.
>>
>> I also ran a Fastpath automated git bisection for the m 16, t 16 p99
>> request latency regression between v7.2 and v7.3-rc1, which identified
>> this commit as the first bad commit. The original regression was ~30.9%,
>> & the bisection reproducibility check showed ~27.8% regression with 20
>> repeats across 2 boot sessions.
>>
>> This commit looks relevant since it changes EEVDF runqueue organization
>> and is intended, in part, to address latency issues with hierarchical fair/
>> cgroup scheduling. I also noticed that there have been fixes on top of this
>> change, including 68e37487810a ("sched/fair: Fix flat hierarchy"), which
>> fixes vruntime handling and has a Fixes: 85570f10a4c6 tag.
>>
>> Given that the overall schbench result is mostly neutral while some p99
>> configurations move considerably in both directions, I wanted to check
>> whether this could be an expected trade-off from the single-runqueue
>> change, or whether the p99 regressions are worth investigating further.
>>
>> I can run additional tests or provide the full results/bisection data if
>> useful.
>>
>>
> 
> Would disabling WA_WEIGHT help? I happened to find that this workaround works
> for me to restore netperf performance (though it is a throughput score rather
> than a latency one), it seems that task_h_load() becomes much
> larger(also on sched/task_h_load):
> echo NO_WA_WEIGHT > /sys/kernel/debug/sched/features
> https://lore.kernel.org/lkml/aoxah90s0bQ4tcUW@three-body/ 

In addition to Chenyu's suggestion, I would also suggest trying out
different cgroup modes via "/sys/kernel/debug/sched/cgroup_mode" if
NO_WA_WEIGHT doesn't resolve your issue.
(P.S. Peter has a branch with some fixes for the task_h_load() at
https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=sched/task_h_load
in case you can try a custom kernel build on your setup)

In my testing, I did see some regression in schbench with the default
"concur" mode for RPS metric but the "smp" mode which used the older
formula for weight calculation seemed to be running on par with the
hierarchical pick.
https://lore.kernel.org/lkml/65980119-a26a-40d6-bec8-dd597833a6e8@amd.com/

-- 
Thanks and Regards,
Prateek

Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Mike Galbraith]

On Thu, 2026-09-24 at 00:48 +0800, Chen Yu wrote:
> 
> Would disabling WA_WEIGHT help? I happened to find that this workaround works
> for me to restore netperf performance (though it is a throughput score rather
> than a latency one), it seems that task_h_load() becomes much
> larger(also on sched/task_h_load):
> echo NO_WA_WEIGHT > /sys/kernel/debug/sched/features
> https://lore.kernel.org/lkml/aoxah90s0bQ4tcUW@three-body/ 

A long while back, I caught WA_WEIGHT causing firefox's event thread
pool to stack on a single CPU.  In that particular case, the victims
(32 of 'em in 8 rq box!) did nothing but go back to sleep, so no real
harm was done. However, I also measured modest real stacking impact, so
gave task_h_load() a floor on GP.

Perhaps weightless burst stacking is a bigger deal for some than it was
here?  That firefox event thread burst showed potential.
 
	-Mike

Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Chen Yu]

Hi Mike, Prateek,

On Thu, Sep 24, 2026 at 10:19:35AM +0200, Mike Galbraith wrote:
> On Thu, 2026-09-24 at 00:48 +0800, Chen Yu wrote:
> > 
> > Would disabling WA_WEIGHT help? I happened to find that this workaround works
> > for me to restore netperf performance (though it is a throughput score rather
> > than a latency one), it seems that task_h_load() becomes much
> > larger(also on sched/task_h_load):
> > echo NO_WA_WEIGHT > /sys/kernel/debug/sched/features
> > https://lore.kernel.org/lkml/aoxah90s0bQ4tcUW@three-body/ 
> 
> A long while back, I caught WA_WEIGHT causing firefox's event thread
> pool to stack on a single CPU.  

I just realized that what I suggested - disabling WA_WEIGHT - does the opposite.
Actually, WA_WEIGHT does help in the netperf scenario. Disabling WA_WEIGHT makes
the performance of both "smp" and "concur" modes drop to the same poor level, which
makes it look like the regression disappears, but actually it does not.

According to my test last week, stacking the waker and wakee on the same CPU brings
a big improvement on my machine, iff the memory bandwidth is saturated. By comparison,
if the memory bandwidth is low, stacking the wakee on top of the waker causes harm.

The machine I used has 2 NUMA nodes; each node has 96 cores, and these 96 cores share
L3, while every 4 cores share L2. The L2 miss rate is much lower if the waker and wakee
are on the same CPU than if the wakee is put on other idle CPUs in the same L2 domain.
Yes, the same L2 - which goes against intuition. How could the L2 miss rate rise even
if the waker and wakee are in the same L2 domain?

I guess this is because stacking the waker and wakee on the same CPU and letting them
run alternately would be the most efficient way to consume the data without evicting
L2 cache lines too much, when the memory bandwidth is saturated.

> In that particular case, the victims
> (32 of 'em in 8 rq box!) did nothing but go back to sleep, so no real
> harm was done. However, I also measured modest real stacking impact, so
> gave task_h_load() a floor on GP.
> 
> Perhaps weightless burst stacking is a bigger deal for some than it was
> here?  That firefox event thread burst showed potential.
>  

It depends on the workload and the memory bandwidth of the system, I suppose. 
If the workload is a producer-consumer type, and the system is under memory bandwidth
pressure, stacking the producer and consumer would bring good data locality - in terms
of L2 locality, per my test.

In summary, the "regression" of flat cgroup task-pick on netperf (or, producer-consumer
type workloads) seems to be the expected consequence. Maybe we need to make WA_WEIGHT
consider more factors here.

I'll collect more data and provide the information later.

thanks,
Chenyu
> 	-Mike

Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Mike Galbraith]

On Mon, 2026-09-28 at 11:12 +0800, Chen Yu wrote:
> 
> According to my test last week, stacking the waker and wakee on the same CPU brings
> a big improvement on my machine, iff the memory bandwidth is saturated. By comparison,
> if the memory bandwidth is low, stacking the wakee on top of the waker causes harm.

Ditto CPU wise.  The impact of even modest waker/wakee concurrency is
considerable.  For much of netperf, the post-wakeup tail becomes a CPU
service latency injection for the wakee when unnecessarily stacked. 
While cache misses sting, those hurt.

As box approaches saturation, stacking of somewhat synchronous stuff
becomes a better bet.  Aiming for having only waker's tail between
wakee and CPU has decent chance of being a winner in a box full of
wider obstacles.

	-Mike

Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Aishwarya Rambhadran]

Hi all,

Thanks for the suggestions. I ran a couple of follow-up tests on the
same AWS Graviton3 (m7g.metal) setup with v7.3-rc4:

1. Disable WA_WEIGHT:
echo NO_WA_WEIGHT > /sys/kernel/debug/sched/features

2. Use the "smp" cgroup mode:
echo smp > /sys/kernel/debug/sched/cgroup_mode

Both tests were run with 20 repeats across 2 boot sessions. Below are
the request latency p99 results for schbench configurations discussed
earlier (usec, smaller is better):

message-thread:16, worker-thread:16
v7.2 = 662869
v7.3-rc4 = 969557
NO_WA_WEIGHT = 958541
smp = 723098

message-thread:64, worker-thread:4
v7.2 = 730453
v7.3-rc4 = 942251
NO_WA_WEIGHT = 740454
smp = 818483

message-thread:32, worker-thread:4
v7.2 = 74251
v7.3-rc4 = 56725
NO_WA_WEIGHT = 59008
smp = 82053

For the m:16 t:16 case that I originally bisected, disabling WA_WEIGHT
does not materially change the regression. Switching to cgroup_mode=smp,
however, recovers most of it.

The m:64 t:4 case behaves differently. Disabling WA_WEIGHT brings the
p99 latency almost back to the v7.2 value, while using smp mode gives
a partial recovery.

There is also an interesting result with m:32 t:4, which was one of the
configurations that improved with v7.3-rc4. That improvement is mostly
preserved with NO_WA_WEIGHT, but disappears with cgroup_mode=smp.

So this does not seem to point to WA_WEIGHT alone as the cause of the
original regression. The results seem consistent with the workload/load-
dependent behavior discussed in this thread, where weight calculation
and resulting placement decisions can help some schbench configurations
while hurting others.

For the original m:16 t:16 regression in particular, the larger recovery
with cgroup_mode=smp also seems consistent with Prateek's observation
that the older weight calculation performs closer to the hierarchical
pick.

Overall, the results suggest that the latency changes are sensitive to
the workload configuration and the weight calculation being used, rather
than being a uniform regression from the single-runqueue change.
It would be interesting to understand whether this variation across
load points is an expected consequence of the new weight calculation,
or whether there is still scope to avoid the larger p99 regressions.

Thanks,
Aishwarya

On 28/09/26 6:30 PM, Mike Galbraith wrote:
> On Mon, 2026-09-28 at 11:12 +0800, Chen Yu wrote:
>> According to my test last week, stacking the waker and wakee on the same CPU brings
>> a big improvement on my machine, iff the memory bandwidth is saturated. By comparison,
>> if the memory bandwidth is low, stacking the wakee on top of the waker causes harm.
> Ditto CPU wise.  The impact of even modest waker/wakee concurrency is
> considerable.  For much of netperf, the post-wakeup tail becomes a CPU
> service latency injection for the wakee when unnecessarily stacked.
> While cache misses sting, those hurt.
>
> As box approaches saturation, stacking of somewhat synchronous stuff
> becomes a better bet.  Aiming for having only waker's tail between
> wakee and CPU has decent chance of being a winner in a box full of
> wider obstacles.
>
> 	-Mike

Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Mike Galbraith]

On Wed, 2026-10-07 at 17:35 +0530, Aishwarya Rambhadran wrote:
> Hi all,
> 
> Thanks for the suggestions. I ran a couple of follow-up tests on the
> same AWS Graviton3 (m7g.metal) setup with v7.3-rc4:
> 
> 1. Disable WA_WEIGHT:
> echo NO_WA_WEIGHT > /sys/kernel/debug/sched/features
> 
> 2. Use the "smp" cgroup mode:
> echo smp > /sys/kernel/debug/sched/cgroup_mode
> 
> Both tests were run with 20 repeats across 2 boot sessions. Below are
> the request latency p99 results for schbench configurations discussed
> earlier (usec, smaller is better):
> 
> message-thread:16, worker-thread:16
> v7.2 = 662869
> v7.3-rc4 = 969557
> NO_WA_WEIGHT = 958541
> smp = 723098
> 
> message-thread:64, worker-thread:4
> v7.2 = 730453
> v7.3-rc4 = 942251
> NO_WA_WEIGHT = 740454
> smp = 818483
> 
> message-thread:32, worker-thread:4
> v7.2 = 74251
> v7.3-rc4 = 56725
> NO_WA_WEIGHT = 59008
> smp = 82053
> 
> For the m:16 t:16 case that I originally bisected, disabling WA_WEIGHT
> does not materially change the regression. Switching to cgroup_mode=smp,
> however, recovers most of it.

I took flat back to 7.2 (to test only that) and did a little mixed load
7.2-virgin vs 7.2+flat testing with defaults. CPU distribution delta
when switching competitor load configuration from 1 group of 8 to 4
groups of 2 vs a fixed 4 client netperf load made it clear that numbers
for anything cgroup vs cgroup will definitely wiggle more than usual.

	-Mike

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Jakub Okoński]

Hi Peter,

On 6/5/26 14:40, Peter Zijlstra wrote:

> Change fair/cgroup to a single runqueue.
>
> [...]
>
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r
>  	if (!sched_feat(WAKEUP_PREEMPTION))
>  		return;
>  
> -	find_matching_se(&se, &pse);
>  	WARN_ON_ONCE(!pse);
>  
>  	cse_is_idle = se_is_idle(se);

I think this changes how cpu.idle affects wakeup preemption.

With the single runqueue, find_matching_se() is gone and se_is_idle()
reduces to task_has_idle_policy(). Wouldn't this mean that SCHED_NORMAL
tasks in a cpu.idle cgroup are no longer treated as idle for
wakeup_preempt_fair()?

I tested my existing 7.2.9 desktop against 7.3-rc6 with the setup below.
cyclictest doesn't reproduce it because it just sleeps and wakes, no CPU
burn and therefore its accrued EEVDF lag keeps the latency low?

cc -O2 -o wakelat wakelat.c
systemd-run --user --collect --unit=spin -p AllowedCPUs=7 -p CPUWeight=idle sh -c 'while :; do :; done'
systemd-run --user --collect --wait --pipe --same-dir -p AllowedCPUs=7 ./wakelat 1000 15 200 |
  mlr --inidx --ifs space --opprint --barred --right label lat \
    then stats1 -a count,p50,p90,p99,max -f lat then rename -r '^lat_,' \
    then put -q 'emit mapsum({"kernel": system("uname -r")}, $*)'
systemctl --user stop spin

wakelat wakes on a fixed 1 ms schedule, records how late each wakeup
was, then busy-loops for 200 micros (wall clock).

+-----------+-------+-----+-----+------+------+
|    kernel | count | p50 | p90 |  p99 |  max |
+-----------+-------+-----+-----+------+------+
|     7.2.9 | 14998 |  11 |  11 |   12 | 1869 |
| 7.3.0-rc6 | 13853 |  11 |  12 | 2029 | 2116 |
+-----------+-------+-----+-----+------+------+

count is out of 15000 periods. rc6 had 1147 wakeups delayed by over 1 ms.


wakelat.c:

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/prctl.h>
#include <time.h>

static long long now_ns(void)
{
	struct timespec t;
	clock_gettime(CLOCK_MONOTONIC, &t);
	return t.tv_sec * 1000000000LL + t.tv_nsec;
}

int main(int argc, char **argv)
{
	if (argc != 4)
		return 2;
	long long interval = atoll(argv[1]) * 1000, dur = atoll(argv[2]) * 1000000000LL;
	long long work = atoll(argv[3]) * 1000;
	long max = dur / interval + 1, n = 0;
	long *lat = malloc(max * sizeof(*lat));

	prctl(PR_SET_TIMERSLACK, 1UL);
	long long start = now_ns(), next = start + interval;
	while (next - start < dur && n < max) {
		struct timespec ts = { next / 1000000000LL, next % 1000000000LL };
		clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &ts, NULL);
		long long now = now_ns();
		lat[n++] = (now - next) / 1000;
		while (now_ns() - now < work)
			;
		do next += interval; while (next <= now);
	}
	for (long i = 0; i < n; i++)
		printf("%ld\n", lat[i]);
	return 0;
}

Regards,
Jakub

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Mike Galbraith]

Greetings!

On Fri, 2026-10-16 at 15:18 +0200, Jakub Okoński wrote:
> Hi Peter,
> 
> On 6/5/26 14:40, Peter Zijlstra wrote:
> 
> > Change fair/cgroup to a single runqueue.
> > 
> > [...]
> > 
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r
> >  	if (!sched_feat(WAKEUP_PREEMPTION))
> >  		return;
> >  
> > -	find_matching_se(&se, &pse);
> >  	WARN_ON_ONCE(!pse);
> >  
> >  	cse_is_idle = se_is_idle(se);
> 
> I think this changes how cpu.idle affects wakeup preemption.

My box claims this to be unrelated to the jitter you're seeing, as it
is present in 7.2.11, with and without the sched-flat set applied.

> With the single runqueue, find_matching_se() is gone and se_is_idle()
> reduces to task_has_idle_policy(). Wouldn't this mean that SCHED_NORMAL
> tasks in a cpu.idle cgroup are no longer treated as idle for
> wakeup_preempt_fair()?
> 
> I tested my existing 7.2.9 desktop against 7.3-rc6 with the setup below.
> cyclictest doesn't reproduce it because it just sleeps and wakes, no CPU
> burn and therefore its accrued EEVDF lag keeps the latency low?
> 
> cc -O2 -o wakelat wakelat.c
> systemd-run --user --collect --unit=spin -p AllowedCPUs=7 -p CPUWeight=idle sh -c 'while :; do :; done'
> systemd-run --user --collect --wait --pipe --same-dir -p AllowedCPUs=7 ./wakelat 1000 15 200 |
>   mlr --inidx --ifs space --opprint --barred --right label lat \
>     then stats1 -a count,p50,p90,p99,max -f lat then rename -r '^lat_,' \
>     then put -q 'emit mapsum({"kernel": system("uname -r")}, $*)'
> systemctl --user stop spin
> 
> wakelat wakes on a fixed 1 ms schedule, records how late each wakeup
> was, then busy-loops for 200 micros (wall clock).
> 
> +-----------+-------+-----+-----+------+------+
> >    kernel | count | p50 | p90 |  p99 |  max |
> +-----------+-------+-----+-----+------+------+
> >     7.2.9 | 14998 |  11 |  11 |   12 | 1869 |
> > 7.3.0-rc6 | 13853 |  11 |  12 | 2029 | 2116 |
> +-----------+-------+-----+-----+------+------+
> 
> count is out of 15000 periods. rc6 had 1147 wakeups delayed by over 1 ms.

Thanks a bunch for the proggy. I've been looking for a clear signal
with various loads, and stuffing wakelat+awk into one of my scripts
provided a perfectly clear signal, which I bisected and confirmed.

Note: below is box pointing it's little silicon finger at the source of
aforementioned jitter, not necessarily at breakage.

Reverting the fingered commit removed the jitter from 7.2.11, and force
reverting 68e37487810a + 333238da9a19 removed it from master as well.

333238da9a193ffc58792995f3e951e4cb87bfd2 is the first 'bad' commit
commit 333238da9a193ffc58792995f3e951e4cb87bfd2
Author: Vincent Guittot <vincent.guittot@linaro.org>
Date:   Fri Aug 14 15:52:41 2026 +0200

    sched: Update time before requeueing delayed entities
    
    In order to compute the right lag, it is required to update time to 'now'.
    Without this, the delayed entity might appear younger than it really is and
    receive less compensation for having waited.
    
    Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
    Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>

 kernel/sched/fair.c | 2 ++
 1 file changed, 2 insertions(+)

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Vincent Guittot]

Le dimanche 18 oct. 2026 à 08:49:29 (+0200), Mike Galbraith a écrit :
> Greetings!
> 
> On Fri, 2026-10-16 at 15:18 +0200, Jakub Okoński wrote:
> > Hi Peter,
> > 
> > On 6/5/26 14:40, Peter Zijlstra wrote:
> > 
> > > Change fair/cgroup to a single runqueue.
> > > 
> > > [...]
> > > 
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r
> > >  	if (!sched_feat(WAKEUP_PREEMPTION))
> > >  		return;
> > >  
> > > -	find_matching_se(&se, &pse);
> > >  	WARN_ON_ONCE(!pse);
> > >  
> > >  	cse_is_idle = se_is_idle(se);
> > 
> > I think this changes how cpu.idle affects wakeup preemption.
> 
> My box claims this to be unrelated to the jitter you're seeing, as it
> is present in 7.2.11, with and without the sched-flat set applied.

Yes, we don't take into account whether the task group is sched_idle

Jakub, Mike,
Could you try the patch below ? It solves the problem on my setup

---
 kernel/sched/fair.c | 80 ++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 68 insertions(+), 12 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 0f2364479f9d..d17e88ad12a5 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -433,6 +433,38 @@ static inline struct sched_entity *parent_entity(const struct sched_entity *se)
 	return se->parent;
 }
 
+static void
+find_matching_se(struct sched_entity **se, struct sched_entity **pse)
+{
+	int se_depth, pse_depth;
+
+	/*
+	 * preemption test can be made between sibling entities who are in the
+	 * same cfs_rq i.e who have a common parent. Walk up the hierarchy of
+	 * both tasks until we find their ancestors who are siblings of common
+	 * parent.
+	 */
+
+	/* First walk up until both entities are at same depth */
+	se_depth = (*se)->depth;
+	pse_depth = (*pse)->depth;
+
+	while (se_depth > pse_depth) {
+		se_depth--;
+		*se = parent_entity(*se);
+	}
+
+	while (pse_depth > se_depth) {
+		pse_depth--;
+		*pse = parent_entity(*pse);
+	}
+
+	while (!is_same_group(*se, *pse)) {
+		*se = parent_entity(*se);
+		*pse = parent_entity(*pse);
+	}
+}
+
 static int tg_is_idle(struct task_group *tg)
 {
 	return tg->idle > 0;
@@ -476,6 +508,11 @@ static inline struct sched_entity *parent_entity(struct sched_entity *se)
 	return NULL;
 }
 
+static inline void
+find_matching_se(struct sched_entity **se, struct sched_entity **pse)
+{
+}
+
 static inline int tg_is_idle(struct task_group *tg)
 {
 	return 0;
@@ -10129,7 +10166,34 @@ preempt_sync(struct rq *rq, int wake_flags,
 }
 
 /*
- * Preempt the current task with a newly woken task if needed:
+ * Preempt an idle entity in favor of a non-idle entity (and don't preempt
+ * in the inverse case).
+ */
+static bool wakeup_preempt_sched_idle(struct sched_entity *cse, struct sched_entity *pse)
+{
+	int cse_is_idle, pse_is_idle;
+
+	find_matching_se(&cse, &pse);
+	WARN_ON_ONCE(!pse);
+
+	cse_is_idle = se_is_idle(cse);
+	pse_is_idle = se_is_idle(pse);
+
+	/*
+	 * Preempt an idle entity in favor of a non-idle entity (and don't preempt
+	 * in the inverse case).
+	 */
+	if (cse_is_idle && !pse_is_idle)
+		return PREEMPT_WAKEUP_RESCHED;
+
+	if (cse_is_idle != pse_is_idle)
+		return PREEMPT_WAKEUP_NONE;
+
+	return PREEMPT_WAKEUP_PICK;
+}
+
+/*
+ * Preempt the current task with a newly woken task if needed.
  */
 static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_flags)
 {
@@ -10174,22 +10238,14 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	if (!sched_feat(WAKEUP_PREEMPTION))
 		return;
 
-	WARN_ON_ONCE(!pse);
-
-	cse_is_idle = se_is_idle(se);
-	pse_is_idle = se_is_idle(pse);
+	preempt_action = wakeup_preempt_sched_idle(se, pse);
 
-	nse = se;
-	/*
-	 * Preempt an idle entity in favor of a non-idle entity (and don't preempt
-	 * in the inverse case).
-	 */
-	if (cse_is_idle && !pse_is_idle)
+	if (preempt_action == PREEMPT_WAKEUP_RESCHED)
 		goto preempt;
 
 	update_curr_fair(rq);
 
-	if (cse_is_idle != pse_is_idle)
+	if (preempt_action == PREEMPT_WAKEUP_NONE)
 		goto update;
 
 	/*
-- 
2.53.0


> 
> > With the single runqueue, find_matching_se() is gone and se_is_idle()
> > reduces to task_has_idle_policy(). Wouldn't this mean that SCHED_NORMAL
> > tasks in a cpu.idle cgroup are no longer treated as idle for
> > wakeup_preempt_fair()?
> >
> > 
> > I tested my existing 7.2.9 desktop against 7.3-rc6 with the setup below.
> > cyclictest doesn't reproduce it because it just sleeps and wakes, no CPU
> > burn and therefore its accrued EEVDF lag keeps the latency low?
> > 
> > cc -O2 -o wakelat wakelat.c
> > systemd-run --user --collect --unit=spin -p AllowedCPUs=7 -p CPUWeight=idle sh -c 'while :; do :; done'
> > systemd-run --user --collect --wait --pipe --same-dir -p AllowedCPUs=7 ./wakelat 1000 15 200 |
> >   mlr --inidx --ifs space --opprint --barred --right label lat \
> >     then stats1 -a count,p50,p90,p99,max -f lat then rename -r '^lat_,' \
> >     then put -q 'emit mapsum({"kernel": system("uname -r")}, $*)'
> > systemctl --user stop spin
> > 
> > wakelat wakes on a fixed 1 ms schedule, records how late each wakeup
> > was, then busy-loops for 200 micros (wall clock).
> > 
> > +-----------+-------+-----+-----+------+------+
> > >    kernel | count | p50 | p90 |  p99 |  max |
> > +-----------+-------+-----+-----+------+------+
> > >     7.2.9 | 14998 |  11 |  11 |   12 | 1869 |
> > > 7.3.0-rc6 | 13853 |  11 |  12 | 2029 | 2116 |
> > +-----------+-------+-----+-----+------+------+
> > 
> > count is out of 15000 periods. rc6 had 1147 wakeups delayed by over 1 ms.
> 
> Thanks a bunch for the proggy. I've been looking for a clear signal
> with various loads, and stuffing wakelat+awk into one of my scripts
> provided a perfectly clear signal, which I bisected and confirmed.
> 
> Note: below is box pointing it's little silicon finger at the source of
> aforementioned jitter, not necessarily at breakage.
> 
> Reverting the fingered commit removed the jitter from 7.2.11, and force
> reverting 68e37487810a + 333238da9a19 removed it from master as well.
> 
> 333238da9a193ffc58792995f3e951e4cb87bfd2 is the first 'bad' commit
> commit 333238da9a193ffc58792995f3e951e4cb87bfd2
> Author: Vincent Guittot <vincent.guittot@linaro.org>
> Date:   Fri Aug 14 15:52:41 2026 +0200
> 
>     sched: Update time before requeueing delayed entities
>     
>     In order to compute the right lag, it is required to update time to 'now'.
>     Without this, the delayed entity might appear younger than it really is and
>     receive less compensation for having waited.
>     
>     Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
>     Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> 
>  kernel/sched/fair.c | 2 ++
>  1 file changed, 2 insertions(+)
> 

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Jakub Okoński]

Hi Vincent,

Your patch fixes my cpu.idle test case.

On 10/19/26 19:01, Vincent Guittot wrote:
>  /*
> - * Preempt the current task with a newly woken task if needed:
> + * Preempt an idle entity in favor of a non-idle entity (and don't preempt
> + * in the inverse case).
> + */
> +static bool wakeup_preempt_sched_idle(struct sched_entity *cse, struct sched_entity *pse)

This should return enum preempt_wakeup_action. But with that change, on
top of 7.3.0, it still regresses from 7.2.9 if you test with a nice 19
spinner in the same cgroup. I think it's incomplete.

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Vincent Guittot]

On Mon, 19 Oct 2026 at 19:36, Jakub Okoński <jakub@okonski.org> wrote:
>
> Hi Vincent,
>
> Your patch fixes my cpu.idle test case.
>
> On 10/19/26 19:01, Vincent Guittot wrote:
> >  /*
> > - * Preempt the current task with a newly woken task if needed:
> > + * Preempt an idle entity in favor of a non-idle entity (and don't preempt
> > + * in the inverse case).
> > + */
> > +static bool wakeup_preempt_sched_idle(struct sched_entity *cse, struct sched_entity *pse)
>
> This should return enum preempt_wakeup_action. But with that change, on

Argh! Yes I moved from a binary to a ternary value but forgot to
update the prototype

> top of 7.3.0, it still regresses from 7.2.9 if you test with a nice 19
> spinner in the same cgroup. I think it's incomplete.

This patch only aims to fix the sched_idle task group case. I will
have a closer look at the nice 19 case. Have you checked the different
cgroup_mode with nice 19 ?

Vincent

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Jakub Okoński]

On 10/19/26 20:04, Vincent Guittot wrote:
> On Mon, 19 Oct 2026 at 19:36, Jakub Okoński <jakub@okonski.org> wrote:
>>
>> top of 7.3.0, it still regresses from 7.2.9 if you test with a nice 19
>> spinner in the same cgroup. I think it's incomplete.
> 
> This patch only aims to fix the sched_idle task group case. I will
> have a closer look at the nice 19 case. Have you checked the different
> cgroup_mode with nice 19 ?
> 
> Vincent

Same result in my nice 19 test for all of up/smp/concur/max/tasks.

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Vincent Guittot]

On Mon, 19 Oct 2026 at 20:24, Jakub Okoński <jakub@okonski.org> wrote:
>
> On 10/19/26 20:04, Vincent Guittot wrote:
> > On Mon, 19 Oct 2026 at 19:36, Jakub Okoński <jakub@okonski.org> wrote:
> >>
> >> top of 7.3.0, it still regresses from 7.2.9 if you test with a nice 19
> >> spinner in the same cgroup. I think it's incomplete.
> >
> > This patch only aims to fix the sched_idle task group case. I will
> > have a closer look at the nice 19 case. Have you checked the different
> > cgroup_mode with nice 19 ?
> >
> > Vincent
>
> Same result in my nice 19 test for all of up/smp/concur/max/tasks.

Okay
Thanks for the tests results

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Vincent Guittot]

On Mon, 19 Oct 2026 at 20:26, Vincent Guittot
<vincent.guittot@linaro.org> wrote:
>
> On Mon, 19 Oct 2026 at 20:24, Jakub Okoński <jakub@okonski.org> wrote:
> >
> > On 10/19/26 20:04, Vincent Guittot wrote:
> > > On Mon, 19 Oct 2026 at 19:36, Jakub Okoński <jakub@okonski.org> wrote:
> > >>
> > >> top of 7.3.0, it still regresses from 7.2.9 if you test with a nice 19
> > >> spinner in the same cgroup. I think it's incomplete.
> > >
> > > This patch only aims to fix the sched_idle task group case. I will
> > > have a closer look at the nice 19 case. Have you checked the different
> > > cgroup_mode with nice 19 ?
> > >
> > > Vincent
> >
> > Same result in my nice 19 test for all of up/smp/concur/max/tasks.
>
> Okay
> Thanks for the tests results

I ran more tests on 7.2.9 and 7.3. I confirm that there is a
difference between 7.2.9 and 7.3. With flat hierarchy, we change the
weight used to compute vruntime, deadline and vlag. The positive vlag
elapses differently with the flat hierarchy and slice protection
becomes more active. I  used rt-app for the short task that wakes up
periodically in my test

    - v7.2.9
Metric               | Value (us)
-----------------------------------
Total Samples        | 16983
Total outlier        | 17
Average              | 54
Median (P50)         | 53
90th Percentile      | 53
99th Percentile      | 54
99.9th Percentile    | 1582
Maximum              | 2248
-----------------------------------

    - v7.3
Metric               | Value (us)
-----------------------------------
Total Samples        | 16368
Total outlier        | 632
Average              | 145
Median (P50)         | 53
90th Percentile      | 53
99th Percentile      | 2039
99.9th Percentile    | 2132
Maximum              | 3367
-----------------------------------

That being said, there is a trend to make the lag to elapse quicker
and an app can't expect to maintain its lag forever while sleeping.
Instead, the app should set its slice to notify the scheduler that it
would like to run 1st as long as fairness is not broken. If I set a
slice of 1ms for the short task I have :

    - v7.2.9
Metric               | Value (us)
-----------------------------------
Total Samples        | 17000
Total outlier        | 16
Average              | 52
Median (P50)         | 53
90th Percentile      | 53
99th Percentile      | 55
99.9th Percentile    | 55
Maximum              | 56
-----------------------------------

    - v7.3
Metric               | Value (us)
-----------------------------------
Total Samples        | 17000
Total outlier        | 11
Average              | 52
Median (P50)         | 53
90th Percentile      | 53
99th Percentile      | 54
99.9th Percentile    | 55
Maximum              | 57
-----------------------------------

Which is the correct way to ask scheduler to make a task to run 1st

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Jakub Okoński]

On 10/21/26 13:55, Vincent Guittot wrote:
> Which is the correct way to ask scheduler to make a task to run 1st

I totally get that and will try to make the changes in my workloads,
starting with https://invent.kde.org/plasma/kwin/-/merge_requests/10192

That said, I hope both the cpu.idle and nice 19 issues get resolved and
I'll be watching those conversations. It's too disruptive otherwise!

Regards,
Jakub

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Chengfeng Lin]

Hi Vincent, Mike, Peter,

Sorry, my earlier report was sent without the reply headers. It contains
additional will-it-scale pipe-handoff results for 333238da9a19:

https://lists.openwall.net/linux-kernel/2026/10/30/1506

I'm linking it here to keep it connected to Mike's report in this thread.
The measurements and reproduction instructions are unchanged.

Thanks,
Chengfeng

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Mike Galbraith]

On Fri, 2026-10-16 at 15:18 +0200, Jakub Okoński wrote:
> Hi Peter,
> 
> On 6/5/26 14:40, Peter Zijlstra wrote:
> 
> > Change fair/cgroup to a single runqueue.
> > 
> > [...]
> > 
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r
> >  	if (!sched_feat(WAKEUP_PREEMPTION))
> >  		return;
> >  
> > -	find_matching_se(&se, &pse);
> >  	WARN_ON_ONCE(!pse);
> >  
> >  	cse_is_idle = se_is_idle(se);
> 
> I think this changes how cpu.idle affects wakeup preemption.
> 
> With the single runqueue, find_matching_se() is gone and se_is_idle()
> reduces to task_has_idle_policy(). Wouldn't this mean that SCHED_NORMAL
> tasks in a cpu.idle cgroup are no longer treated as idle for
> wakeup_preempt_fair()?

Am I perhaps testing wrong?

What I do to produce the perf sched data below is stuff pinned hogs in
one cgroup, pinned wakelat in another, diddle the hog populated cgroup
cpu.idle... and watch wakelat blast straight through the hog between it
and the CPU just fine, regardless of cpu.idle state.

WRT the data, ignore perf's max below, take awk processed wakelat
output as gospel.  I can make the perf numbers match nearly perfects by
running all players in the background and terminating via a time event
but didn't bother here.  The important bits are accumulated runtime,
ctx switch count, and avg delay for the bottom two lines of each
report. Ignore report format, that's a local hacklet, and also ignore
the wakelat runtime delta between tested kernels when cpu.idle=0 in the
hog group, that's a DELAY_DEQUEUE issue I've verified to be the source
of the jitter I can reproduce.

BEGIN 7.3.0.g14c89774-master

7.3.0.g14c89774-master cgroupA-hogs-(cpu.idle=0)-vs-groupB-wakelat
wakeup latency >1ms: 4568  max: 2.4ms  stddev: 554.882
  ----------------------------------------------------------------------------------------------------------
  Task                    |    Runtime     |  Count   |    Avg delay    |    Max delay    |    Sum delay    |
  ----------------------------------------------------------------------------------------------------------
  cpuhog:1267             |      60.048 s  |       40 |        6.186 us |       43.523 us |      247.445 us |
  cpuhog:1269             |      60.047 s  |      102 |        8.867 us |      144.492 us |      904.512 us |
  cpuhog:1268             |      60.047 s  |      126 |       10.899 us |      774.840 us |        1.373 ms |
  cpuhog:1266             |      60.045 s  |       58 |       19.230 us |      531.945 us |        1.115 ms |
  cpuhog:1270             |      60.044 s  |       35 |        4.708 us |       32.297 us |      164.798 us |
  cpuhog:1264             |      60.033 s  |      492 |       27.200 us |        2.383 ms |       13.383 ms |
  cpuhog:1265             |      60.030 s  |      783 |       23.775 us |       83.871 us |       18.617 ms |
  cpuhog:1271             |      49.519 s  |    51947 |      202.691 us |        5.607 ms |       10.529 s  |
  wakelat:1277            |      10.512 s  |    52011 |      171.849 us |       10.416 ms |        8.938 s  |
  ----------------------------------------------------------------------------------------------------------
  TOTAL:                  |     480.376 s  |   107581 |                 |       10.416 ms |       19.627 s  |
  ----------------------------------------------------------------------------------------------------------

7.3.0.g14c89774-master cgroupA-hogs-(cpu.idle=1)-vs-groupB-wakelat
wakeup latency >1ms: 0  max: 0.0ms  stddev: 0.500
  ----------------------------------------------------------------------------------------------------------
  Task                    |    Runtime     |  Count   |    Avg delay    |    Max delay    |    Sum delay    |
  ----------------------------------------------------------------------------------------------------------
  cpuhog:1297             |      60.026 s  |       43 |        9.577 us |       83.461 us |      411.831 us |
  cpuhog:1298             |      60.024 s  |       27 |        4.094 us |       31.306 us |      110.547 us |
  cpuhog:1294             |      60.023 s  |       60 |       10.620 us |       52.234 us |      637.217 us |
  cpuhog:1296             |      60.022 s  |      131 |       15.621 us |      908.988 us |        2.046 ms |
  cpuhog:1293             |      60.019 s  |      224 |        6.861 us |      593.440 us |        1.537 ms |
  cpuhog:1292             |      60.006 s  |      690 |       28.075 us |      286.517 us |       19.372 ms |
  cpuhog:1295             |      60.005 s  |       26 |        4.119 us |       32.429 us |      107.111 us |
  cpuhog:1299             |      47.903 s  |    60216 |      201.386 us |        5.270 ms |       12.127 s  |
  wakelat:1305            |      12.124 s  |    60023 |        1.963 us |        2.156 ms |      117.862 ms |
  ----------------------------------------------------------------------------------------------------------
  TOTAL:                  |     480.196 s  |   122918 |                 |        5.270 ms |       12.394 s  |
  ----------------------------------------------------------------------------------------------------------

BEGIN 7.2.11-ag-flat+hack
Note: this tree has virgin flat applied, and the bisected commit reverted.

7.2.11-ag-flat+hack cgroupA-hogs-(cpu.idle=0)-vs-groupB-wakelat
wakeup latency >1ms: 0  max: 0.0ms  stddev: 0.438
  ----------------------------------------------------------------------------------------------------------
  Task                    |    Runtime     |  Count   |    Avg delay    |    Max delay    |    Sum delay    |
  ----------------------------------------------------------------------------------------------------------
  cpuhog:1272             |      60.040 s  |       44 |        6.139 us |       40.163 us |      270.136 us |
  cpuhog:1275             |      60.039 s  |      139 |        3.768 us |      185.461 us |      523.864 us |
  cpuhog:1276             |      60.039 s  |      140 |        5.432 us |       28.697 us |      760.493 us |
  cpuhog:1274             |      60.039 s  |      104 |       12.441 us |      814.146 us |        1.294 ms |
  cpuhog:1270             |      60.037 s  |      271 |       13.571 us |      637.071 us |        3.678 ms |
  cpuhog:1273             |      60.036 s  |       51 |       17.658 us |      439.987 us |      900.594 us |
  cpuhog:1271             |      60.018 s  |      636 |       28.960 us |      362.859 us |       18.419 ms |
  cpuhog:1277             |      47.902 s  |    60081 |      202.120 us |        9.812 ms |       12.144 s  |
  wakelat:1283            |      12.122 s  |    60016 |        1.905 us |        2.801 ms |      114.368 ms |
  ----------------------------------------------------------------------------------------------------------
  TOTAL:                  |     480.316 s  |   123011 |                 |        9.812 ms |       12.352 s  |
  ----------------------------------------------------------------------------------------------------------

7.2.11-ag-flat+hack cgroupA-hogs-(cpu.idle=1)-vs-groupB-wakelat
wakeup latency >1ms: 0  max: 0.0ms  stddev: 0.455
  ----------------------------------------------------------------------------------------------------------
  Task                    |    Runtime     |  Count   |    Avg delay    |    Max delay    |    Sum delay    |
  ----------------------------------------------------------------------------------------------------------
  cpuhog:1298             |      60.021 s  |      307 |        8.860 us |      272.423 us |        2.720 ms |
  cpuhog:1301             |      60.020 s  |       20 |        3.661 us |       30.994 us |       73.236 us |
  cpuhog:1303             |      60.020 s  |      135 |        2.070 us |       28.715 us |      279.571 us |
  cpuhog:1302             |      60.019 s  |       80 |       14.390 us |      782.611 us |        1.151 ms |
  cpuhog:1300             |      60.017 s  |       46 |       42.518 us |        1.656 ms |        1.956 ms |
  cpuhog:1299             |      60.005 s  |       22 |        6.165 us |       62.913 us |      135.646 us |
  cpuhog:1304             |      60.001 s  |      666 |       27.965 us |       92.954 us |       18.625 ms |
  cpuhog:1305             |      47.900 s  |    60084 |      201.849 us |        4.224 ms |       12.128 s  |
  wakelat:1311            |      12.124 s  |    60009 |        1.977 us |        1.440 ms |      118.685 ms |
  ----------------------------------------------------------------------------------------------------------
  TOTAL:                  |     480.165 s  |   122809 |                 |        4.224 ms |       12.364 s  |
  ----------------------------------------------------------------------------------------------------------

Re: [PATCH v3 0/7] sched: Flatten the pick

[K Prateek Nayak]

Hello Peter,

On 6/5/2026 6:10 PM, Peter Zijlstra wrote:
> Can also be had:
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat

Here are early performance numbers at commit 7eb85c33dd20 ("sched/eevdf:
Move to a single runqueue").

tl;dr

  Apart form a regression in the tbench and schbench at super high
  utilization, all numbers for concur mode looks good so 7.2-rc1 target
  should be good.

  P.S. I won't be able to look into this until Thursday; Sorry in
  advance.

Note: I haven't gotten around to analyzing anything in depth; These are
just early numbers on microbenchmarks and DeathStarBench. Please take
them with a grain of salt.

tip was at commit f666241e6bd5 ("sched/fair: Unify cfs_rq throttling via
account_cfs_rq_runtime()") and each label are the individual cgroup
mode togged on at queue:sched/flat:

Machine:

o 4th Generation EPYC system (Zen4c)
o 2 x 128C/256T (32LLCs)
o Boost enabled
o C2 disabled; MWAIT based C1 and POLL remained enabled

Benchmark numbers:


  ==================================================================
  Test          : hackbench
  Units         : Normalized time in seconds
  Interpretation: Lower is better
  Statistic     : AMean
  ==================================================================
  Case:           tip[pct imp](CV)            up[pct imp](CV)           smp[pct imp](CV)     max[pct imp](CV)        concur[pct imp](CV)      tasks[pct imp](CV)
   1-groups     1.00 [ -0.00](11.18)     1.01 [ -0.60](12.84)     1.12 [-12.08](12.50)     1.16 [-15.71]( 9.46)     1.07 [ -7.25](11.46)     0.99 [  1.21](15.44)
   2-groups     1.00 [ -0.00]( 4.73)     1.01 [ -1.00]( 9.29)     2.88 [-188.25](63.11)    0.97 [  2.75](11.62)     0.95 [  5.00](12.47)     0.96 [  4.00](11.72)
   4-groups     1.00 [ -0.00]( 2.90)     0.99 [  0.96]( 2.48)     0.99 [  0.72]( 2.38)     1.00 [  0.48]( 1.78)     0.99 [  0.96]( 6.61)     1.00 [ -0.00]( 5.23)
   8-groups     1.00 [ -0.00]( 1.82)     1.03 [ -2.51]( 2.81)     0.99 [  0.91]( 3.33)     0.99 [  0.91]( 2.40)     1.01 [ -0.68]( 2.44)     1.02 [ -1.60]( 2.45)
  16-groups     1.00 [ -0.00]( 3.05)     1.03 [ -2.96]( 1.97)     1.23 [-22.82](22.67)     1.01 [ -1.31]( 2.03)     0.99 [  0.99]( 2.48)     1.01 [ -0.66]( 2.76)

  Note: For smp variant runs, I think there was some system noise form
  an unrelated job that stated by mistake. I have to go back and
  rerun to confirm if the regression holds.
  
  ==================================================================
  Test          : tbench
  Units         : Normalized throughput
  Interpretation: Higher is better
  Statistic     : AMean
  ==================================================================
  Clients:    tip[pct imp](CV)          up[pct imp](CV)         smp[pct imp](CV)         max[pct imp](CV)       concur[pct imp](CV)       tasks[pct imp](CV)
      1     1.00 [  0.00]( 0.23)     1.02 [  1.53]( 0.10)     1.02 [  1.80]( 0.96)     1.00 [  0.19]( 0.17)     0.99 [ -0.58]( 0.18)     1.01 [  1.13]( 0.18)
      2     1.00 [  0.00]( 0.11)     1.01 [  1.13]( 0.04)     1.01 [  1.26]( 0.08)     1.00 [ -0.12]( 0.21)     0.99 [ -1.00]( 0.29)     1.01 [  0.51]( 0.02)
      4     1.00 [  0.00]( 0.11)     1.01 [  1.30]( 0.16)     1.02 [  2.26]( 0.48)     1.00 [  0.06]( 0.37)     0.99 [ -0.74]( 0.52)     1.01 [  1.35]( 0.35)
      8     1.00 [  0.00]( 0.24)     1.01 [  1.19]( 0.81)     1.02 [  2.17]( 0.52)     1.00 [  0.45]( 0.36)     0.99 [ -0.80]( 0.05)     1.01 [  0.92]( 0.39)
     16     1.00 [  0.00]( 0.15)     1.01 [  0.87]( 0.70)     1.03 [  2.56]( 0.18)     1.00 [ -0.41]( 0.46)     0.99 [ -0.99]( 0.36)     1.01 [  0.55]( 1.02)
     32     1.00 [  0.00]( 1.02)     1.02 [  2.42]( 0.36)     1.04 [  3.76]( 0.94)     1.01 [  1.20]( 0.19)     1.00 [ -0.10]( 0.44)     1.02 [  1.61]( 0.27)
     64     1.00 [  0.00]( 0.36)     1.02 [  1.92]( 1.71)     1.03 [  2.59]( 1.15)     0.99 [ -0.51]( 0.88)     1.01 [  1.19]( 0.29)     1.02 [  2.42]( 0.57)
    128     1.00 [  0.00]( 0.45)     1.01 [  1.11]( 1.37)     1.05 [  4.64]( 1.05)     1.01 [  0.98]( 2.47)     1.01 [  0.84]( 1.97)     1.03 [  2.56]( 1.22)
    256     1.00 [  0.00]( 0.06)     1.02 [  2.23]( 1.11)     1.02 [  2.17]( 0.69)     1.03 [  2.87]( 0.46)     1.03 [  2.57]( 0.41)     1.03 [  2.99]( 0.84)
    512     1.00 [  0.00]( 1.50)     0.92 [ -7.62]( 6.42)     1.02 [  1.94]( 2.12)     1.01 [  0.74]( 6.70)     0.94 [ -6.09]( 5.19)     0.98 [ -2.40]( 3.00)
   1024     1.00 [  0.00]( 0.07)     0.98 [ -1.51]( 0.30)     1.02 [  1.66]( 0.13)     0.97 [ -2.97]( 0.76)     0.94 [ -6.33]( 0.26)     0.95 [ -4.63]( 0.47)
   2048     1.00 [  0.00]( 0.25)     0.98 [ -1.57]( 0.59)     1.02 [  1.81]( 0.20)     0.98 [ -1.94]( 0.38)     0.94 [ -5.54]( 0.17)     0.95 [ -4.56]( 0.77)
  
  
  ==================================================================
  Test          : stream-10
  Units         : Normalized Bandwidth, MB/s
  Interpretation: Higher is better
  Statistic     : HMean
  ==================================================================
  Test:       tip[pct imp](CV)          up[pct imp](CV)         smp[pct imp](CV)         max[pct imp](CV)        concur[pct imp](CV)     tasks[pct imp](CV)
   Copy     1.00 [  0.00]( 0.51)     0.92 [ -7.61](12.64)     0.99 [ -0.79]( 0.30)     0.85 [-15.33](21.73)     0.99 [ -0.61]( 0.35)     0.99 [ -0.90]( 0.38)
  Scale     1.00 [  0.00]( 0.35)     0.90 [ -9.96](14.86)     0.99 [ -1.45]( 0.76)     0.85 [-14.73](21.14)     0.99 [ -1.12]( 0.86)     0.99 [ -1.13]( 0.71)
    Add     1.00 [  0.00]( 0.21)     0.93 [ -6.95](10.26)     0.98 [ -1.51]( 0.80)     0.95 [ -4.91]( 9.84)     0.99 [ -0.98]( 0.69)     0.99 [ -1.37]( 0.67)
  Triad     1.00 [  0.00]( 0.24)     0.89 [-10.61](15.55)     0.99 [ -1.46]( 0.72)     0.95 [ -4.87]( 9.50)     0.99 [ -0.90]( 0.62)     0.99 [ -1.44]( 0.61)
  
  
  ==================================================================
  Test          : stream-100
  Units         : Normalized Bandwidth, MB/s
  Interpretation: Higher is better
  Statistic     : HMean
  ==================================================================
  Test:       tip[pct imp](CV)          up[pct imp](CV)         smp[pct imp](CV)         max[pct imp](CV)        concur[pct imp](CV)     tasks[pct imp](CV)
   Copy     1.00 [  0.00]( 1.52)     1.00 [ -0.16]( 0.56)     0.97 [ -2.54]( 2.30)     1.00 [  0.09]( 0.18)     1.00 [  0.17]( 0.44)     0.99 [ -0.74]( 1.49)
  Scale     1.00 [  0.00]( 1.43)     0.99 [ -0.63]( 0.55)     0.97 [ -2.72]( 2.44)     1.00 [  0.20]( 0.16)     0.99 [ -0.67]( 0.43)     0.99 [ -1.39]( 1.45)
    Add     1.00 [  0.00]( 1.06)     0.99 [ -1.27]( 0.42)     0.97 [ -3.08]( 1.95)     1.00 [ -0.13]( 0.19)     0.99 [ -1.19]( 0.34)     0.98 [ -1.76]( 1.07)
  Triad     1.00 [  0.00]( 1.10)     0.98 [ -1.55]( 0.41)     0.97 [ -3.40]( 1.95)     1.00 [ -0.42]( 0.16)     0.99 [ -1.49]( 0.35)     0.98 [ -2.08]( 1.11)
  
  
  ==================================================================
  Test          : netperf
  Units         : Normalized Througput
  Interpretation: Higher is better
  Statistic     : AMean
  ==================================================================
  Clients:           tip[pct imp](CV)          up[pct imp](CV)         smp[pct imp](CV)         max[pct imp](CV)       concur[pct imp](CV)       tasks[pct imp](CV)
     1-clients     1.00 [  0.00]( 0.15)     1.02 [  2.24]( 0.12)     1.02 [  1.50]( 0.22)     1.07 [  6.86]( 2.15)     1.00 [  0.27]( 0.45)     1.02 [  1.86]( 0.15)
     2-clients     1.00 [  0.00]( 0.37)     1.03 [  2.50]( 0.80)     1.03 [  3.11]( 0.78)     1.07 [  7.15]( 0.96)     1.00 [ -0.43]( 0.54)     1.02 [  1.67]( 0.69)
     4-clients     1.00 [  0.00]( 0.26)     1.02 [  2.26]( 0.33)     1.02 [  2.02]( 0.37)     1.07 [  7.37]( 0.78)     1.00 [ -0.24]( 0.42)     1.02 [  1.84]( 0.22)
     8-clients     1.00 [  0.00]( 0.21)     1.02 [  2.47]( 0.55)     1.02 [  2.13]( 0.48)     1.08 [  7.60]( 0.57)     1.00 [ -0.15]( 0.37)     1.02 [  1.87]( 0.29)
    16-clients     1.00 [  0.00]( 0.23)     1.02 [  2.07]( 0.69)     1.02 [  1.87]( 0.42)     1.07 [  7.38]( 0.50)     0.99 [ -0.55]( 0.48)     1.02 [  1.78]( 0.25)
    32-clients     1.00 [  0.00]( 0.47)     1.02 [  2.14]( 0.63)     1.02 [  1.81]( 0.75)     1.07 [  7.44]( 0.94)     1.00 [ -0.38]( 0.53)     1.02 [  1.76]( 0.43)
    64-clients     1.00 [  0.00]( 0.91)     1.02 [  2.00]( 0.81)     1.02 [  1.74]( 0.96)     1.07 [  7.12]( 1.06)     1.00 [ -0.20]( 0.68)     1.02 [  1.63]( 0.76)
   128-clients     1.00 [  0.00]( 1.19)     1.01 [  1.36]( 1.28)     1.01 [  1.34]( 1.18)     1.06 [  6.37]( 1.47)     1.00 [ -0.49]( 1.15)     1.01 [  1.06]( 1.09)
   256-clients     1.00 [  0.00]( 1.00)     1.02 [  1.70]( 1.15)     1.02 [  1.64]( 1.18)     1.07 [  7.17]( 1.87)     1.00 [ -0.31]( 1.19)     1.01 [  1.29]( 1.12)
   512-clients     1.00 [  0.00]( 5.16)     1.00 [  0.02]( 6.48)     0.99 [ -0.60]( 4.31)     1.04 [  4.08]( 3.83)     1.00 [  0.02]( 2.52)     1.00 [  0.48]( 2.86)
   768-clients     1.00 [  0.00](34.61)     1.03 [  2.84](62.48)     1.00 [  0.26](30.91)     0.98 [ -2.12](13.41)     0.94 [ -6.34](11.15)     0.95 [ -4.82](10.61)
  1024-clients     1.00 [  0.00](41.78)     1.04 [  3.95](76.45)     1.01 [  0.99](40.23)     0.97 [ -2.58](11.79)     0.95 [ -5.36](11.21)     0.96 [ -4.01](12.98)
  
  
  ==================================================================
  Test          : schbench
  Units         : Normalized 99th percentile latency in us
  Interpretation: Lower is better
  Statistic     : Median
  ==================================================================
  #workers:  tip[pct imp](CV)          up[pct imp](CV)         smp[pct imp](CV)         max[pct imp](CV)        concur[pct imp](CV)     tasks[pct imp](CV)
     1     1.00 [ -0.00]( 5.88)     0.94 [  5.88](19.25)     0.88 [ 11.76]( 7.37)     0.88 [ 11.76](34.02)     1.06 [ -5.88](11.11)     0.94 [  5.88]( 3.69)
     2     1.00 [ -0.00](36.56)     0.97 [  3.03]( 9.52)     0.97 [  3.03]( 4.82)     1.00 [ -0.00](26.93)     0.97 [  3.03](14.53)     0.82 [ 18.18](13.87)
     4     1.00 [ -0.00]( 9.35)     0.97 [  3.12](18.25)     0.94 [  6.25]( 9.12)     1.00 [ -0.00](20.82)     1.00 [ -0.00]( 4.82)     0.88 [ 12.50](15.93)
     8     1.00 [ -0.00](23.38)     0.94 [  6.45](23.26)     1.26 [-25.81](14.43)     1.26 [-25.81](26.13)     1.06 [ -6.45]( 4.68)     0.97 [  3.23](10.00)
    16     1.00 [ -0.00]( 2.71)     1.04 [ -3.57]( 2.62)     1.00 [ -0.00]( 2.04)     0.98 [  1.79]( 2.76)     1.04 [ -3.57]( 0.00)     1.02 [ -1.79]( 2.70)
    32     1.00 [ -0.00]( 0.72)     1.00 [ -0.00]( 0.72)     0.96 [  3.75]( 1.30)     1.01 [ -1.25]( 0.71)     1.01 [ -1.25]( 1.88)     1.02 [ -2.50]( 1.86)
    64     1.00 [ -0.00]( 1.52)     0.96 [  4.41]( 1.18)     0.97 [  2.94]( 4.49)     0.94 [  5.88]( 1.19)     0.95 [  5.15]( 0.45)     0.96 [  4.41]( 0.45)
   128     1.00 [ -0.00]( 3.24)     0.96 [  3.80]( 0.91)     0.96 [  3.80]( 1.53)     0.95 [  4.64]( 0.26)     0.96 [  3.80]( 0.44)     0.97 [  2.53]( 1.32)
   256     1.00 [ -0.00]( 0.90)     0.92 [  7.60]( 0.75)     0.95 [  5.40]( 0.61)     0.92 [  7.80]( 0.55)     0.93 [  6.60]( 6.23)     1.00 [ -0.00]( 0.61)
   512     1.00 [ -0.00]( 1.76)     1.10 [-10.36]( 0.48)     1.04 [ -4.15]( 1.46)     0.97 [  3.45]( 7.07)     1.07 [ -6.56]( 2.54)     0.93 [  6.56]( 2.45)
   768     1.00 [ -0.00]( 2.79)     0.95 [  5.10]( 6.65)     1.71 [-70.87]( 4.67)     0.80 [ 20.13]( 1.81)     0.78 [ 21.80]( 1.19)     0.78 [ 22.36]( 1.46)
  1024     1.00 [ -0.00]( 0.86)     0.35 [ 64.79](18.07)     1.16 [-15.56]( 1.17)     1.00 [  0.32]( 3.65)     0.95 [  4.54]( 3.10)     0.90 [  9.72]( 2.33)
  
  
  ==================================================================
  Test          : new-schbench-requests-per-second
  Units         : Normalized Requests per second
  Interpretation: Higher is better
  Statistic     : Median
  ==================================================================
  #workers:  tip[pct imp](CV)          up[pct imp](CV)         smp[pct imp](CV)         max[pct imp](CV)       concur[pct imp](CV)       tasks[pct imp](CV)
     1     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)     1.00 [ -0.30]( 0.00)     1.00 [ -0.30]( 0.31)     0.99 [ -0.59]( 0.15)     1.00 [  0.00]( 0.00)
     2     1.00 [  0.00]( 0.00)     1.00 [  0.30]( 0.00)     1.00 [  0.00]( 0.61)     1.00 [  0.30]( 0.15)     1.00 [ -0.30]( 0.70)     1.00 [  0.00]( 0.61)
     4     1.00 [  0.00]( 0.00)     1.00 [  0.29]( 0.30)     1.00 [  0.00]( 0.15)     1.00 [  0.29]( 0.00)     1.00 [ -0.29]( 0.15)     0.99 [ -0.59]( 0.31)
     8     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)     1.00 [ -0.29]( 0.00)     1.00 [  0.00]( 0.00)     1.00 [ -0.29]( 0.00)     1.00 [ -0.29]( 0.15)
    16     1.00 [  0.00]( 0.00)     1.00 [  0.29]( 0.15)     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.15)     1.00 [ -0.29]( 0.00)     1.00 [ -0.29]( 0.15)
    32     1.00 [  0.00]( 0.15)     1.00 [  0.29]( 0.15)     1.00 [  0.29]( 0.15)     1.00 [  0.29]( 0.00)     1.00 [  0.00]( 0.15)     1.00 [  0.29]( 0.15)
    64     1.00 [  0.00]( 0.15)     1.00 [  0.29]( 0.00)     1.00 [  0.00]( 0.00)     1.00 [  0.29]( 0.00)     1.00 [  0.00]( 0.00)     1.00 [  0.29]( 0.15)
   128     1.00 [  0.00](17.05)     1.01 [  0.59](14.45)     1.00 [  0.30]( 0.00)     1.01 [  0.59](10.49)     1.00 [  0.00]( 9.79)     1.00 [  0.30]( 0.31)
   256     1.00 [  0.00]( 0.59)     1.00 [  0.28]( 0.59)     1.00 [  0.28]( 0.44)     1.00 [ -0.28]( 0.82)     0.99 [ -0.57]( 0.39)     1.00 [  0.00]( 0.67)
   512     1.00 [  0.00]( 0.69)     1.01 [  1.50]( 1.66)     1.00 [  0.00]( 0.33)     0.99 [ -1.50]( 0.39)     0.99 [ -1.12]( 0.58)     1.00 [  0.00]( 0.78)
   768     1.00 [  0.00]( 1.13)     1.12 [ 11.51]( 9.06)     0.99 [ -1.15]( 0.21)     0.91 [ -8.52]( 0.47)     0.91 [ -8.98]( 0.47)     0.93 [ -6.67]( 0.84)
  1024     1.00 [  0.00]( 1.23)     1.12 [ 12.41]( 4.32)     1.01 [  1.10]( 0.71)     0.87 [-13.24]( 0.59)     0.87 [-12.97]( 0.49)     0.86 [-14.07]( 0.00)
  
  
  ==================================================================
  Test          : new-schbench-wakeup-latency
  Units         : Normalized 99th percentile latency in us
  Interpretation: Lower is better
  Statistic     : Median
  ==================================================================
  #workers:  tip[pct imp](CV)          up[pct imp](CV)         smp[pct imp](CV)         max[pct imp](CV)        concur[pct imp](CV)      tasks[pct imp](CV)
     1     1.00 [ -0.00](21.48)     2.75 [-175.00](14.06)    3.00 [-200.00](12.91)    3.62 [-262.50](29.64)    3.62 [-262.50](21.84)    2.25 [-125.00](12.45)
     2     1.00 [ -0.00]( 5.96)     1.33 [-33.33](24.78)     1.33 [-33.33](18.25)     1.11 [-11.11](25.37)     1.11 [-11.11]( 9.68)     1.11 [-11.11]( 5.00)
     4     1.00 [ -0.00]( 6.20)     1.38 [-37.50]( 4.84)     1.12 [-12.50]( 0.00)     1.25 [-25.00](21.51)     1.12 [-12.50](19.99)     1.38 [-37.50](12.81)
     8     1.00 [ -0.00]( 5.34)     1.20 [-20.00]( 0.00)     1.00 [ -0.00]( 5.00)     1.20 [-20.00]( 7.45)     1.20 [-20.00]( 0.00)     1.20 [-20.00]( 4.19)
    16     1.00 [ -0.00]( 5.53)     1.22 [-22.22]( 0.00)     1.11 [-11.11]( 0.00)     1.11 [-11.11]( 9.68)     1.33 [-33.33]( 0.00)     1.33 [-33.33]( 4.43)
    32     1.00 [ -0.00]( 5.53)     1.11 [-11.11]( 5.00)     1.00 [ -0.00]( 0.00)     1.11 [-11.11]( 5.00)     1.11 [-11.11]( 0.00)     1.11 [-11.11]( 0.00)
    64     1.00 [ -0.00](12.81)     1.00 [ -0.00]( 0.00)     0.91 [  9.09]( 5.34)     0.91 [  9.09]( 5.00)     0.91 [  9.09]( 5.00)     0.82 [ 18.18](10.68)
   128     1.00 [ -0.00](12.14)     1.12 [-12.50]( 4.97)     0.81 [ 18.75](13.62)     1.06 [ -6.25]( 5.26)     0.94 [  6.25](14.68)     0.81 [ 18.75]( 6.88)
   256     1.00 [ -0.00]( 4.83)     0.98 [  1.90]( 7.97)     0.98 [  2.37]( 2.41)     0.95 [  4.74]( 5.31)     0.95 [  5.21]( 3.25)     0.97 [  2.84]( 2.84)
   512     1.00 [ -0.00]( 0.00)     0.98 [  2.00]( 5.86)     0.96 [  3.99]( 1.61)     0.83 [ 16.83]( 1.08)     0.85 [ 14.84]( 7.53)     0.97 [  3.42]( 7.91)
   768     1.00 [ -0.00]( 0.71)     0.96 [  3.63](14.27)     0.96 [  3.73]( 1.19)     0.67 [ 32.98]( 0.00)     0.67 [ 32.98]( 0.00)     0.67 [ 32.98]( 0.15)
  1024     1.00 [ -0.00]( 2.55)     0.85 [ 14.99](14.23)     0.98 [  1.80]( 0.82)     0.79 [ 21.29]( 0.00)     0.79 [ 21.29]( 0.00)     0.79 [ 21.29]( 0.00)
  
  
  ==================================================================
  Test          : new-schbench-request-latency
  Units         : Normalized 99th percentile latency in us
  Interpretation: Lower is better
  Statistic     : Median
  ==================================================================
  #workers:  tip[pct imp](CV)          up[pct imp](CV)         smp[pct imp](CV)         max[pct imp](CV)       concur[pct imp](CV)       tasks[pct imp](CV)
     1     1.00 [ -0.00]( 0.14)     1.00 [ -0.00]( 0.00)     1.00 [ -0.00]( 0.14)     1.00 [ -0.26]( 0.27)     1.01 [ -0.53]( 0.27)     1.00 [ -0.00]( 0.14)
     2     1.00 [ -0.00]( 0.00)     1.00 [  0.26]( 0.14)     1.02 [ -2.36]( 1.51)     0.99 [  0.52]( 0.14)     1.00 [ -0.00]( 2.00)     1.00 [ -0.00]( 1.74)
     4     1.00 [ -0.00]( 0.00)     1.00 [  0.26]( 1.82)     1.03 [ -2.63]( 0.23)     0.99 [  0.53]( 0.14)     1.00 [ -0.26]( 1.61)     1.03 [ -3.42]( 1.79)
     8     1.00 [ -0.00]( 0.14)     0.99 [  0.79]( 0.00)     1.00 [ -0.00]( 0.00)     0.99 [  1.05]( 0.14)     1.00 [ -0.00]( 0.00)     1.00 [ -0.00]( 0.00)
    16     1.00 [ -0.00]( 0.14)     1.00 [ -0.00]( 1.16)     1.00 [ -0.26]( 1.28)     0.99 [  0.79]( 1.97)     1.00 [ -0.26]( 0.00)     1.02 [ -2.38]( 1.15)
    32     1.00 [ -0.00]( 0.23)     0.99 [  1.04]( 0.24)     1.00 [ -0.00]( 0.67)     0.99 [  1.04]( 0.14)     0.99 [  0.52]( 0.49)     0.99 [  1.30]( 0.95)
    64     1.00 [ -0.00](12.39)     1.72 [-71.69](26.18)     1.01 [ -1.29](28.90)     0.97 [  3.09]( 0.27)     0.98 [  2.06]( 0.41)     0.98 [  2.32]( 1.22)
   128     1.00 [ -0.00]( 4.75)     0.98 [  2.07](12.01)     0.99 [  0.59]( 0.41)     1.00 [  0.30]( 3.02)     1.00 [ -0.00]( 2.72)     0.99 [  0.59]( 0.77)
   256     1.00 [ -0.00]( 0.13)     0.99 [  0.76]( 0.13)     1.00 [ -0.00]( 0.13)     0.99 [  0.76]( 0.00)     0.99 [  0.51]( 0.13)     0.99 [  0.51]( 0.13)
   512     1.00 [ -0.00]( 7.49)     1.04 [ -4.36](28.15)     0.89 [ 10.50]( 4.15)     0.42 [ 57.93]( 9.03)     0.48 [ 52.23](25.18)     0.71 [ 28.83](25.82)
   768     1.00 [ -0.00]( 3.13)     1.31 [-31.19](10.91)     1.10 [-10.13]( 1.12)     0.71 [ 28.69]( 3.69)     0.70 [ 29.59]( 3.58)     0.84 [ 15.65]( 4.74)
  1024     1.00 [ -0.00]( 2.28)     1.44 [-44.48](16.05)     1.13 [-13.09]( 0.88)     0.85 [ 15.39]( 6.64)     0.85 [ 15.03]( 0.39)     0.81 [ 19.39]( 1.36)
---

DeathStarBench was run on a 3rd Generation EPYC system (2 x 64C/128T)
with boost enabled and C2 disabled:

  ==================================================================
  Test          : DeathStarBench
  Units         : %diff compared to tip
  Interpretation: Higher is better
  Statistic     : Average throughput
  ==================================================================

  Scaling\cg_mode    up      smp      max    concur    tasks
      2x            1.63%  -0.32%   -0.03%   -0.65%    0.62%
      4x           -4.94%   1.38%   -6.59%    1.09%   -6.11%
      6x            1.83%   0.16%   -0.87%    2.24%   -0.23%

---

I'll get to look any deeper into any of the regressions until Thursday
but overall the concur mode seems good in my testing for most part.

-- 
Thanks and Regards,
Prateek

Re: [PATCH v3 0/7] sched: Flatten the pick

[Shubhang Kaushik]

Hello Peter,

I applied the `sched/flat` patchset from your tree on top of the
`tip/sched/core` base commit (9ebe5c3c29f62)/(7.1-rc2)

The evaluation was performed on an 80-core, Ampere Altra
system running Fedora Linux 41.

Benchmark Runs:-

1. Hackbench (Execution time in seconds: lower is better)
     The data reveals a clear architectural pivot point at 4 tasks:
     - Low Concurrency (< 4 tasks): Regresses by +1.8% to +4.0%.
       Removing cgroup isolation boundaries expands the idle CPU search
       adding slight overhead to the wake-up path.
       * 1 Thread:  (+1.8%)
       * 2 Threads: (+4.0%)
       * 2 Procs:   (+3.3%)
     - Tipping Point (4 tasks): Performance is completely flat.
       * 4 Threads: (+0.03%)
       * 4 Procs:   (+0.1%)
     - High Concurrency (>= 8 tasks): Improves by -0.7% to -2.3%.
       Collapsing the tree structure down to a flat layout removes
       multi-layer load tracking updates (update_load_avg), saving cycles
       under load.
       * 8 Threads:  (-0.7%)
       * 16 Threads: (-1.8%)
       * 8 Procs:    (-1.2%)
       * 16 Procs:   (-2.3%)
       * 32 Procs:   (-1.6%)

2. Schbench (Wakeup Tail Latency)
     - 16 Threads (128kb footprint): 99.9th percentile tail latency drops
       significantly by -12.21% (us). Operating on a unified runqueue layer
       prevents induced group-level throttling.
     - 32 Threads (128kb footprint): 99.9th percentile tail latency
       regresses by +5.50% (us). Eliminating nested queues increases lock
       contention during heavy simultaneous wakeups.

3. Sysbench
     - Sysbench RAM: Throughput increases by +1.55% (MiB/sec). Fewer tree
       traversals reduce cache-line bouncing, freeing up cycles.

The patchset trades minor low-load performance for better scaling and
tighter tail latencies under distributed load. However, the majority of
these deltas remain small and sit near the measurement noise floor (<=
4%).

Regards,
Shubhang Kaushik

On Fri, 5 Jun 2026, Peter Zijlstra wrote:

>
> Hi!
>
> New version, same story [1]. TL;DR:
>
> - Adds new cgroup_mode knob and implements new policies to address the
>   hierarchy level weight mismatch.
>
> - Builds upon that base to create a flat / single runqueue scheduler where the
>   cgroup hierarchy is expressed through dynamic weight management.
>
> I'm hoping to be able to merge these patches early in the next cycle (after
> 7.2-rc1).
>
> Random benchmark:
>
> Game vs 'for ((i=0; i<8; i++)) do nice ./spin.sh; done':
>
>  Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper)
>  Intel Core i7-2600K
>  AMD Radeon RX 580
>
>  Shadows Awakening (GOG)
>
> 	  default slice(*)
>
>  FPS min   4.0   29.0
>      avg  47.5   59.2
>      max  83.7   83.7
>
>  FT  min   9.3   10.2
>      avg  34.0   17.0
>      max 121.2   30.0
>
>  FPS (Frames Per Second)
>  FT  (FrameTime)
>
>  [*] Command prefix: 'chrt -o --sched-runtime 100000 0'
>
>
> Changes since v2:
>
> - merged debug and prep patches
> - fixed update_entity_lag() on dequeue (Vincent)
> - fixed throttle vs tick (Prateek)
> - fixed wakeup_preempt_fair()
> - rebased on tip/sched/core
> - rewritten cgroup_mode changelogs
> - reworked cgroup_mode concur
> - added cgroup_mode tasks
> - changed default cgroup_mode
>
>
> [1] - https://lore.kernel.org/r/20260511113104.563854162@infradead.org
>
> Can also be had:
>
>  git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat
>
> include/linux/cpuset.h |    6
> include/linux/sched.h  |    1
> kernel/cgroup/cpuset.c |   15
> kernel/sched/core.c    |    5
> kernel/sched/debug.c   |   89 ++++
> kernel/sched/fair.c    |  943 ++++++++++++++++++++++++-------------------------
> kernel/sched/pelt.c    |    6
> kernel/sched/sched.h   |   30 -
> 8 files changed, 607 insertions(+), 488 deletions(-)
>
>

Re: [PATCH v3 0/7] sched: Flatten the pick

[Szabina Korbai]

Hello Peter,

We ran the same benchmarks (schbench, sysbench, hackbench) as Shubhang
has on s390 on an LPAR running fedora 43 with 32 vCPUs.

We ran the benchmarks for each of the cgroup modes, and for the
baseline, we chose the commit prior to the patches (f666241e6bd5 -
sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() ).

We have also tried running stress-ng in parallel with the benchmarks
(set to generate 50% or 90% utilization for each vCPU).

Compared to simply running the benchmarks on their own, this has
revealed some performance trade-offs that the move to a single runqueue
can introduce.

=======================================================================
  HACKBENCH (via phoronix-test-suite pts/hackbench)
=======================================================================

The hackbench results were quite consistent, CV was under 2.5% in most
cases, a '*' marks the cases where they weren't.

What we've found is that if there is no other workload running, the
results were generally favorable, especially for higher number of
threads/processes.

However, with stress-ng also running in parallel, while the high
thread/process count cases showed even greater improvement, the lower-
count cases actually started to regress, which got worse at higher CPU
utilization.

To compound this problem, the stress-ng results also showed regression
(the stress-ng figures were recorded over the whole hackbench stress-ng
run for a specific mode, so currently there is no higher granularity
data for the 32 process case for example).

(lower = better)

Hackbench % diff from baseline by mode:

[s-00] Hackbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+
| Hackbench arg     | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| 1 thread          | -0.40%  | +0.81%  | +0.40%  | +0.37%  | -2.82%  |
| 2 thread          | -0.50%  | -0.96%  | -0.65%  | -1.68%  | -1.55%  |
| 4 thread          | -3.67%  | -3.95%  | -4.27%  | -3.67%  | -5.38%  |
| 8 thread          | -6.21%  | -6.62%  | -6.86%  | -6.30%  | -9.55%  |
| 16 thread         | -8.90%  | -9.09%  | -9.21%  | -8.58%  | -8.39%  |
| 32 thread         | -6.67%  | -6.57%  | -6.84%  | -6.25%  | -5.64%  |
| 1 process         | +1.41%  | +0.84%  | +0.19%  | +2.36%  | -3.50%  |
| 2 process         | +0.23%  | +0.91%  | +0.00%  | +1.49%  | -0.75%  |
| 4 process         | -3.21%  | -2.98%  | -3.54%  | -3.31%  | -5.57%  |
| 8 process         | -5.55%  | -5.40%  | -5.53%  | -5.56%  | -8.47%  |
| 16 process        | -7.77%  | -7.75%  | -8.33%  | -7.47%  | -7.89%  |
| 32 process        | -7.44%  | -6.92%  | -7.19%  | -4.76%  | -6.89%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Hackbench — % diff from baseline by mode
+------------------+---------+---------+---------+---------+----------+
| Hackbench arg    | concur  | max     | smp     | tasks   | up       |
+------------------+---------+---------+---------+---------+----------+
| 1 thread         | +5.73%  | +0.98%  | +0.65%  | +0.77%  | -0.46%   |
| 2 thread         | +3.24%  | -1.82%  | -2.04%  | -2.11%  | -0.65%   |
| 4 thread         | -2.51%  | -4.50%  | -4.49%  | -4.18%  | -8.75%   |
| 8 thread         | -9.29%  | -18.48% | -18.05% | -18.33% | -21.06%  |
| 16 thread        | -14.77% | -35.28% | -34.42% | -35.24% | -28.20%* |
| 32 thread        | -5.79%  | -39.03% | -38.58% | -39.16% | -27.75%  |
| 1 process        | +6.35%  | +1.20%  | +1.01%  | +0.87%  | +0.34%   |
| 2 process        | +3.91%  | -1.87%  | -1.90%  | -2.26%  | -0.54%   |
| 4 process        | -1.71%  | -3.54%  | -3.95%  | -3.69%  | -7.42%   |
| 8 process        | -8.96%  | -17.83% | -17.35% | -17.82% | -21.43%  |
| 16 process       | -14.86% | -34.80% | -33.98% | -35.37% | -30.54%  |
| 32 process       | -6.72%  | -38.85% | -38.81% | -39.50% | -33.24%  |
+------------------+---------+---------+---------+---------+----------+


[s-90] Hackbench — % diff from baseline by mode
+-----------------+----------+---------+---------+---------+----------+
| Hackbench arg   | concur   | max     | smp     | tasks   | up       |
+-----------------+----------+---------+---------+---------+----------+
| 1 thread        | +17.81%  | +18.24% | +17.29% | +17.40% | +31.11%  |
| 2 thread        | +5.81%   | +5.43%  | +6.34%  | +5.59%  | +11.62%  |
| 4 thread        | -8.69%   | -8.65%  | -7.40%  | -8.80%  | -5.85%   |
| 8 thread        | -22.75%  | -22.69% | -21.47% | -22.88% | -23.04%  |
| 16 thread       | -37.81%  | -37.71% | -36.49% | -37.84% | -28.13%  |
| 32 thread       | -39.75%  | -39.97% | -39.45% | -40.06% | -32.54%  |
| 1 process       | +23.63%* | +22.55% | +22.17% | +21.17% | +34.84%  |
| 2 process       | +7.54%   | +7.44%  | +7.34%  | +7.47%  | +13.74%  |
| 4 process       | -6.61%   | -6.87%  | -5.53%  | -6.45%  | -4.03%   |
| 8 process       | -21.92%  | -21.70% | -20.40% | -21.79% | -21.90%  |
| 16 process      | -37.17%  | -36.92% | -35.98% | -37.31% | -30.01%* |
| 32 process      | -39.20%  | -38.32% | -39.03% | -39.19% | -25.16%* |
+-----------------+----------+---------+---------+---------+----------+

stress-ng bogo-ops/s statistics by stress level (Hackbench):

(higher = better)

[s-50] Hackbench — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| concur   | -3.35%     | ±0.15%  | 0.15% |
| max      | -29.48%    | ±0.75%  | 0.75% |
| smp      | -29.57%    | ±0.68%  | 0.68% |
| tasks    | -30.01%    | ±0.42%  | 0.42% |
| up       | -24.68%    | ±2.27%  | 2.26% |
+----------+------------+---------+-------+


[s-90] Hackbench — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| concur   | -13.55%    | ±0.23%  | 0.23% |
| max      | -20.19%    | ±0.44%  | 0.44% |
| smp      | -19.72%    | ±0.57%  | 0.57% |
| tasks    | -20.61%    | ±0.33%  | 0.33% |
| up       | -25.52%*   | ±5.86%  | 5.86% |
+----------+------------+---------+-------+


=======================================================================
  SYSBENCH (via phoronix-test-suite ciunas/sysbench) - throughput
=======================================================================

The sysbench results showed small improvements, but more interesting
were the stress-ng results, showing regressions across all modes at 50%
CPU utilization, but improvements at 90%.

RAM/Memory testcase

(higher = better)

Sysbench % diff from baseline by mode:

[s-00] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+
|                   | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| memory run        | -3.91%  | +0.02%  | -4.05%  | +0.44%  | -4.11%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+
|                   | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| memory run        | +2.84%  | +1.09%  | +0.73%  | +2.62%  | +1.91%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+
|                   | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| memory run        | +1.83%  | +4.03%  | +2.64%  | +1.16%  | -12.52% |
+-------------------+---------+---------+---------+---------+---------+


stress-ng bogo-ops/s statistics by stress level (Sysbench):

(higher = better)

[s-50] Sysbench — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| baseline | 588.34     | ±1.72%  | 1.72% |
+----------+------------+---------+-------+
| concur   | -10.77%    | ±2.52%  | 2.52% |
| max      | -6.24%     | ±2.21%  | 2.21% |
| smp      | -6.16%     | ±1.93%  | 1.93% |
| tasks    | -8.51%     | ±3.21%  | 3.21% |
| up       | -10.88%    | ±3.25%  | 3.25% |
+----------+------------+---------+-------+


[s-90] Sysbench — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| baseline | 726.96     | ±2.00%  | 2.00% |
+----------+------------+---------+-------+
| concur   | +4.22%     | ±1.24%  | 1.24% |
| max      | +4.00%     | ±0.90%  | 0.90% |
| smp      | +3.10%     | ±1.73%  | 1.73% |
| tasks    | +4.28%     | ±1.83%  | 1.83% |
| up       | +15.61%    | ±5.10%  | 5.10% |
+----------+------------+---------+-------+

=======================================================================
  SCHBENCH
=======================================================================

All testruns were done with footprint set to 128kb.

There is some variability between the different test environments
(especially between the 32 threads-locking-yes case and the others),
but there is a general pattern.

Without stress-ng running in parallel, over the different test cases,
we did generally see a reduction in tail latency while other metrics
remained largely unchanged.

At 50% cpu utilization, tail latency is still improved (or close to the
noise floor), but average latency shows a more significant regression.

Then, at 90% cpu utilization, all latency components show regression,
tail latency the most of all.

RPS is not impacted as strongly.

Throughout this the stress-ng benchmark results show improvement at 90%
cpu utilization, while at 50% the changes are quite close to the noise
floor.


=======================================================================
  SCHBENCH 16T -- LOCKING: NO
=======================================================================

Schbench % diff from baseline by mode:

(Request latency percentiles: lower = better; RPS: higher = better)

[s-00] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.00%  |
| Req p90  (us)     | -0.23%  | +0.00%  | +0.00%  | +0.00%  | -0.23%  |
| Req p99.9(us)     | -7.23%  | -5.16%  | -3.10%  | -5.85%  | -5.51%  |
| RPS p50  (req)    | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.35%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +13.51% | +15.57% | +15.27% | +13.80% | +14.68% |
| Req p90  (us)     | -1.48%  | +2.32%  | +2.53%  | -2.11%  | +1.48%  |
| Req p99.9(us)     | -8.99%  | -7.53%  | -5.35%  | -8.75%  | +4.86%  |
| RPS p50  (req)    | +3.65%  | +1.00%  | +1.00%  | +3.65%  | +0.66%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +7.72%  | +7.72%  | +7.46%  | +7.72%  | +8.49%  |
| Req p90  (us)     | +4.76%  | +4.36%  | +4.76%  | +4.76%  | +19.35% |
| Req p99.9(us)     | +34.35% | +34.35% | +34.35% | +34.35% | +38.10% |
| RPS p50  (req)    | -1.09%  | +0.00%  | +0.00%  | -1.09%  | -7.56%  |
+-------------------+---------+---------+---------+---------+---------+


stress-ng bogo-ops/s statistics by stress level (schbench):

[s-50] Schbench 16t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| concur   | -2.17%     | ±1.50%  | 1.50% |
| max      | +1.08%     | ±1.61%  | 1.61% |
| smp      | +0.37%     | ±1.63%  | 1.63% |
| tasks    | -2.20%     | ±1.00%  | 1.00% |
| up       | +2.27%     | ±1.56%  | 1.56% |
+----------+------------+---------+-------+


[s-90] Schbench 16t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| concur   | +3.23%     | ±1.05%  | 1.05% |
| max      | -0.25%     | ±1.12%  | 1.12% |
| smp      | +0.93%     | ±2.70%  | 2.70% |
| tasks    | +1.33%     | ±1.62%  | 1.62% |
| up       | +8.09%     | ±5.19%  | 5.18% |
+----------+------------+---------+-------+



=======================================================================
  SCHBENCH 16T -- LOCKING: YES
=======================================================================

Schbench % diff from baseline by mode:

(Request latency percentiles: lower = better; RPS: higher = better)

[s-00] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.00%  |
| Req p90  (us)     | +0.23%  | +0.23%  | +0.23%  | +0.23%  | +0.00%  |
| Req p99.9(us)     | -12.86% | -15.25% | -2.39%  | -11.66% | -8.67%  |
| RPS p50  (req)    | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.35%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +10.16% | +11.85% | +11.85% | +10.16% | +11.00% |
| Req p90  (us)     | -0.36%  | +4.63%  | +5.35%  | -0.36%  | +2.50%  |
| Req p99.9(us)     | +0.58%  | +0.87%  | +4.05%  | -1.45%  | +2.03%  |
| RPS p50  (req)    | +3.21%  | -0.36%  | +0.00%  | +2.85%  | +0.71%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +7.70%  | +7.45%  | +7.45%  | +7.70%  | +9.24%  |
| Req p90  (us)     | +4.13%  | +1.50%  | +3.38%  | +3.75%  | +102.2… |
| Req p99.9(us)     | +18.21% | +13.94% | +16.22% | +13.66% | +306.8… |
| RPS p50  (req)    | -1.52%  | -0.38%  | -0.38%  | -1.52%  | -26.95% |
+-------------------+---------+---------+---------+---------+---------+

stress-ng bogo-ops/s statistics by stress level (schbench):

[s-50] Schbench 16t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| concur   | -1.29%     | ±1.16%  | 1.16% |
| max      | +1.26%     | ±1.18%  | 1.17% |
| smp      | +1.18%     | ±1.15%  | 1.15% |
| tasks    | +0.02%     | ±2.05%  | 2.04% |
| up       | +2.31%     | ±2.56%  | 2.56% |
+----------+------------+---------+-------+


[s-90] Schbench 16t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| concur   | +6.16%     | ±1.47%  | 1.47% |
| max      | +3.10%     | ±1.89%  | 1.89% |
| smp      | +3.91%     | ±1.83%  | 1.83% |
| tasks    | +5.68%     | ±1.87%  | 1.87% |
| up       | +16.60%    | ±8.60%  | 8.60% |
+----------+------------+---------+-------+


=======================================================================
  SCHBENCH 32T -- LOCKING: NO
=======================================================================

Schbench % diff from baseline by mode:

(Request latency percentiles: lower = better; RPS: higher = better)

[s-00] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.00%  |
| Req p90  (us)     | +0.00%  | +0.00%  | -0.23%  | +0.00%  | -0.23%  |
| Req p99.9(us)     | -4.80%  | -3.77%  | -6.17%  | -7.55%  | -10.98% |
| RPS p50  (req)    | +0.35%  | +0.35%  | +0.35%  | +0.35%  | +0.35%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +11.76% | +13.77% | +13.49% | +11.76% | +12.63% |
| Req p90  (us)     | -2.52%  | +2.31%  | +2.31%  | -2.73%  | -0.63%  |
| Req p99.9(us)     | -8.99%  | -6.80%  | -4.13%  | -9.96%  | +2.43%  |
| RPS p50  (req)    | +2.61%  | -0.65%  | -0.33%  | +2.61%  | +0.65%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +5.55%  | +5.55%  | +5.55%  | +5.55%  | +7.06%  |
| Req p90  (us)     | +2.62%  | +2.62%  | +2.62%  | +2.62%  | +32.73% |
| Req p99.9(us)     | +31.57% | +31.11% | +31.57% | +31.57% | +34.33% |
| RPS p50  (req)    | +1.46%  | +2.55%  | +2.55%  | +1.46%  | -10.66% |
+-------------------+---------+---------+---------+---------+---------+


stress-ng bogo-ops/s statistics by stress level (schbench):

[s-50] Schbench 32t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| baseline | 478.27     | ±1.98%  | 1.99% |
+----------+------------+---------+-------+
| concur   | -2.85%     | ±1.45%  | 1.45% |
| max      | +0.12%     | ±1.95%  | 1.95% |
| smp      | -0.23%     | ±1.22%  | 1.22% |
| tasks    | -2.39%     | ±1.96%  | 1.96% |
| up       | -0.27%     | ±1.59%  | 1.59% |
+----------+------------+---------+-------+


[s-90] Schbench 32t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| baseline | 577.45     | ±0.69%  | 0.70% |
+----------+------------+---------+-------+
| concur   | +2.17%     | ±1.54%  | 1.54% |
| max      | +0.25%     | ±1.28%  | 1.28% |
| smp      | -0.33%     | ±1.42%  | 1.42% |
| tasks    | +1.07%     | ±1.17%  | 1.17% |
| up       | +10.70%    | ±5.47%  | 5.47% |
+----------+------------+---------+-------+


=======================================================================
  SCHBENCH 32T -- LOCKING: YES
=======================================================================

Schbench % diff from baseline by mode:

(Request latency percentiles: lower = better; RPS: higher = better)

[s-00] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.00%  |
| Req p90  (us)     | +0.00%  | +0.00%  | +0.00%  | +0.00%  | -0.23%  |
| Req p99.9(us)     | +7.31%  | +10.05% | +13.70% | +10.35% | -16.74% |
| RPS p50  (req)    | +0.00%  | +0.00%  | +0.00%  | +0.35%  | +0.35%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +8.30%  | +9.96%  | +9.96%  | +8.30%  | +9.41%  |
| Req p90  (us)     | -0.71%  | +3.54%  | +4.60%  | -0.71%  | +0.00%  |
| Req p99.9(us)     | -4.05%  | +3.76%  | +3.47%  | -2.89%  | -4.92%  |
| RPS p50  (req)    | +2.11%  | -1.05%  | -1.41%  | +2.11%  | +1.05%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +5.55%  | +5.55%  | +5.55%  | +5.55%  | +6.81%  |
| Req p90  (us)     | -0.77%  | -1.16%  | -0.77%  | -0.77%  | +55.49% |
| Req p99.9(us)     | +11.00% | +11.28% | +14.67% | +19.18% | +92.67% |
| RPS p50  (req)    | +1.51%  | +2.64%  | +2.26%  | +1.51%  | -15.16% |
+-------------------+---------+---------+---------+---------+---------+


stress-ng bogo-ops/s statistics by stress level (schbench):

[s-50] Schbench 32t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| baseline | 506.91     | ±1.74%  | 1.74% |
+----------+------------+---------+-------+
| concur   | -6.03%     | ±1.28%  | 1.28% |
| max      | -3.98%     | ±1.68%  | 1.68% |
| smp      | -4.23%     | ±1.31%  | 1.31% |
| tasks    | -5.75%     | ±1.47%  | 1.47% |
| up       | -3.84%     | ±2.10%  | 2.10% |
+----------+------------+---------+-------+


[s-90] Schbench 32t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| baseline | 555.39     | ±1.03%  | 1.03% |
+----------+------------+---------+-------+
| concur   | +6.59%     | ±1.38%  | 1.38% |
| max      | +5.05%     | ±1.25%  | 1.25% |
| smp      | +7.28%     | ±1.90%  | 1.90% |
| tasks    | +7.07%     | ±1.46%  | 1.46% |
| up       | +18.88%    | ±7.68%  | 7.68% |
+----------+------------+---------+-------+

Regards,

-- 
Szabina Korbai

Linux on Z development
Software Labs Campus Unlimited Company
25 North Wall Quay,
Dublin 1,
D01 H104,
Ireland

szkorbai@linux.ibm.com


IBM

Re: [PATCH v3 0/7] sched: Flatten the pick

[K Prateek Nayak]

Hello Szabina,

On 8/17/2026 9:35 PM, Szabina Korbai wrote:
> Hello Peter,
> 
> We ran the same benchmarks (schbench, sysbench, hackbench) as Shubhang
> has on s390 on an LPAR running fedora 43 with 32 vCPUs.
> 
> We ran the benchmarks for each of the cgroup modes, and for the
> baseline, we chose the commit prior to the patches (f666241e6bd5 -
> sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() ).
> 
> We have also tried running stress-ng in parallel with the benchmarks
> (set to generate 50% or 90% utilization for each vCPU).
> 
> Compared to simply running the benchmarks on their own, this has
> revealed some performance trade-offs that the move to a single runqueue
> can introduce.

Are you using tip:sched/core at commit 68e3748781 ("sched/fair: Fix flat
hierarchy") for the flat_cg numbers or did you checkout at 85570f10a4c6
("sched/eevdf: Move to a single runqueue")?

There are a couple fixes for vruntime update and Vincent's optimizations
for preemption bits which might make a difference to the overall
results.

-- 
Thanks and Regards,
Prateek

Re: [PATCH v3 0/7] sched: Flatten the pick

[Szabina Korbai]

On Mon, 2026-08-17 at 22:05 +0530, K Prateek Nayak wrote:
> Hello Szabina,
> 
> On 8/17/2026 9:35 PM, Szabina Korbai wrote:
> > Hello Peter,
> > 
> > We ran the same benchmarks (schbench, sysbench, hackbench) as
> > Shubhang
> > has on s390 on an LPAR running fedora 43 with 32 vCPUs.
> > 
> > We ran the benchmarks for each of the cgroup modes, and for the
> > baseline, we chose the commit prior to the patches (f666241e6bd5 -
> > sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() ).
> > 
> > We have also tried running stress-ng in parallel with the
> > benchmarks
> > (set to generate 50% or 90% utilization for each vCPU).
> > 
> > Compared to simply running the benchmarks on their own, this has
> > revealed some performance trade-offs that the move to a single
> > runqueue
> > can introduce.
> 
> Are you using tip:sched/core at commit 68e3748781 ("sched/fair: Fix
> flat
> hierarchy") for the flat_cg numbers or did you checkout at
> 85570f10a4c6
> ("sched/eevdf: Move to a single runqueue")?
> 
> There are a couple fixes for vruntime update and Vincent's
> optimizations
> for preemption bits which might make a difference to the overall
> results.


Hi Prateek,

thank you, that's a good call. I did checkout at "Move to a single
runqueue". Let me try it with the fix included, see how the results are
affected.

Regards,

-- 
Szabina Korbai

Linux on Z development
Software Labs Campus Unlimited Company
25 North Wall Quay,
Dublin 1,
D01 H104,
Ireland

szkorbai@linux.ibm.com


IBM

Re: [PATCH v3 0/7] sched: Flatten the pick

[Peter Zijlstra]

On Tue, Aug 18, 2026 at 10:04:57AM +0100, Szabina Korbai wrote:
> On Mon, 2026-08-17 at 22:05 +0530, K Prateek Nayak wrote:
> > Hello Szabina,
> > 
> > On 8/17/2026 9:35 PM, Szabina Korbai wrote:
> > > Hello Peter,
> > > 
> > > We ran the same benchmarks (schbench, sysbench, hackbench) as
> > > Shubhang
> > > has on s390 on an LPAR running fedora 43 with 32 vCPUs.
> > > 
> > > We ran the benchmarks for each of the cgroup modes, and for the
> > > baseline, we chose the commit prior to the patches (f666241e6bd5 -
> > > sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() ).
> > > 
> > > We have also tried running stress-ng in parallel with the
> > > benchmarks
> > > (set to generate 50% or 90% utilization for each vCPU).
> > > 
> > > Compared to simply running the benchmarks on their own, this has
> > > revealed some performance trade-offs that the move to a single
> > > runqueue
> > > can introduce.
> > 
> > Are you using tip:sched/core at commit 68e3748781 ("sched/fair: Fix
> > flat
> > hierarchy") for the flat_cg numbers or did you checkout at
> > 85570f10a4c6
> > ("sched/eevdf: Move to a single runqueue")?
> > 
> > There are a couple fixes for vruntime update and Vincent's
> > optimizations
> > for preemption bits which might make a difference to the overall
> > results.
> 
> 
> Hi Prateek,
> 
> thank you, that's a good call. I did checkout at "Move to a single
> runqueue". Let me try it with the fix included, see how the results are
> affected.

I've not yet managed to digest your various benchmark results, but also
double check that patch 6/7 from this series is not to 'blame' for the
some of the changes.

The 0day robot fingered that patch for at least one issue.

In that case the benchmark threads ended up 'heavier' than before, which
resulted in less preemptions. Probably ksoftirqd getting ran less and
causing a regression in network throughput for that thing.

I did suggest trying to change the slice of ksoftirqd down, such that it
might be ran more readily, but I'm not sure that ever got tried.

Re: [PATCH v3 0/7] sched: Flatten the pick

[Szabina Korbai]

On Tue, 2026-08-18 at 11:16 +0200, Peter Zijlstra wrote:
> On Tue, Aug 18, 2026 at 10:04:57AM +0100, Szabina Korbai wrote:
> > On Mon, 2026-08-17 at 22:05 +0530, K Prateek Nayak wrote:
> > > Hello Szabina,
> > > 
> > > On 8/17/2026 9:35 PM, Szabina Korbai wrote:
> > > > Hello Peter,
> > > > 
> > > > We ran the same benchmarks (schbench, sysbench, hackbench) as
> > > > Shubhang
> > > > has on s390 on an LPAR running fedora 43 with 32 vCPUs.
> > > > 
> > > > We ran the benchmarks for each of the cgroup modes, and for the
> > > > baseline, we chose the commit prior to the patches
> > > > (f666241e6bd5 -
> > > > sched/fair: Unify cfs_rq throttling via
> > > > account_cfs_rq_runtime() ).
> > > > 
> > > > We have also tried running stress-ng in parallel with the
> > > > benchmarks
> > > > (set to generate 50% or 90% utilization for each vCPU).
> > > > 
> > > > Compared to simply running the benchmarks on their own, this
> > > > has
> > > > revealed some performance trade-offs that the move to a single
> > > > runqueue
> > > > can introduce.
> > > 
> > > Are you using tip:sched/core at commit 68e3748781 ("sched/fair:
> > > Fix
> > > flat
> > > hierarchy") for the flat_cg numbers or did you checkout at
> > > 85570f10a4c6
> > > ("sched/eevdf: Move to a single runqueue")?
> > > 
> > > There are a couple fixes for vruntime update and Vincent's
> > > optimizations
> > > for preemption bits which might make a difference to the overall
> > > results.
> > 
> > 
> > Hi Prateek,
> > 
> > thank you, that's a good call. I did checkout at "Move to a single
> > runqueue". Let me try it with the fix included, see how the results
> > are
> > affected.
> 
> I've not yet managed to digest your various benchmark results, but
> also
> double check that patch 6/7 from this series is not to 'blame' for
> the
> some of the changes.
> 
> The 0day robot fingered that patch for at least one issue.
> 
> In that case the benchmark threads ended up 'heavier' than before,
> which
> resulted in less preemptions. Probably ksoftirqd getting ran less and
> causing a regression in network throughput for that thing.
> 
> I did suggest trying to change the slice of ksoftirqd down, such that
> it
> might be ran more readily, but I'm not sure that ever got tried.

Hello Peter,

I've repeated the benchmark runs by checking out at tip:sched/core
85570f10a4c6 - "sched/eevdf: Move to a single runqueue" and cherry-
picking the fix (tip:sched/core at commit 68e3748781 - "sched/fair: Fix
flat hierarchy") on top of it. The baseline was 4f166adb5cb0 -
sched/fair: Fix overflow in update_tg_cfs_runnable().

It does look like most of the regressions/inconsistencies I saw earlier
were addressed by applying Vincent's fix.

Overall the effect of the patch series on system performance is now
more positive than negative, though there are still some
regressions in some cases. (Up mode remains something of an outlier in
this regard.)

At this time I haven't yet applied Jake Steinman's patch (sched/fair:
floor tg_cpus() at 1), but neither did the system experience a kernel
panic.

Observations by benchmark:

HACKBENCH

More parallel threads/processes benefit more from the patch, while
fewer ones show regression - without an additional stress-ng workload.
This is the same trend that Shubhang's benchmark results showed.

With an additional stress-ng workload running in parallel, there is
regression in the hackbench benchmark, but the bogo-ops/second metric
of stress-ng does improve to a much greater degree (for example 1.30%
regression with 32 threads for concur mode - and a 66.22% improvement
to the stress-ng workload).

There was quite a high variance to the stress-ng results though,
especially to the baseline: worst was the 32 processes case where the
CV was 15.83%.

SCHBENCH

The improvement in p99.9 tail latency is now more consistent even at
90% CPU utilization. One exception is the 32 threads with no locking
case, which shows a ~5% regression in concur mode without the stress-ng
workload.

Additionally, with the stress-ng load set to 90% the p50 and p90
latency of the hackbench workload still does show regression under some
conditions.

Aside from up mode, schbench RPS and stress-ng bogo-ops/s values showed
more minor changes (<2% difference) individually, and usually balanced
each other out.

SYSBENCH

The two (sysbench and stress-ng) workloads together showed overall
performance improvement (~2%), but not sysbench on its own.


Numbers below:

=======================================================================
  HACKBENCH RESULTS
=======================================================================

* - CV over 5%

(lower = better)

Hackbench % diff from baseline by mode:

[s-00] Hackbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+
| Argument          | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| 1 thread          | +2.07%  | +2.55%  | +2.14%  | +3.11%  | -5.10%  |
| 2 thread          | +2.65%  | +2.84%  | +2.16%  | +1.34%  | -1.37%  |
| 4 thread          | -1.98%  | -1.63%  | -1.62%  | -2.55%  | -5.29%  |
| 8 thread          | -5.71%  | -5.09%  | -5.19%  | -6.46%  | -8.83%  |
| 16 thread         | -9.62%  | -8.66%  | -9.09%  | -10.38% | -10.75% |
| 32 thread         | -7.47%  | -6.08%  | -6.11%  | -7.94%  | -1.15%* |
| 1 process         | +3.00%  | +3.28%  | +3.39%  | +2.61%  | -1.95%  |
| 2 process         | +2.22%  | +2.46%  | +2.70%  | +2.49%  | -0.51%  |
| 4 process         | -1.65%  | -0.90%  | -1.65%  | -2.28%  | -3.50%  |
| 8 process         | -4.45%  | -3.76%  | -4.66%  | -5.49%  | -8.01%  |
| 16 process        | -8.58%  | -7.75%  | -8.81%  | -9.53%  | -9.59%  |
| 32 process        | -6.56%  | -5.26%  | -5.86%  | -7.53%  | -0.57%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Hackbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+
| Argument          | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| 1 thread          | +2.88%  | +3.44%  | +3.23%  | +3.25%  | +2.29%  |
| 2 thread          | -0.57%  | +0.16%  | -0.26%  | -0.88%  | +1.35%  |
| 4 thread          | +1.38%  | +1.77%  | +1.57%  | +1.08%  | -2.44%  |
| 8 thread          | +7.57%  | +8.20%  | +8.69%  | +7.22%  | +2.94%  |
| 16 thread         | -0.32%  | +0.36%  | +1.54%  | -1.13%  | +8.17%  |
| 32 thread         | -1.65%  | -0.61%  | -0.70%  | -2.50%  | +12.28% |
| 1 process         | +3.27%  | +2.87%  | +2.83%  | +3.11%  | +2.24%  |
| 2 process         | +0.09%  | +0.46%  | -0.10%  | -0.40%  | +1.80%  |
| 4 process         | +1.03%  | +1.66%  | +1.24%  | +0.79%  | -2.38%  |
| 8 process         | +7.82%  | +8.33%  | +9.05%  | +7.89%  | +4.85%* |
| 16 process        | +0.80%  | +1.55%  | +3.47%  | +0.23%  | +6.80%* |
| 32 process        | -1.09%  | +0.06%  | -0.12%  | -1.86%  | +17.00% |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Hackbench — % diff from baseline by mode
+------------------+---------+---------+---------+---------+----------+
| Argument         | concur  | max     | smp     | tasks   | up       |
+------------------+---------+---------+---------+---------+----------+
| 1 thread         | +7.32%  | +7.55%  | +7.03%  | +5.90%  | +15.32%  |
| 2 thread         | +9.16%  | +9.38%  | +8.95%  | +8.19%  | +13.79%  |
| 4 thread         | +18.34% | +18.44% | +18.34% | +17.76% | +21.62%  |
| 8 thread         | +17.14% | +17.64% | +19.50% | +16.86% | +21.07%* |
| 16 thread        | +7.39%  | +8.01%  | +10.73% | +6.20%  | +26.18%  |
| 32 thread        | +0.14%  | +0.58%  | +0.87%  | -0.55%  | +22.58%* |
| 1 process        | +6.12%  | +7.08%  | +6.46%  | +5.84%  | +19.06%  |
| 2 process        | +8.79%  | +8.31%  | +8.60%  | +8.09%  | +14.79%  |
| 4 process        | +18.76% | +19.15% | +20.63% | +18.46% | +21.42%  |
| 8 process        | +17.55% | +17.99% | +19.75% | +16.96% | +19.02%* |
| 16 process       | +7.49%  | +8.22%  | +10.65% | +6.81%  | +25.28%  |
| 32 process       | +1.30%  | +2.95%  | +1.77%  | +0.50%  | +23.78%* |
+------------------+---------+---------+---------+---------+----------+


stress-ng (int64 stressor) mean % diff from baseline by stress level
(Hackbench):

(higher = better)

[s-50] Hackbench — stress-ng mean % diff from baseline
+----------------+---------+----------+----------+---------+----------+
| Argument       | concur  | max      | smp      | tasks   | up       |
+----------------+---------+----------+----------+---------+----------+
| 1 thread       | -3.16%  | -2.78%   | -3.38%   | -2.90%  | -2.76%   |
| 2 thread       | -0.32%  | -0.48%   | -0.64%   | -0.23%  | -0.61%   |
| 4 thread       | +14.47% | +14.50%  | +14.76%  | +14.82% | +14.61%  |
| 8 thread       | +38.73% | +38.45%  | +39.97%  | +39.00% | +41.90%  |
| 16 thread      | +42.93% | +44.01%  | +46.30%* | +42.41% | +71.38%* |
| 32 thread      | +47.79% | +50.61%* | +50.44%* | +49.75% | +129.7…* |
| 1 process      | -2.19%  | -2.80%   | -2.94%   | -2.56%  | -2.72%   |
| 2 process      | -0.30%  | -0.58%   | -0.76%   | -0.61%  | -0.49%   |
| 4 process      | +12.84% | +12.70%  | +12.97%  | +12.98% | +12.90%  |
| 8 process      | +37.22% | +36.93%  | +38.34%  | +37.60% | +40.19%  |
| 16 process     | +44.33% | +44.00%  | +48.00%* | +44.04% | +61.75%* |
| 32 process     | +51.70% | +51.24%* | +50.52%* | +51.62% | +142.8…* |
+----------------+---------+----------+----------+---------+----------+


[s-90] Hackbench — stress-ng mean % diff from baseline
+--------------+----------+----------+----------+----------+----------+
| Argument     | concur   | max      | smp      | tasks    | up       |
+----- --------+----------+----------+----------+----------+----------+
| 1 thread     | +2.68%   | +2.61%   | +2.63%   | +2.48%   | +5.19%   |
| 2 thread     | +5.85%   | +6.20%*  | +6.06%   | +5.92%   | +10.09%  |
| 4 thread     | +23.55%* | +23.53%* | +23.64%* | +23.79%* | +34.50%* |
| 8 thread     | +47.11%  | +47.47%* | +49.44%  | +47.59%  | +61.43%  |
| 16 thread    | +56.40%  | +58.08%* | +63.22%* | +55.92%  | +102.2…* |
| 32 thread    | +58.54%  | +59.72%* | +60.52%* | +58.75%* | +164.9…* |
| 1 process    | +2.16%   | +2.50%   | +2.44%   | +2.46%   | +5.99%   |
| 2 process    | +5.48%   | +5.37%   | +5.58%   | +5.70%   | +10.38%* |
| 4 process    | +23.87%* | +23.74%* | +25.94%* | +24.97%* | +33.32%* |
| 8 process    | +45.77%  | +46.32%  | +48.47%  | +46.12%  | +57.70%  |
| 16 process   | +56.41%  | +57.84%  | +62.58%* | +56.79%  | +100.9…* |
| 32 process   | +66.22%  | +72.01%* | +65.52%* | +68.43%* | +178.1…* |
+--------------+----------+----------+----------+----------+----------+


=======================================================================
  SCHBENCH 16T -- LOCKING: NO
=======================================================================

Schbench % diff from baseline by mode:

(Latency: lower = better, RPS: higher = better)

[s-00] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.00%  |
| Req p90  (us)     | +0.23%  | +0.23%  | +0.23%  | +0.23%  | -0.45%  |
| Req p99.9(us)     | -9.57%  | -6.84%  | -4.79%  | -6.84%  | -7.18%  |
| RPS p50  (req)    | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.35%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | -3.54%  | -3.81%  | -2.99%  | -3.27%  | -3.54%  |
| Req p90  (us)     | -2.19%  | -1.97%  | -1.75%  | -2.41%  | -3.29%  |
| Req p99.9(us)     | -3.11%  | -2.59%  | -0.52%  | -3.63%  | +1.82%  |
| RPS p50  (req)    | +0.96%  | +0.96%  | +1.28%  | +0.96%  | +0.64%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +3.52%  | +3.52%  | +3.52%  | +3.52%  | +3.52%  |
| Req p90  (us)     | +6.04%  | +6.24%  | +6.24%  | +6.04%  | +25.91% |
| Req p99.9(us)     | -16.82% | -14.59% | -14.18% | -17.63% | -3.24%  |
| RPS p50  (req)    | +0.37%  | +0.37%  | +0.37%  | +0.37%  | -4.08%  |
+-------------------+---------+---------+---------+---------+---------+


stress-ng (int64 stressor) bogo-ops/s statistics by stress level
(schbench):

(higher = better)

[s-50] Schbench 16t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| baseline | 442.34     | ±0.48%  | 0.48% |
+----------+------------+---------+-------+
| concur   | -1.32%     | ±0.76%  | 0.76% |
| max      | -1.28%     | ±0.86%  | 0.86% |
| smp      | -1.37%     | ±0.77%  | 0.77% |
| tasks    | -1.23%     | ±0.75%  | 0.75% |
| up       | -1.07%     | ±0.45%  | 0.45% |
+----------+------------+---------+-------+


[s-90] Schbench 16t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+--------+
| Variant  | Mean %diff | StdDev± | CV     |
+----------+------------+---------+--------+
| baseline | 524.18     | ±1.51%  | 1.50%  |
+----------+------------+---------+--------+
| concur   | +0.03%     | ±1.19%  | 1.19%  |
| max      | +0.17%     | ±1.57%  | 1.57%  |
| smp      | +0.18%     | ±1.16%  | 1.16%  |
| tasks    | +0.14%     | ±1.25%  | 1.25%  |
| up       | +4.94%     | ±13.42% | 13.42% |
+----------+------------+---------+--------+


=======================================================================
  SCHBENCH 16T -- LOCKING: YES
=======================================================================

Schbench % diff from baseline by mode:

(Latency: lower = better, RPS: higher = better)

[s-00] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.00%  |
| Req p90  (us)     | +0.23%  | +0.23%  | +0.00%  | +0.23%  | -0.23%  |
| Req p99.9(us)     | -10.29% | -13.01% | -10.59% | -13.92% | -17.55% |
| RPS p50  (req)    | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.35%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | -3.21%  | -2.95%  | -2.68%  | -2.95%  | -5.35%  |
| Req p90  (us)     | -3.33%  | -3.33%  | -2.22%  | -3.70%  | -6.56%  |
| Req p99.9(us)     | -3.15%  | -5.15%  | -5.72%  | -5.72%  | -9.44%  |
| RPS p50  (req)    | +1.37%  | +1.37%  | +1.03%  | +1.37%  | +2.39%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +3.51%  | +3.51%  | +3.26%  | +3.51%  | +2.01%  |
| Req p90  (us)     | -0.76%  | -0.38%  | -0.38%  | -0.76%  | +9.56%  |
| Req p99.9(us)     | -7.08%  | -2.10%  | -0.26%  | -6.82%  | +32.77% |
| RPS p50  (req)    | +0.77%  | +0.77%  | +0.77%  | +0.77%  | -3.56%  |
+-------------------+---------+---------+---------+---------+---------+

stress-ng (int64 stressor) bogo-ops/s statistics by stress level
(schbench):

(higher = better)

[s-50] Schbench 16t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| baseline | 453.53     | ±0.38%  | 0.38% |
+----------+------------+---------+-------+
| concur   | -1.54%     | ±0.53%  | 0.53% |
| max      | -1.57%     | ±0.60%  | 0.60% |
| smp      | -1.59%     | ±0.68%  | 0.68% |
| tasks    | -1.59%     | ±0.55%  | 0.55% |
| up       | -1.92%     | ±0.58%  | 0.59% |
+----------+------------+---------+-------+


[s-90] Schbench 16t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+--------+
| Variant  | Mean %diff | StdDev± | CV     |
+----------+------------+---------+--------+
| baseline | 531.40     | ±1.73%  | 1.73%  |
+----------+------------+---------+--------+
| concur   | -0.26%     | ±1.14%  | 1.14%  |
| max      | -0.16%     | ±1.71%  | 1.71%  |
| smp      | -0.13%     | ±1.49%  | 1.49%  |
| tasks    | -0.32%     | ±1.19%  | 1.19%  |
| up       | +1.91%     | ±10.60% | 10.60% |
+----------+------------+---------+--------+

=======================================================================
  SCHBENCH 32T -- LOCKING: NO
=======================================================================

Schbench % diff from baseline by mode:

(Latency: lower = better, RPS: higher = better)

[s-00] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.00%  |
| Req p90  (us)     | -0.23%  | +0.00%  | +0.00%  | -0.45%  | -0.23%  |
| Req p99.9(us)     | +5.14%  | +1.47%  | +6.97%  | +3.30%  | -1.10%  |
| RPS p50  (req)    | +0.00%  | +0.00%  | -0.35%  | +0.00%  | +0.00%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | -2.70%  | -2.70%  | -2.43%  | -2.16%  | -1.62%  |
| Req p90  (us)     | -2.20%  | -1.98%  | -2.20%  | -2.20%  | -3.51%  |
| Req p99.9(us)     | -3.40%  | -1.05%  | -1.57%  | -2.61%  | +0.00%  |
| RPS p50  (req)    | +1.27%  | +1.27%  | +1.27%  | +1.27%  | +1.58%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +3.27%  | +3.27%  | +3.02%  | +3.27%  | +3.27%  |
| Req p90  (us)     | +12.35% | +12.35% | +12.13% | +12.13% | +32.08% |
| Req p99.9(us)     | -16.45% | -16.06% | -17.04% | -16.06% | -6.07%  |
| RPS p50  (req)    | +1.11%  | +1.11%  | +1.11%  | +1.11%  | -3.34%  |
+-------------------+---------+---------+---------+---------+---------+

stress-ng (int64 stressor) bogo-ops/s statistics by stress level
(schbench):

(higher = better)

[s-50] Schbench 32t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| baseline | 438.90     | ±0.49%  | 0.48% |
+----------+------------+---------+-------+
| concur   | -1.55%     | ±0.63%  | 0.63% |
| max      | -1.59%     | ±0.71%  | 0.71% |
| smp      | -1.50%     | ±0.79%  | 0.79% |
| tasks    | -1.49%     | ±0.63%  | 0.63% |
| up       | -1.86%     | ±0.62%  | 0.62% |
+----------+------------+---------+-------+


[s-90] Schbench 32t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+--------+
| Variant  | Mean %diff | StdDev± | CV     |
+----------+------------+---------+--------+
| baseline | 523.09     | ±1.70%  | 1.70%  |
+----------+------------+---------+--------+
| concur   | -0.33%     | ±1.21%  | 1.21%  |
| max      | -0.29%     | ±1.36%  | 1.36%  |
| smp      | -0.40%     | ±1.24%  | 1.24%  |
| tasks    | -0.27%     | ±1.31%  | 1.31%  |
| up       | +3.62%     | ±11.49% | 11.49% |
+----------+------------+---------+--------+

=======================================================================
  SCHBENCH 32T -- LOCKING: YES
=======================================================================

Schbench % diff from baseline by mode:

(Latency: lower = better, RPS: higher = better)

[s-00] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.00%  |
| Req p90  (us)     | +0.00%  | +0.23%  | +0.23%  | +0.23%  | -0.45%  |
| Req p99.9(us)     | -19.66% | -11.98% | -18.98% | -24.41% | -36.38% |
| RPS p50  (req)    | +0.00%  | +0.00%  | +0.00%  | +0.00%  | +0.35%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | -3.18%  | -3.18%  | -3.44%  | -3.18%  | -3.18%  |
| Req p90  (us)     | -2.59%  | -2.22%  | -1.85%  | -2.96%  | -5.45%  |
| Req p99.9(us)     | -2.97%  | -1.49%  | -2.38%  | -3.27%  | -6.54%  |
| RPS p50  (req)    | +1.70%  | +1.36%  | +1.36%  | +1.36%  | +3.06%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+
| Metric            | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| Req p50  (us)     | +3.27%  | +3.27%  | +3.02%  | +3.27%  | +1.76%  |
| Req p90  (us)     | +2.69%  | +2.69%  | +2.69%  | +2.69%  | +47.36% |
| Req p99.9(us)     | -8.60%  | -8.60%  | -2.28%  | -8.60%  | +41.34% |
| RPS p50  (req)    | +1.54%  | +1.54%  | +1.54%  | +1.54%  | -7.39%  |
+-------------------+---------+---------+---------+---------+---------+

stress-ng (int64 stressor) bogo-ops/s statistics by stress level
(schbench):

(higher = better)

[s-50] Schbench 32t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+
| Variant  | Mean %diff | StdDev± | CV    |
+----------+------------+---------+-------+
| baseline | 451.63     | ±0.40%  | 0.40% |
+----------+------------+---------+-------+
| concur   | -1.82%     | ±0.45%  | 0.45% |
| max      | -1.80%     | ±0.45%  | 0.45% |
| smp      | -1.72%     | ±0.55%  | 0.55% |
| tasks    | -1.76%     | ±0.46%  | 0.47% |
| up       | -2.61%     | ±0.87%  | 0.87% |
+----------+------------+---------+-------+


[s-90] Schbench 32t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+--------+
| Variant  | Mean %diff | StdDev± | CV     |
+----------+------------+---------+--------+
| baseline | 529.80     | ±1.78%  | 1.78%  |
+----------+------------+---------+--------+
| concur   | -0.75%     | ±1.12%  | 1.12%  |
| max      | -0.72%     | ±1.14%  | 1.14%  |
| smp      | -0.66%     | ±0.96%  | 0.96%  |
| tasks    | -0.67%     | ±1.19%  | 1.19%  |
| up       | +5.03%     | ±15.81% | 15.81% |
+----------+------------+---------+--------+


=======================================================================
  SYSBENCH RESULTS
=======================================================================


Sysbench throughput % diff from baseline by mode:

(higher = better)

[s-00] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+
| Argument          | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| memory            | -0.18%  | +0.12%  | -0.36%  | -0.18%  | +0.00%  |
+-------------------+---------+---------+---------+---------+---------+


[s-50] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+
| Argument          | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| memory            | -1.71%  | -0.43%  | -0.71%  | -1.07%  | -2.21%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+
| Argument          | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| memory            | +2.94%  | +3.97%  | +3.46%  | +4.34%  | -6.76%  |
+-------------------+---------+---------+---------+---------+---------+



stress-ng (int64 stressor) mean % diff from baseline by stress level
(Sysbench):

(higher = better)

[s-50] Sysbench — stress-ng mean % diff from baseline
+-------------------+---------+---------+---------+---------+---------+
| Argument          | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| memory            | +3.59%  | +2.78%  | +4.36%  | +3.60%  | +1.46%  |
+-------------------+---------+---------+---------+---------+---------+


[s-90] Sysbench — stress-ng mean % diff from baseline
+-------------------+---------+---------+---------+---------+---------+
| Argument          | concur  | max     | smp     | tasks   | up      |
+-------------------+---------+---------+---------+---------+---------+
| memory            | +0.85%  | -1.35%  | -0.00%  | -1.79%  | +18.47% |
+-------------------+---------+---------+---------+---------+---------+


Regards,

-- 
Szabina Korbai

Linux on Z development
Software Labs Campus Unlimited Company
25 North Wall Quay,
Dublin 1,
D01 H104,
Ireland

szkorbai@linux.ibm.com


IBM

Re: [PATCH v3 0/7] sched: Flatten the pick

[Chen Yu]

On Tue, Aug 18, 2026 at 11:16:49AM +0200, Peter Zijlstra wrote:
> > > 
> > > Are you using tip:sched/core at commit 68e3748781 ("sched/fair: Fix
> > > flat
> > > hierarchy") for the flat_cg numbers or did you checkout at
> > > 85570f10a4c6
> > > ("sched/eevdf: Move to a single runqueue")?
> > > 
> > > There are a couple fixes for vruntime update and Vincent's
> > > optimizations
> > > for preemption bits which might make a difference to the overall
> > > results.
> > 
> > 
> > Hi Prateek,
> > 
> > thank you, that's a good call. I did checkout at "Move to a single
> > runqueue". Let me try it with the fix included, see how the results are
> > affected.
> 
> I've not yet managed to digest your various benchmark results, but also
> double check that patch 6/7 from this series is not to 'blame' for the
> some of the changes.
> 
> The 0day robot fingered that patch for at least one issue.
> 
> In that case the benchmark threads ended up 'heavier' than before, which
> resulted in less preemptions. Probably ksoftirqd getting ran less and
> causing a regression in network throughput for that thing.
> 

The 0day' regression was triggered by running netperf/netserver using loopback,
so ksoftirqd was not really running very frequently IMO.

> I did suggest trying to change the slice of ksoftirqd down, such that it
> might be ran more readily, but I'm not sure that ever got tried.

The 0day regression was triggered by running netperf/netserver over loopback,
so ksoftirqd was not running very frequently IIUC.

I borrowed a similar machine(no-SMT, every 4 Cores share L2, 192 Cores) as 0day used.
I asked AI to generate a simple test script based on 0day's reproducer
(attached at the end of this email), and successfully reproduced the regression.

TL;DR,
The decision made by wake_affine_weight() seems to be affected by the increased
weight of the wakee under concur mode. As a result, wake_affine_weight() became less
likely to select this_cpu. This reduced preference for this_cpu leads to a higher
L2 miss rate on this platform, and consequently lower throughput.

Test summary:

| Config | Throughput (Mbps) | **L2 miss%** | **LLC miss%** |
| smp + WA_WEIGHT | **470249** | **19.37%** | 60.08% |
| smp + NO_WA_WEIGHT | 305360 | 32.67% | 60.46% |
| concur + WA_WEIGHT | 338770 | 31.45% | 62.85% |
| concur + NO_WA_WEIGHT | 259358 | 36.24% | 61.95% |

We can see if WA_WEIGHT is disabled in smp mode, the performance drops to concur. One
possible reason is that task_h_load(p) returns a very small value in smp mode(just the
issue that concur wants to fix), whereas it returns a "normal" value in concur mode.

In smp mode, the extremly small value returned by task_h_load(p) can be effectively ignored
in the comparison in wake_affine_weight(). As a result, the condition for returning this_cpu
becomes:

cpu_load(cpu_rq(this_cpu)) * 100 < cpu_load(cpu_rq(prev_cpu) * 108

As a result, this_cpu has a higher chance of being selected in smp mode. In concur mode,
task_h_load(p) is on par with cpu_load(), so it cannot be simply ignored and provides a fairer
basis for choosing between this_cpu and prev_cpu. Since netperf/netserver is a sync wakeup, it
prefers to be co-located on the same CPU/L2 domain for better L2 cache locality.

I'm not sure if this is the expected behavior before I further digging into the details,
a hack patch could restore part of the performance, because it increases the ratio to
return this_cpu:

thanks,
Chenyu

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 6d881e530f89..8bff8b6698a9 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8385,6 +8385,7 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p,
        unsigned long task_load;
 
        this_eff_load = cpu_load(cpu_rq(this_cpu));
+       task_load = task_h_load(p);
 
        if (sync) {
                unsigned long current_load = task_h_load(current);
@@ -8392,11 +8393,13 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p,
                if (current_load > this_eff_load)
                        return this_cpu;
 
+               /* replace current load with wakee load does no harm */
+               if (sched_feat(WA_SYNC_WEIGHT) && current_load >= task_load)
+                       return this_cpu;
+
                this_eff_load -= current_load;
        }
 
-       task_load = task_h_load(p);
-
        this_eff_load += task_load;
        if (sched_feat(WA_BIAS))
                this_eff_load *= 100;
diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index 8f0dee8fc475..99fabe53fb22 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -129,6 +129,7 @@ SCHED_FEAT(ATTACH_AGE_LOAD, true)
 SCHED_FEAT(WA_IDLE, true)
 SCHED_FEAT(WA_WEIGHT, true)
 SCHED_FEAT(WA_BIAS, true)
+SCHED_FEAT(WA_SYNC_WEIGHT, true)


test command:

NR=$(( $(nproc) * 2 ))
run() {
local mode=$1 waw=$2 tag=$3
echo $mode | sudo -n tee /sys/kernel/debug/sched/cgroup_mode >/dev/null
echo $waw  | sudo -n tee /sys/kernel/debug/sched/features    >/dev/null
sudo -n pkill -x netserver; sleep 1
netserver -4 >/dev/null 2>&1; sleep 1
rm -f /tmp/m4.$tag
for i in $(seq $NR); do
netperf -4 -H 127.0.0.1 -t TCP_MAERTS -l 70 -P 0 >> /tmp/m4.$tag 
2>/dev/null &
done
sleep 15
sudo -n ./tools/perf/perf stat -a \
-e 
l2_request.hit,l2_request.miss,longest_lat_cache.reference,longest_lat_cache.miss 
\
-x, -o /tmp/p4.$tag -- sleep 20
wait
} >/dev/null 2>&1
run smp    WA_WEIGHT    smp_waw
run smp    NO_WA_WEIGHT smp_nowaw
run concur WA_WEIGHT    con_waw
run concur NO_WA_WEIGHT con_nowaw
echo WA_WEIGHT | sudo -n tee /sys/kernel/debug/sched/features >/dev/null


printf "%-12s %10s %8s %10s %10s\n" CONFIG Mbps streams L2miss% LLCmiss%
for t in smp_waw smp_nowaw con_waw con_nowaw; do
   read TP NS < <(awk '{s+=$NF;n++} END{print s, n}' /tmp/m4.$t)
   awk -F, -v t=$t -v tp=$TP -v ns=$NS '
     $3=="l2_request.hit"{h=$1} $3=="l2_request.miss"{m=$1}
     $3=="longest_lat_cache.reference"{r=$1} 
$3=="longest_lat_cache.miss"{lm=$1}
     END{printf "%-12s %10.0f %8d %9.2f%% %9.2f%%\n", t, tp, ns, 
100*m/(h+m), 100*lm/r}' /tmp/p4.$t
done

Re: [PATCH v3 0/7] sched: Flatten the pick

[Peter Zijlstra]

On Mon, Aug 24, 2026 at 10:51:51PM +0800, Chen Yu wrote:
> TL;DR,
> The decision made by wake_affine_weight() seems to be affected by the increased
> weight of the wakee under concur mode. As a result, wake_affine_weight() became less
> likely to select this_cpu. This reduced preference for this_cpu leads to a higher
> L2 miss rate on this platform, and consequently lower throughput.
> 
> Test summary:
> 
> | Config | Throughput (Mbps) | **L2 miss%** | **LLC miss%** |
> | smp + WA_WEIGHT | **470249** | **19.37%** | 60.08% |
> | smp + NO_WA_WEIGHT | 305360 | 32.67% | 60.46% |
> | concur + WA_WEIGHT | 338770 | 31.45% | 62.85% |
> | concur + NO_WA_WEIGHT | 259358 | 36.24% | 61.95% |

(I'm thinking it would've been easier to read if all those '|' were
properly aligned?)

> We can see if WA_WEIGHT is disabled in smp mode, the performance drops to concur. One
> possible reason is that task_h_load(p) returns a very small value in smp mode(just the
> issue that concur wants to fix), whereas it returns a "normal" value in concur mode.
> 
> In smp mode, the extremly small value returned by task_h_load(p) can be effectively ignored
> in the comparison in wake_affine_weight(). As a result, the condition for returning this_cpu
> becomes:
> 
> cpu_load(cpu_rq(this_cpu)) * 100 < cpu_load(cpu_rq(prev_cpu) * 108
> 
> As a result, this_cpu has a higher chance of being selected in smp mode. In concur mode,
> task_h_load(p) is on par with cpu_load(), so it cannot be simply ignored and provides a fairer
> basis for choosing between this_cpu and prev_cpu. Since netperf/netserver is a sync wakeup, it
> prefers to be co-located on the same CPU/L2 domain for better L2 cache locality.

There is something wonky with task_h_load(), it isn't behaving properly.
I've not managed to put a finger on it yet. Let me continue poking at it.

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Chengfeng Lin]

[-- Attachment #1: Type: text/plain, Size: 4452 bytes --]

Hi Vincent, Mike, Peter,

I have additional pipe-handoff throughput results for 333238da9a19
("sched: Update time before requeueing delayed entities"), which Mike
identified in his wakeup-jitter test [1]. With 15 workers, will-it-scale
throughput on v7.3 is 15.69% lower than v7.1.3 in thread mode and 14.82%
lower in process mode. Reverting this commit on v7.2 in a separate controlled
comparison improves 15-worker thread throughput by 13.65%; the 2-worker
control changes by -0.48%.

I found this while running the existing MMTests recipe for will-it-scale's
context_switch1 test [2], not a generated workload.
Each worker exchanges one-byte messages with its own helper over two pipes,
counting two handoffs per round trip. I kept the default CORE affinity,
restricted to the i7-12700KF's eight P-cores, with both logical CPUs per core
available. Helpers inherit their worker's affinity mask; above eight
workers, cores are reused. All tasks ran at SCHED_OTHER/nice 0, with no extra
CPU-bound competitors or custom time slices. The benchmark loop was unchanged.

After each boot, I ran one warmup run followed by nine measured runs for
each condition. Each run excluded about six seconds of built-in warmup,
then measured throughput for 15 seconds. All kernels were built with
GCC 15.2.0 and run with full preemption, the powersave governor,
balance_performance EPP and Turbo enabled.

The v7.1.3 -> v7.3 -> v7.1.3 comparison used the recipe's worker counts,
plus 1/4/8-worker controls. Changes below are relative to the average of
the two baseline means:

  Workers       Thread mode       Process mode
   1/2/4/6/8    -0.70%..-4.52%    -1.32%..-4.59%
      11           -10.54%            -9.96%
      15           -15.69%           -14.82%
      20           -14.81%           -14.43%
      26           -12.69%           -13.55%
      33            -8.75%            -7.33%
      39            -7.39%            -6.46%
      40            -7.19%            -6.51%

For example, 15-worker thread throughput is 10,875,243 -> 9,168,152 ->
10,872,945 handoffs/s, about 1.71 million fewer handoffs/s on v7.3.
The within-boot coefficient of variation (CV) was at most 2.01%; the two
baseline means differed by at most 1.39%.

The matched v7.2 revert comparison used the same CORE affinity (handoffs/s):

  Thread workers     v7.2 A       v7.2 reverted        v7.2 B
        2          1,178,295         1,174,770       1,182,545
       15          9,413,892        10,695,877       9,409,225

At 15 workers the gain is about 1.28 million handoffs/s. The second baseline
was 0.05% lower than the first, and the maximum CV was 0.11%.

In separate PMU runs, I counted instructions and handoffs over the same
15-second post-warmup window. At 15 workers, kernel instructions per
handoff rose from 8,338 on v7.1.3 to 10,423 on v7.3 (+25.00%). Reverting
the commit on v7.2 reduced this from 10,239 to 8,355 (-18.40%). Context
switches stayed at about one per handoff in both comparisons, pointing to
more kernel work per handoff rather than more switches.

Separate entry probes over the same 15-second window observed about 1.0
delayed requeue per 100 handoffs at 2 workers and 92.6 at 15 workers on v7.2.
Almost every delayed requeue included the added update_curr() call.
The revert removed that call, but also reduced the 15-worker requeue rate
to 55.3 per 100 handoffs. With 15 workers, this accounting path was taken
much more often; reverting the commit also changed how often requeues
occurred.

CPU-location samples showed the same broad placement pattern across the
compared kernels.

These results point to additional work in the delayed-requeue path.
Updating time is needed for correct lag compensation, so the revert is
diagnostic, not a proposed fix. Can this overhead be reduced while
preserving correct lag accounting?

Reproduction details are attached; measurements and diagnostic summaries
are available at [3].

[1] https://lists.openwall.net/linux-kernel/2026/10/18/311
[2] https://github.com/antonblanchard/will-it-scale/blob/a34a85cc1e9b9b74e94fdd3ecc479019da610e6a/tests/context_switch1.c
[3] https://github.com/lcf0399/linux-regression-evidence/tree/8022fe1fb0256e65f63b4bde530b346027f63059/kernel/sched/sched-fair-delayed-requeue-handoff-333238da

LLMs assisted with experiment orchestration, analysis and drafting.
The workload is upstream will-it-scale; results are bare-metal measurements.

Assisted-by: LLM

Thanks,
Chengfeng

[-- Attachment #2: reproduce-wis.txt --]
[-- Type: text/plain, Size: 9043 bytes --]

Native CORE will-it-scale pipe-handoff reproduction
==================================================

This describes the native CORE measurements in the accompanying report.
Run on a dedicated test host.
CPU numbers describe the measured i7-12700KF; adapt them to the other host's
physical topology, then keep the chosen topology identical across kernels.

Pinned inputs
-------------
will-it-scale: https://github.com/antonblanchard/will-it-scale
  a34a85cc1e9b9b74e94fdd3ecc479019da610e6a
MMTests: https://github.com/gormanm/mmtests
  638401e07e70d67fa5a1e6295e1315ae0af831b2
Kernel peeled commits (not annotated-tag object IDs):
  v7.1.3  199c9959d3a9b53f346c221757fc7ac507fbac50
  v7.2    8d3ae59288f1e7d58d76558a6ee96d533bc5019f
  v7.3    c03c1a14507dfaf8b856c2608f9ac216021fc222
Diagnostic v7.2 variant: reverse 333238da9a193ffc58792995f3e951e4cb87bfd2.
It removes update_curr(cfs_rq) from requeue_delayed_entity and is NOT a
proposed fix. The intervention was not performed on v7.3.

Build
-----
Dependencies: C compiler, make, patch, hwloc development files, GNU time,
systemd and cgroup-v2 cpuset support. Tested GCC 15.2.0 and hwloc 2.13.0.
Use clean adjacent will-it-scale and mmtests checkouts. In will-it-scale:

  git checkout --detach a34a85cc1e9b9b74e94fdd3ecc479019da610e6a
  git -C ../mmtests checkout --detach 638401e07e70d67fa5a1e6295e1315ae0af831b2
  sed -n '/^==== BEGIN align-alloc.patch ====$/,/^==== END align-alloc.patch ====$/p' \
    ../mmtests/shellpack_src/src/wis/wis-install | sed '1d;$d' | patch -p1
  sed -i 's|/tmp/shared_area|shared_area|g' main.c
  make -j2 context_switch1_threads context_switch1_processes \
    CFLAGS='-Wall -O2 -g -std=gnu11'

The alignment patch and shared_area relocation are from the pinned MMTests
install recipe. tests/context_switch1.c is unchanged. Other file-path edits
in the recipe affect file-backed tests, not this pipe test.

Our executable SHA-256 values (different build environments can differ):
  threads:
    6fd5c63e4caec9916cbd3942b1ad037f738d24b1232c18711f96b72bc5f248d6
  processes:
    e0e9cf0017bdc0e5f276d7625ecfc0f26b1af1593bca1c50aafd9b1aa7497867

Placement and one invocation
----------------------------
Check lscpu -e=CPU,CORE,SOCKET,NODE. On this host CPU0-15 are eight P-cores;
CORE masks are 0,1; 2,3; ...; 14,15. CPU16-19 are E-cores and excluded from
the test group, but all 20 logical CPUs remain online.

Keep both SMT CPUs in each CORE mask. Native hwloc binding is enabled;
do not add -n or -m. Helpers inherit their worker's mask. Above eight
workers, cores are reused in native order. The coordinator retains CPU0-15.
This is not one worker/helper pair forced onto two separate physical cores.

Example for 15 workers in thread mode (replace mode with processes):
Use an ordinary account with sudo for service creation. Keep the checkout
outside /tmp and /var/tmp because PrivateTmp would hide it.

  workers=15
  mode=threads
  binary="$PWD/context_switch1_$mode"
  work=$(mktemp -d "$PWD/wis-run.XXXXXX")
  sudo systemd-run --quiet --wait --pipe --collect --service-type=exec \
    --unit="wis-handoff-$$" \
    -p "User=$(id -un)" -p "Group=$(id -gn)" \
    -p 'CPUAffinity=0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15' \
    -p AllowedCPUs=0-15 -p MemoryMax=2G -p MemorySwapMax=0 \
    -p "TasksMax=$((2 * workers + 16))" \
    -p RuntimeMaxSec=45 -p TimeoutStopSec=5 -p KillMode=control-group \
    -p NoNewPrivileges=yes -p Environment=LC_ALL=C \
    -p "WorkingDirectory=$work" -p "ReadWritePaths=$work" \
    -p PrivateTmp=yes -p PrivateDevices=yes \
    -p ProtectSystem=strict -p ProtectHome=read-only \
    -p TemporaryFileSystem=/dev/shm:rw,size=16M,mode=1777 \
    -p LimitCORE=0 -p LimitFSIZE=536870912 -p LimitNOFILE=8192 \
    -- /usr/bin/time -v -o "$work/time.txt" \
    "$binary" -t "$workers" -s 15 > "$work/native.txt"

Separately validate affinity/clone inheritance using strace -f with
sched_setaffinity, clone and clone3 before clean timing. Check the effective
mask after intersection with the cgroup, not merely the requested mask.
At 15 workers, 30 worker/helper tasks use the eight CORE masks; the first
seven masks have four tasks each and the last has two. The coordinator is
additional. In process mode these are processes, not the same thread group.
Do not use strace or perf during timing. Preserve failed/incomplete runs.

Separate actual-placement check (not clean timing): two inherited 199-Hz
CPU-clock records per condition on v7.1.3, v7.3, v7.2 and the diagnostic
revert. In each record, 14/15 worker/helper pairs at 15 workers had all
captured samples on one logical CPU per pair; the remaining pair used the
CPU14/15 core. At 2 workers, both pairs had samples across their core's
SMT CPUs. Thus an allowed CORE mask does not imply the pair actually runs
on separate logical CPUs. The pattern was unchanged across the four kernels.
Pairs were reconstructed from fork events, with independently checked
affinity masks. The sampled window starts six seconds after the last test
thread is created and ends at the first test-thread exit. It approximates
the measured interval, not an exact benchmark marker or a complete schedule;
unsampled migrations and other timing effects cannot be excluded.

Parameters, runtime and sampling
--------------------------------
Native recipe worker list on this host:
  2, 6, 11, 15, 20, 26, 33, 39, 40
Also measured separately: 1, 4, 8. Both thread and process modes were tested.
This does not cover every possible MMTests configuration.

Bare metal, i7-12700KF, 32 GiB RAM; GCC 15.2.0, runtime preempt=full.
SCHED_OTHER/nice 0, no custom sched_runtime/slice, no extra CPU-bound spinner.
powersave governor, balance_performance EPP, Turbo enabled at every point.
No CPU bandwidth quota; 2 GiB memory cap and no swap for the test group.
RT budget remains 950000 us per 1000000 us; these are not RT tasks.
The matched v7.2 builds differ only in the diagnostic revert and LOCALVERSION.
Keep other settings matched and restore temporary changes afterward.
No benchmark, kernel build, coverage observer or profiler runs concurrently.

For each condition in each boot: one placement diagnostic, one complete
warmup run, then nine measured runs without instrumentation. Each run has
six native warmup intervals, fifteen
measurement intervals and average:N. The average is the integer floor of
the sum of fifteen measured totals divided by fifteen.

The main metric is aggregate native handoffs/s, two increments per
write/read round trip. It is not a PMU context-switch count. GNU time records
whole-run user/system CPU and elapsed time separately. Runs last about
21 seconds; lower throughput does not mean a correspondingly longer run.

Independent comparisons:
  24 conditions: v7.1.3 -> v7.3 -> v7.1.3.
  Thread workers 2/15: v7.2 A -> diagnostic revert -> v7.2 B.
Nine runs within one boot are not nine independent boots. Keep comparisons
and boot positions separate; do not pool batches. Compare the middle mean
to the mean of the first/return means. CV uses sample standard deviation;
return drift is return / first - 1.

Per-run measurements and raw-output verification are retained locally and
available on request; no measurement CSV accompanies the initial report.

Separate post-warmup diagnostics
-------------------------------
Counter diagnostics use the same fifteen native intervals as the throughput
denominator, not the whole 21-second invocation. A diagnostic-only preload
reads inherited task-scoped perf descriptors at the original "measurement"
and "average" output boundaries; the benchmark executable and hot loop are
unchanged. Subtract the two readings and divide by the sum of the fifteen
native handoff totals. Record enabled/running time and boundary read times;
these are short sequential snapshots, not a simultaneous stop of all workers.

There were three counter runs per condition at v7.1.3, v7.3, v7.2 and the
matched revert, for 2/15 workers. Kernel instructions/cycles cover the test
task family, not update_curr alone; interrupt work in that context can count.
They are diagnostics, never additional clean-throughput samples.

A different batch used two entry-probe runs per condition on v7.2 and the
revert, with the same native window markers. Count requeue_delayed_entity
entries/returns, update_curr entries, and update_curr inside the requeue,
restricted to the marked benchmark thread group. Check entry/return bounds
and per-object count conservation; the first arguments are sched_entity and
cfs_rq respectively, not a common runqueue identifier. Exact tested symbols
and the presence/absence of the internal update_curr call were checked in
both vmlinux files. Probe overhead changes execution, so no PMU or throughput
values from these instrumented runs are used as performance evidence. The counts
do not reconstruct every lag value, hierarchy or scheduling decision.

Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue

[Mike Galbraith]

On Fri, 2026-10-30 at 21:03 +0800, Chengfeng Lin wrote:
> Hi Vincent, Mike, Peter,
> 
> I have additional pipe-handoff throughput results for 333238da9a19
> ("sched: Update time before requeueing delayed entities"), which Mike
> identified in his wakeup-jitter test [1]. With 15 workers, will-it-scale
> throughput on v7.3 is 15.69% lower than v7.1.3 in thread mode and 14.82%
> lower in process mode. Reverting this commit on v7.2 in a separate controlled
> comparison improves 15-worker thread throughput by 13.65%; the 2-worker
> control changes by -0.48%.

After much crawling over trees etc, and way more yacking than intended
for a datapoint, I think I know what's up, and it's simple: there's a
good reason those "now" updates are event driven.  That one was added
in the DELAY_DEQUEUE machinery, where it expends and *banks* precious
fastpath cycles, shifting bean counting enough to make DELAY_DEQUEUE
measurably less effective.  It looks completely broken with a 20% util
probe, but more measurement shows that it's not dead, just a bit sad.

Test probe sent another signal in virgin flat in __dequeue_task(), and
I think it's valid for that update_curr() too. It seems to be in
keeping with placement strategy, but with flat changes, we just updated
the cfs_rq where all the action is, and hierarchy update is right below
it.. looks like redundant overhead.

	-Mike