[PATCH v3] sched/core: Skip rq->avg_idle update without a valid idle_stamp

[PATCH v3] sched/core: Skip rq->avg_idle update without a valid idle_stamp

[Shubhang Kaushik (Ampere)]

Commit 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved
to an idle CPU") moved rq->avg_idle accounting out of the wakeup path and
into put_prev_task_idle(), so that the idle interval is consumed whenever
the idle task is switched out.

The wakeup-side accounting that it replaced only updated rq->avg_idle
when rq->idle_stamp was non-zero. The new helper lost that validity
check and unconditionally computes:

	rq_clock(rq) - rq->idle_stamp

If rq->idle_stamp is zero, this uses rq_clock(rq) as the sample. That is
not a valid idle duration and can immediately drive rq->avg_idle to its
clamp.

This can happen when sched_balance_newidle() returns before setting
rq->idle_stamp, for example when this_rq->ttwu_pending is set. In that
case the rq can switch to the idle task with idle_stamp still zero and
leave idle again when the pending wakeup is processed.

Other paths can also switch to the idle task without setting
rq->idle_stamp via newidle_balance(), for example find_proxy_task() or
force-idling.

Restore the idle_stamp validity check in update_rq_avg_idle() and skip
the rq->avg_idle update when there is no measured idle interval.

Fixes: 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU")
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Acked-by: John Stultz <jstultz@google.com>
Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org>
---
Temporary tracing under hackbench load confirmed that
update_rq_avg_idle() can be reached with rq->idle_stamp == 0.
Hackbench showed no material regression versus v7.2-rc5 mainline.

Related discussion:
  https://lore.kernel.org/r/20260423023322.1293923-1-firelzrd@gmail.com

This is a narrower variant of the earlier proposal. It keeps the
rq->idle_stamp guard in update_rq_avg_idle(), but intentionally does not
stamp idle entry from set_next_task_idle(), preserving the existing
newidle accounting model and avoiding force-idle/proxy-exec accounting
concerns.
---
Changes in v3:
  - Describe the sched_balance_newidle()/ttwu_pending path as an
    example of entering idle without a valid rq->idle_stamp.
  - Drop unlikely() from the idle_stamp check.
  - Add Acked-by from John Stultz.

Link to v2: https://lore.kernel.org/r/20260806-master-v2-1-e1f3a1a0c903@gentwo.org

Changes in v2:
  - Add Reviewed-by from Prateek.
  - Mention find_proxy_task() and force-idling as examples of paths that
    can switch to the idle task without a valid rq->idle_stamp.
  - Cc John Stultz.

Link to v1: https://lore.kernel.org/r/20260728-master-v1-1-f95d9b0147d2@gentwo.org
---
 kernel/sched/core.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 96226707c2f6135341aa779b8262f113e103d8ad..68fa724cd668fb6c4cead329d05fa95e2f1ea5db 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3732,11 +3732,17 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
 
 void update_rq_avg_idle(struct rq *rq)
 {
-	u64 delta = rq_clock(rq) - rq->idle_stamp;
-	u64 max = 2*rq->max_idle_balance_cost;
+	u64 idle_stamp = rq->idle_stamp;
+	u64 delta, max;
+
+	if (!idle_stamp)
+		return;
+
+	delta = rq_clock(rq) - idle_stamp;
 
 	update_avg(&rq->avg_idle, delta);
 
+	max = 2 * rq->max_idle_balance_cost;
 	if (rq->avg_idle > max)
 		rq->avg_idle = max;
 	rq->idle_stamp = 0;

---
base-commit: 3f008280327ba5ad132965abab0c7846283cef0c
change-id: 20260728-master-55cd7cc13290

Best regards,
-- 
Shubhang Kaushik (Ampere) <sh@gentwo.org>

Re: [PATCH v3] sched/core: Skip rq->avg_idle update without a valid idle_stamp

[Shubhang]

Hello all,

Gentle ping for v3. I repeated the Hackbench comparison with multiple runs 
per kernel. The reported thread and process cases stayed within about 1.5% 
mean delta, with similar baseline run-to-run variation.

Please let me know if there are further concerns with this approach.

Thanks,
Shubhang Kaushik

On Fri, 7 Aug 2026, Shubhang Kaushik (Ampere) wrote:

> Fixes: 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU")
> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
> Acked-by: John Stultz <jstultz@google.com>
> Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org>
> ---
> Temporary tracing under hackbench load confirmed that
> update_rq_avg_idle() can be reached with rq->idle_stamp == 0.
> Hackbench showed no material regression versus v7.2-rc5 mainline.
>
> Related discussion:
>  https://lore.kernel.org/r/20260423023322.1293923-1-firelzrd@gmail.com
>
> This is a narrower variant of the earlier proposal. It keeps the
> rq->idle_stamp guard in update_rq_avg_idle(), but intentionally does not
> stamp idle entry from set_next_task_idle(), preserving the existing
> newidle accounting model and avoiding force-idle/proxy-exec accounting
> concerns.
> ---
> Changes in v3:
>  - Describe the sched_balance_newidle()/ttwu_pending path as an
>    example of entering idle without a valid rq->idle_stamp.
>  - Drop unlikely() from the idle_stamp check.
>  - Add Acked-by from John Stultz.
>
> Link to v2: https://lore.kernel.org/r/20260806-master-v2-1-e1f3a1a0c903@gentwo.org

Re: [PATCH v3] sched/core: Skip rq->avg_idle update without a valid idle_stamp

[Vincent Guittot]

On Fri, 7 Aug 2026 at 22:47, Shubhang Kaushik (Ampere) <sh@gentwo.org> wrote:
>
> Commit 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved
> to an idle CPU") moved rq->avg_idle accounting out of the wakeup path and
> into put_prev_task_idle(), so that the idle interval is consumed whenever
> the idle task is switched out.
>
> The wakeup-side accounting that it replaced only updated rq->avg_idle
> when rq->idle_stamp was non-zero. The new helper lost that validity
> check and unconditionally computes:
>
>         rq_clock(rq) - rq->idle_stamp
>
> If rq->idle_stamp is zero, this uses rq_clock(rq) as the sample. That is
> not a valid idle duration and can immediately drive rq->avg_idle to its
> clamp.
>
> This can happen when sched_balance_newidle() returns before setting
> rq->idle_stamp, for example when this_rq->ttwu_pending is set. In that
> case the rq can switch to the idle task with idle_stamp still zero and
> leave idle again when the pending wakeup is processed.
>
> Other paths can also switch to the idle task without setting
> rq->idle_stamp via newidle_balance(), for example find_proxy_task() or
> force-idling.
>
> Restore the idle_stamp validity check in update_rq_avg_idle() and skip
> the rq->avg_idle update when there is no measured idle interval.
>
> Fixes: 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU")
> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
> Acked-by: John Stultz <jstultz@google.com>
> Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org>

Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>

> ---
> Temporary tracing under hackbench load confirmed that
> update_rq_avg_idle() can be reached with rq->idle_stamp == 0.
> Hackbench showed no material regression versus v7.2-rc5 mainline.
>
> Related discussion:
>   https://lore.kernel.org/r/20260423023322.1293923-1-firelzrd@gmail.com
>
> This is a narrower variant of the earlier proposal. It keeps the
> rq->idle_stamp guard in update_rq_avg_idle(), but intentionally does not
> stamp idle entry from set_next_task_idle(), preserving the existing
> newidle accounting model and avoiding force-idle/proxy-exec accounting
> concerns.
> ---
> Changes in v3:
>   - Describe the sched_balance_newidle()/ttwu_pending path as an
>     example of entering idle without a valid rq->idle_stamp.
>   - Drop unlikely() from the idle_stamp check.
>   - Add Acked-by from John Stultz.
>
> Link to v2: https://lore.kernel.org/r/20260806-master-v2-1-e1f3a1a0c903@gentwo.org
>
> Changes in v2:
>   - Add Reviewed-by from Prateek.
>   - Mention find_proxy_task() and force-idling as examples of paths that
>     can switch to the idle task without a valid rq->idle_stamp.
>   - Cc John Stultz.
>
> Link to v1: https://lore.kernel.org/r/20260728-master-v1-1-f95d9b0147d2@gentwo.org
> ---
>  kernel/sched/core.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 96226707c2f6135341aa779b8262f113e103d8ad..68fa724cd668fb6c4cead329d05fa95e2f1ea5db 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3732,11 +3732,17 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
>
>  void update_rq_avg_idle(struct rq *rq)
>  {
> -       u64 delta = rq_clock(rq) - rq->idle_stamp;
> -       u64 max = 2*rq->max_idle_balance_cost;
> +       u64 idle_stamp = rq->idle_stamp;
> +       u64 delta, max;
> +
> +       if (!idle_stamp)
> +               return;
> +
> +       delta = rq_clock(rq) - idle_stamp;
>
>         update_avg(&rq->avg_idle, delta);
>
> +       max = 2 * rq->max_idle_balance_cost;
>         if (rq->avg_idle > max)
>                 rq->avg_idle = max;
>         rq->idle_stamp = 0;
>
> ---
> base-commit: 3f008280327ba5ad132965abab0c7846283cef0c
> change-id: 20260728-master-55cd7cc13290
>
> Best regards,
> --
> Shubhang Kaushik (Ampere) <sh@gentwo.org>
>
>

Re: [PATCH v3] sched/core: Skip rq->avg_idle update without a valid idle_stamp

[Shubhang]

Hi,

Could you please let me know whether it is suitable for the sched/core
tree for v7.3, or whether further work is needed?

Thanks,
Shubhang Kaushik

On Fri, 7 Aug 2026, Shubhang Kaushik (Ampere) wrote:

> Commit 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved
> to an idle CPU") moved rq->avg_idle accounting out of the wakeup path and
> into put_prev_task_idle(), so that the idle interval is consumed whenever
> the idle task is switched out.
>
> The wakeup-side accounting that it replaced only updated rq->avg_idle
> when rq->idle_stamp was non-zero. The new helper lost that validity
> check and unconditionally computes:
>
> 	rq_clock(rq) - rq->idle_stamp
>
> If rq->idle_stamp is zero, this uses rq_clock(rq) as the sample. That is
> not a valid idle duration and can immediately drive rq->avg_idle to its
> clamp.
>
> This can happen when sched_balance_newidle() returns before setting
> rq->idle_stamp, for example when this_rq->ttwu_pending is set. In that
> case the rq can switch to the idle task with idle_stamp still zero and
> leave idle again when the pending wakeup is processed.
>
> Other paths can also switch to the idle task without setting
> rq->idle_stamp via newidle_balance(), for example find_proxy_task() or
> force-idling.
>
> Restore the idle_stamp validity check in update_rq_avg_idle() and skip
> the rq->avg_idle update when there is no measured idle interval.
>
> Fixes: 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU")
> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
> Acked-by: John Stultz <jstultz@google.com>
> Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org>
> ---
> Temporary tracing under hackbench load confirmed that
> update_rq_avg_idle() can be reached with rq->idle_stamp == 0.
> Hackbench showed no material regression versus v7.2-rc5 mainline.
>
> Related discussion:
>  https://lore.kernel.org/r/20260423023322.1293923-1-firelzrd@gmail.com
>
> This is a narrower variant of the earlier proposal. It keeps the
> rq->idle_stamp guard in update_rq_avg_idle(), but intentionally does not
> stamp idle entry from set_next_task_idle(), preserving the existing
> newidle accounting model and avoiding force-idle/proxy-exec accounting
> concerns.
> ---
> Changes in v3:
>  - Describe the sched_balance_newidle()/ttwu_pending path as an
>    example of entering idle without a valid rq->idle_stamp.
>  - Drop unlikely() from the idle_stamp check.
>  - Add Acked-by from John Stultz.
>
> Link to v2: https://lore.kernel.org/r/20260806-master-v2-1-e1f3a1a0c903@gentwo.org
>
> Changes in v2:
>  - Add Reviewed-by from Prateek.
>  - Mention find_proxy_task() and force-idling as examples of paths that
>    can switch to the idle task without a valid rq->idle_stamp.
>  - Cc John Stultz.
>
> Link to v1: https://lore.kernel.org/r/20260728-master-v1-1-f95d9b0147d2@gentwo.org
> ---
> kernel/sched/core.c | 10 ++++++++--
> 1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 96226707c2f6135341aa779b8262f113e103d8ad..68fa724cd668fb6c4cead329d05fa95e2f1ea5db 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3732,11 +3732,17 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
>
> void update_rq_avg_idle(struct rq *rq)
> {
> -	u64 delta = rq_clock(rq) - rq->idle_stamp;
> -	u64 max = 2*rq->max_idle_balance_cost;
> +	u64 idle_stamp = rq->idle_stamp;
> +	u64 delta, max;
> +
> +	if (!idle_stamp)
> +		return;
> +
> +	delta = rq_clock(rq) - idle_stamp;
>
> 	update_avg(&rq->avg_idle, delta);
>
> +	max = 2 * rq->max_idle_balance_cost;
> 	if (rq->avg_idle > max)
> 		rq->avg_idle = max;
> 	rq->idle_stamp = 0;
>
> ---
> base-commit: 3f008280327ba5ad132965abab0c7846283cef0c
> change-id: 20260728-master-55cd7cc13290
>
> Best regards,
> -- 
> Shubhang Kaushik (Ampere) <sh@gentwo.org>
>

Re: [PATCH v3] sched/core: Skip rq->avg_idle update without a valid idle_stamp

[Zhan Xusheng]

The diff of 4b603f1551a73 is its own evidence, which may be quicker to
look at than the description:

  -	if (rq->idle_stamp) {
  -		u64 delta = rq_clock(rq) - rq->idle_stamp;
  +void update_rq_avg_idle(struct rq *rq)
  +{
  +	u64 delta = rq_clock(rq) - rq->idle_stamp;

The trigger moved on purpose, from the wakeup to idle switch-out.  The
guard came off with it, and nothing about that fails to compile.

Two things about it that the changelog leaves narrower than they are.

rq->idle_stamp has exactly one setter, fair.c:14563, and it sits below the
ttwu_pending bail-out at 14555.  So the reachable set is not the three
examples but any path into idle that misses that one line, and since
update_rq_avg_idle() zeroes the field on the way out, every such entry
starts from zero again.

The sample is not merely invalid, it saturates on the first hit.
update_avg() adds diff/8, so a rq_clock() of ~1e12 on a few minutes of
uptime moves avg_idle by ~1.25e11, against a clamp of
2*max_idle_balance_cost in the 1e4..1e5 range.  Both places that consult
avg_idle to decide whether an idle interval was long enough to be worth
balancing, fair.c:14504 and 14584, then stop holding anything back.  It
only ever adds newidle balancing, never removes it, which fits latency
reports rather than wrong results.

I also went looking for a stale stamp rather than a zero one and did not
find one: 14674 clears it when newidle actually pulled a task, so no idle
follows, and the other write is sched_init().  The field is either zero or
this idle period's rq_clock, which is what the restored guard splits on.

Reviewed-by: Zhan Xusheng <zhanxusheng@xiaomi.com>

Thanks,
Zhan Xusheng

[tip: sched/urgent] sched/core: Skip rq->avg_idle update without a valid idle_stamp

[tip-bot2 for Shubhang Kaushik (Ampere)]

The following commit has been merged into the sched/urgent branch of tip:

Commit-ID:     c6dcd97c8be75f052a1ca52cf79b03e7292962f1
Gitweb:        https://git.kernel.org/tip/c6dcd97c8be75f052a1ca52cf79b03e7292962f1
Author:        Shubhang Kaushik (Ampere) <sh@gentwo.org>
AuthorDate:    Fri, 07 Aug 2026 13:38:52 -07:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Wed, 02 Sep 2026 09:17:49 +02:00

sched/core: Skip rq->avg_idle update without a valid idle_stamp

Commit 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved
to an idle CPU") moved rq->avg_idle accounting out of the wakeup path and
into put_prev_task_idle(), so that the idle interval is consumed whenever
the idle task is switched out.

The wakeup-side accounting that it replaced only updated rq->avg_idle
when rq->idle_stamp was non-zero. The new helper lost that validity
check and unconditionally computes:

	rq_clock(rq) - rq->idle_stamp

If rq->idle_stamp is zero, this uses rq_clock(rq) as the sample. That is
not a valid idle duration and can immediately drive rq->avg_idle to its
clamp.

This can happen when sched_balance_newidle() returns before setting
rq->idle_stamp, for example when this_rq->ttwu_pending is set. In that
case the rq can switch to the idle task with idle_stamp still zero and
leave idle again when the pending wakeup is processed.

Other paths can also switch to the idle task without setting
rq->idle_stamp via newidle_balance(), for example find_proxy_task() or
force-idling.

Restore the idle_stamp validity check in update_rq_avg_idle() and skip
the rq->avg_idle update when there is no measured idle interval.

Fixes: 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU")
Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Acked-by: John Stultz <jstultz@google.com>
Link: https://patch.msgid.link/20260807-master-v3-1-c328354efed3@gentwo.org
---
 kernel/sched/core.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index f782751..7472450 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3742,11 +3742,17 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
 
 void update_rq_avg_idle(struct rq *rq)
 {
-	u64 delta = rq_clock(rq) - rq->idle_stamp;
-	u64 max = 2*rq->max_idle_balance_cost;
+	u64 idle_stamp = rq->idle_stamp;
+	u64 delta, max;
+
+	if (!idle_stamp)
+		return;
+
+	delta = rq_clock(rq) - idle_stamp;
 
 	update_avg(&rq->avg_idle, delta);
 
+	max = 2 * rq->max_idle_balance_cost;
 	if (rq->avg_idle > max)
 		rq->avg_idle = max;
 	rq->idle_stamp = 0;