[PATCH 0/1] sched/psi: skip irqtime accounting when no new irq time has elapsed

[PATCH 0/1] sched/psi: skip irqtime accounting when no new irq time has elapsed

[Usama Arif]

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative
counter and only bails out when the delta vs. the previously accounted
amount is negative.  A delta of exactly zero is treated as "do the
work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on
x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor
chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a
context switch crosses a PSI group boundary on a CPU that hasn't
serviced an interrupt between the two switches.

To find out how often this actually fires in the wild, I attached a
bpftrace probe to psi_account_irqtime() on a 176-thread AMD EPYC 9D64
server running an compute intensive workload.
The probe also reads irq_time_read(cpu) directly from the per-CPU
cpu_irqtime variable so it can separate delta == 0 from delta < 0.
The bpftrace script was generated by claude and is at the end
of the coverletter.

Over a 30 s window under steady-state load:

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

15.6 % of all psi_account_irqtime() calls -- and 29.7 % of the calls
that get past the early returns -- hit the delta == 0 case.  delta < 0
did not occur once in the 30 s window.  That works out to ~89 k calls/sec
on this host that today take the full seqcount write + cpu_clock() +
cgroup-chain walk purely to add 0 to every group's PSI_IRQ_FULL counter.

Extend the early-return to also cover delta == 0.  rq->psi_irq_time
does not need updating in that case (it would store the same value
back) and no PSI bucket would change.  The existing behaviour for
delta > 0 is untouched.

--------- psi_delta_exact.bt -------
#!/usr/bin/env bpftrace

#include <linux/sched.h>

kprobe:psi_account_irqtime
{
	$rq = (struct rq *)arg0;
	$curr = (struct task_struct *)arg1;
	$prev = (struct task_struct *)arg2;

	@total = count();

	if ($curr->pid == 0) {
		@ret_curr_swapper = count();
		return;
	}

	if ($prev != 0) {
		$pg_curr = $curr->cgroups->dfl_cgrp;
		$pg_prev = $prev->cgroups->dfl_cgrp;
		if ($pg_curr == $pg_prev) {
			@ret_samegrp = count();
			return;
		}
	}

	@reached_delta = count();

	$pcpu_off  = *(uint64 *)(kaddr("__per_cpu_offset") + cpu * 8);
	$irq_time  = *(uint64 *)(kaddr("cpu_irqtime") + $pcpu_off);
	$prev_time = $rq->psi_irq_time;
	$delta     = (int64)($irq_time - $prev_time);

	if ($delta == 0) {
		@delta_zero = count();
	} else if ($delta > 0) {
		@delta_positive    = count();
	} else {
		@delta_negative = count();
	}
}

interval:s:30 { exit(); }
--------- end bpftrace ---------
 
Usama Arif (1):
  sched/psi: skip irqtime accounting when no new irq time has elapsed

 kernel/sched/psi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.53.0-Meta

[PATCH 1/1] sched/psi: skip irqtime accounting when no new irq time has elapsed

[Usama Arif]

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative
counter and only bails out when the delta vs. the previously accounted
amount is negative.  A delta of exactly zero is treated as "do the
work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on
x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor
chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a
context switch crosses a PSI group boundary on a CPU that hasn't
serviced an interrupt between the two switches.

Measured on a 176-thread AMD EPYC 9D64 server running a compute
intensive production workload, instrumented with bpftrace over a 30s
window (irq_time_read() read directly from the per-CPU cpu_irqtime so
that delta == 0 and delta < 0 could be separated):

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the
calls that get past the early returns - hit the delta == 0 case;
delta < 0 did not occur once in the 30 s window.  Under the current
code each of those ~89 k calls per second performs the full seqcount
write + cpu_clock() read + cgroup-chain walk just to add 0 to every
group's PSI_IRQ_FULL counter.

Extend the early-return to also cover delta == 0.  rq->psi_irq_time
does not need updating in that case (it would store the same value
back) and no PSI bucket would change.  The existing behaviour for
delta > 0 is untouched.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 kernel/sched/psi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index d9c9d9480a45..848955f8893d 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -1023,7 +1023,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st
 
 	irq = irq_time_read(cpu);
 	delta = (s64)(irq - rq->psi_irq_time);
-	if (delta < 0)
+	if (delta <= 0)
 		return;
 	rq->psi_irq_time = irq;
 
-- 
2.53.0-Meta

Re: [PATCH 1/1] sched/psi: skip irqtime accounting when no new irq time has elapsed

[Johannes Weiner]

On Wed, Jun 17, 2026 at 10:50:06AM -0700, Usama Arif wrote:
> psi_account_irqtime() reads irq_time_read() into a per-rq cumulative
> counter and only bails out when the delta vs. the previously accounted
> amount is negative.  A delta of exactly zero is treated as "do the
> work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on
> x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor
> chain is walked to add zero to every group's PSI_IRQ_FULL bucket.
> 
> The zero-delta case is common in practice -- it fires every time a
> context switch crosses a PSI group boundary on a CPU that hasn't
> serviced an interrupt between the two switches.
> 
> Measured on a 176-thread AMD EPYC 9D64 server running a compute
> intensive production workload, instrumented with bpftrace over a 30s
> window (irq_time_read() read directly from the per-CPU cpu_irqtime so
> that delta == 0 and delta < 0 could be separated):
> 
>   @total              17,229,311    (100.0%)
>   @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
>   @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
>   @reached_delta       9,041,817    ( 52.5%)
>     @delta_positive    6,358,192    ( 36.9%)  real work
>     @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
>     @delta_negative          (0)    (  0.0%)  monotonic clock
> 
> So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the
> calls that get past the early returns - hit the delta == 0 case;
> delta < 0 did not occur once in the 30 s window.  Under the current
> code each of those ~89 k calls per second performs the full seqcount
> write + cpu_clock() read + cgroup-chain walk just to add 0 to every
> group's PSI_IRQ_FULL counter.
> 
> Extend the early-return to also cover delta == 0.  rq->psi_irq_time
> does not need updating in that case (it would store the same value
> back) and no PSI bucket would change.  The existing behaviour for
> delta > 0 is untouched.
> 
> Signed-off-by: Usama Arif <usama.arif@linux.dev>

Acked-by: Johannes Weiner <hannes@cmpxchg.org>

CCing Chengming as well, quote untrimmed.

> ---
>  kernel/sched/psi.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
> index d9c9d9480a45..848955f8893d 100644
> --- a/kernel/sched/psi.c
> +++ b/kernel/sched/psi.c
> @@ -1023,7 +1023,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st
>  
>  	irq = irq_time_read(cpu);
>  	delta = (s64)(irq - rq->psi_irq_time);
> -	if (delta < 0)
> +	if (delta <= 0)
>  		return;
>  	rq->psi_irq_time = irq;
>  
> -- 
> 2.53.0-Meta
> 

Re: [PATCH 1/1] sched/psi: skip irqtime accounting when no new irq time has elapsed

[Shakeel Butt]

On Wed, Jun 17, 2026 at 10:50:06AM -0700, Usama Arif wrote:
> psi_account_irqtime() reads irq_time_read() into a per-rq cumulative
> counter and only bails out when the delta vs. the previously accounted
> amount is negative.  A delta of exactly zero is treated as "do the
> work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on
> x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor
> chain is walked to add zero to every group's PSI_IRQ_FULL bucket.
> 
> The zero-delta case is common in practice -- it fires every time a
> context switch crosses a PSI group boundary on a CPU that hasn't
> serviced an interrupt between the two switches.
> 
> Measured on a 176-thread AMD EPYC 9D64 server running a compute
> intensive production workload, instrumented with bpftrace over a 30s
> window (irq_time_read() read directly from the per-CPU cpu_irqtime so
> that delta == 0 and delta < 0 could be separated):
> 
>   @total              17,229,311    (100.0%)
>   @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
>   @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
>   @reached_delta       9,041,817    ( 52.5%)
>     @delta_positive    6,358,192    ( 36.9%)  real work
>     @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
>     @delta_negative          (0)    (  0.0%)  monotonic clock
> 
> So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the
> calls that get past the early returns - hit the delta == 0 case;
> delta < 0 did not occur once in the 30 s window.  Under the current
> code each of those ~89 k calls per second performs the full seqcount
> write + cpu_clock() read + cgroup-chain walk just to add 0 to every
> group's PSI_IRQ_FULL counter.
> 
> Extend the early-return to also cover delta == 0.  rq->psi_irq_time
> does not need updating in that case (it would store the same value
> back) and no PSI bucket would change.  The existing behaviour for
> delta > 0 is untouched.
> 
> Signed-off-by: Usama Arif <usama.arif@linux.dev>

Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>

[tip: sched/core] sched/psi: skip irqtime accounting when no new irq time has elapsed

[tip-bot2 for Usama Arif]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     b0bc1c75f304099347cfc0cc880b556966b54138
Gitweb:        https://git.kernel.org/tip/b0bc1c75f304099347cfc0cc880b556966b54138
Author:        Usama Arif <usama.arif@linux.dev>
AuthorDate:    Wed, 17 Jun 2026 10:50:06 -07:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 30 Jun 2026 10:56:56 +02:00

sched/psi: skip irqtime accounting when no new irq time has elapsed

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative
counter and only bails out when the delta vs. the previously accounted
amount is negative.  A delta of exactly zero is treated as "do the
work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on
x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor
chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a
context switch crosses a PSI group boundary on a CPU that hasn't
serviced an interrupt between the two switches.

Measured on a 176-thread AMD EPYC 9D64 server running a compute
intensive production workload, instrumented with bpftrace over a 30s
window (irq_time_read() read directly from the per-CPU cpu_irqtime so
that delta == 0 and delta < 0 could be separated):

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the
calls that get past the early returns - hit the delta == 0 case;
delta < 0 did not occur once in the 30 s window.  Under the current
code each of those ~89 k calls per second performs the full seqcount
write + cpu_clock() read + cgroup-chain walk just to add 0 to every
group's PSI_IRQ_FULL counter.

Extend the early-return to also cover delta == 0.  rq->psi_irq_time
does not need updating in that case (it would store the same value
back) and no PSI bucket would change.  The existing behaviour for
delta > 0 is untouched.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Link: https://patch.msgid.link/20260617175219.2494857-2-usama.arif@linux.dev
---
 kernel/sched/psi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index d9c9d94..848955f 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -1023,7 +1023,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st
 
 	irq = irq_time_read(cpu);
 	delta = (s64)(irq - rq->psi_irq_time);
-	if (delta < 0)
+	if (delta <= 0)
 		return;
 	rq->psi_irq_time = irq;