* Re: [BUG] sched/delayacct: blkio_start recorded on the wrong task when a delayed task is blocked from pick
2026-11-03 6:01 [BUG] sched/delayacct: blkio_start recorded on the wrong task when a delayed task is blocked from pick kiln
2026-11-03 6:10 ` K Prateek Nayak
@ 2026-11-03 8:21 ` Christian Loehle
2026-11-03 9:56 ` Peter Zijlstra
2026-11-03 8:28 ` [PATCH] delayacct: Record blkio start on the correct task Jie Du
2 siblings, 1 reply; 6+ messages in thread
From: Christian Loehle @ 2026-11-03 8:21 UTC (permalink / raw)
To: kiln, Peter Zijlstra, Ingo Molnar, Balbir Singh, Yang Yang
Cc: Vincent Guittot, K Prateek Nayak, linux-kernel
On 11/3/26 06:01, kiln wrote:
> Hi,
>
> I think there's a delay accounting bug that came in with DELAY_DEQUEUE.
> __block_task() calls delayacct_blkio_start(), and that helper always
> writes current->delays->blkio_start. That's fine when __block_task() runs
> from __schedule() for prev, but when the task was delayed and gets
> blocked later from pick_next_task_fair() -> pick_next_entity() ->
> dequeue_entities() -> __block_task(), current is whoever is scheduling on
> that CPU, not p. So p's blkio_start is never set (or is left over from
> an older sleep), and when the I/O completes ttwu() calls
> delayacct_blkio_end(p), which adds local_clock() - blkio_start to p's
> blkio_delay. For a fresh thread blkio_start is 0, so the thread gets
> charged the whole time since boot in one go.
>
> How I ran into it: per-thread blkio delay (taskstats, and field 42 of
> /proc/<pid>/task/<tid>/stat) on some hosts showed threads that were a
> couple of seconds old with ~7,070,000 s of block I/O delay, which was
> exactly the host uptime. Always freshly created threads of a headless
> Chrome (ThreadPoolForeg, Chrome_IOThread, Compositor and so on); Chrome
> creates lots of short-lived threads that soon wait on ext4 journal I/O.
>
> Kernel: a distro 6.6.102 kernel that carries the EEVDF / DELAY_DEQUEUE
> backport (Alibaba Cloud Linux 4, 6.6.102-7.alnx4). I have not reproduced
> it on mainline yet, but as far as I can tell the code there is the same:
> __block_task() in kernel/sched/sched.h still calls delayacct_blkio_start()
> with no task argument, in v6.12 and current master.
>
> What I measured with bpftrace while starting Chrome a few times
> (roughly 70 s per run):
>
> kprobe:dequeue_entities { @in_de[tid] = 1; }
> kretprobe:dequeue_entities { delete(@in_de[tid]); }
> kprobe:__delayacct_blkio_start {
> @start[@in_de[tid] ? "from dequeue_entities" : "from __schedule",
> curtask->in_iowait] = count();
> }
> kprobe:__delayacct_blkio_end {
> $p = (struct task_struct *)arg0;
> if ($p->delays->blkio_start == 0) { @zero[$p->comm] = count(); }
> }
>
> With DELAY_DEQUEUE on (two runs):
> from __schedule, current in_iowait=1: 11081 / 8691
> from __schedule, current in_iowait=0: 0 / 0
> from dequeue_entities, current in_iowait=1: 50 / 61
> from dequeue_entities, current in_iowait=0: 52 / 103
> blkio_end with blkio_start == 0: 3 / 10 (all Chrome threads)
>
> With NO_DELAY_DEQUEUE (two runs, same workload):
> from dequeue_entities: 0 / 2
> blkio_end with blkio_start == 0: 0 / 0
> starts vs ends: 12057 vs 12056, 9612 vs 9610
>
> So on the dequeue_entities path about half of the starts land on a task
> that isn't even in iowait. A kernel stack for one of the zero-start ends:
>
> __delayacct_blkio_end+1
> try_to_wake_up+1509
> wake_bit_function+57
> __wake_up_common+118
> __wake_up_common_lock+129
> wake_up_bit+127
> journal_end_buffer_io_sync+48
> end_bio_bh_io_sync+38
>
> This looks like the mirror image of c96f5471ce7d ("delayacct: Account
> blkio completion on the correct task"), which fixed the end side the same
> way. The obvious fix would be to pass p down, something like
> delayacct_blkio_start(p) setting p->delays->blkio_start (checking
> p->delays rather than current->delays). Happy to send a patch if that
> direction is okay.
>
> Thanks
> kiln
>
This should be fixed by
https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/commit/?h=sched/urgent&id=5d010dd27be92a68eaac3af4f5ef0555bcde6031
(please give it a try!). It being in queue/sched/urgent should mean it
comes in with -rc2?
Peter any reason this isn't in tip yet?
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH] delayacct: Record blkio start on the correct task
2026-11-03 6:01 [BUG] sched/delayacct: blkio_start recorded on the wrong task when a delayed task is blocked from pick kiln
2026-11-03 6:10 ` K Prateek Nayak
2026-11-03 8:21 ` Christian Loehle
@ 2026-11-03 8:28 ` Jie Du
2026-11-03 9:07 ` Jie Du
2 siblings, 1 reply; 6+ messages in thread
From: Jie Du @ 2026-11-03 8:28 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Balbir Singh, Yang Yang, Juri Lelli,
Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Andrew Morton, linux-kernel
__block_task() calls delayacct_blkio_start(), which stamps
current->delays->blkio_start. That is only correct when the task being
blocked is current, as it is when __schedule() blocks prev.
With DELAY_DEQUEUE, a task that blocks while not eligible stays on the
runqueue as sched_delayed. If it gets picked before it is woken,
pick_next_entity() dequeues it and __block_task() runs on behalf of
whatever task is scheduling on that CPU. Proxy execution can also block a
task that is not current. In these cases blkio_start is written to
current, and p keeps whatever blkio_start it had before: the start of an
older sleep, or 0 for a thread that never blocked on I/O. When the I/O
completes, ttwu() calls delayacct_blkio_end(p), which charges p with
local_clock() - blkio_start. For a new thread that is the whole time
since boot. current's own blkio_start, if it is sleeping on I/O itself,
is moved forward at the same time.
This shows up as freshly created threads reporting days of block I/O
delay through taskstats and field 42 (delayacct_blkio_ticks) of
/proc/<pid>/stat, in steps of exactly the system uptime. Anything summing
those numbers per cgroup or per host reports nonsense.
Pass the task down to delayacct_blkio_start(), like commit c96f5471ce7d
("delayacct: Account blkio completion on the correct task") did for the
completion side. __block_task() still sets blkio_start before it clears
p->on_rq, so a concurrent ttwu() cannot reach delayacct_blkio_end(p)
before the start is recorded.
Tested with a small reproducer that keeps one CPU busy with two spinning
threads and keeps creating short-lived threads that spin for 2ms and then
do O_DIRECT reads; a thread is counted as bad when its
delayacct_blkio_ticks is larger than its age. Under QEMU (TCG, 2 vCPUs),
4 rounds of 60s each, bad threads:
v7.4-rc1 v7.4-rc1 + this patch
DELAY_DEQUEUE 20 of 14718 0 of 19383
NO_DELAY_DEQUEUE 0 of 15766 0 of 20020
Each of the 20 was charged roughly the uptime at that point.
On a 6.6-based distribution kernel that carries the EEVDF backport the
same reproducer gives about 690 bad threads out of 2000 in 20s with
DELAY_DEQUEUE and none with NO_DELAY_DEQUEUE.
The problem was investigated, and this patch and its reproducer were
written, with help from an AI coding assistant (Claude). I have tested
it as described above.
Fixes: 152e11f6df29 ("sched/fair: Implement delayed dequeue")
Closes: https://lore.kernel.org/all/tencent_6CFB6E9B26A308DC2800D4059DF93FEC9706@qq.com/
Cc: stable@vger.kernel.org
Signed-off-by: Jie Du <806144828@qq.com>
---
Reproducer used above (gcc -O2 -pthread; run as
./blkio-repro <file or block device of at least 64MB> [seconds]):
// gcc -O2 -pthread -o blkio-repro blkio-repro.c
// ./blkio-repro <file or block device, >= 64MB> [seconds]
#define _GNU_SOURCE
#include <fcntl.h>
#include <pthread.h>
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
static int fd;
static volatile int stop;
static unsigned long nr, bad;
static double now(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec + ts.tv_nsec / 1e9;
}
/* delayacct_blkio_ticks, field 42 of /proc/thread-self/stat */
static unsigned long long blkio_ticks(void)
{
char buf[1024], *p;
FILE *f = fopen("/proc/thread-self/stat", "r");
int i;
if (!f || !fgets(buf, sizeof(buf), f))
exit(1);
fclose(f);
p = strrchr(buf, ')') + 2;
for (i = 3; i < 42; i++)
p = strchr(p, ' ') + 1;
return strtoull(p, NULL, 10);
}
static void *hog(void *arg)
{
while (!stop)
;
return NULL;
}
/* burn 2ms so we are not eligible, then block on synchronous reads */
static void *worker(void *arg)
{
unsigned int seed = (unsigned long)arg;
double born = now();
void *buf;
int i;
if (posix_memalign(&buf, 4096, 4096))
exit(1);
while (now() - born < 0.002)
;
for (i = 0; i < 8; i++)
if (pread(fd, buf, 4096, (rand_r(&seed) % 16384) * 4096L) != 4096)
exit(1);
if (blkio_ticks() > (now() - born) * sysconf(_SC_CLK_TCK) + 1)
bad++;
nr++;
free(buf);
return NULL;
}
int main(int argc, char **argv)
{
double end = now() + (argc > 2 ? atoi(argv[2]) : 30);
pthread_t h[2], t;
unsigned long n = 0;
cpu_set_t set;
fd = open(argv[1], O_RDONLY | O_DIRECT);
if (fd < 0)
return 1;
/* everything on one CPU */
CPU_ZERO(&set);
CPU_SET(0, &set);
sched_setaffinity(0, sizeof(set), &set);
pthread_create(&h[0], NULL, hog, NULL);
pthread_create(&h[1], NULL, hog, NULL);
while (now() < end) {
pthread_create(&t, NULL, worker, (void *)++n);
pthread_join(t, NULL);
}
stop = 1;
printf("threads %lu, blkio delay longer than thread age: %lu\n", nr, bad);
return 0;
}
include/linux/delayacct.h | 10 +++++-----
kernel/delayacct.c | 8 ++++++--
kernel/sched/sched.h | 2 +-
3 files changed, 12 insertions(+), 8 deletions(-)
diff --git a/include/linux/delayacct.h b/include/linux/delayacct.h
index ecb06f16d2..4bb01a1f86 100644
--- a/include/linux/delayacct.h
+++ b/include/linux/delayacct.h
@@ -92,7 +92,7 @@ extern void delayacct_init(void);
extern void __delayacct_tsk_init(struct task_struct *);
extern void __delayacct_tsk_exit(struct task_struct *);
-extern void __delayacct_blkio_start(void);
+void __delayacct_blkio_start(struct task_struct *p);
extern void __delayacct_blkio_end(struct task_struct *);
extern int delayacct_add_tsk(struct taskstats *, struct task_struct *);
extern __u64 __delayacct_blkio_ticks(struct task_struct *);
@@ -126,13 +126,13 @@ static inline void delayacct_tsk_free(struct task_struct *tsk)
tsk->delays = NULL;
}
-static inline void delayacct_blkio_start(void)
+static inline void delayacct_blkio_start(struct task_struct *p)
{
if (!static_branch_unlikely(&delayacct_key))
return;
- if (current->delays)
- __delayacct_blkio_start();
+ if (p->delays)
+ __delayacct_blkio_start(p);
}
static inline void delayacct_blkio_end(struct task_struct *p)
@@ -257,7 +257,7 @@ static inline void delayacct_tsk_init(struct task_struct *tsk)
{}
static inline void delayacct_tsk_free(struct task_struct *tsk)
{}
-static inline void delayacct_blkio_start(void)
+static inline void delayacct_blkio_start(struct task_struct *p)
{}
static inline void delayacct_blkio_end(struct task_struct *p)
{}
diff --git a/kernel/delayacct.c b/kernel/delayacct.c
index 479e860aff..1d735d58e1 100644
--- a/kernel/delayacct.c
+++ b/kernel/delayacct.c
@@ -126,9 +126,13 @@ static void delayacct_end(raw_spinlock_t *lock, u64 *start, u64 *total, u32 *cou
}
}
-void __delayacct_blkio_start(void)
+/*
+ * @p is not necessarily current: a sched_delayed task can be blocked by
+ * another task picking it, see __block_task().
+ */
+void __delayacct_blkio_start(struct task_struct *p)
{
- current->delays->blkio_start = local_clock();
+ p->delays->blkio_start = local_clock();
}
/*
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 6c7c563352..4ca7eac523 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3126,7 +3126,7 @@ static inline void __block_task(struct rq *rq, struct task_struct *p)
if (p->in_iowait) {
atomic_inc(&rq->nr_iowait);
- delayacct_blkio_start();
+ delayacct_blkio_start(p);
}
ASSERT_EXCLUSIVE_WRITER(p->on_rq);
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
^ permalink raw reply [flat|nested] 6+ messages in thread