Re: [PATCH] io_uring: don't sleep on a claimed DEFER_TASKRUN wakeup

[Jens Axboe <axboe@kernel.dk>]

On 10/19/26 9:06 AM, Daniel Byrne wrote:
> A producer that stalls between publishing local task work and
> running the cq_wait_nr wake protocol can claim an arm from a later
> wait cycle than the one its work belongs to. The waiter may already
> have drained that work and re-armed for its next sleep. The claimed
> wake then fires through wake_up_state() while the waiter is still
> TASK_RUNNING, before it has set its task state for the sleep, and
> is a no-op. The arm is spent and nothing restores it in time, so
> every later producer reads a non-positive cq_wait_nr and skips the
> wake. No producer wakes the waiter, which sleeps while the
> completions it waits for pile up on ctx->work_list.
> 
> Three waiters arm cq_wait_nr. io_cqring_wait() and the BPF loop
> waiter io_loop_wait() arm it a few instructions before
> set_current_state(). The third waiter is the cancel wait in
> io_uring_cancel_generic(), where exec waits for all of a task's
> inflight requests and exit waits for its tracked ones, such as
> futex waits. There, io_uring_try_cancel_requests() arms cq_wait_nr
> at the start of each cancel pass. The task state is set only after
> the whole pass, so the window spans the io-wq cancel, the local
> work run and the cancel walkers. That wait has no timeout. A
> stranded exec sleeps until a signal arrives, and a stranded exit of
> the last thread, which no longer receives signals, can sleep for
> good.
> 
> A prototype that busy-polls device completions in TASK_RUNNING from
> io_cqring_wait_schedule() while armed (to be posted separately, as part of
> the io_uring DMA copy offload) widens the io_cqring_wait() window to
> tens of microseconds, and those completions push local work from many
> CPUs, so what is a narrow race in upstream io_cqring_wait() wedged fio
> reliably within seconds there. Every ring was parked in
> io_cqring_wait() with exactly its queue depth of completions stranded
> on ctx->work_list and cq_wait_nr at IO_CQ_WAKE_INIT.
> 
> Add io_cq_wait_arm_spent() and refuse to sleep in all three waiters
> once the arm is spent. io_cqring_wait() runs its wait loop again,
> and io_loop_wait() returns to the loop in __io_run_loop(). Both
> loops drain anything pending and re-arm. The cancel wait runs
> another pass, which re-arms. Each check follows the setting of the
> task state, so a later claim can no longer be lost. The cancel wait
> checks only the rings whose deferred work the task runs, because an
> arm wakes only ctx->submitter_task. wait.h documents the rule next
> to IO_CQ_WAKE_INIT.
> 
> The check relies on nothing else restoring a spent arm before it
> runs. The min_wait timer stores an arm only during the timed sleep,
> which starts after the check and cancels the timer before the
> waiter checks again. io_uring_try_cancel_requests() also arms the
> ring when io_ring_exit_work() runs it or when a task that does not
> own the ring runs it, such as the creator of an R_DISABLED ring
> that another task enabled. Such a store that lands between the
> owner's test for pending local work and its check hides a spent
> arm, and the owner sleeps on work that no later producer wakes it
> for. Arm only in the owner's passes, the ones that also run the
> ring's local work, as the owner is the only task an arm wakes.
> io_ring_exit_work() never sleeps on the arm and runs stranded work
> itself, so it needs no arm.

This should already be fixed by proper ordering in my
for-7.4/io_uring branch - see:

8257f6ec45ee ("io_uring: don't lose the DEFER_TASKRUN wake up")

and

76a82c666abf ("io_uring/cancel: rearm local work wake up before sleep")

Please take a look and see if you agree.

-- 
Jens Axboe