[PATCH] io_uring: clear IO_RING_F_DRAIN_ACTIVE on drain completion flush

[PATCH] io_uring: clear IO_RING_F_DRAIN_ACTIVE on drain completion flush

[Wei Gao]

When running a test that repeatedly submits requests followed by a
drained request across iterations on the same ring (such as liburing's
test/multicqes_drain.t), the test occasionally deadlocks and times out
waiting for completions (-ETIME).

Test mechanism (multicqes_drain.t):
The test executes the following sequence in each iteration on the same
ring:
1. Submits 2 poll requests on pipes:
   - sqe[0]: multishot poll on pipe1 (user_data = 10 / 0xa)
   - sqe[1]: oneshot poll on pipe2 (user_data = 20 / 0x14)
2. Triggers events via trigger_event():
   - Writes "foo" to pipe1 twice, sleeps 1ms, and reads "foo" back.
     Generates 2 CQEs for multishot poll #10 (IORING_CQE_F_MORE).
   - Writes "foo" to pipe2 once, sleeps 1ms, and reads "foo" back.
     Generates 1 CQE for oneshot poll #20.
3. Submits poll_remove and a drained NOP:
   - sqe[0]: poll_remove #10 (user_data = 30 / 0x1e)
   - sqe[1]: NOP with IOSQE_IO_DRAIN (user_data = 40 / 0x28)
   Produces 1 CQE for poll_remove (ret=0), 1 CQE for multishot
   cancel (ret=-ECANCELED), and 1 CQE for the drained NOP.
4. Harvests all 6 expected CQEs with io_uring_wait_cqe_timeout().
5. Loops back to step 1 on the same ring for the next iteration.

Why the failure occurs:
Prior to commit 8fb7aee05591 ("io_uring: drain based on allocates reqs"),
io_queue_deferred() checked CQ sequence numbers:
(seq + ctx->cq_extra != ctx->cached_cq_tail). When the prior drain request
posted its CQE, sequence numbers matched immediately. Even if
IO_RING_F_DRAIN_ACTIVE was not cleared upon completion, the next
iteration's non-drain request dequeued right away, emptied
ctx->defer_list, and cleared the flag via io_drain_req():

    if (!drain && list_empty(&ctx->defer_list))
        ctx->int_flags &= ~IO_RING_F_DRAIN_ACTIVE;

However, commit 8fb7aee05591 ("io_uring: drain based on allocates reqs")
changed the defer check:

-   if ((drain_seen || first) && io_drain_defer_seq(de->req, de->seq))
-       break;
+   if ((drain_seen || first) && ctx->nr_req_allocated != ctx->nr_drained)
+       return;

This change severed the self-healing mechanism:
1. Originally, the check verified whether all CQEs had been posted
   (seq + cq_extra == cached_cq_tail, e.g. 6 == 6). Even if the io-wq
   worker thread was delayed in freeing request memory, cached_cq_tail had
   already advanced, allowing the request to dequeue and self-heal.
2. The commit changed the check to whether slab memory has been freed
   (nr_req_allocated != nr_drained). Drained requests run via io-wq
   (due to forced async) and have an asynchronous refcount teardown: the
   CQE is reaped by userspace, but the io-wq worker thread drops its
   reference slightly later via io_wq_free_work() -> io_free_req().
3. If iteration N+1 submits before io_wq_free_work() runs, the allocation
   counter still accounts for the prior NOP (allocated=2, drained=1).
4. Seeing 2 != 1, io_queue_deferred() aborts early via return. This
   completely bypasses list_del_init() and the self-healing clearance in
   io_drain_req(), stranding the new poll requests on ctx->defer_list.
5. While polls are stranded on ctx->defer_list, userspace writes to
   and reads from the pipe. When polls are finally released later, the
   pipe data is already gone. The multishot poll permanently misses the
   event, producing fewer CQEs and causing a timeout (-ETIME).

Trace comparison (Passing vs Failing iteration):

Kprobes registered for tracing:
  p:drain_check io_queue_deferred+0xd7 \
      allocated=+0x464:u32 drained=+0x460:u32
  p:drain_blocked io_queue_deferred+0xeb \
      allocated=+0x464:u32 drained=+0x460:u32
  p:drain_dequeued io_queue_deferred+0x88 drained=+0x460:u32
  p:free_work io_wq_free_work work=%di
  p:flush_compl __io_submit_flush_completions \
      allocated=+0x464:u32 int_flags=+0x4:u32

[Passing iteration - iou-wrk frees slab before next submission]:
1. Prior drained NOP completes, slab memory freed in time (1 == 1):
   400.411558: flush_compl: allocated=1 int_flags=208
   400.411558: io_uring_complete: user_data 0x28, result 0, cflags 0x0
2. New poll requests enter io_drain_req(); check matches (1 == 1):
   400.411563: drain_check: allocated=1 drained=1
   400.411564: drain_dequeued: drained=1
   400.411566: free_work: work freed
3. Polls arm on pipes immediately before userspace writes; events caught:
   400.412638: io_uring_complete: user_data 0xa, result 1, cflags 0x2
   400.412638: flush_compl: allocated=9 int_flags=192
   400.413703: io_uring_complete: user_data 0xa, result 1, cflags 0x2
   400.413703: flush_compl: allocated=9 int_flags=192
   400.414768: flush_compl: allocated=9 int_flags=192
   400.414768: io_uring_complete: user_data 0x14, result 1, cflags 0x0
4. poll_remove and drained NOP submitted; drained NOP waits:
   400.414769: drain_check: allocated=3 drained=1
   400.414770: drain_blocked: allocated=3 drained=1
   400.414770: flush_compl: allocated=3 int_flags=208
   400.414770: io_uring_complete: user_data 0x1e, result 0, cflags 0x0
   400.414770: drain_check: allocated=2 drained=1
   400.414770: drain_blocked: allocated=2 drained=1
   400.414771: flush_compl: allocated=2 int_flags=208
   400.414771: io_uring_complete: user_data 0xa, result -125, cflags 0x0
5. All prior requests done (1 == 1); drained NOP dequeues and completes:
   400.414771: drain_check: allocated=1 drained=1
   400.414771: drain_dequeued: drained=1
   400.414773: free_work: work freed
   400.414774: flush_compl: allocated=1 int_flags=208
   400.414774: io_uring_complete: user_data 0x28, result 0, cflags 0x0
   => All 6 CQEs reaped successfully (PASS).

[Failing iteration - iou-wrk delayed by 15 us -> FAIL]:
1. Iteration N NOP completes, but iou-wrk free_work has not run yet:
   400.424650: flush_compl: allocated=1 int_flags=208
   400.424650: io_uring_complete: user_data 0x28, result 0, cflags 0x0
2. 10 us later, Iteration N+1 submits sqe[0] (POLL_ADD) and sqe[1].
   Because DRAIN_ACTIVE was not cleared, requests enter io_drain_req().
   Because slab memory for the prior NOP is not freed yet, counters
   mismatch (2 != 1):
   400.424660: drain_check: allocated=2 drained=1
   400.424661: drain_blocked: allocated=2 drained=1
   400.424662: drain_check: allocated=3 drained=2
   400.424662: drain_blocked: allocated=3 drained=2
3. 5 us later, iou-wrk finally frees old NOP (polls already blocked):
   400.424665: free_work: work freed
4. While polls are trapped on defer_list, userspace writes and reads pipe.
   When polls are finally released 1ms later, the pipe is already empty:
   400.425726: flush_compl: allocated=3 int_flags=208
   400.425727: drain_check: allocated=2 drained=2
   400.425727: drain_dequeued: drained=2
   400.425727: drain_dequeued: drained=1
5. Pipe event permanently missed; 1.0s later, wait_cqe times out (-ETIME):
   401.428202: flush_compl: allocated=3 int_flags=208
   401.428203: io_uring_complete: user_data 0xa, result -125, cflags 0x0
   401.428207: drain_check: allocated=2 drained=2
   401.428208: drain_dequeued: drained=2
   401.428210: drain_blocked: allocated=2 drained=1
   401.428213: flush_compl: allocated=2 int_flags=208
   401.428213: io_uring_complete: user_data 0x1e, result -14, cflags 0x0
   401.428213: io_uring_complete: user_data 0x28, result -125, cflags 0x0
   => wait_cqe timed out: ret=-62 (-ETIME)

Fix:
Make io_queue_deferred() return a bool indicating whether any drain
request was active or deferred. In __io_submit_flush_completions(), track
whether any completing request had REQ_F_IO_DRAIN. Clear
IO_RING_F_DRAIN_ACTIVE only when a drained request has completed
(has_drain), no drain was active or deferred (!drain_dispatched), and
ctx->defer_list is empty. This prevents premature clearance if an
unrelated completion flushes while a drain request is still executing
in io-wq.

All 10,000 iterations of reproduce_drain and liburing multicqes_drain.t
pass reliably with this fix.

Fixes: 8fb7aee05591 ("io_uring: drain based on allocates reqs")
Signed-off-by: Wei Gao <wegao@suse.com>
---
 io_uring/io_uring.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index cf5218d100d9..4f7b054e6f97 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -445,7 +445,7 @@ unsigned io_linked_nr(struct io_kiocb *req)
 	return nr;
 }
 
-static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx)
+static __cold noinline bool io_queue_deferred(struct io_ring_ctx *ctx)
 {
 	bool drain_seen = false, first = true;
 
@@ -458,7 +458,7 @@ static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx)
 
 		drain_seen |= de->req->flags & REQ_F_IO_DRAIN;
 		if ((drain_seen || first) && ctx->nr_req_allocated != ctx->nr_drained)
-			return;
+			return true;
 
 		list_del_init(&de->list);
 		ctx->nr_drained -= io_linked_nr(de->req);
@@ -466,6 +466,8 @@ static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx)
 		kfree(de);
 		first = false;
 	}
+
+	return drain_seen;
 }
 
 void __io_commit_cqring_flush(struct io_ring_ctx *ctx)
@@ -1131,12 +1133,16 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx)
 {
 	struct io_submit_state *state = &ctx->submit_state;
 	struct io_wq_work_node *node;
+	bool has_drain = false;
 
 	__io_cq_lock(ctx);
 	__wq_list_for_each(node, &state->compl_reqs) {
 		struct io_kiocb *req = container_of(node, struct io_kiocb,
 					    comp_list);
 
+		if (req->flags & REQ_F_IO_DRAIN)
+			has_drain = true;
+
 		/*
 		 * Requests marked with REQUEUE should not post a CQE, they
 		 * will go through the io-wq retry machinery and post one
@@ -1157,8 +1163,12 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx)
 		INIT_WQ_LIST(&state->compl_reqs);
 	}
 
-	if (unlikely(ctx->int_flags & IO_RING_F_DRAIN_ACTIVE))
-		io_queue_deferred(ctx);
+	if (unlikely(ctx->int_flags & IO_RING_F_DRAIN_ACTIVE)) {
+		bool drain_dispatched = io_queue_deferred(ctx);
+
+		if (has_drain && !drain_dispatched && list_empty(&ctx->defer_list))
+			ctx->int_flags &= ~IO_RING_F_DRAIN_ACTIVE;
+	}
 
 	ctx->submit_state.cq_flush = false;
 }
-- 
2.55.0

Re: [PATCH] io_uring: clear IO_RING_F_DRAIN_ACTIVE on drain completion flush

[Jens Axboe]

On 10/18/26 1:01 AM, Wei Gao wrote:
> Fix:
> Make io_queue_deferred() return a bool indicating whether any drain
> request was active or deferred. In __io_submit_flush_completions(), track
> whether any completing request had REQ_F_IO_DRAIN. Clear
> IO_RING_F_DRAIN_ACTIVE only when a drained request has completed
> (has_drain), no drain was active or deferred (!drain_dispatched), and
> ctx->defer_list is empty. This prevents premature clearance if an
> unrelated completion flushes while a drain request is still executing
> in io-wq.

Didn't quite feel right to me, below is with some LLM assistance
to try and get to the bottom of it.

Thanks for digging into this. I don't think it's just a race with a
slow io-wq free, though. DRAIN_ACTIVE is still set when the next request gets initialized, so it's forced async and goes through the defer list
every time, even if the drain request is freed in time.een read.

Basing the clear off any REQ_F_IO_DRAIN request in the flush is not
enough:

- A drain request marked REQ_F_REISSUE counts as has_drain, but
  io_free_batch_list() punts it back to io-wq in that same flush. With
  an empty defer list, we clear DRAIN_ACTIVE while the drain is still
  pending.

- A second IOSQE_IO_DRAIN request that fails init after io_init_drain()
  has run gets completed via io_req_defer_failed() right away. If that
  happens while an earlier drain is still running in io-wq, we end the
  drain, and the next request is issued before the earlier drain
  completes.

It also only works if the drain request completes through the flush.
Without DEFER_TASKRUN, io-wq posts the CQE directly from
io_req_complete_post(), so the application can see the CQE and submit
before anything clears the flag.

A drain request dispatched from the defer list always runs alone, so
we can just remember which one it is and end the drain when its CQE is
flushed, and have io-wq complete drain requests via task_work so that
happens under ->uring_lock. Something like the below. Can you give it
a spin with your reproducer?

diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h
index 7c5d99a8f75d..2e5a786f2a2a 100644
--- a/include/linux/io_uring_types.h
+++ b/include/linux/io_uring_types.h
@@ -524,6 +524,8 @@ struct io_ring_ctx {
 	atomic_t			poll_wq_rcu_pending;
 	struct list_head		defer_list;
 	unsigned			nr_drained;
+	/* drain request dispatched from ->defer_list, until it completes */
+	struct io_kiocb			*drain_req;
 
 	/* protected by ->completion_lock */
 	unsigned			nr_req_allocated;
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index e6569c204917..8ffe0c9a4e29 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -480,6 +480,9 @@ static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx)
 
 	lockdep_assert_held(&ctx->uring_lock);
 	__io_req_caches_free(ctx);
+	/* nothing outside of ->defer_list is alive */
+	if (ctx->nr_req_allocated == ctx->nr_drained)
+		ctx->drain_req = NULL;
 
 	while (!list_empty(&ctx->defer_list)) {
 		struct io_defer_entry *de = list_first_entry(&ctx->defer_list,
@@ -491,6 +494,9 @@ static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx)
 
 		list_del_init(&de->list);
 		ctx->nr_drained -= io_linked_nr(de->req);
+		/* it runs alone, track its completion unless it's a link */
+		if ((de->req->flags & REQ_F_IO_DRAIN) && !de->req->link)
+			ctx->drain_req = de->req;
 		io_req_task_queue(de->req);
 		kfree(de);
 		first = false;
@@ -980,9 +986,11 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags)
 
 	/*
 	 * Handle special CQ sync cases via task_work. DEFER_TASKRUN requires
-	 * the submitter task context, IOPOLL protects with uring_lock.
+	 * the submitter task context, IOPOLL protects with uring_lock. A drain
+	 * request ends the drain under uring_lock as its CQE is posted.
 	 */
-	if ((ctx->int_flags & IO_RING_F_LOCKLESS_CQ) || (req->flags & REQ_F_REISSUE)) {
+	if ((ctx->int_flags & IO_RING_F_LOCKLESS_CQ) ||
+	    (req->flags & (REQ_F_REISSUE | REQ_F_IO_DRAIN))) {
 defer_complete:
 		req->io_task_work.func = io_req_task_complete;
 		io_req_task_work_add(req);
@@ -1185,11 +1193,33 @@ static void io_free_batch_list(struct io_ring_ctx *ctx,
 	} while (node);
 }
 
+/*
+ * The dispatched drain request ran alone, and anything submitted after it was
+ * deferred. Once it completed, new requests may run if nothing is waiting.
+ */
+static __cold bool io_drain_req_completed(struct io_ring_ctx *ctx)
+{
+	struct io_wq_work_node *node;
+
+	__wq_list_for_each(node, &ctx->submit_state.compl_reqs) {
+		struct io_kiocb *req = container_of(node, struct io_kiocb,
+					    comp_list);
+
+		/* a requeued request hasn't completed */
+		if (req == ctx->drain_req && !(req->flags & REQ_F_REISSUE)) {
+			ctx->drain_req = NULL;
+			return true;
+		}
+	}
+	return false;
+}
+
 void __io_submit_flush_completions(struct io_ring_ctx *ctx)
 	__must_hold(&ctx->uring_lock)
 {
 	struct io_submit_state *state = &ctx->submit_state;
 	struct io_wq_work_node *node;
+	bool drain_done = false;
 
 	/*
 	 * Drop the files before the CQEs are posted, so they're released by
@@ -1224,13 +1254,19 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx)
 	}
 	__io_cq_unlock_post(ctx);
 
+	if (unlikely(ctx->int_flags & IO_RING_F_DRAIN_ACTIVE) && ctx->drain_req)
+		drain_done = io_drain_req_completed(ctx);
+
 	if (!wq_list_empty(&state->compl_reqs)) {
 		io_free_batch_list(ctx, state->compl_reqs.first);
 		INIT_WQ_LIST(&state->compl_reqs);
 	}
 
-	if (unlikely(ctx->int_flags & IO_RING_F_DRAIN_ACTIVE))
+	if (unlikely(ctx->int_flags & IO_RING_F_DRAIN_ACTIVE)) {
 		io_queue_deferred(ctx);
+		if (drain_done && list_empty(&ctx->defer_list))
+			ctx->int_flags &= ~IO_RING_F_DRAIN_ACTIVE;
+	}
 
 	ctx->submit_state.cq_flush = false;
 }

-- 
Jens Axboe

Re: [PATCH] io_uring: clear IO_RING_F_DRAIN_ACTIVE on drain completion flush

[Wei Gao]

On Mon, Oct 19, 2026 at 02:50:16PM -0600, Jens Axboe wrote:
> On 10/18/26 1:01 AM, Wei Gao wrote:
> > Fix:
> > Make io_queue_deferred() return a bool indicating whether any drain
> > request was active or deferred. In __io_submit_flush_completions(), track
> > whether any completing request had REQ_F_IO_DRAIN. Clear
> > IO_RING_F_DRAIN_ACTIVE only when a drained request has completed
> > (has_drain), no drain was active or deferred (!drain_dispatched), and
> > ctx->defer_list is empty. This prevents premature clearance if an
> > unrelated completion flushes while a drain request is still executing
> > in io-wq.
> 
> Didn't quite feel right to me, below is with some LLM assistance
> to try and get to the bottom of it.
> 
> Thanks for digging into this. I don't think it's just a race with a
> slow io-wq free, though. DRAIN_ACTIVE is still set when the next request gets initialized, so it's forced async and goes through the defer list
> every time, even if the drain request is freed in time.een read.
> 
> Basing the clear off any REQ_F_IO_DRAIN request in the flush is not
> enough:
> 
> - A drain request marked REQ_F_REISSUE counts as has_drain, but
>   io_free_batch_list() punts it back to io-wq in that same flush. With
>   an empty defer list, we clear DRAIN_ACTIVE while the drain is still
>   pending.
> 
> - A second IOSQE_IO_DRAIN request that fails init after io_init_drain()
>   has run gets completed via io_req_defer_failed() right away. If that
>   happens while an earlier drain is still running in io-wq, we end the
>   drain, and the next request is issued before the earlier drain
>   completes.
> 
> It also only works if the drain request completes through the flush.
> Without DEFER_TASKRUN, io-wq posts the CQE directly from
> io_req_complete_post(), so the application can see the CQE and submit
> before anything clears the flag.
> 
> A drain request dispatched from the defer list always runs alone, so
> we can just remember which one it is and end the drain when its CQE is
> flushed, and have io-wq complete drain requests via task_work so that
> happens under ->uring_lock. Something like the below. Can you give it
> a spin with your reproducer?
Thanks a lot for the thorough explanation and pointing out the edge cases.
Your proposed fix works as expected. I ran 10,000 iterations of test with 
your patch applied, and all tests passed without hitting the timeout.

Feel free to add:
Tested-by: Wei Gao wegao@suse.com
> 
> 
> -- 
> Jens Axboe

Re: [PATCH] io_uring: clear IO_RING_F_DRAIN_ACTIVE on drain completion flush

[Jens Axboe]

On 10/19/26 8:46 PM, Wei Gao wrote:
> On Mon, Oct 19, 2026 at 02:50:16PM -0600, Jens Axboe wrote:
>> On 10/18/26 1:01 AM, Wei Gao wrote:
>>> Fix:
>>> Make io_queue_deferred() return a bool indicating whether any drain
>>> request was active or deferred. In __io_submit_flush_completions(), track
>>> whether any completing request had REQ_F_IO_DRAIN. Clear
>>> IO_RING_F_DRAIN_ACTIVE only when a drained request has completed
>>> (has_drain), no drain was active or deferred (!drain_dispatched), and
>>> ctx->defer_list is empty. This prevents premature clearance if an
>>> unrelated completion flushes while a drain request is still executing
>>> in io-wq.
>>
>> Didn't quite feel right to me, below is with some LLM assistance
>> to try and get to the bottom of it.
>>
>> Thanks for digging into this. I don't think it's just a race with a
>> slow io-wq free, though. DRAIN_ACTIVE is still set when the next request gets initialized, so it's forced async and goes through the defer list
>> every time, even if the drain request is freed in time.een read.
>>
>> Basing the clear off any REQ_F_IO_DRAIN request in the flush is not
>> enough:
>>
>> - A drain request marked REQ_F_REISSUE counts as has_drain, but
>>   io_free_batch_list() punts it back to io-wq in that same flush. With
>>   an empty defer list, we clear DRAIN_ACTIVE while the drain is still
>>   pending.
>>
>> - A second IOSQE_IO_DRAIN request that fails init after io_init_drain()
>>   has run gets completed via io_req_defer_failed() right away. If that
>>   happens while an earlier drain is still running in io-wq, we end the
>>   drain, and the next request is issued before the earlier drain
>>   completes.
>>
>> It also only works if the drain request completes through the flush.
>> Without DEFER_TASKRUN, io-wq posts the CQE directly from
>> io_req_complete_post(), so the application can see the CQE and submit
>> before anything clears the flag.
>>
>> A drain request dispatched from the defer list always runs alone, so
>> we can just remember which one it is and end the drain when its CQE is
>> flushed, and have io-wq complete drain requests via task_work so that
>> happens under ->uring_lock. Something like the below. Can you give it
>> a spin with your reproducer?
> Thanks a lot for the thorough explanation and pointing out the edge cases.
> Your proposed fix works as expected. I ran 10,000 iterations of test with 
> your patch applied, and all tests passed without hitting the timeout.
> 
> Feel free to add:
> Tested-by: Wei Gao wegao@suse.com

Great thanks, done. Thanks for the report and initial patch
and the testing!

-- 
Jens Axboe