[PATCH 0/2] io_uring: add schedule points to request freeing loops

[PATCH 0/2] io_uring: add schedule points to request freeing loops

[Yuanhe Shu]

An I/O stress test on a production v6.6-based debug kernel (stress-ng -a 1
--class io -t 7h on a 384-CPU machine) hit the soft lockup watchdog
twice.  The kernel was built with KASAN, kmemleak, lockdep and
CONFIG_PREEMPT_NONE=y.  Both lockups occurred while a CPU was busy
in an io_uring request freeing loop that has no scheduling point:

1) A stress-ng worker looped in io_free_batch_list(), reached from
   exit-time task_work, until the machine panicked on "softlockup:
   hung tasks".

2) A completion fallback work item held uring_lock for 671 seconds
   in that same free loop, until the watchdog fired again
   ("BUG: soft lockup - CPU#61 stuck for 671s").

The field kernel is v6.6-based, so to check whether the same
weakness exists in current mainline, I tried to reproduce the
pattern on v7.3-rc7 on a 192-CPU debug kernel (KASAN, kmemleak,
lockdep; preempt=none - see the note below), with a test
that submits 32K requests per ring and then kills the rings.  I
don't know whether real workloads hit these exact loops on
mainline as hard as our test does, but the loops clearly have the
same unbounded shape:

- A single io_free_batch_list() call freeing a 32K completion batch
  took 443ms with no scheduling point in between.

- Killing 48 rings at once made one __io_req_caches_free() call
  drain a single cache for up to 13 seconds, with 8 kworkers each
  stuck for over 12 seconds; a separate run with 96 rings produced
  97 soft lockup warnings during teardown (the stack is in patch 2).

With this series applied, the same 96-ring run produces zero
kworker soft lockup warnings: all 97 drain-path warnings are gone.
This series targets the request freeing and cache draining paths;
the submission loop itself may need a separate treatment and is
intentionally left out of this series.

The free loops are the remaining gap: the CQ overflow flush already
drops the lock and reschedules (eac2ca2d682f), and io_add_buffers(),
io_remove_buffers_legacy(), the fdinfo dump loops and the tctx
teardown all iterate with a cond_resched() in the loop.  These two
request freeing loops have none.  The debug options amplify the
per-request cost, but the loops are otherwise bounded only by the
number of completed or cached requests.  No functional change is
intended; the patches only add scheduling points to long loops,
and cond_resched() only invokes the scheduler if need_resched()
is set.  Unlike the overflow flush resched point, these do not drop
the lock or leave any critical section, so they introduce no
reordering concerns.

Note: the mainline test was booted with preempt=none to match
the production kernel above.  On x86 this required a local Kconfig
adjustment to disable PREEMPT_LAZY; this is a test-environment
detail and does not change the fix.  cond_resched() is the standard
schedule point on PREEMPT_NONE/VOLUNTARY kernels.

Tested with CONFIG_DEBUG_ATOMIC_SLEEP and CONFIG_PROVE_LOCKING
enabled; no might_sleep or scheduling-while-atomic warnings during
the reproduction runs.

Yuanhe Shu (2):
  io_uring: add a schedule point in io_free_batch_list()
  io_uring: add a schedule point in __io_req_caches_free()

 io_uring/io_uring.c | 2 ++
 1 file changed, 2 insertions(+)

base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
-- 
2.43.7

[PATCH 1/2] io_uring: add a schedule point in io_free_batch_list()

[Yuanhe Shu]

Completed requests accumulate in submit_state.compl_reqs - deferred
completions queued by task_work handlers via io_req_complete_defer(),
plus everything io_do_iopoll() reaps - and are then all freed in one
io_free_batch_list() pass under ctx->uring_lock.  The loop has no
scheduling point.

Each iteration does real work: io_clean_op() freeing request
payloads, file, task and rsrc puts.  With debug features like KASAN
or kmemleak enabled, that work gets expensive enough that a large
batch can hold the CPU well beyond the soft lockup threshold.

Caught by io stress testing (stress-ng -a 1 --class io -t 7h) on a
384 CPU machine running a v6.6-based debug kernel (KASAN, kmemleak,
lockdep; CONFIG_PREEMPT_NONE=y): a stress-ng worker was looping
here past the watchdog threshold, entered from exit-time task_work,
until the machine panicked on "softlockup: hung tasks".  Frames
from the dump:

    io_clean_op
    io_free_batch_list
    __io_submit_flush_completions
    ctx_flush_and_put
    tctx_task_work
    task_work_run
    get_signal

The same test later caught a completion fallback work item holding
uring_lock for 671 seconds in this same loop ("BUG: soft lockup -
CPU#61 stuck for 671s").

Also measured on mainline v7.3-rc7 (192-CPU debug kernel with the
same debug features): a single pass freeing a 32K completion batch
took 443ms with no scheduling point in between.

Insert a cond_resched() in the loop, like other potentially long
io_uring loops already have.  The loop runs in process context with
ctx->uring_lock held, which is a mutex, so scheduling is allowed.

Assisted-by: LLM
Signed-off-by: Yuanhe Shu <xiangzao@linux.alibaba.com>
---
 io_uring/io_uring.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index cf5218d100d9..26cd91a91aa2 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1123,6 +1123,7 @@ static void io_free_batch_list(struct io_ring_ctx *ctx,
 
 		node = req->comp_list.next;
 		io_req_add_to_cache(req, ctx);
+		cond_resched();
 	} while (node);
 }
 
-- 
2.43.7

[PATCH 2/2] io_uring: add a schedule point in __io_req_caches_free()

[Yuanhe Shu]

The request cache can grow to the workload's in-flight high-water
mark.  __io_req_caches_free() drains it with one
kmem_cache_free() per request under ctx->uring_lock, with no
scheduling point in between.  With debug features like KASAN or
kmemleak enabled, each free gets expensive, and draining a large
cache can hold the CPU well beyond the soft lockup threshold.

Reproduced on mainline v7.3-rc7 on a 192-CPU debug kernel
(preempt=none), with a test that submits 32K requests per
ring and then kills the rings.  Killing 48 rings at once made one
drain take up to 13 seconds with no scheduling point, with 8
kworkers each stuck for over 12 seconds; a separate run with 96
rings produced 97 soft lockup warnings during teardown, of the
form:

  watchdog: BUG: soft lockup - CPU#2 stuck for 12s! [kworker/u769:5]
  Workqueue: iou_exit io_ring_exit_work
  Call Trace:
    stack_trace_save
    kasan_save_stack
    kmem_cache_free+0x3cf/0x6a0
    __io_req_caches_free+0x196/0x4f0
    io_ring_exit_work+0x287/0x880
    process_one_work

Regardless of the per-free cost, the loop has no bound other than
the workload's in-flight high-water mark, so it belongs with the
other potentially long io_uring loops.

Insert a cond_resched() in the drain loop.  The function is __cold;
all callers run in process context and hold only ctx->uring_lock,
a mutex, so scheduling is allowed.

Assisted-by: LLM
Signed-off-by: Yuanhe Shu <xiangzao@linux.alibaba.com>
---
 io_uring/io_uring.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 26cd91a91aa2..dffce8ce55d5 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2142,6 +2142,7 @@ static __cold void __io_req_caches_free(struct io_ring_ctx *ctx)
 		io_poison_req(req);
 		kmem_cache_free(req_cachep, req);
 		nr++;
+		cond_resched();
 	}
 	if (nr) {
 		ctx->nr_req_allocated -= nr;
-- 
2.43.7