[PATCH 0/2] io_uring/cancel: cancel all poll matches in one pass

[PATCH 0/2] io_uring/cancel: cancel all poll matches in one pass

[Shang En Sim]

IORING_OP_ASYNC_CANCEL with IORING_ASYNC_CANCEL_ALL cancels poll
requests one at a time, and every lookup starts over at the beginning
of the poll hash. Cancelled requests stay hashed until their task work
runs, so each lookup also walks past the ones already cancelled. One
cancel that matches K of N pending polls costs about K*N/2 visits in a
single io_uring_enter(), with uring_lock held.

Patch 2 makes poll cancel all its matches in one pass, as futex and
waitid already do through io_cancel_remove(). Patch 1 is a prerequisite
and a fix on its own: __io_async_cancel() counts each io_try_cancel()
call as one, so CANCEL_ALL on futex or waitid requests reports 1 no
matter how many it cancelled. Patch 2 needs it to keep the poll count
right.

Reproducer, 80000 polls on one eventfd and a single CANCEL_FD|ALL:

  #include <liburing.h>
  #include <poll.h>
  #include <sys/eventfd.h>

  int main(void)
  {
  	struct io_uring ring;
  	int fd = eventfd(0, 0), i;

  	io_uring_queue_init(4096, &ring, 0);
  	for (i = 0; i < 80000; i++) {
  		struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);

  		if (!sqe) {
  			io_uring_submit(&ring);
  			sqe = io_uring_get_sqe(&ring);
  		}
  		io_uring_prep_poll_add(sqe, fd, POLLIN);
  	}
  	io_uring_prep_cancel_fd(io_uring_get_sqe(&ring), fd,
  				IORING_ASYNC_CANCEL_ALL);
  	io_uring_submit(&ring);
  	return 0;
  }

In a 4 vCPU KVM guest, "time ./repro" takes 17.06s of system time on
next-20261029 (lazy preemption) and 0.02s with these patches. On
v6.18.59 with voluntary preemption it takes 36.24s, and the soft lockup
watchdog fires during the cancel.

Testing, on next-20261029 and v6.18.59: liburing's cancel tests
(poll-cancel-all, poll-cancel, poll-cancel-ton, cancel-fd-userdata,
io-cancel, sync-cancel, cancel-race, futex, waitid), plus three new
tests that I'll send to liburing separately: a futex CANCEL_ALL count
(fails before patch 1), and poll CANCEL_ALL by user_data alone and with
1000 polls on one fd. All pass, except waitid's pidfd test on
v6.18.59, which fails with and without these patches. Not tested:
lockdep, KASAN, and SQPOLL or DEFER_TASKRUN rings for the new cases.


Shang En Sim (2):
  io_uring/cancel: count all requests that a CANCEL_ALL cancels
  io_uring/poll: cancel all matching requests in one pass

 io_uring/cancel.c |  3 ++-
 io_uring/poll.c   | 39 +++++++++++++++++++++++++++++++++++++--
 2 files changed, 39 insertions(+), 3 deletions(-)


base-commit: 70866cfa169d5ee149364267289e71d50f9a7ae9
-- 
2.56.0

[PATCH 1/2] io_uring/cancel: count all requests that a CANCEL_ALL cancels

[Shang En Sim]

With IORING_ASYNC_CANCEL_ALL, __io_async_cancel() counts one for each
io_try_cancel() call that cancels something. io_futex_cancel() and
io_waitid_cancel() cancel every match in one call and return how many,
so four futex waits cancelled with CANCEL_ALL give a result of 1.

Add the number that they return.

Cc: stable@vger.kernel.org
Fixes: 194bb58c6090 ("io_uring: add support for futex wake and wait")
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Signed-off-by: Shang En Sim <sim@shangen.org>
---
 io_uring/cancel.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/io_uring/cancel.c b/io_uring/cancel.c
index 79b5c133319b..50a65fbc6ff0 100644
--- a/io_uring/cancel.c
+++ b/io_uring/cancel.c
@@ -198,7 +198,8 @@ static int __io_async_cancel(struct io_cancel_data *cd,
 			break;
 		if (!all)
 			return ret;
-		nr++;
+		/* futex and waitid return how many they cancelled */
+		nr += max(ret, 1);
 	} while (1);
 
 	/* slow path, try all io-wq's */
-- 
2.56.0

[PATCH 2/2] io_uring/poll: cancel all matching requests in one pass

[Shang En Sim]

With IORING_ASYNC_CANCEL_ALL, __io_async_cancel() calls io_try_cancel()
until nothing is found, and io_poll_cancel() cancels one request per
call, starting each lookup at the beginning of the hash. A cancelled
request stays hashed until its task work runs, so every lookup walks
past the ones already cancelled. Cancelling K of N pending polls visits
about K*N/2 requests in one io_uring_enter(), with uring_lock held.

A single CANCEL_FD|ALL on 80000 polls on one eventfd takes 17.06s,
0.02s with this change. With voluntary preemption the soft lockup
watchdog fires:

  watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [repro:100]
  RIP: 0010:io_cancel_req_match+0x19/0xa0
   io_poll_cancel+0x73/0x170
   io_try_cancel+0x7b/0xf0
   __io_async_cancel+0x41/0x110

Walk the hash once and cancel every match, as io_cancel_remove() does
for futex and waitid. Cancel by user_data only needs its own bucket.
__io_async_cancel() still loops until nothing is found, so a request
that io-wq parks in the poll hash meanwhile is still cancelled.

A cancel that matches one request now walks the whole hash twice instead
of stopping at the match.

Cc: stable@vger.kernel.org
Fixes: 8e29da69fead ("io_uring: add support for IORING_ASYNC_CANCEL_ALL")
Signed-off-by: Shang En Sim <sim@shangen.org>
---
 io_uring/cancel.c |  2 +-
 io_uring/poll.c   | 39 +++++++++++++++++++++++++++++++++++++--
 2 files changed, 38 insertions(+), 3 deletions(-)

diff --git a/io_uring/cancel.c b/io_uring/cancel.c
index 50a65fbc6ff0..e3d9fe8fc982 100644
--- a/io_uring/cancel.c
+++ b/io_uring/cancel.c
@@ -198,7 +198,7 @@ static int __io_async_cancel(struct io_cancel_data *cd,
 			break;
 		if (!all)
 			return ret;
-		/* futex and waitid return how many they cancelled */
+		/* poll, futex and waitid return how many they cancelled */
 		nr += max(ret, 1);
 	} while (1);
 
diff --git a/io_uring/poll.c b/io_uring/poll.c
index 4f04bda42f7f..9b73b0d88a0e 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -827,12 +827,47 @@ static int io_poll_disarm(struct io_kiocb *req)
 	return 0;
 }
 
+/*
+ * Cancel every request matching @cd in one pass. Cancelled requests stay
+ * hashed until their task work runs, so finding them one at a time would
+ * walk past all the earlier ones on each lookup.
+ */
+static int io_poll_cancel_all(struct io_ring_ctx *ctx, struct io_cancel_data *cd)
+{
+	unsigned int nr_buckets = 1U << ctx->cancel_table.hash_bits;
+	unsigned int i = 0;
+	struct hlist_node *tmp;
+	struct io_kiocb *req;
+	int nr = 0;
+
+	if (!(cd->flags & (IORING_ASYNC_CANCEL_FD | IORING_ASYNC_CANCEL_OP |
+			   IORING_ASYNC_CANCEL_ANY))) {
+		/* keyed by user_data, so only one bucket can match */
+		i = hash_long(cd->data, ctx->cancel_table.hash_bits);
+		nr_buckets = i + 1;
+	}
+
+	for (; i < nr_buckets; i++) {
+		struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i];
+
+		hlist_for_each_entry_safe(req, tmp, &hb->list, hash_node) {
+			if (!io_cancel_req_match(req, cd))
+				continue;
+			io_poll_cancel_req(req);
+			nr++;
+		}
+	}
+	return nr ?: -ENOENT;
+}
+
 static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd)
 {
 	struct io_kiocb *req;
 
-	if (cd->flags & (IORING_ASYNC_CANCEL_FD | IORING_ASYNC_CANCEL_OP |
-			 IORING_ASYNC_CANCEL_ANY))
+	if (cd->flags & (IORING_ASYNC_CANCEL_ALL | IORING_ASYNC_CANCEL_ANY))
+		return io_poll_cancel_all(ctx, cd);
+
+	if (cd->flags & (IORING_ASYNC_CANCEL_FD | IORING_ASYNC_CANCEL_OP))
 		req = io_poll_file_find(ctx, cd);
 	else
 		req = io_poll_find(ctx, false, cd);
-- 
2.56.0