From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yx1-f99.google.com (mail-yx1-f99.google.com [74.125.224.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE8CE3D902B for ; Fri, 27 Feb 2026 22:35:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.99 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772231711; cv=none; b=pioshhH3y9ECQrz89/qlxSHpQPRybbbWPfYRTujZDOtrlJORFlXVN8x7joN9FrZ3hJUgge7IBdABnfyD1hDouBOOs3o2FLb3Fh6GyNFqeNti7CnDyTk+5gVvRtmcaOJPGv/vQ94lCDNls9uPdVUCyY+JKxr+bHxdGEE+Ai3wHxo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772231711; c=relaxed/simple; bh=x/lgR7lfxoNQ8NgcIqgi/nGp9L0HR70DpaM2xD91Dbc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NpyeU2aOt9UyT4wHSLpjI/awzu6FMB3Kr5lc71mKXzewXTQluQnjUbCIX1PxraSC+hN8+J7/GuYuQkXSfAKAFKcS9lYLZvgmQbwwoJmYn333RXFAOM0BLw3v4700YtBfeZTN+6yPm2hrSrzWtz6IY9+mEnmXhBLcll6+yMn/uNg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=WlCXbMQo; arc=none smtp.client-ip=74.125.224.99 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="WlCXbMQo" Received: by mail-yx1-f99.google.com with SMTP id 956f58d0204a3-64c9cbec9b0so252140d50.1 for ; Fri, 27 Feb 2026 14:35:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1772231708; x=1772836508; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=doNIdz9wMn9gx8zF7Ky2pC6GD/iifRgNWdZEMqpkv1c=; b=WlCXbMQohgfDevE37gEl1e8EgNC0Z6xLKsdJVlurSnx9q07v1xhQ1vGJIoXwEHDw8u mSyt/WEGenHvRUmx+Y/pgo39YFJIVlBX1NYOVDzbJzREH3ou9RGIk3UW167282iiDg2x zCSf/6bzmz7+G7CJeGNhek4fUNAoVt/0dWwk0Z7798Z5IcZMppJv2C0cVGA20FXVNwMX iU25QNuYl79/PrDr0HV0GDUCdIiyx3kcL+++fGndqxExKebAZSyv9IsiK+E5gB5E5YFM uI3rWl1EJqUafrTJipmQmS2qKLDT6NBV1To+F350xkAlqpfFsASriT04RtR65Ert6TpC svgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772231708; x=1772836508; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=doNIdz9wMn9gx8zF7Ky2pC6GD/iifRgNWdZEMqpkv1c=; b=hkdgxwXwjhpyqYI+2Mtc6OuWepwSbgohg50i4UiKAWvHVMJJTB1pcQpLQcTVzrYmlQ iYwgM+RilymjqtQIqqNm5gPK7q4DRNnLFBTXXnka7HrKXzTZ+U5yy9AQfNeUUJkNxUO1 Bcmv2fUV/oXvCNUhPjyd8qxtgIbtYxHQL1ANNCTRJXeU6mBBBqC6DFmBz2rVagBFFjO4 mvi++A/0lGDAcQ6ggAP3OBVakEbtahDpT4/7RZbqZFn6geDyVZNEKFs5AVwktnE2lBvJ 91mefwMH9N9q4OqOWkNzJ4+6h2+cNAx9R0cZRIuUmIJ4XulUDrHECmMB4HG3OdzbM36O pZ3w== X-Forwarded-Encrypted: i=1; AJvYcCUW+vRRoeS4sqspnLgKSZImHdY5kYvT/+lisGmcxDoEZdyQMwIm4th5zUosMqsx3gtawuY5HRMjsZzdrY0=@vger.kernel.org X-Gm-Message-State: AOJu0YxPVxfdZXszA8DrOiMkoGaSCb/ueSkQXeOVxNQpVhTVoOCcuhI8 P+qOoI+J09CiNnC2sop7FOPrQkJE/Nayu0l1zJf23fMVI1NuIGzGeXL/gyLgKoFvrmeekt+s8Nx atOFDzBjPG3CxbPf5vjWm+ELZFqR5O0JE6tXi X-Gm-Gg: ATEYQzzxLT0eMLVoZGhoFSfG3D5sv3thjcvQuNxCYgKe0wWQrVlPECJovnAX7PvsT/5 Q5uf5DOcMdp6yHrK846l2dAs3WBPW/wFp3XeNuo1edNWqaBWd40MdTKmryyvX4zgb7Bk35BPs5P 7tGCmC+DxWRIskalIetZQeFVxlE9uu6VxaxY2d30sAbiEiHfIvNxPEOO14Qw2I27h5P/k8Gu+Rx W3hmETvdO1ArKO/HG+qmKGwPpczkLeDAHU0fgY6+4R07j127zKDUBYjd27rS8dmZQ9AWdfv4tg5 aUeZ/QBLEZ4xdlHRPs1GMj0KhEz+i5dwq+DKu0AzukxbFoMAm0kVNBEVdFwXqbIXr8WskG5md9U 5Un1iCuVNyOgU5z4XAbwgTIt9xl2jhF4Pq3ABULtMGDNPcf4MZYNRCg== X-Received: by 2002:a05:690c:c4fa:b0:797:a57b:d200 with SMTP id 00721157ae682-798850f351bmr35803657b3.0.1772231707838; Fri, 27 Feb 2026 14:35:07 -0800 (PST) Received: from c7-smtp-2023.dev.purestorage.com ([208.88.159.128]) by smtp-relay.gmail.com with ESMTPS id 00721157ae682-79876ab7b69sm6884587b3.1.2026.02.27.14.35.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Feb 2026 14:35:07 -0800 (PST) X-Relaying-Domain: purestorage.com Received: from dev-csander.dev.purestorage.com (dev-csander.dev.purestorage.com [10.112.29.101]) by c7-smtp-2023.dev.purestorage.com (Postfix) with ESMTP id 255C9341DB3; Fri, 27 Feb 2026 15:35:07 -0700 (MST) Received: by dev-csander.dev.purestorage.com (Postfix, from userid 1557716354) id 213C2E420D8; Fri, 27 Feb 2026 15:35:07 -0700 (MST) From: Caleb Sander Mateos To: Jens Axboe , Christoph Hellwig , Keith Busch , Sagi Grimberg Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Anuj Gupta , Kanchan Joshi , Ming Lei , Caleb Sander Mateos Subject: [PATCH v4 3/5] io_uring: count CQEs in io_iopoll_check() Date: Fri, 27 Feb 2026 15:35:01 -0700 Message-ID: <20260227223504.1162421-4-csander@purestorage.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20260227223504.1162421-1-csander@purestorage.com> References: <20260227223504.1162421-1-csander@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit A subsequent commit will allow uring_cmds that don't use iopoll on IORING_SETUP_IOPOLL io_urings. As a result, CQEs can be posted without setting the iopoll_completed flag for a request in iopoll_list or going through task work. For example, a UBLK_U_IO_FETCH_IO_CMDS command could call io_uring_mshot_cmd_post_cqe() to directly post a CQE. The io_iopoll_check() loop currently only counts completions posted in io_do_iopoll() when determining whether the min_events threshold has been met. It also exits early if there are any existing CQEs before polling, or if any CQEs are posted while running task work. CQEs posted via io_uring_mshot_cmd_post_cqe() or other mechanisms won't be counted against min_events. Explicitly check the available CQEs in each io_iopoll_check() loop iteration to account for CQEs posted in any fashion. Signed-off-by: Caleb Sander Mateos --- io_uring/io_uring.c | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 46f39831d27c..5f694052f501 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1184,11 +1184,10 @@ __cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx) io_move_task_work_from_local(ctx); } static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) { - unsigned int nr_events = 0; unsigned long check_cq; min_events = min(min_events, ctx->cq_entries); lockdep_assert_held(&ctx->uring_lock); @@ -1205,19 +1204,12 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) * dropped CQE. */ if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) return -EBADR; } - /* - * Don't enter poll loop if we already have events pending. - * If we do, we can potentially be spinning for commands that - * already triggered a CQE (eg in error). - */ - if (io_cqring_events(ctx)) - return 0; - do { + while (io_cqring_events(ctx) < min_events) { int ret = 0; /* * If a submit got punted to a workqueue, we can have the * application entering polling for a command before it gets @@ -1227,34 +1219,30 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) * the poll to the issued list. Otherwise we can spin here * forever, while the workqueue is stuck trying to acquire the * very same mutex. */ if (list_empty(&ctx->iopoll_list) || io_task_work_pending(ctx)) { - u32 tail = ctx->cached_cq_tail; - (void) io_run_local_work_locked(ctx, min_events); if (task_work_pending(current) || list_empty(&ctx->iopoll_list)) { mutex_unlock(&ctx->uring_lock); io_run_task_work(); mutex_lock(&ctx->uring_lock); } /* some requests don't go through iopoll_list */ - if (tail != ctx->cached_cq_tail || list_empty(&ctx->iopoll_list)) + if (list_empty(&ctx->iopoll_list)) break; } ret = io_do_iopoll(ctx, !min_events); if (unlikely(ret < 0)) return ret; if (task_sigpending(current)) return -EINTR; if (need_resched()) break; - - nr_events += ret; - } while (nr_events < min_events); + } return 0; } void io_req_task_complete(struct io_tw_req tw_req, io_tw_token_t tw) -- 2.45.2