From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oo1-f101.google.com (mail-oo1-f101.google.com [209.85.161.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B046033C182 for ; Thu, 25 Jun 2026 21:27:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.101 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782422855; cv=none; b=E5fmOYxxixxmslm+oGay3BpCzKDSjvrlcOC7kALf9snRMIY1jrn4e1mCL8uBfR8r7Wq2Yg67ez7dpgtkjk+D5FAywiITDwj0DCVjXWnrfMdp8G+Ht/iEO9bwGtC963b2pm3c/hhJNG5yP59W5GIBw5uM0yG88AYJrgvet81wnpQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782422855; c=relaxed/simple; bh=9Re3mpf4DtHvXmDVI7ahD21DvjEmYpJNaDbWr1qjieg=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=h8RQigMwLTyOy1vkUybtBfp+P/T2zS4plzyjab7TvhB8NxkB//xDZLLP/zonwAEH1cJ8WLPvWBWtQer2X5RuP8T1gGY1yPqWzK7rvolfDR/Uejvq26rZyjOeTj1109jHgELUmX4/rUWntSYpqO+3m1TYXdZ0dnJYvDYq+CXjtJY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=purestorage.com; spf=pass smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=UmHQhIiO; arc=none smtp.client-ip=209.85.161.101 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="UmHQhIiO" Received: by mail-oo1-f101.google.com with SMTP id 006d021491bc7-6a0a0b46cd9so109766eaf.1 for ; Thu, 25 Jun 2026 14:27:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1782422853; x=1783027653; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=u6g8GnQBUk2ne5BaPEaScB3+h0qp9XYYdT6ZfiBb2l4=; b=UmHQhIiO9vJ19pNsI8iY+WyNiHgbsG7Doq9VsMS8XXrtKUEZd02af1icwrlpL0ewNz o1jpKuzJ97grnhF7w/cR4LW9qt1ay26yxzXXEofBS2xoCzyDKDXszeU/sc56OxhTqwaV wAJtXI5P3FuV46kA+GY9z5J3ptV5Ug4CrbhEDgmid+YMdMKPDvUDXMNLbh4pBYO2c5+P 6K2JAhA5THBJYHQPQct2J4h0JK0g1MXqAsjQOVS8jG729ZvGjHsDbmMQTnvIHWIb7QsK oeqtHOJbaBDYKygqcyAlutiU/48xUzPfVfmf4swaSKF1xxyGf/daPHwowvzqtYiv9jF4 EA3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782422853; x=1783027653; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=u6g8GnQBUk2ne5BaPEaScB3+h0qp9XYYdT6ZfiBb2l4=; b=Z0nDHTrCt14Ool3FVpnX/as2r7WZsg1EDtj0AOo4MoZLJLrnTLpMKWGvyFofvT0JmI T6Ut/Q/8/5WrTlDUWGq3+AKXJBp458q5ZD42OLksIJTG+tqVxeRsHD5W7DRNE7c+w0oC rUiUprUdZAUiN/Azq99dFuV1v6LyTiIiym642LfaupyX0cb7m0mWo+cFv+clvaOse0M7 iQHn7A7l7N7NP1N/3u8Dfd00TyAERFxw1iaWm1NqYUmh1XzFfoEIYD7hbKzrwmozwoe3 H8TOkOOA7CPjcS1Es43T6CzUS8/TOlv718li1WkK/mhIAiq1uz+onUDmwv0y6AvsoLu4 NwBw== X-Forwarded-Encrypted: i=1; AFNElJ+N5tM62HOj0JszCf5HKu7WgF2nh4DuM7kzBOglNyCLh7wRUVace6R+/rgIBdMm8CFAP8mFeUHb3jDfSqI=@vger.kernel.org X-Gm-Message-State: AOJu0YxETv8dKrEo7F2akE5rwh0OMD5acN/avtRM6zmH2ZoYeVbmLmSn Xh7GuCltx5R0iHVL915a6RXzSGmY7P7FczCNyzUdFDDiH6XBAtoPY9R/K63W3GCNyC1QBLVwd95 +Ho7RMEmzSVctoyRlrrqHq5x8tNiku4AW9EzF X-Gm-Gg: AfdE7ck7HDWEAHqjkiTxx/ne5pO/1uhVBIrm1is5XvEM03nZZRVmFuvOoFuhQB5ckFZ KDEixTxWvFKNpw13b5W/CXS9e1HB2PIUUZWqzjpHXQ60WFjKMxmw/jnPnJOK74Uu43xwdrNDuRp y5bTs+bi4z75woLFPbUr5hNIe1eKqRBYUE4ZvuNnb++Gqm3Zs88Co0SJfFOpgwS3Zfj+VscADcG zoM6BMdhMPHbkucGNVSh6VmZkVbRwxNs85z/K5xcKc0bWXBvJqxOkHb513fx8SEynnr0xAI8SCP UODciy1pHWeEdAZblFlNQ5EFMGFTY4x6zSN99XMTKUfyjT+gbrOg/ewmgk4ZHJ8AK0Zn38MABIN DaUw0a5oGE4oCuncYc1bJEW6XCQothiOqmVVBiWahqA== X-Received: by 2002:a05:6820:1b14:b0:69d:f7bf:c53f with SMTP id 006d021491bc7-6a13524882bmr3311653eaf.39.1782422853559; Thu, 25 Jun 2026 14:27:33 -0700 (PDT) Received: from c7-smtp-2026.dev.purestorage.com ([208.88.159.129]) by smtp-relay.gmail.com with ESMTPS id 586e51a60fabf-4472f162514sm1474497fac.12.2026.06.25.14.27.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Jun 2026 14:27:33 -0700 (PDT) X-Relaying-Domain: purestorage.com Received: from dev-sgogte.dev.purestorage.com (bond0.slc5-n22m24-k8s.dev.purestorage.com [IPv6:2620:125:9025:20::a31:429]) by c7-smtp-2026.dev.purestorage.com (Postfix) with ESMTP id 171DA401FD; Thu, 25 Jun 2026 15:27:33 -0600 (MDT) Received: by dev-sgogte.dev.purestorage.com (Postfix, from userid 1557734945) id 1352051222; Thu, 25 Jun 2026 15:27:33 -0600 (MDT) From: Surabhi Gogte To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg Cc: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, mkhalfella@purestorage.com, randyj@purestorage.com, adailey@purestorage.com, Surabhi Gogte Subject: [PATCH v3 0/2] nvme-rdma: parallelize I/O queue setup Date: Thu, 25 Jun 2026 15:27:20 -0600 Message-ID: <20260625212722.1302344-1-sgogte@purestorage.com> X-Mailer: git-send-email 2.54.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit This patch series parallelizes nvme-rdma connection and reconnection by setting up I/O queues in parallel instead of sequentially. Allocation and startup of each queue are combined into a single per-queue async work item, so per-queue connection latency overlaps across all queues. This matters most on high-core-count hosts with many I/O queues, where serial setup dominates connect time. Patch 1 is a preparatory refactor: nvme_rdma_alloc_queue() takes a queue pointer so allocation and startup can be folded into a single async worker. Patch 2 contains the implementation for async setup of I/O queues. Testing on a 64-core host with 64 I/O queues shows nvme-rdma connection time reduced from ~1.4s to 416ms. Signed-off-by: Surabhi Gogte --- Changes from v2->v3: - Split the series into two patches: extract the nvme_rdma_alloc_queue() refactor into a separate preparatory patch. - Replace the atomic error flag in struct nvme_rdma_ctrl with a per-work nvme_rdma_setup_ctx { queue, err } struct. - Fix formatting changes regarding line overflow indentation and nesting. Changes from v1->v2: - Remove separate workqueue and use the async API instead. Previous versions: v1: https://lore.kernel.org/all/20260529001354.1003640-1-sgogte@purestorage.com/ v2: https://lore.kernel.org/all/20260604195321.2232838-1-sgogte@purestorage.com/ Surabhi Gogte (2): nvme-rdma: refactor nvme_rdma_alloc_queue() to take a queue pointer nvme-rdma: parallelize I/O queue allocation and startup drivers/nvme/host/rdma.c | 135 ++++++++++++++++++++++++--------------- 1 file changed, 82 insertions(+), 53 deletions(-) -- 2.54.0