From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f102.google.com (mail-ej1-f102.google.com [209.85.218.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC332E555 for ; Sat, 27 Jun 2026 04:16:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.102 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782533801; cv=none; b=kG/fspAIwDgb/i5CNRoIdsvUpz72P7PoidMuhyUd5OOcqYEptEdarP7vXBJSyju6B8SdTpieBlIYl+3Nv+HYhJjCodqCGL3Q7JfG+xbGntWuTlxhS7QUjVKJtrfAgNUA3srFrI6+a0ctJ6Oby/VNpu9Rn07N9/BoH14qus78crI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782533801; c=relaxed/simple; bh=1m8ig/9tYjKAqvH/rmEQNxjYip2IJS1u2zLrZxm+2gg=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=HAF3ReMWupr8G0lRYM289Kg7cTsgUbD+1q25BpkcKbFgC/vhtSSuGqa4AEipZF10VhuoukSOYFSO8oOqIpq4dSJELqYeqxuhx+iXuB0ZuVUF9jo+QlvXWOigYDosudW42BGpqM0QDu9SoSyMlbgEGaWx8IbO+6mogN19o6dyTpk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=purestorage.com; spf=pass smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=V21CGBbq; arc=none smtp.client-ip=209.85.218.102 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="V21CGBbq" Received: by mail-ej1-f102.google.com with SMTP id a640c23a62f3a-c07c67ad9f2so194879466b.0 for ; Fri, 26 Jun 2026 21:16:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1782533797; x=1783138597; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=W3BlDeepFbwZ4oE1Vsnyo+V98vHfS2bZA1E4/dojlN4=; b=V21CGBbqt9aSYvmwVc/PGcqVBhV4MOZoa41I2473AuxW6YDULDdOGTJ43sPsqjAT8a FYXIncQufkRouqyCLGQ4jAKCMSOQ4LTKNYnzaghjtKT2a7duOEn+MsATnOqxIqeEGL0T RKzCtAHZ79xbJYNrUF9Fp6rXuJbYeLcA29gIQmhGfYUwJJI9JvuzkAm6JZOzyXWtuJEW s5kB+yf94YAOkNdn2C4RwZUQtGQqHtBbAnzsOY5aGHsdlF783PNflAFwgjC0+Ee/uU42 TZKXun+DYkKgNVhqT7jINNSDQDy8ZNm8eZvos58uqhbZgrEfEArvWWQ+FKbPa3NULE01 A9ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782533797; x=1783138597; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=W3BlDeepFbwZ4oE1Vsnyo+V98vHfS2bZA1E4/dojlN4=; b=k9X1mHxoKx8+Rjv2UULgSGSrQiuLdVHj2i4y6hOOlCFt4pMUxCarHq3Um/DBNh49qb KwMfZkqCdwNpigFyRXTE+EbdmV3JVMpkpMI3hujbwXPLRyol/gRXTcLNgJJYgE7VO/6m n5oPtefKoCeFaYI86TSOd5re3yLssyzCbPx6HJFtlI7puWhy3QysX/o/CX9pwZLu47ba 5MBTshFIXF5yl0hWQmYikhA8ChVr8fPA27c1M/BWCIKBMCI/kZ5tfqoWok4DyEsbULuf GJq2x2WpO0AT8sQ5nxkZnBuKr6R56WvdM7UeDQbybfCO8lumNF85rt1DFKnarND0e49z Ey1g== X-Forwarded-Encrypted: i=1; AHgh+RpI46hStMJMurXXl2p5I0p1KtIIUmlAfoZp7Cgns7LhHHCACnk9YQD2a1oJlVs0gOUOpUpH4FDQX3mfiac=@vger.kernel.org X-Gm-Message-State: AOJu0YxdH5NFX8bgv67rcNo8b2eCxpbTIJpRFcMyWNT3p9LMls7u4/nH mEExwF81LQsqUL4K4mA6/2+V8h7IxsPiUUKJFOYTd56t66DwODlPyp7rem7qH+i9/M5NbOSW4Ly ToCe1ks31EBUjZgL4gBcUvct7iENd7o3M0RKPD/VvW2FdXZJPNM/R X-Gm-Gg: AfdE7cnQ2jrIk7pA1eec0iJu7N2K4f9jO2h2+VFvI+G6nO0IN/1T8r4sNE7FCefRWr/ TuD0baMwcwAzo28JGehXKaatqe8L71Xjac9NGPtaCHsOFkJ1n8ZOug8g2OpEBgCyEU0FqDsz3Jt 8nVQCciGlgBoY7Hgr5YILXcrqFY0hxQhJx25DiaL/N/jDYSPE+CG9odtfDPuJP8zNUjJFixNueN oFvm8UEDBO/mdE2GDdmoIY8aGmv9MvKlV9KHHGy3UfbMSYI5VaNtZ2I2+7jfw6ghKmlnX3bYjKm 8ZHHlGukoutMoLXgRsqDkFv+BXbzW2v4b4vZajnsyVoCWfLwc+sQTelD2Fm/A2iSxa9qpdBGyFA CqzZUf98/jle3nrlC14yuF0TwiYY= X-Received: by 2002:a17:907:d30c:b0:c0f:bf3b:1793 with SMTP id a640c23a62f3a-c120446a4camr569437966b.21.1782533797198; Fri, 26 Jun 2026 21:16:37 -0700 (PDT) Received: from c7-smtp-2026.dev.purestorage.com ([2620:125:9017:12:36:3:6:0]) by smtp-relay.gmail.com with ESMTPS id a640c23a62f3a-c1230ac0218sm13606566b.35.2026.06.26.21.16.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jun 2026 21:16:37 -0700 (PDT) X-Relaying-Domain: purestorage.com Received: from dev-sgogte.dev.purestorage.com (bond0.slc5-n22m24-k8s.dev.purestorage.com [IPv6:2620:125:9025:20::a31:429]) by c7-smtp-2026.dev.purestorage.com (Postfix) with ESMTP id 17A9540146; Fri, 26 Jun 2026 22:16:35 -0600 (MDT) Received: by dev-sgogte.dev.purestorage.com (Postfix, from userid 1557734945) id 11F2551219; Fri, 26 Jun 2026 22:16:35 -0600 (MDT) From: Surabhi Gogte To: Christoph Hellwig , Keith Busch , Jens Axboe , Sagi Grimberg Cc: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, mkhalfella@purestorage.com, randyj@purestorage.com, adailey@purestorage.com, Surabhi Gogte Subject: [PATCH v4 0/2] nvme-rdma: parallelize I/O queue setup Date: Fri, 26 Jun 2026 22:15:49 -0600 Message-ID: <20260627041551.1981256-1-sgogte@purestorage.com> X-Mailer: git-send-email 2.54.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit This patch series parallelizes nvme-rdma connection and reconnection by setting up I/O queues in parallel instead of sequentially. Allocation and startup of each queue are combined into a single per-queue async work item, so per-queue connection latency overlaps across all queues. This matters most on high-core-count hosts with many I/O queues, where serial setup dominates connect time. Patch 1 is a preparatory refactor: nvme_rdma_alloc_queue() takes a queue pointer so allocation and startup can be folded into a single async worker. Patch 2 contains the implementation for async setup of I/O queues. Testing on a 64-core host with 64 I/O queues shows nvme-rdma connection time reduced from ~1.4s to 416ms. Signed-off-by: Surabhi Gogte --- Changes from v3->v4: - Fixed formatting. - Replaced kmalloc_array with kmalloc_objs. Changes from v2->v3: - Split the series into two patches: extract the nvme_rdma_alloc_queue() refactor into a separate preparatory patch. - Replace the atomic error flag in struct nvme_rdma_ctrl with a per-work nvme_rdma_setup_ctx { queue, err } struct. - Fix formatting changes regarding line overflow indentation and nesting. Changes from v1->v2: - Remove separate workqueue and use the async API instead. Previous versions: v1: https://lore.kernel.org/all/20260529001354.1003640-1-sgogte@purestorage.com/ v2: https://lore.kernel.org/all/20260604195321.2232838-1-sgogte@purestorage.com/ v3: https://lore.kernel.org/all/20260625212722.1302344-1-sgogte@purestorage.com/ Surabhi Gogte (2): nvme-rdma: refactor nvme_rdma_alloc_queue() to take a queue pointer nvme-rdma: parallelize I/O queue allocation and startup drivers/nvme/host/rdma.c | 136 ++++++++++++++++++++++++--------------- 1 file changed, 83 insertions(+), 53 deletions(-) -- 2.54.0