[PATCH 0/1] nvme-rdma: post the command capsule inline when it fits

[PATCH 0/1] nvme-rdma: post the command capsule inline when it fits

[Andrei-Alexandru Bleortu]

The host posts each 64-byte command capsule by reference, so the HCA
reads it from host memory before it sends the command. This patch
asks for inline data at QP creation and posts capsules that carry no
in-capsule data with IB_SEND_INLINE. The commit message has the
details and the measurements.

The target is left alone. For a read it posts the response capsule in
the same WR chain as the data RDMA Write. In the measured runs, inlining
the response alone made no measurable difference, and inlining both
capsules was about the same as inlining the command alone.

It builds with W=1 on nvme-7.4 without new warnings. I tested it in a
QEMU guest with rdma_rxe on a dummy netdev and nvmet-rdma looped back
to a file-backed namespace. 16 MiB of O_DIRECT writes and reads at
4 KiB and 128 KiB compared equal. The QP got 80 bytes of inline data.
4371 capsules went inline and 4098 by reference, mostly 4 KiB writes,
which carry in-capsule data. With rxe changed to refuse inline data,
QP creation fell back and the data still compared equal. The guest ran
a 7.3-rc4 based kernel. Its drivers/nvme/host/rdma.c is the same as
nvme-7.4's except for one later fix, 909307cea08a ("nvme-rdma: don't
complete a request after a bogus remote invalidation").

I wrote this patch with an LLM coding assistant (Claude Code), which
also drafted the commit message and ran the build and the VM test.

Andrei-Alexandru Bleortu (1):
  nvme-rdma: post the command capsule inline when it fits

 drivers/nvme/host/rdma.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)


base-commit: bbca7c312ab25efd0024eb87545c51a76ab2ea39
-- 
2.56.0

[PATCH 1/1] nvme-rdma: post the command capsule inline when it fits

[Andrei-Alexandru Bleortu]

nvme_rdma_post_send() posts the 64-byte command capsule by reference,
so the HCA has to DMA-read the capsule from host memory before it can
send the command. That read is on the latency path of every I/O.

Ask for sizeof(struct nvme_command) bytes of inline data when creating
the queue pair, and create it without inline data if the device refuses.
Post a capsule that has no in-capsule data with IB_SEND_INLINE when it
fits in the granted inline size. The CPU then copies the capsule into
the send WQE and the HCA no longer reads it.

Providers copy inline data from sg_list[].addr as a kernel virtual
address, but nvme-rdma keeps the capsule's DMA address there. On the
inline path the SGE therefore points at qe->data. The capsule stays
DMA-mapped for the by-reference path.

An earlier version of this patch was measured with fio doing 4 KiB
random reads at queue depth 1 against a null_blk namespace exported by
nvmet-rdma. Both host pairs ran a 6.8 kernel over RoCEv2 at 100 Gb/s.
The mean completion latency fell by 0.998 us (95% CI 0.943-1.049)
between two AMD EPYC 7452 hosts with ConnectX-5 Ex, and by 0.589 us
(0.558-0.618) between two Xeon Platinum 8360Y hosts with ConnectX-6 Dx.
IOPS rose by 3.3% on the EPYC pair. A run there that asked for inline
data but kept posting by reference showed no change. That version asked
for 220 bytes and had no fallback, but it inlined the same capsules.

Assisted-by: LLM
Signed-off-by: Andrei-Alexandru Bleortu <me@andrei-z.com>
---
 drivers/nvme/host/rdma.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 5b9b540693b5..517098ebffb1 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -106,6 +106,7 @@ struct nvme_rdma_queue {
 	bool			pi_support;
 	int			cq_size;
 	struct mutex		queue_lock;
+	u32			max_inline_data;
 };
 
 struct nvme_rdma_setup_ctx {
@@ -265,7 +266,7 @@ static int nvme_rdma_wait_for_cm(struct nvme_rdma_queue *queue)
 static int nvme_rdma_create_qp(struct nvme_rdma_queue *queue, const int factor)
 {
 	struct nvme_rdma_device *dev = queue->device;
-	struct ib_qp_init_attr init_attr;
+	struct ib_qp_init_attr init_attr, attr;
 	int ret;
 
 	memset(&init_attr, 0, sizeof(init_attr));
@@ -284,9 +285,21 @@ static int nvme_rdma_create_qp(struct nvme_rdma_queue *queue, const int factor)
 		init_attr.create_flags |= IB_QP_CREATE_INTEGRITY_EN;
 	init_attr.qp_context = queue;
 
-	ret = rdma_create_qp(queue->cm_id, dev->pd, &init_attr);
+	/*
+	 * Ask for enough inline data to carry the command capsule in the send
+	 * WQE (see nvme_rdma_post_send()).  If the device cannot provide it,
+	 * create the QP without inline data and post the capsule by reference.
+	 */
+	attr = init_attr;
+	attr.cap.max_inline_data = sizeof(struct nvme_command);
+	ret = rdma_create_qp(queue->cm_id, dev->pd, &attr);
+	if (ret) {
+		attr = init_attr;
+		ret = rdma_create_qp(queue->cm_id, dev->pd, &attr);
+	}
 
 	queue->qp = queue->cm_id->qp;
+	queue->max_inline_data = ret ? 0 : attr.cap.max_inline_data;
 	return ret;
 }
 
@@ -1641,6 +1654,17 @@ static int nvme_rdma_post_send(struct nvme_rdma_queue *queue,
 	wr.opcode     = IB_WR_SEND;
 	wr.send_flags = IB_SEND_SIGNALED;
 
+	/*
+	 * Post a capsule without in-capsule data inline if it fits in the
+	 * granted inline size, so the HCA does not have to DMA-read it from
+	 * host memory.  The CPU copies inline data from a kernel virtual
+	 * address, so point the SGE at qe->data.
+	 */
+	if (num_sge == 1 && sge->length <= queue->max_inline_data) {
+		sge->addr = (uintptr_t)qe->data;
+		wr.send_flags |= IB_SEND_INLINE;
+	}
+
 	if (first)
 		first->next = &wr;
 	else
-- 
2.56.0