[PATCH 0/1] xprtrdma: Post small RPC Calls inline

[PATCH 0/1] xprtrdma: Post small RPC Calls inline

[Andrei-Alexandru Bleortu]

The client posts every RPC Call by reference, so the HCA reads it from
host memory before transmitting. This patch asks for 256 bytes of
inline data at QP creation and posts small Calls with IB_SEND_INLINE.
If the device refuses, it asks for less. The commit message has the
details and the measurements.

The matching server change, "svcrdma: Post small Replies inline", is
going to the NFSD maintainers on its own. The two do not depend on
each other.

It builds with W=1 on Anna's linux-next without new warnings. I tested
it in a QEMU guest with rdma_rxe on a dummy netdev. The NFS client and
server ran on the same host over RDMA, using NFSv3 and NFSv4.1.
8 MiB of 4 KiB O_DIRECT writes and reads, and 200 small files,
compared equal. 9836 Calls went inline and 214 by reference.

I also made rxe refuse inline requests above a limit and report a
chosen vendor ID. The data still compared equal in each case:

  vendor ID   limit        QP got
  Intel       216/101/48   216/101/48 bytes
  Alibaba     96           96 bytes
  rxe         101          96 bytes

At 216 bytes 5632 Calls went inline; at 96 or 101 only 3 did, as most
Calls in this test are larger. The guest kernel also carried the
svcrdma patch. Its xprtrdma client code is linux-next's without two
later commits, 0b5bb4e586f7 and 9ee534a75912, which do not change how
Calls are posted.

I wrote this patch with an LLM coding assistant (Claude Code), which
also drafted the commit message and ran the build and the VM test.

Andrei-Alexandru Bleortu (1):
  xprtrdma: Post small RPC Calls inline

 net/sunrpc/xprtrdma/rpc_rdma.c | 30 ++++++++++++++++++
 net/sunrpc/xprtrdma/verbs.c    | 58 +++++++++++++++++++++++++++++++++-
 2 files changed, 87 insertions(+), 1 deletion(-)


base-commit: 9ee534a75912361308b1f956bc3e7895a082a422
-- 
2.56.0

[PATCH 1/1] xprtrdma: Post small RPC Calls inline

[Andrei-Alexandru Bleortu]

rpcrdma_ep_create() asks for no inline data, so every RPC Call is
posted by reference. Before the HCA can transmit a Call it has to
DMA-read the transport header and the RPC message from host memory,
and for a small Call that read is on the latency path of every RPC.

Ask for 256 bytes of inline data per Send WR when creating the QP. If
the device refuses, ask for less. Intel irdma has a fixed limit of 216,
101 or 48 bytes, depending on the generation, and Alibaba erdma one of
96. On those devices, found by vendor ID, try these limits first, then
step down 16 bytes at a time to none. libfabric's verbs provider also
probes the inline limit by trial QP creation.

Post a Call with IB_SEND_INLINE when its SGEs cover only the
persistently mapped transport header and head buffers and it fits in the
granted size. Calls with page or tail SGEs, and larger Calls, are posted
by reference as before. IB_SEND_INLINE is unrelated to the RPC-over-RDMA
inline threshold.

Providers copy inline data from sg_list[].addr as a kernel virtual
address, so on the inline path the SGEs point at the regbufs' virtual
addresses. These SGEs are rebuilt for every Send and are not used for
unmapping: rpcrdma_sendctx_dma_unmap() starts at the third SGE, and the
regbufs stay mapped.

fio ran 4 KiB direct I/O at queue depth 1 on an NFSv3 mount of a tmpfs
export, between two Xeon Platinum 8360Y hosts with ConnectX-6 Dx over
RoCEv2 at 100 Gb/s on a 7.0 kernel. An earlier version of this patch
lowered the mean completion latency of random writes by 0.1923 us
(95% CI 0.1139-0.2769) and of random reads by 0.2705 us (0.1908-0.3502).
That version made the same check in frwr_send() and asked for 512
bytes. The largest Call in those runs was 216 bytes.

Assisted-by: LLM
Signed-off-by: Andrei-Alexandru Bleortu <me@andrei-z.com>
---
 net/sunrpc/xprtrdma/rpc_rdma.c | 30 ++++++++++++++++++
 net/sunrpc/xprtrdma/verbs.c    | 58 +++++++++++++++++++++++++++++++++-
 2 files changed, 87 insertions(+), 1 deletion(-)

diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
index 7e5713367291..9c0ef7303439 100644
--- a/net/sunrpc/xprtrdma/rpc_rdma.c
+++ b/net/sunrpc/xprtrdma/rpc_rdma.c
@@ -732,6 +732,33 @@ static bool rpcrdma_prepare_readch(struct rpcrdma_xprt *r_xprt,
 	return true;
 }
 
+/* Post a Send inline when its SGEs cover only the transport header
+ * and head buffers and it fits in the granted inline size. The CPU
+ * then copies it into the send WQE and the HCA does not DMA-read it.
+ * Inline data is copied from kernel virtual addresses, so point the
+ * SGEs there. The buffers stay DMA-mapped and are not unmapped
+ * through these SGEs.
+ */
+static void rpcrdma_prepare_send_inline(struct rpcrdma_xprt *r_xprt,
+					struct rpcrdma_req *req)
+{
+	struct ib_sge *sge = req->rl_wr.sg_list;
+	unsigned int len = 0;
+	int i;
+
+	if (req->rl_sendctx->sc_unmap_count)
+		return;
+	for (i = 0; i < req->rl_wr.num_sge; i++)
+		len += sge[i].length;
+	if (len > r_xprt->rx_ep->re_attr.cap.max_inline_data)
+		return;
+
+	sge[0].addr = (uintptr_t)rdmab_data(req->rl_rdmabuf);
+	if (req->rl_wr.num_sge > 1)
+		sge[1].addr = (uintptr_t)rdmab_data(req->rl_sendbuf);
+	req->rl_wr.send_flags |= IB_SEND_INLINE;
+}
+
 /**
  * rpcrdma_prepare_send_sges - Construct SGEs for a Send WR
  * @r_xprt: controlling transport
@@ -760,6 +787,7 @@ inline int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt,
 	req->rl_wr.sg_list = req->rl_sendctx->sc_sges;
 	req->rl_wr.num_sge = 0;
 	req->rl_wr.opcode = IB_WR_SEND;
+	req->rl_wr.send_flags &= ~IB_SEND_INLINE;
 
 	rpcrdma_prepare_hdr_sge(r_xprt, req, hdrlen);
 
@@ -783,6 +811,8 @@ inline int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt,
 		goto out_unmap;
 	}
 
+	rpcrdma_prepare_send_inline(r_xprt, req);
+
 	/* The Send-side owner releases this reference when the
 	 * Send has completed.
 	 */
diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
index 54957295a631..2e8a7fb6b1a6 100644
--- a/net/sunrpc/xprtrdma/verbs.c
+++ b/net/sunrpc/xprtrdma/verbs.c
@@ -55,6 +55,7 @@
 #include <linux/sunrpc/addr.h>
 #include <linux/sunrpc/svc_rdma.h>
 #include <linux/log2.h>
+#include <linux/pci_ids.h>
 
 #include <asm/barrier.h>
 
@@ -63,6 +64,44 @@
 #include "xprt_rdma.h"
 #include <trace/events/rpcrdma.h>
 
+/* Bytes of inline data (IB_SEND_INLINE) to ask for on each Send WR.
+ * This is unrelated to the RPC-over-RDMA inline threshold.
+ */
+#define RPCRDMA_WQE_INLINE_SIZE	(256)
+#define RPCRDMA_WQE_INLINE_STEP	(16)
+
+/* The verbs API cannot report a device's inline data limit. Some
+ * drivers have a fixed one: Intel irdma 216, 101 or 48 depending on
+ * the generation, Alibaba erdma 96. Try those first, by vendor ID,
+ * then step down RPCRDMA_WQE_INLINE_STEP bytes at a time to zero.
+ * libfabric's verbs provider also probes the limit
+ * (vrb_find_max_inline()).
+ */
+static const struct {
+	u32	vendor_id;
+	u32	max_inline;
+} rpcrdma_inline_limits[] = {
+	{ PCI_VENDOR_ID_INTEL, 216 },
+	{ PCI_VENDOR_ID_INTEL, 101 },
+	{ PCI_VENDOR_ID_INTEL, 48 },
+	{ PCI_VENDOR_ID_ALIBABA, 96 },
+};
+
+/* Inline data size to ask for: on the first try, @size capped at the
+ * vendor's first known limit; after a refusal, the vendor's next
+ * smaller known limit, else one step smaller.
+ */
+static u32 rpcrdma_inline_size(u32 vendor_id, u32 size, bool refused)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(rpcrdma_inline_limits); i++)
+		if (rpcrdma_inline_limits[i].vendor_id == vendor_id &&
+		    (!refused || rpcrdma_inline_limits[i].max_inline < size))
+			return min(rpcrdma_inline_limits[i].max_inline, size);
+	return refused ? size - min(size, RPCRDMA_WQE_INLINE_STEP) : size;
+}
+
 static int rpcrdma_sendctxs_create(struct rpcrdma_xprt *r_xprt);
 static void rpcrdma_sendctxs_destroy(struct rpcrdma_xprt *r_xprt);
 static unsigned long rpcrdma_sendctx_next(struct rpcrdma_buffer *buf,
@@ -392,8 +431,10 @@ static int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt)
 {
 	struct rpcrdma_connect_private *pmsg;
 	struct ib_device *device;
+	struct ib_qp_init_attr attr;
 	struct rdma_cm_id *id;
 	struct rpcrdma_ep *ep;
+	u32 inline_size;
 	int rc;
 
 	ep = kzalloc_obj(*ep, XPRTRDMA_GFP_FLAGS);
@@ -487,7 +528,22 @@ static int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt)
 		goto out_destroy;
 	}
 
-	rc = rdma_create_qp(id, ep->re_pd, &ep->re_attr);
+	/* Ask for enough inline data to copy small Sends into the send
+	 * WQE (see rpcrdma_prepare_send_inline()), or for as much as the
+	 * device accepts.
+	 */
+	attr = ep->re_attr;
+	inline_size = rpcrdma_inline_size(device->attrs.vendor_id,
+					  RPCRDMA_WQE_INLINE_SIZE, false);
+	for (;;) {
+		ep->re_attr = attr;
+		ep->re_attr.cap.max_inline_data = inline_size;
+		rc = rdma_create_qp(id, ep->re_pd, &ep->re_attr);
+		if (!rc || !inline_size)
+			break;
+		inline_size = rpcrdma_inline_size(device->attrs.vendor_id,
+						  inline_size, true);
+	}
 	if (rc)
 		goto out_destroy;
 
-- 
2.56.0

Re: [PATCH 1/1] xprtrdma: Post small RPC Calls inline

[Chuck Lever]

On Thu, Oct 15, 2026, at 8:48 AM, Andrei-Alexandru Bleortu wrote:
> rpcrdma_ep_create() asks for no inline data, so every RPC Call is
> posted by reference. Before the HCA can transmit a Call it has to
> DMA-read the transport header and the RPC message from host memory,
> and for a small Call that read is on the latency path of every RPC.
>
> Ask for 256 bytes of inline data per Send WR when creating the QP. If
> the device refuses, ask for less. Intel irdma has a fixed limit of 216,
> 101 or 48 bytes, depending on the generation, and Alibaba erdma one of
> 96. On those devices, found by vendor ID, try these limits first, then
> step down 16 bytes at a time to none. libfabric's verbs provider also
> probes the inline limit by trial QP creation.
>
> Post a Call with IB_SEND_INLINE when its SGEs cover only the
> persistently mapped transport header and head buffers and it fits in the
> granted size. Calls with page or tail SGEs, and larger Calls, are posted
> by reference as before. IB_SEND_INLINE is unrelated to the RPC-over-RDMA
> inline threshold.
>
> Providers copy inline data from sg_list[].addr as a kernel virtual
> address, so on the inline path the SGEs point at the regbufs' virtual
> addresses. These SGEs are rebuilt for every Send and are not used for
> unmapping: rpcrdma_sendctx_dma_unmap() starts at the third SGE, and the
> regbufs stay mapped.
>
> fio ran 4 KiB direct I/O at queue depth 1 on an NFSv3 mount of a tmpfs
> export, between two Xeon Platinum 8360Y hosts with ConnectX-6 Dx over
> RoCEv2 at 100 Gb/s on a 7.0 kernel. An earlier version of this patch
> lowered the mean completion latency of random writes by 0.1923 us
> (95% CI 0.1139-0.2769) and of random reads by 0.2705 us (0.1908-0.3502).
> That version made the same check in frwr_send() and asked for 512
> bytes. The largest Call in those runs was 216 bytes.

These mean completion latency gains are noise compared to the client's
own RPC tasking machinery. Perhaps the additional code complexity is
not worth it yet.


> Assisted-by: LLM
> Signed-off-by: Andrei-Alexandru Bleortu <me@andrei-z.com>

> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 54957295a631..2e8a7fb6b1a6 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -55,6 +55,7 @@
>  #include <linux/sunrpc/addr.h>
>  #include <linux/sunrpc/svc_rdma.h>
>  #include <linux/log2.h>
> +#include <linux/pci_ids.h>
> 
>  #include <asm/barrier.h>
> 
> @@ -63,6 +64,44 @@
>  #include "xprt_rdma.h"
>  #include <trace/events/rpcrdma.h>
> 
> +/* Bytes of inline data (IB_SEND_INLINE) to ask for on each Send WR.
> + * This is unrelated to the RPC-over-RDMA inline threshold.
> + */
> +#define RPCRDMA_WQE_INLINE_SIZE	(256)
> +#define RPCRDMA_WQE_INLINE_STEP	(16)
> +
> +/* The verbs API cannot report a device's inline data limit. Some
> + * drivers have a fixed one: Intel irdma 216, 101 or 48 depending on
> + * the generation, Alibaba erdma 96.

The table itself already states this, so the above comment does not
add value.


>                                       Try those first, by vendor ID,
> + * then step down RPCRDMA_WQE_INLINE_STEP bytes at a time to zero.
> + * libfabric's verbs provider also probes the limit
> + * (vrb_find_max_inline()).
> + */
> +static const struct {
> +	u32	vendor_id;
> +	u32	max_inline;
> +} rpcrdma_inline_limits[] = {
> +	{ PCI_VENDOR_ID_INTEL, 216 },
> +	{ PCI_VENDOR_ID_INTEL, 101 },
> +	{ PCI_VENDOR_ID_INTEL, 48 },
> +	{ PCI_VENDOR_ID_ALIBABA, 96 },
> +};

General policy is not to put device-specific details like this
in ULP code. So, NACK on holding these numbers in a table in
the rpcrdma.ko module.

I didn't look at the other ULPs you modified, but I expect that
they have a similar need to obtain this value. That suggests you
should first add a helper function to the IB core, or provide
the value in the qp attrs.

The retry-based approach means a connect can take over a dozen
QP creation retries on a device that does not support
IB_SEND_INLINE. To avoid the retry, you might consider adding a
new QP creation flag, say IB_QP_CREATE_INLINE_CLAMP, that changes
one rule of ib_create_qp(): with the flag set, a driver that
cannot grant the requested cap.max_inline_data grants the largest
value this QP can support instead of failing with -EINVAL, and
reports it back in cap.max_inline_data as it already does for
the rounded-up value. Zero is a legal answer, which is what mlx4,
pvrdma and rdmavt already return for kernel QPs.

The smaller sizes these devices provide are barely larger than
the RPC/RDMA transport header. It might be best to set a lower
bound (don't use inline at all for devices with inline limits
that can't cover the RPC/RDMA and RPC headers).

Also, at least for mlx5, the inline limit trades directly
against send queue depth and the QP's host memory footprint.

TL;DR: I'm not convinced adding several dozen lines of code
pays for a tenth of a microsecond latency improvement.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)

Re: [PATCH 1/1] xprtrdma: Post small RPC Calls inline

[Andrei-Alexandru Bleortu]

On Thu, Oct 15, 2026, Chuck Lever wrote:
> These mean completion latency gains are noise compared to the client's
> own RPC tasking machinery. Perhaps the additional code complexity is
> not worth it yet.

Thanks for looking. The gain is real but small. Over 40 paired runs
its interval excludes zero, and a comment-only patch moved the result
by -0.0139 us. But the 4 KiB write took 48.4 us on average, so the Call
saves 0.4 % of it, and 0.8 % of the 35.7 us read.

> General policy is not to put device-specific details like this
> in ULP code. So, NACK on holding these numbers in a table in
> the rpcrdma.ko module.

Understood. Bart asked for the same on the SRP patch, and I proposed
the create flag you describe on linux-rdma:
https://lore.kernel.org/all/20261015212227.3338582-1-me@andrei-z.com/

I will drop this patch and the svcrdma one for now. If the flag goes
in, the client change shrinks to setting the flag and the check in the
send path, with your lower bound. You can judge then whether it is
worth it.