Re: [PATCH 1/1] xprtrdma: Post small RPC Calls inline

["Chuck Lever" <cel@kernel.org>]

On Thu, Oct 15, 2026, at 8:48 AM, Andrei-Alexandru Bleortu wrote:
> rpcrdma_ep_create() asks for no inline data, so every RPC Call is
> posted by reference. Before the HCA can transmit a Call it has to
> DMA-read the transport header and the RPC message from host memory,
> and for a small Call that read is on the latency path of every RPC.
>
> Ask for 256 bytes of inline data per Send WR when creating the QP. If
> the device refuses, ask for less. Intel irdma has a fixed limit of 216,
> 101 or 48 bytes, depending on the generation, and Alibaba erdma one of
> 96. On those devices, found by vendor ID, try these limits first, then
> step down 16 bytes at a time to none. libfabric's verbs provider also
> probes the inline limit by trial QP creation.
>
> Post a Call with IB_SEND_INLINE when its SGEs cover only the
> persistently mapped transport header and head buffers and it fits in the
> granted size. Calls with page or tail SGEs, and larger Calls, are posted
> by reference as before. IB_SEND_INLINE is unrelated to the RPC-over-RDMA
> inline threshold.
>
> Providers copy inline data from sg_list[].addr as a kernel virtual
> address, so on the inline path the SGEs point at the regbufs' virtual
> addresses. These SGEs are rebuilt for every Send and are not used for
> unmapping: rpcrdma_sendctx_dma_unmap() starts at the third SGE, and the
> regbufs stay mapped.
>
> fio ran 4 KiB direct I/O at queue depth 1 on an NFSv3 mount of a tmpfs
> export, between two Xeon Platinum 8360Y hosts with ConnectX-6 Dx over
> RoCEv2 at 100 Gb/s on a 7.0 kernel. An earlier version of this patch
> lowered the mean completion latency of random writes by 0.1923 us
> (95% CI 0.1139-0.2769) and of random reads by 0.2705 us (0.1908-0.3502).
> That version made the same check in frwr_send() and asked for 512
> bytes. The largest Call in those runs was 216 bytes.

These mean completion latency gains are noise compared to the client's
own RPC tasking machinery. Perhaps the additional code complexity is
not worth it yet.


> Assisted-by: LLM
> Signed-off-by: Andrei-Alexandru Bleortu <me@andrei-z.com>

> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 54957295a631..2e8a7fb6b1a6 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -55,6 +55,7 @@
>  #include <linux/sunrpc/addr.h>
>  #include <linux/sunrpc/svc_rdma.h>
>  #include <linux/log2.h>
> +#include <linux/pci_ids.h>
> 
>  #include <asm/barrier.h>
> 
> @@ -63,6 +64,44 @@
>  #include "xprt_rdma.h"
>  #include <trace/events/rpcrdma.h>
> 
> +/* Bytes of inline data (IB_SEND_INLINE) to ask for on each Send WR.
> + * This is unrelated to the RPC-over-RDMA inline threshold.
> + */
> +#define RPCRDMA_WQE_INLINE_SIZE	(256)
> +#define RPCRDMA_WQE_INLINE_STEP	(16)
> +
> +/* The verbs API cannot report a device's inline data limit. Some
> + * drivers have a fixed one: Intel irdma 216, 101 or 48 depending on
> + * the generation, Alibaba erdma 96.

The table itself already states this, so the above comment does not
add value.


>                                       Try those first, by vendor ID,
> + * then step down RPCRDMA_WQE_INLINE_STEP bytes at a time to zero.
> + * libfabric's verbs provider also probes the limit
> + * (vrb_find_max_inline()).
> + */
> +static const struct {
> +	u32	vendor_id;
> +	u32	max_inline;
> +} rpcrdma_inline_limits[] = {
> +	{ PCI_VENDOR_ID_INTEL, 216 },
> +	{ PCI_VENDOR_ID_INTEL, 101 },
> +	{ PCI_VENDOR_ID_INTEL, 48 },
> +	{ PCI_VENDOR_ID_ALIBABA, 96 },
> +};

General policy is not to put device-specific details like this
in ULP code. So, NACK on holding these numbers in a table in
the rpcrdma.ko module.

I didn't look at the other ULPs you modified, but I expect that
they have a similar need to obtain this value. That suggests you
should first add a helper function to the IB core, or provide
the value in the qp attrs.

The retry-based approach means a connect can take over a dozen
QP creation retries on a device that does not support
IB_SEND_INLINE. To avoid the retry, you might consider adding a
new QP creation flag, say IB_QP_CREATE_INLINE_CLAMP, that changes
one rule of ib_create_qp(): with the flag set, a driver that
cannot grant the requested cap.max_inline_data grants the largest
value this QP can support instead of failing with -EINVAL, and
reports it back in cap.max_inline_data as it already does for
the rounded-up value. Zero is a legal answer, which is what mlx4,
pvrdma and rdmavt already return for kernel QPs.

The smaller sizes these devices provide are barely larger than
the RPC/RDMA transport header. It might be best to set a lower
bound (don't use inline at all for devices with inline limits
that can't cover the RPC/RDMA and RPC headers).

Also, at least for mlx5, the inline limit trades directly
against send queue depth and the QP's host memory footprint.

TL;DR: I'm not convinced adding several dozen lines of code
pays for a tenth of a microsecond latency improvement.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)