[PATCH net-next v2] net/mlx5e: Prefetch the packet a few strides ahead in the linear MPWQE path

[PATCH net-next v2] net/mlx5e: Prefetch the packet a few strides ahead in the linear MPWQE path

[Antonio Quartulli]

Without cache injection (DDIO/SDCI), the first accesses to a received
packet's headers and skb_shared_info are DRAM misses, and the
net_prefetch() in mlx5e_skb_from_cqe_mpwrq_linear() is too late to hide
them. This caps the packet rate of a queue.

In the linear striding RQ path a packet takes one stride and the device
fills the strides in order, so the packet MLX5E_RX_PREFETCH_DIST (4)
strides ahead is known without its CQE: prefetch its headers and page,
and the line its skb_shared_info will use if it has the length of the
current packet, as in bulk flows. That line matters when every packet
becomes its own skb: TCP inside an ovpn UDP tunnel gains 34% with it and
4% without.

AMD Ryzen 9 9950X, ConnectX-5 Ex, one RX queue on a dedicated CPU,
MTU 1500, mean of 3 runs:

                                   before       after
  UDP 512B flood, pktgen         2.52 Mpps    3.44 Mpps   +36%
  TCP, 1 stream                  44.6 Gb/s    48.0 Gb/s    +8%
    NAPI CPU cycles/packet            1334          715   -46%
  XDP_DROP reading the packet,
    NAPI CPU cycles/packet             409          280   -32%
  XDP_TX, NAPI CPU cycles/packet       829          554   -33%

With the change, the XDP packet rates were limited by the sender, so
cycles per packet are compared.

Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
---
v2:
- prefetch the packet MLX5E_RX_PREFETCH_DIST strides ahead instead of
  looking at the CQE ahead (Alexei); no CQE peek, RQ state bit or wq.h
  helper any more, and it also works with CQE compression (with it on:
  UDP floods +26%, TCP +6%, where v1 did nothing)
- also prefetch the skb_shared_info line, guessing the length from the
  current packet: without it, TCP inside a VPN tunnel (one skb per
  record) gained 4% instead of 34%
- XDP numbers added; XDP_DROP without touching the packet is unchanged,
  and the XDP packet rates were capped by the sender (12-16 Mpps), so
  cycles per packet are quoted
v1: https://lore.kernel.org/netdev/20261103235823.536706-1-antonio@openvpn.net/

Notes:
- Distance: with v1, 4, 6 and 8 were within 3% of each other.
- The prefetches are issued before the stride ahead is synced for the
  CPU; if the device hasn't written it yet (low rate), they are wasted.
- No prefetch for the first 4 strides of a WQE.
- Measured on one system only (Zen 5 desktop, no cache injection,
  CX-5 Ex); numbers from DDIO/SDCI systems and CX-6/CX-7 would be
  welcome.

 .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 22fea4739a2c..57da30b1bffe 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -2094,6 +2094,36 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
 	return skb;
 }
 
+#define MLX5E_RX_PREFETCH_DIST	4
+
+/* A packet that fits the MTU takes one stride of a linear MPWQE, and the
+ * device fills the strides in order: prefetch the packet
+ * MLX5E_RX_PREFETCH_DIST strides ahead, and the line its skb_shared_info
+ * will use if it has the length of the current one, as in a bulk flow.
+ */
+static void mlx5e_mpwrq_linear_prefetch(struct mlx5e_rq *rq,
+					struct mlx5e_mpw_info *wi,
+					u32 head_offset, u32 page_idx,
+					u16 cqe_bcnt)
+{
+	struct mlx5e_frag_page *frag_page;
+	void *va;
+
+	head_offset += MLX5E_RX_PREFETCH_DIST << rq->mpwqe.log_stride_sz;
+	page_idx += head_offset >> rq->mpwqe.page_shift;
+	if (page_idx >= rq->mpwqe.pages_per_wqe)
+		return;
+
+	head_offset &= (1 << rq->mpwqe.page_shift) - 1;
+	frag_page = &wi->alloc_units.frag_pages[page_idx];
+	va = netmem_address(frag_page->netmem) + head_offset;
+	prefetch(netmem_to_page(frag_page->netmem));
+	net_prefetch(va + rq->buff.headroom);
+	/* with XDP the packet may not become an skb */
+	if (!rcu_access_pointer(rq->xdp_prog))
+		prefetchw(va + SKB_DATA_ALIGN(rq->buff.headroom + cqe_bcnt));
+}
+
 static struct sk_buff *
 mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
 				struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset,
@@ -2114,6 +2144,8 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
 		return NULL;
 	}
 
+	mlx5e_mpwrq_linear_prefetch(rq, wi, head_offset, page_idx, cqe_bcnt);
+
 	va             = netmem_address(frag_page->netmem) + head_offset;
 	data           = va + rx_headroom;
 	frag_size      = MLX5_SKB_FRAG_SZ(rx_headroom + cqe_bcnt);
-- 
2.56.0

Re: [PATCH net-next v2] net/mlx5e: Prefetch the packet a few strides ahead in the linear MPWQE path

[Dragos Tatulea]

Hi,

On 04.11.26 20:52, Antonio Quartulli wrote:
> Without cache injection (DDIO/SDCI), the first accesses to a received
> packet's headers and skb_shared_info are DRAM misses, and the
> net_prefetch() in mlx5e_skb_from_cqe_mpwrq_linear() is too late to hide
> them. This caps the packet rate of a queue.
> 
> In the linear striding RQ path a packet takes one stride and the device
> fills the strides in order, so the packet MLX5E_RX_PREFETCH_DIST (4)
> strides ahead is known without its CQE: prefetch its headers and page,
> and the line its skb_shared_info will use if it has the length of the
> current packet, as in bulk flows. That line matters when every packet
> becomes its own skb: TCP inside an ovpn UDP tunnel gains 34% with it and
> 4% without.
> 
> AMD Ryzen 9 9950X, ConnectX-5 Ex, one RX queue on a dedicated CPU,
> MTU 1500, mean of 3 runs:
> 
>                                    before       after
>   UDP 512B flood, pktgen         2.52 Mpps    3.44 Mpps   +36%
>   TCP, 1 stream                  44.6 Gb/s    48.0 Gb/s    +8%
>     NAPI CPU cycles/packet            1334          715   -46%
>   XDP_DROP reading the packet,
>     NAPI CPU cycles/packet             409          280   -32%
>   XDP_TX, NAPI CPU cycles/packet       829          554   -33%
> 
> With the change, the XDP packet rates were limited by the sender, so
> cycles per packet are compared.
> 
Thanks for your patch! These look indeed like enticing numbers. I'd
like to take some time to try this at scale and on other NICs (CX8)
and CPUs (Grace) and also check latency.

Thanks,
Dragos