[PATCH net-next v2] net/mlx5e: Prefetch the packet a few strides ahead in the linear MPWQE path

[PATCH net-next v2] net/mlx5e: Prefetch the packet a few strides ahead in the linear MPWQE path

[Antonio Quartulli]

Without cache injection (DDIO/SDCI), the first accesses to a received
packet's headers and skb_shared_info are DRAM misses, and the
net_prefetch() in mlx5e_skb_from_cqe_mpwrq_linear() is too late to hide
them. This caps the packet rate of a queue.

In the linear striding RQ path a packet takes one stride and the device
fills the strides in order, so the packet MLX5E_RX_PREFETCH_DIST (4)
strides ahead is known without its CQE: prefetch its headers and page,
and the line its skb_shared_info will use if it has the length of the
current packet, as in bulk flows. That line matters when every packet
becomes its own skb: TCP inside an ovpn UDP tunnel gains 34% with it and
4% without.

AMD Ryzen 9 9950X, ConnectX-5 Ex, one RX queue on a dedicated CPU,
MTU 1500, mean of 3 runs:

                                   before       after
  UDP 512B flood, pktgen         2.52 Mpps    3.44 Mpps   +36%
  TCP, 1 stream                  44.6 Gb/s    48.0 Gb/s    +8%
    NAPI CPU cycles/packet            1334          715   -46%
  XDP_DROP reading the packet,
    NAPI CPU cycles/packet             409          280   -32%
  XDP_TX, NAPI CPU cycles/packet       829          554   -33%

With the change, the XDP packet rates were limited by the sender, so
cycles per packet are compared.

Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
---
v2:
- prefetch the packet MLX5E_RX_PREFETCH_DIST strides ahead instead of
  looking at the CQE ahead (Alexei); no CQE peek, RQ state bit or wq.h
  helper any more, and it also works with CQE compression (with it on:
  UDP floods +26%, TCP +6%, where v1 did nothing)
- also prefetch the skb_shared_info line, guessing the length from the
  current packet: without it, TCP inside a VPN tunnel (one skb per
  record) gained 4% instead of 34%
- XDP numbers added; XDP_DROP without touching the packet is unchanged,
  and the XDP packet rates were capped by the sender (12-16 Mpps), so
  cycles per packet are quoted
v1: https://lore.kernel.org/netdev/20261103235823.536706-1-antonio@openvpn.net/

Notes:
- Distance: with v1, 4, 6 and 8 were within 3% of each other.
- The prefetches are issued before the stride ahead is synced for the
  CPU; if the device hasn't written it yet (low rate), they are wasted.
- No prefetch for the first 4 strides of a WQE.
- Measured on one system only (Zen 5 desktop, no cache injection,
  CX-5 Ex); numbers from DDIO/SDCI systems and CX-6/CX-7 would be
  welcome.

 .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 22fea4739a2c..57da30b1bffe 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -2094,6 +2094,36 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
 	return skb;
 }
 
+#define MLX5E_RX_PREFETCH_DIST	4
+
+/* A packet that fits the MTU takes one stride of a linear MPWQE, and the
+ * device fills the strides in order: prefetch the packet
+ * MLX5E_RX_PREFETCH_DIST strides ahead, and the line its skb_shared_info
+ * will use if it has the length of the current one, as in a bulk flow.
+ */
+static void mlx5e_mpwrq_linear_prefetch(struct mlx5e_rq *rq,
+					struct mlx5e_mpw_info *wi,
+					u32 head_offset, u32 page_idx,
+					u16 cqe_bcnt)
+{
+	struct mlx5e_frag_page *frag_page;
+	void *va;
+
+	head_offset += MLX5E_RX_PREFETCH_DIST << rq->mpwqe.log_stride_sz;
+	page_idx += head_offset >> rq->mpwqe.page_shift;
+	if (page_idx >= rq->mpwqe.pages_per_wqe)
+		return;
+
+	head_offset &= (1 << rq->mpwqe.page_shift) - 1;
+	frag_page = &wi->alloc_units.frag_pages[page_idx];
+	va = netmem_address(frag_page->netmem) + head_offset;
+	prefetch(netmem_to_page(frag_page->netmem));
+	net_prefetch(va + rq->buff.headroom);
+	/* with XDP the packet may not become an skb */
+	if (!rcu_access_pointer(rq->xdp_prog))
+		prefetchw(va + SKB_DATA_ALIGN(rq->buff.headroom + cqe_bcnt));
+}
+
 static struct sk_buff *
 mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
 				struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset,
@@ -2114,6 +2144,8 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
 		return NULL;
 	}
 
+	mlx5e_mpwrq_linear_prefetch(rq, wi, head_offset, page_idx, cqe_bcnt);
+
 	va             = netmem_address(frag_page->netmem) + head_offset;
 	data           = va + rx_headroom;
 	frag_size      = MLX5_SKB_FRAG_SZ(rx_headroom + cqe_bcnt);
-- 
2.56.0