Re: [PATCH net-next] net/mlx5e: Prefetch packets of upcoming CQEs in the linear MPWQE path

["Alexei Starovoitov" <alexei.starovoitov@gmail.com>]

On Tue, Nov 03, 2026 at 03:58 PM Antonio Quartulli <antonio@openvpn.net> wrote:
> While processing a CQE, look at the CQE MLX5E_RX_PREFETCH_DIST (4) entries
> ahead and, if it is completed, prefetch its packet headers, skb_shared_info
> and page. Also prefetch the CQE twice as far ahead.
>
> Looking ahead requires uncompressed CQEs, so this is enabled only for
> linear striding RQs without CQE compression.

can we prefetch 4 strides ahead without the CQE, like gve_rx() ?
In linear mode a packet takes one stride and HW fills the strides in order.
mlx5e_alloc_rx_mpwqe() allocates all pages of the WQE.
So mlx5e_skb_from_cqe_mpwrq_linear() can compute the address of the packet
4 strides ahead from head_offset and page_idx.

  per packet                 the patch     4 strides ahead
  -------------------------  ------------  ------------------
  CQE reads                  2             1
  dma_rmb()                  2             1
  mlx5e_get_mpw_info()       2             1
  headers, page              prefetch      prefetch
  skb_shared_info            prefetch      no, needs cqe_bcnt
  next CQEs                  prefetch      no
  first 4 strides of a WQE   prefetch      no
  stride is not filled yet   no prefetch   wasted prefetch
  CQE compression            not covered   covered

Then no need for MLX5E_RQ_STATE_PREFETCH_NEXT and mlx5_cqwq_get_cqe_ctr().

Something like this (not even compile tested):

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 22fea47..31d5bc4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -2094,6 +2094,29 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
 	return skb;
 }

+#define MLX5E_RX_PREFETCH_DIST	4
+
+/* A packet takes one stride of a linear MPWQE. The address of the packet
+ * MLX5E_RX_PREFETCH_DIST strides ahead is known without its CQE.
+ */
+static void mlx5e_mpwrq_linear_prefetch(struct mlx5e_rq *rq,
+					struct mlx5e_mpw_info *wi,
+					u32 head_offset, u32 page_idx)
+{
+	struct mlx5e_frag_page *frag_page;
+
+	head_offset += MLX5E_RX_PREFETCH_DIST << rq->mpwqe.log_stride_sz;
+	page_idx += head_offset >> rq->mpwqe.page_shift;
+	if (page_idx >= rq->mpwqe.pages_per_wqe)
+		return;
+
+	head_offset &= BIT(rq->mpwqe.page_shift) - 1;
+	frag_page = &wi->alloc_units.frag_pages[page_idx];
+	prefetch(netmem_to_page(frag_page->netmem));
+	net_prefetch(netmem_address(frag_page->netmem) + head_offset +
+		     rq->buff.headroom);
+}
+
 static struct sk_buff *
 mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
 				struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset,
@@ -2114,6 +2137,8 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
 		return NULL;
 	}

+	mlx5e_mpwrq_linear_prefetch(rq, wi, head_offset, page_idx);
+
 	va             = netmem_address(frag_page->netmem) + head_offset;
 	data           = va + rx_headroom;
 	frag_size      = MLX5_SKB_FRAG_SZ(rx_headroom + cqe_bcnt);

> - With XDP, skb_shared_info is not prefetched. XDP_DROP rate, low-rate
>   traffic and multiple queues were not measured.

Pls add XDP_DROP and XDP_TX numbers.