From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD8F21EB9E3 for ; Wed, 12 Nov 2025 13:55:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.179 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762955723; cv=none; b=Sz30jjUvM8Y7Jar9leUuIVUR6QPrsYSlwfsQnr9ZyHgak6KznMNi9aTnHWQ50b/9888feHtEtrbd2D81UYX02JpbjLvD29mg9Oo5OLvcLRBhNETVsjxsrFF0+cT1M5zcqYSaomB3DQMrW4pK73n8+znRuqYV3CO+IjDWDGNJXvQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762955723; c=relaxed/simple; bh=mQTO3qMBrrbrec+vOaxyEQHikAOjUWJdth7psxqKQd8=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=FXqj9XSz0l19Qj8muCdj8MnAQDyQDkOh5wkqZPe6zRfCxqE3Sa/zwEghkX56MDoiBvRFMSPjPTWDcKLziPB/sHojsJrxrmXq2rRotdTyyg5z+yR/8jjI6//EzWUW76T/wUS/gVHUKl9+m7bepp1ikvL6YPZFwrzm0YK5mAic3eM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rkBDN33T; arc=none smtp.client-ip=209.85.160.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rkBDN33T" Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-4ed9c1924adso7362401cf.1 for ; Wed, 12 Nov 2025 05:55:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1762955720; x=1763560520; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=olHOzv1HDFcRoKcRPLEDHq3xryx1qVQZjsYC+KidFkM=; b=rkBDN33TmH8m8SbC49wKLYrQ8IUuqrxk8bJFNuT9qB91Timhuqs4SthYuzx/hXV5is nXunJbot2eU0+D3yuMCLcmK74VkT+rl1iC/qcEN45oe3f9L3UV/w5FY0Y6XXpbKP1TU1 r7RWuyaC9B2xZtO2HdmQS1Pw38bQVxi187CHoeeyZ0AJVb+strUJZlnf7IkLeAS0RM4w pf8UC/BV58GqIxqFoy8hNf86YFaVl2gh6Mtct6OHn35FqX3PySe0jKZ5z8Ew6lfnYlM2 b85J/EFpExGXhOvgt6sXajOE7ncZfIx7PJolb8+N5m81ZfgRHJSXDBkXQshiaZzT8VzT PFPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762955720; x=1763560520; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=olHOzv1HDFcRoKcRPLEDHq3xryx1qVQZjsYC+KidFkM=; b=XbkFfcuC6F/V3I3CZN9pHJ5BSWpW/0lba2EdJG/1fdftrVETYQwQc516BLq4kPeYKT PWYiG5z5wi7YgMOmiinPbJf+VIt1Ys+TEbL/9XHAAScni+lzTmGsfPe/k9VQ13uoXUKq bJBeHP32JKylzUZf0AYUQGOR+cvJYGDPikHFUsX8Gdgx5DKI1BFXXkY1NJ2fnaR61R8d HJPdFsqE3xvWOTDUmp6+zvCALjL90bn2SywtqUAdZjtmw0rWWevWdfaNLkPKppf2ZGj1 KBlIXNUGdbYe4sE5eIKHCiHmEA5+xhD9ctXfaj3wb79uKlBFutsl/Vn6ZH9agl5eWGan zO/g== X-Forwarded-Encrypted: i=1; AJvYcCVCj0HCpTfSapMnk0sZt6SOQP1yY0CtauDh/hC0VoZt6cuGoRTqOmUSbfm14bfRlWyGXG4P2pHPC5/9s/Q=@vger.kernel.org X-Gm-Message-State: AOJu0YzFRmkSgXnlNDaIYjS7GBxd4zX3rh2yg+/N+4NMhlZ5HIx5uXj7 tCPP3SijOM90IvoPoO8tsg7i8wR0faUyWAhtWtlA6+N1TUVMJ/P7Hd8ABXs+nFpIwELWwGfZIq2 ahAUTRcZZHaucVvnllup6tHcowsEHQYy68E01VhM6 X-Gm-Gg: ASbGncur1Mp0Np1vhVCAA+xjbWRUj/+cZz6OxPNsiYFZvFVOazs+SLqG+x3chePDVZ3 B0GQ+DZs3aHbvW0TdzXoLDiirZhXzR0lTNJ9sovsv4AKDNTvcNTrZct+2Hsw6U0oCw6HTH0yquz JuS0iLLT4Lr+7cmMkU2uEIpFN8E1B2b/OTQgrZfrQRjx+CMdAoZ4TkFsGKWt+Qncw3RIG13jWDp MctSu0xfSMszBkOBOpNCSrlxGSSFKMfWn2VimuxY1xR1Wu40lxI569At8Lms1kp63MjkPAU X-Google-Smtp-Source: AGHT+IH5mtOTw5ql/3tyCPfaJIqRYelbwEuY2GxwU9SpSn2Er1u7sqEWL7WpBlKzBs9sxmpYzFu7SfnMl4N6pUgTzCk= X-Received: by 2002:a05:622a:54e:b0:4ec:f07c:3e73 with SMTP id d75a77b69052e-4eddbe1c28amr37326511cf.76.1762955720119; Wed, 12 Nov 2025 05:55:20 -0800 (PST) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <1762952506-23593-1-git-send-email-gargaditya@linux.microsoft.com> <1762952506-23593-2-git-send-email-gargaditya@linux.microsoft.com> In-Reply-To: <1762952506-23593-2-git-send-email-gargaditya@linux.microsoft.com> From: Eric Dumazet Date: Wed, 12 Nov 2025 05:55:07 -0800 X-Gm-Features: AWmQ_bluu1-AEflxU_3suMcewGFMw8kM6f7K2JJf4BGWU2evk1xj6w6Ga79O0hY Message-ID: Subject: Re: [PATCH net-next v4 1/2] net: mana: Handle SKB if TX SGEs exceed hardware limit To: Aditya Garg Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, andrew+netdev@lunn.ch, davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, longli@microsoft.com, kotaranov@microsoft.com, horms@kernel.org, shradhagupta@linux.microsoft.com, ssengar@linux.microsoft.com, ernis@linux.microsoft.com, dipayanroy@linux.microsoft.com, shirazsaleem@microsoft.com, leon@kernel.org, mlevitsk@redhat.com, yury.norov@gmail.com, sbhatta@marvell.com, linux-hyperv@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, gargaditya@microsoft.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Nov 12, 2025 at 5:11=E2=80=AFAM Aditya Garg wrote: > > The MANA hardware supports a maximum of 30 scatter-gather entries (SGEs) > per TX WQE. Exceeding this limit can cause TX failures. > Add ndo_features_check() callback to validate SKB layout before > transmission. For GSO SKBs that would exceed the hardware SGE limit, clea= r > NETIF_F_GSO_MASK to enforce software segmentation in the stack. > Add a fallback in mana_start_xmit() to linearize non-GSO SKBs that still > exceed the SGE limit. > > Also, Add ethtool counter for SKBs linearized > > Co-developed-by: Dipayaan Roy > Signed-off-by: Dipayaan Roy > Signed-off-by: Aditya Garg > --- > drivers/net/ethernet/microsoft/mana/mana_en.c | 37 ++++++++++++++++++- > .../ethernet/microsoft/mana/mana_ethtool.c | 2 + > include/net/mana/gdma.h | 6 ++- > include/net/mana/mana.h | 1 + > 4 files changed, 43 insertions(+), 3 deletions(-) > > diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/= ethernet/microsoft/mana/mana_en.c > index cccd5b63cee6..67ae5421f9ee 100644 > --- a/drivers/net/ethernet/microsoft/mana/mana_en.c > +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c > @@ -11,6 +11,7 @@ > #include > #include > #include > +#include > > #include > #include > @@ -329,6 +330,20 @@ netdev_tx_t mana_start_xmit(struct sk_buff *skb, str= uct net_device *ndev) > cq =3D &apc->tx_qp[txq_idx].tx_cq; > tx_stats =3D &txq->stats; > > + if (MAX_SKB_FRAGS + 2 > MAX_TX_WQE_SGL_ENTRIES && > + skb_shinfo(skb)->nr_frags + 2 > MAX_TX_WQE_SGL_ENTRIES) { > + /* GSO skb with Hardware SGE limit exceeded is not expect= ed here > + * as they are handled in mana_features_check() callback > + */ > + if (skb_linearize(skb)) { > + netdev_warn_once(ndev, "Failed to linearize skb w= ith nr_frags=3D%d and is_gso=3D%d\n", > + skb_shinfo(skb)->nr_frags, > + skb_is_gso(skb)); > + goto tx_drop_count; > + } > + apc->eth_stats.linear_pkt_tx_cnt++; > + } > + > pkg.tx_oob.s_oob.vcq_num =3D cq->gdma_id; > pkg.tx_oob.s_oob.vsq_frame =3D txq->vsq_frame; > > @@ -442,8 +457,6 @@ netdev_tx_t mana_start_xmit(struct sk_buff *skb, stru= ct net_device *ndev) > } > } > > - WARN_ON_ONCE(pkg.wqe_req.num_sge > MAX_TX_WQE_SGL_ENTRIES); > - > if (pkg.wqe_req.num_sge <=3D ARRAY_SIZE(pkg.sgl_array)) { > pkg.wqe_req.sgl =3D pkg.sgl_array; > } else { > @@ -518,6 +531,25 @@ netdev_tx_t mana_start_xmit(struct sk_buff *skb, str= uct net_device *ndev) > return NETDEV_TX_OK; > } > #if MAX_SKB_FRAGS + 2 > MAX_TX_WQE_SGL_ENTRIES > +static netdev_features_t mana_features_check(struct sk_buff *skb, > + struct net_device *ndev, > + netdev_features_t features) > +{ > + if (MAX_SKB_FRAGS + 2 > MAX_TX_WQE_SGL_ENTRIES && > + skb_shinfo(skb)->nr_frags + 2 > MAX_TX_WQE_SGL_ENTRIES) { > + /* Exceeds HW SGE limit. > + * GSO case: > + * Disable GSO so the stack will software-segment the s= kb > + * into smaller skbs that fit the SGE budget. > + * Non-GSO case: > + * The xmit path will attempt skb_linearize() as a fall= back. > + */ > + if (skb_is_gso(skb)) No need to test skb_is_gso(skb), you can clear bits, this will be a NOP if the packet is non GSO anyway. > + features &=3D ~NETIF_F_GSO_MASK; > + } > + return features; > +} #endif > + > static void mana_get_stats64(struct net_device *ndev, > struct rtnl_link_stats64 *st) > { > @@ -878,6 +910,7 @@ static const struct net_device_ops mana_devops =3D { > .ndo_open =3D mana_open, > .ndo_stop =3D mana_close, > .ndo_select_queue =3D mana_select_queue, > + .ndo_features_check =3D mana_features_check, Note that if your mana_features_check() is a nop if MAX_SKB_FRAGS is small enough, you could set a non NULL .ndo_features_check based on a preprocessor condit= ion #if MAX_SKB_FRAGS + 2 > MAX_TX_WQE_SGL_ENTRIES .ndo_features_check =3D .... #endif This would avoid an expensive indirect call when possible. > .ndo_start_xmit =3D mana_start_xmit, > .ndo_validate_addr =3D eth_validate_addr, > .ndo_get_stats64 =3D mana_get_stats64, > diff --git a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c b/drivers= /net/ethernet/microsoft/mana/mana_ethtool.c > index a1afa75a9463..fa5e1a2f06a9 100644 > --- a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c > +++ b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c > @@ -71,6 +71,8 @@ static const struct mana_stats_desc mana_eth_stats[] = =3D { > {"tx_cq_err", offsetof(struct mana_ethtool_stats, tx_cqe_err)}, > {"tx_cqe_unknown_type", offsetof(struct mana_ethtool_stats, > tx_cqe_unknown_type)}, > + {"linear_pkt_tx_cnt", offsetof(struct mana_ethtool_stats, > + linear_pkt_tx_cnt)}, > {"rx_coalesced_err", offsetof(struct mana_ethtool_stats, > rx_coalesced_err)}, > {"rx_cqe_unknown_type", offsetof(struct mana_ethtool_stats, > diff --git a/include/net/mana/gdma.h b/include/net/mana/gdma.h > index 637f42485dba..84614ebe0f4c 100644 > --- a/include/net/mana/gdma.h > +++ b/include/net/mana/gdma.h > @@ -592,6 +592,9 @@ enum { > #define GDMA_DRV_CAP_FLAG_1_HANDLE_RECONFIG_EQE BIT(17) > #define GDMA_DRV_CAP_FLAG_1_HW_VPORT_LINK_AWARE BIT(6) > > +/* Driver supports linearizing the skb when num_sge exceeds hardware lim= it */ > +#define GDMA_DRV_CAP_FLAG_1_SKB_LINEARIZE BIT(20) > + > #define GDMA_DRV_CAP_FLAGS1 \ > (GDMA_DRV_CAP_FLAG_1_EQ_SHARING_MULTI_VPORT | \ > GDMA_DRV_CAP_FLAG_1_NAPI_WKDONE_FIX | \ > @@ -601,7 +604,8 @@ enum { > GDMA_DRV_CAP_FLAG_1_DYNAMIC_IRQ_ALLOC_SUPPORT | \ > GDMA_DRV_CAP_FLAG_1_SELF_RESET_ON_EQE | \ > GDMA_DRV_CAP_FLAG_1_HANDLE_RECONFIG_EQE | \ > - GDMA_DRV_CAP_FLAG_1_HW_VPORT_LINK_AWARE) > + GDMA_DRV_CAP_FLAG_1_HW_VPORT_LINK_AWARE | \ > + GDMA_DRV_CAP_FLAG_1_SKB_LINEARIZE) > > #define GDMA_DRV_CAP_FLAGS2 0 > > diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h > index 8906901535f5..50a532fb30d6 100644 > --- a/include/net/mana/mana.h > +++ b/include/net/mana/mana.h > @@ -404,6 +404,7 @@ struct mana_ethtool_stats { > u64 hc_tx_err_gdma; > u64 tx_cqe_err; > u64 tx_cqe_unknown_type; > + u64 linear_pkt_tx_cnt; > u64 rx_coalesced_err; > u64 rx_cqe_unknown_type; > }; > -- > 2.43.0 >