* [PATCH net v2 0/2] ionic: fix Tx stalls on TSO skbs larger than the Tx ring
@ 2026-11-17 19:18 Nikhil P. Rao
2026-11-17 19:18 ` [PATCH net v2 1/2] ionic: reject Tx ring sizes below the minimum Nikhil P. Rao
2026-11-17 19:18 ` [PATCH net v2 2/2] ionic: limit TSO segments to the minimum Tx ring size Nikhil P. Rao
0 siblings, 2 replies; 5+ messages in thread
From: Nikhil P. Rao @ 2026-11-17 19:18 UTC (permalink / raw)
To: Brett Creeley, Andrew Lunn, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, Breno Leitao, Eric Joyner,
Abhijit Gangurde, stable, Nikhil P. Rao
ionic uses one Tx descriptor per TSO segment but never sets
tso_max_segs, so the stack can build TSO skbs with more segments than
the Tx ring holds. Such an skb can never be posted: the queue stops,
nothing wakes it, and the netdev watchdog fires. Guest TCP traffic with
an MSS below 64 hit this on the default 1024-entry ring.
Patch 1 rejects Tx ring sizes below 64. The completion path wakes a
stopped Tx queue only once 44 descriptors are free, so a smaller ring
never wakes once stopped.
Patch 2 raises the minimum Tx ring size to 128 and sets tso_max_segs
to 127, so every Tx ring can hold the largest TSO skb and larger GSO
skbs are segmented in software.
The two patches fix separate bugs; patch 2 raises the Tx ring minimum
that patch 1 adds from 64 to 128.
Changes in v2:
- Patch 2: use a fixed limit of 127 segments and a 128-entry minimum
Tx ring instead of following the Tx ring size, as suggested by Jakub.
This also covers the hwstamp and kdump Tx rings.
- Add testing notes to both patches, and Cc stable.
- Link to v1: https://lore.kernel.org/all/20261107-b4-ionic-tso-max-segs-v1-0-67c248a5b681@amd.com/
This series was developed with the help of an LLM-based coding
assistant (Claude).
---
Nikhil P. Rao (2):
ionic: reject Tx ring sizes below the minimum
ionic: limit TSO segments to the minimum Tx ring size
drivers/net/ethernet/pensando/ionic/ionic_dev.h | 3 ++-
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c | 7 +++++++
drivers/net/ethernet/pensando/ionic/ionic_lif.c | 10 ++++++----
3 files changed, 15 insertions(+), 5 deletions(-)
---
base-commit: a2fd526d644233d6a892c28173f07c61e21e4c93
change-id: 20261107-b4-ionic-tso-max-segs-057bde9cfe49
Best regards,
--
Nikhil P. Rao <nikhil.rao@amd.com>
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH net v2 1/2] ionic: reject Tx ring sizes below the minimum 2026-11-17 19:18 [PATCH net v2 0/2] ionic: fix Tx stalls on TSO skbs larger than the Tx ring Nikhil P. Rao @ 2026-11-17 19:18 ` Nikhil P. Rao 2026-11-17 19:18 ` [PATCH net v2 2/2] ionic: limit TSO segments to the minimum Tx ring size Nikhil P. Rao 1 sibling, 0 replies; 5+ messages in thread From: Nikhil P. Rao @ 2026-11-17 19:18 UTC (permalink / raw) To: Brett Creeley, Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni Cc: netdev, linux-kernel, Breno Leitao, Eric Joyner, Abhijit Gangurde, stable, Nikhil P. Rao ionic_tx_cq_service() wakes a stopped Tx queue only once IONIC_TSO_DESCS_NEEDED (44) descriptors are free. The threshold is sized for a 64 KB TSO skb at a 1500 byte MSS and assumes the ring can have that many free; a Tx ring of 32 or fewer descriptors cannot, so a stopped queue is never woken and the netdev watchdog fires. ionic_set_ringparam() only checks that the size is a power of 2, and the ethtool core only checks the maximum, so such sizes are accepted. Reject Tx ring sizes below IONIC_MIN_TXRX_DESC. The stall was confirmed from the wake and stop thresholds, not reproduced. Tested on an AMD Pensando NIC: "ethtool -G <dev> tx 32" is rejected and leaves the ring unchanged, and "ethtool -G <dev> tx 64" is accepted. Fixes: 061b9bedbef1 ("ionic: Rework Tx start/stop flow") Cc: stable@vger.kernel.org Reported-by: sashiko-bot@kernel.org # Running on a local machine Signed-off-by: Nikhil P. Rao <nikhil.rao@amd.com> Reviewed-by: Brett Creeley <brett.creeley@amd.com> --- drivers/net/ethernet/pensando/ionic/ionic_ethtool.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/net/ethernet/pensando/ionic/ionic_ethtool.c b/drivers/net/ethernet/pensando/ionic/ionic_ethtool.c index 119ac720c078..3518fbf5882d 100644 --- a/drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +++ b/drivers/net/ethernet/pensando/ionic/ionic_ethtool.c @@ -708,6 +708,13 @@ static int ionic_set_ringparam(struct net_device *netdev, return -EINVAL; } + if (ring->tx_pending < IONIC_MIN_TXRX_DESC) { + NL_SET_ERR_MSG_FMT_MOD(extack, + "Tx descriptor count must be at least %d", + IONIC_MIN_TXRX_DESC); + return -EINVAL; + } + /* if nothing to do return success */ if (ring->tx_pending == lif->ntxq_descs && ring->rx_pending == lif->nrxq_descs && -- 2.43.0 ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH net v2 2/2] ionic: limit TSO segments to the minimum Tx ring size 2026-11-17 19:18 [PATCH net v2 0/2] ionic: fix Tx stalls on TSO skbs larger than the Tx ring Nikhil P. Rao 2026-11-17 19:18 ` [PATCH net v2 1/2] ionic: reject Tx ring sizes below the minimum Nikhil P. Rao @ 2026-11-17 19:18 ` Nikhil P. Rao 2026-11-20 19:58 ` Rao, Nikhil 2026-11-21 19:32 ` netdev-bot+sashiko 1 sibling, 2 replies; 5+ messages in thread From: Nikhil P. Rao @ 2026-11-17 19:18 UTC (permalink / raw) To: Brett Creeley, Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni Cc: netdev, linux-kernel, Breno Leitao, Eric Joyner, Abhijit Gangurde, stable, Nikhil P. Rao ionic uses one Tx descriptor per TSO segment, and a Tx ring holds at most num_descs - 1 of them. The driver does not set tso_max_segs, so the stack can hand it GSO skbs with more segments than the Tx ring can hold. ionic_tx() then stops the queue waiting for descriptors that can never become free, and the netdev watchdog fires repeatedly. This was hit in production: guest TCP traffic with an MSS below 64 produced 64 KB GSO skbs on the default 1024-entry ring. Raise the minimum Tx ring size to IONIC_MIN_TX_DESC (128), for the hwstamp and kdump Tx rings as well, and set tso_max_segs to IONIC_MIN_TX_DESC - 1 so that every Tx ring can hold the largest TSO skb. Larger GSO skbs are segmented in software. 127 segments still covers a 64 KB skb at the 536 byte default MSS. IONIC_MIN_TXRX_DESC now only applies to Rx, so rename it to IONIC_MIN_RX_DESC. Tested on an AMD Pensando NIC with GSO frames of about 1500 segments, injected through a tap with a virtio-net header and redirected to the ionic netdev with tc mirred: without this patch the Tx queue stalls and the watchdog fires, and with it they are sent. Fixes: 0f3154e6bcb3 ("ionic: Add Tx and Rx handling") Cc: stable@vger.kernel.org Suggested-by: Jakub Kicinski <kuba@kernel.org> Link: https://lore.kernel.org/all/20261112082753.6ee90fc8@kernel.org/ Signed-off-by: Nikhil P. Rao <nikhil.rao@amd.com> Reviewed-by: Brett Creeley <brett.creeley@amd.com> --- drivers/net/ethernet/pensando/ionic/ionic_dev.h | 3 ++- drivers/net/ethernet/pensando/ionic/ionic_ethtool.c | 4 ++-- drivers/net/ethernet/pensando/ionic/ionic_lif.c | 10 ++++++---- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/drivers/net/ethernet/pensando/ionic/ionic_dev.h b/drivers/net/ethernet/pensando/ionic/ionic_dev.h index db90e39a1442..4b3e244e6bbb 100644 --- a/drivers/net/ethernet/pensando/ionic/ionic_dev.h +++ b/drivers/net/ethernet/pensando/ionic/ionic_dev.h @@ -16,7 +16,8 @@ #define IONIC_MAX_TX_DESC 8192 #define IONIC_MAX_RX_DESC 16384 -#define IONIC_MIN_TXRX_DESC 64 +#define IONIC_MIN_RX_DESC 64 +#define IONIC_MIN_TX_DESC 128 #define IONIC_DEF_TXRX_DESC 1024 #define IONIC_RX_FILL_THRESHOLD 16 #define IONIC_RX_FILL_DIV 8 diff --git a/drivers/net/ethernet/pensando/ionic/ionic_ethtool.c b/drivers/net/ethernet/pensando/ionic/ionic_ethtool.c index 3518fbf5882d..24fd34b4cfcc 100644 --- a/drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +++ b/drivers/net/ethernet/pensando/ionic/ionic_ethtool.c @@ -708,10 +708,10 @@ static int ionic_set_ringparam(struct net_device *netdev, return -EINVAL; } - if (ring->tx_pending < IONIC_MIN_TXRX_DESC) { + if (ring->tx_pending < IONIC_MIN_TX_DESC) { NL_SET_ERR_MSG_FMT_MOD(extack, "Tx descriptor count must be at least %d", - IONIC_MIN_TXRX_DESC); + IONIC_MIN_TX_DESC); return -EINVAL; } diff --git a/drivers/net/ethernet/pensando/ionic/ionic_lif.c b/drivers/net/ethernet/pensando/ionic/ionic_lif.c index 7f0da2b0defd..1f24c00334bd 100644 --- a/drivers/net/ethernet/pensando/ionic/ionic_lif.c +++ b/drivers/net/ethernet/pensando/ionic/ionic_lif.c @@ -997,7 +997,7 @@ int ionic_lif_create_hwstamp_txq(struct ionic_lif *lif) features = IONIC_Q_F_2X_CQ_DESC | IONIC_TXQ_F_HWSTAMP; - num_desc = IONIC_MIN_TXRX_DESC; + num_desc = IONIC_MIN_TX_DESC; desc_sz = sizeof(struct ionic_txq_desc); comp_sz = 2 * sizeof(struct ionic_txq_comp); @@ -1062,7 +1062,7 @@ int ionic_lif_create_hwstamp_rxq(struct ionic_lif *lif) features = IONIC_Q_F_2X_CQ_DESC | IONIC_RXQ_F_HWSTAMP; - num_desc = IONIC_MIN_TXRX_DESC; + num_desc = IONIC_MIN_RX_DESC; desc_sz = sizeof(struct ionic_rxq_desc); comp_sz = 2 * sizeof(struct ionic_rxq_comp); sg_desc_sz = sizeof(struct ionic_rxq_sg_desc); @@ -1642,6 +1642,8 @@ static int ionic_init_nic_features(struct ionic_lif *lif) netdev->features |= netdev->hw_features; netdev->vlan_features |= netdev->features & ~NETIF_F_VLAN_FEATURES; + netif_set_tso_max_segs(netdev, IONIC_MIN_TX_DESC - 1); + netdev->priv_flags |= IFF_UNICAST_FLT | IFF_LIVE_ADDR_CHANGE; @@ -3284,8 +3286,8 @@ int ionic_lif_alloc(struct ionic *ionic) lif->index = 0; if (is_kdump_kernel()) { - lif->ntxq_descs = IONIC_MIN_TXRX_DESC; - lif->nrxq_descs = IONIC_MIN_TXRX_DESC; + lif->ntxq_descs = IONIC_MIN_TX_DESC; + lif->nrxq_descs = IONIC_MIN_RX_DESC; } else { lif->ntxq_descs = IONIC_DEF_TXRX_DESC; lif->nrxq_descs = IONIC_DEF_TXRX_DESC; -- 2.43.0 ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net v2 2/2] ionic: limit TSO segments to the minimum Tx ring size 2026-11-17 19:18 ` [PATCH net v2 2/2] ionic: limit TSO segments to the minimum Tx ring size Nikhil P. Rao @ 2026-11-20 19:58 ` Rao, Nikhil 2026-11-21 19:32 ` netdev-bot+sashiko 1 sibling, 0 replies; 5+ messages in thread From: Rao, Nikhil @ 2026-11-20 19:58 UTC (permalink / raw) To: Nikhil P. Rao, Brett Creeley, Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni Cc: netdev, linux-kernel, Breno Leitao, Eric Joyner, Abhijit Gangurde, stable On 11/17/2026 11:18 AM, Nikhil P. Rao wrote: > ionic uses one Tx descriptor per TSO segment, and a Tx ring holds at > most num_descs - 1 of them. The driver does not set tso_max_segs, so > the stack can hand it GSO skbs with more segments than the Tx ring > can hold. ionic_tx() then stops the queue waiting for descriptors > that can never become free, and the netdev watchdog fires repeatedly. > > This was hit in production: guest TCP traffic with an MSS below 64 > produced 64 KB GSO skbs on the default 1024-entry ring. > > Raise the minimum Tx ring size to IONIC_MIN_TX_DESC (128), for the > hwstamp and kdump Tx rings as well, and set tso_max_segs to > IONIC_MIN_TX_DESC - 1 so that every Tx ring can hold the largest TSO > skb. Larger GSO skbs are segmented in software. 127 segments still > covers a 64 KB skb at the 536 byte default MSS. > > IONIC_MIN_TXRX_DESC now only applies to Rx, so rename it to > IONIC_MIN_RX_DESC. > > Tested on an AMD Pensando NIC with GSO frames of about 1500 segments, > injected through a tap with a virtio-net header and redirected to the > ionic netdev with tc mirred: without this patch the Tx queue stalls and > the watchdog fires, and with it they are sent. > > Fixes: 0f3154e6bcb3 ("ionic: Add Tx and Rx handling") > Cc: stable@vger.kernel.org > Suggested-by: Jakub Kicinski <kuba@kernel.org> > Link: https://lore.kernel.org/all/20261112082753.6ee90fc8@kernel.org/ > Signed-off-by: Nikhil P. Rao <nikhil.rao@amd.com> > Reviewed-by: Brett Creeley <brett.creeley@amd.com> > --- > drivers/net/ethernet/pensando/ionic/ionic_dev.h | 3 ++- > drivers/net/ethernet/pensando/ionic/ionic_ethtool.c | 4 ++-- > drivers/net/ethernet/pensando/ionic/ionic_lif.c | 10 ++++++---- > 3 files changed, 10 insertions(+), 7 deletions(-) > [...] A note for the stable backport: on kernels before v6.19 this patch alone does not stop the stall for guest traffic. A guest's GSO skb arrives as SKB_GSO_DODGY with gso_segs still 0, so gso_features_check() never sees it exceed gso_max_segs. That is the production case here: guest TCP forwarded through the host to an ionic VF. be1b70ab21cb ("net: init shinfo->gso_segs from qdisc_pkt_len_init()") fills in gso_segs before that check. Tested on v6.8 with an AMD Pensando NIC, using the same tap + tc mirred injection: with this series alone the Tx queue still stalls; with be1b70ab21cb added (a one-line backport there) the frames are sent. So the stable tag on this patch should have been: Cc: stable@vger.kernel.org # 6.18.x and older: be1b70ab21cb: net: init shinfo->gso_segs from qdisc_pkt_len_init() ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net v2 2/2] ionic: limit TSO segments to the minimum Tx ring size 2026-11-17 19:18 ` [PATCH net v2 2/2] ionic: limit TSO segments to the minimum Tx ring size Nikhil P. Rao 2026-11-20 19:58 ` Rao, Nikhil @ 2026-11-21 19:32 ` netdev-bot+sashiko 1 sibling, 0 replies; 5+ messages in thread From: netdev-bot+sashiko @ 2026-11-21 19:32 UTC (permalink / raw) To: nikhil.rao Cc: brett.creeley, andrew+netdev, davem, edumazet, kuba, pabeni, netdev, linux-kernel, leitao, eric.joyner, abhijit.gangurde, stable Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider. Critical: 0 · High: 1 · Medium: 1 · Low: 0 If you believe none of the issues are legitimate, please reply to each issue of Medium severity or higher and explain why the AI is wrong. If any of them are legitimate and you plan to respin, please let patchwork know by including "pw-bot: cr" as a separate line at the end of your reply (one such reply per series is enough). - [High] The fix only works when skb_shinfo(skb)->gso_segs is already correct at the moment the core checks the device limit. - [Medium] tso_max_segs is now IONIC_MIN_TX_DESC - 1 = 127, so on a 128-entry Tx ring a 127-segment TSO skb can only be posted when the… -- ionic: limit TSO segments to the minimum Tx ring size The patch raises the minimum Tx ring size to 128 descriptors, and uses that size for the hwstamp and kdump Tx rings too. It then sets tso_max_segs to IONIC_MIN_TX_DESC - 1 so the stack never hands the driver a TSO skb with more segments than a Tx ring can hold. The Rx minimum is kept at 64 and renamed to IONIC_MIN_RX_DESC. Link: https://lore.kernel.org/all/20261112082753.6ee90fc8@kernel.org/ > diff --git a/drivers/net/ethernet/pensando/ionic/ionic_lif.c b/drivers/net/ethernet/pensando/ionic/ionic_lif.c > index 7f0da2b0defdf..1f24c00334bd0 100644 > --- a/drivers/net/ethernet/pensando/ionic/ionic_lif.c > +++ b/drivers/net/ethernet/pensando/ionic/ionic_lif.c [ ... ] > @@ -1642,6 +1642,8 @@ static int ionic_init_nic_features(struct ionic_lif *lif) > netdev->features |= netdev->hw_features; > netdev->vlan_features |= netdev->features & ~NETIF_F_VLAN_FEATURES; > > + netif_set_tso_max_segs(netdev, IONIC_MIN_TX_DESC - 1); > + [Severity: High] Is this limit enforced on every path that reaches ionic_start_xmit()? The core applies it in gso_within_dev_limits(): if (skb_shinfo(skb)->gso_segs > READ_ONCE(dev->gso_max_segs)) This check only works if gso_segs is already accurate. For SKB_GSO_DODGY skbs, gso_segs is recomputed early by qdisc_pkt_len_segs_init(), and only __dev_queue_xmit() calls that. AF_PACKET with PACKET_QDISC_BYPASS and PACKET_VNET_HDR goes around it: packet_snd() virtio_net_hdr_to_skb() /* SKB_GSO_DODGY, gso_segs = 0 */ packet_xmit() dev_direct_xmit() __dev_direct_xmit() validate_xmit_skb_list() validate_xmit_skb() netif_skb_features() gso_features_check() gso_within_dev_limits() /* 0 <= 127, passes */ __skb_gso_segment() tcp_gso_segment() ionic does not set NETIF_F_GSO_ROBUST, so netif_needs_gso() is true. But tcp_gso_segment() then takes this branch: if (skb_gso_ok(skb, features | NETIF_F_GSO_ROBUST)) { skb_shinfo(skb)->gso_segs = DIV_ROUND_UP(skb->len, mss); segs = NULL; goto out; } So the original skb reaches the driver with its real gso_segs. virtio_net_hdr_to_skb() only clamps gso_size to TCP_MIN_GSO_SIZE, so a 64KB frame can carry thousands of segments. A tc/tcx egress BPF program that calls bpf_skb_adjust_room() or bpf_skb_net_grow() does the same thing. These run in sch_handle_egress() after qdisc_pkt_len_segs_init(), reset gso_segs to 0 and shrink gso_size. In the driver, ionic_tx_descs_needed() returns gso_segs and never compares it with q->num_descs - 1: if (skb_is_gso(skb)) { ndescs = skb_shinfo(skb)->gso_segs; ionic_start_xmit() then does: if (!netif_txq_maybe_stop(q_to_ndq(netdev, q), ionic_q_space_avail(q), ndescs, ndescs)) return NETDEV_TX_BUSY; On the qdisc path the skb is requeued and retried forever. On the direct path, if the ring was idle, no completion arrives to wake the queue. It stays stopped until dev_watchdog() -> ionic_tx_timeout() resets the queues. Isn't that the same stall the commit message says is fixed? The commit message says: so that every Tx ring can hold the largest TSO skb. Larger GSO skbs are segmented in software. Should ionic_start_xmit() also drop or software-segment skbs where ndescs > q->num_descs - 1, or does this need a fix in the core? [Severity: Medium] With tso_max_segs at 127, a 128-entry Tx ring can only accept a 127-segment TSO skb when it is completely empty. 128 is the new minimum and the kdump default. Can a deferred doorbell keep the ring from ever draining? The core produces 127-segment skbs on purpose when it splits oversized guest GSO skbs, since skb_gso_output_max_segs() caps at gso_max_segs. TCP also produces them at an MSS of about 516 or less. Consider a bulk dequeue on the same txq: 1. ionic_tx() posts a non-TSO skb A with ionic_txq_post(q, false). If netdev_xmit_more() is true and the queue is not stopped, __netdev_tx_sent_queue() returns false and the doorbell is deferred: ring_dbell = __netdev_tx_sent_queue(ndq, skb->len, netdev_xmit_more()); 2. The next skb B is a 127-segment TSO skb. ionic_start_xmit() sees avail < gso_segs, stops the queue and returns NETDEV_TX_BUSY. The doorbell for A is never rung. 3. The descriptors that were doorbelled complete, and ionic_tx_cq_service() wakes the queue. B is retried alone with xmit_more false. It fails the space check again before anything is posted, so the doorbell is still not rung. 4. The NIC never fetches A's descriptor, so it never completes. avail stays below 127 and the queue stays stopped. The periodic doorbell poke in ionic_queue_doorbell_check() would recover this, but it starts with: if (!ionic->lif->doorbell_wa) return; ionic_doorbell_wa() only sets doorbell_wa for asic_type 0 or ELBA. On other ASICs the queue stays stalled until the watchdog resets the queues. The NETDEV_TX_BUSY path already skipped the doorbell flush before this patch. But with no slack on the minimum ring, a single deferred descriptor is now enough to trigger the stall. Would it make sense to ring the doorbell before returning NETDEV_TX_BUSY? Or to leave some headroom between tso_max_segs and the minimum Tx ring size? > netdev->priv_flags |= IFF_UNICAST_FLT | > IFF_LIVE_ADDR_CHANGE; > [ ... ] -- Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261117-b4-ionic-tso-max-segs-v2-0-f9792566c5d5%40amd.com ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-21 19:32 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-11-17 19:18 [PATCH net v2 0/2] ionic: fix Tx stalls on TSO skbs larger than the Tx ring Nikhil P. Rao 2026-11-17 19:18 ` [PATCH net v2 1/2] ionic: reject Tx ring sizes below the minimum Nikhil P. Rao 2026-11-17 19:18 ` [PATCH net v2 2/2] ionic: limit TSO segments to the minimum Tx ring size Nikhil P. Rao 2026-11-20 19:58 ` Rao, Nikhil 2026-11-21 19:32 ` netdev-bot+sashiko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®