From: Curtis McBurney <cmcburney@telcobridges.com>
To: netdev@vger.kernel.org
Cc: davem@davemloft.net, edumazet@kernel.org, kuba@kernel.org,
pabeni@redhat.com, horms@kernel.org, andrew+netdev@lunn.ch,
willemb@google.com, toke@redhat.com,
linux-kernel@vger.kernel.org,
Curtis McBurney <cmcburney@telcobridges.com>,
stable@vger.kernel.org
Subject: [PATCH net 1/2] net: keep txq->trans_start on repeated netif_tx_stop_queue()
Date: Mon, 16 Nov 2026 09:37:15 -0500 [thread overview]
Message-ID: <20261116143716.367451-2-cmcburney@telcobridges.com> (raw)
In-Reply-To: <20261116143716.367451-1-cmcburney@telcobridges.com>
Since commit 95ecba62e2fd ("net: fix races in
netdev_tx_sent_queue()/dev_watchdog()"), netif_tx_stop_queue() writes
txq->trans_start on every call, also when the queue is already stopped.
Several drivers stop the queue from their periodic Tx hang check and
leave the recovery to dev_watchdog(). e1000, e1000e, igb and igc call
netif_stop_queue() or netif_stop_subqueue() each time the hang check,
requested every 2 seconds by the driver watchdog, finds the hang again,
and do not wake the queue. Each call now moves trans_start forward, so
dev_watchdog() never sees the queue stopped for longer than
watchdog_timeo (5 seconds for these drivers). ndo_tx_timeout() is never
called and the adapter is never reset: the driver logs "Detected Tx Unit
Hang" every 2 seconds while the NIC stays hung.
Write trans_start only when the driver stops a queue that it had not
stopped yet, i.e. when __QUEUE_STATE_DRV_XOFF is clear. The write still
happens before set_bit(), which keeps the ordering against
dev_watchdog() added by 95ecba62e2fd. A queue stopped only by BQL
(__QUEUE_STATE_STACK_XOFF) still gets a new trans_start when the driver
stops it.
The test_bit() and set_bit() pair is not atomic. If
netif_tx_wake_queue() runs between the two, trans_start keeps the time
of the previous stop and dev_watchdog() measures from that earlier stop.
A timeout is then reported only if both stops together last longer than
watchdog_timeo.
The issue was observed on a system in real use: a hung NIC kept
reporting a Tx unit hang and was never reset. Code inspection then
pointed to 95ecba62e2fd.
I reproduced the issue and tested the fix on an AlmaLinux 10 kernel
(6.12.0-211.68.1.el10_2, which contains 95ecba62e2fd) with the e1000
driver on a VMware-emulated Intel 82545EM. Reproducing required
modifying the driver: a test-only module parameter makes
e1000_clean_tx_irq() report a Tx unit hang on every hang check of one
interface. netif_tx_stop_queue() is inlined into the driver, so the
patched variant was e1000 built out of tree against the 6.12 headers
with this change applied. With the fake hang held for 30 seconds:
unpatched: 15 "Detected Tx Unit Hang", no NETDEV WATCHDOG,
tx_timeout 0 -> 0
patched: first "NETDEV WATCHDOG ... timed out 5054 ms" 5 seconds
after the first stop, followed by an adapter reset;
tx_timeout 0 -> 5
This patch was not booted on the net tree. The KUnit tests added in the
next patch pass with this change and fail without it.
I wrote the fix. Claude Code (Anthropic) reviewed it, built the e1000
reproducer and wrote this changelog.
Fixes: 95ecba62e2fd ("net: fix races in netdev_tx_sent_queue()/dev_watchdog()")
Cc: stable@vger.kernel.org
Signed-off-by: Curtis McBurney <cmcburney@telcobridges.com>
---
include/linux/netdevice.h | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 435be7880d7e..6a488245db1e 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -3805,8 +3805,15 @@ static inline void netif_tx_wake_all_queues(struct net_device *dev)
static __always_inline void netif_tx_stop_queue(struct netdev_queue *dev_queue)
{
- /* Paired with READ_ONCE() from dev_watchdog() */
- WRITE_ONCE(dev_queue->trans_start, jiffies);
+ /* Only write trans_start on the initial state transition
+ * (when the DRV_XOFF bit changes from 0 to 1).
+ * This prevents recursive driver watchdog polls from continuously
+ * resetting the timeout timeline and blinding dev_watchdog().
+ *
+ * Paired with READ_ONCE() from dev_watchdog()
+ */
+ if (!test_bit(__QUEUE_STATE_DRV_XOFF, &dev_queue->state))
+ WRITE_ONCE(dev_queue->trans_start, jiffies);
/* This barrier is paired with smp_mb() from dev_watchdog() */
smp_mb__before_atomic();
base-commit: aa40afa831ebfe45e40a23721711059f2ac584d4
--
2.52.0
next prev parent reply other threads:[~2026-11-16 14:38 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 14:37 [PATCH net 0/2] " Curtis McBurney
2026-11-16 14:37 ` Curtis McBurney [this message]
2026-11-16 14:37 ` [PATCH net 2/2] net: net_test: add tests for netif_tx_stop_queue() trans_start Curtis McBurney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116143716.367451-2-cmcburney@telcobridges.com \
--to=cmcburney@telcobridges.com \
--cc=andrew+netdev@lunn.ch \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=horms@kernel.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=stable@vger.kernel.org \
--cc=toke@redhat.com \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®