mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Curtis McBurney <cmcburney@telcobridges.com>
To: netdev@vger.kernel.org
Cc: davem@davemloft.net, edumazet@kernel.org, kuba@kernel.org,
	pabeni@redhat.com, horms@kernel.org, andrew+netdev@lunn.ch,
	willemb@google.com, toke@redhat.com,
	linux-kernel@vger.kernel.org,
	Curtis McBurney <cmcburney@telcobridges.com>,
	stable@vger.kernel.org
Subject: [PATCH net 1/2] net: keep txq->trans_start on repeated netif_tx_stop_queue()
Date: Mon, 16 Nov 2026 09:37:15 -0500	[thread overview]
Message-ID: <20261116143716.367451-2-cmcburney@telcobridges.com> (raw)
In-Reply-To: <20261116143716.367451-1-cmcburney@telcobridges.com>

Since commit 95ecba62e2fd ("net: fix races in
netdev_tx_sent_queue()/dev_watchdog()"), netif_tx_stop_queue() writes
txq->trans_start on every call, also when the queue is already stopped.

Several drivers stop the queue from their periodic Tx hang check and
leave the recovery to dev_watchdog(). e1000, e1000e, igb and igc call
netif_stop_queue() or netif_stop_subqueue() each time the hang check,
requested every 2 seconds by the driver watchdog, finds the hang again,
and do not wake the queue. Each call now moves trans_start forward, so
dev_watchdog() never sees the queue stopped for longer than
watchdog_timeo (5 seconds for these drivers). ndo_tx_timeout() is never
called and the adapter is never reset: the driver logs "Detected Tx Unit
Hang" every 2 seconds while the NIC stays hung.

Write trans_start only when the driver stops a queue that it had not
stopped yet, i.e. when __QUEUE_STATE_DRV_XOFF is clear. The write still
happens before set_bit(), which keeps the ordering against
dev_watchdog() added by 95ecba62e2fd. A queue stopped only by BQL
(__QUEUE_STATE_STACK_XOFF) still gets a new trans_start when the driver
stops it.

The test_bit() and set_bit() pair is not atomic. If
netif_tx_wake_queue() runs between the two, trans_start keeps the time
of the previous stop and dev_watchdog() measures from that earlier stop.
A timeout is then reported only if both stops together last longer than
watchdog_timeo.

The issue was observed on a system in real use: a hung NIC kept
reporting a Tx unit hang and was never reset. Code inspection then
pointed to 95ecba62e2fd.

I reproduced the issue and tested the fix on an AlmaLinux 10 kernel
(6.12.0-211.68.1.el10_2, which contains 95ecba62e2fd) with the e1000
driver on a VMware-emulated Intel 82545EM. Reproducing required
modifying the driver: a test-only module parameter makes
e1000_clean_tx_irq() report a Tx unit hang on every hang check of one
interface. netif_tx_stop_queue() is inlined into the driver, so the
patched variant was e1000 built out of tree against the 6.12 headers
with this change applied. With the fake hang held for 30 seconds:

  unpatched: 15 "Detected Tx Unit Hang", no NETDEV WATCHDOG,
             tx_timeout 0 -> 0
  patched:   first "NETDEV WATCHDOG ... timed out 5054 ms" 5 seconds
             after the first stop, followed by an adapter reset;
             tx_timeout 0 -> 5

This patch was not booted on the net tree. The KUnit tests added in the
next patch pass with this change and fail without it.

I wrote the fix. Claude Code (Anthropic) reviewed it, built the e1000
reproducer and wrote this changelog.

Fixes: 95ecba62e2fd ("net: fix races in netdev_tx_sent_queue()/dev_watchdog()")
Cc: stable@vger.kernel.org
Signed-off-by: Curtis McBurney <cmcburney@telcobridges.com>
---
 include/linux/netdevice.h | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 435be7880d7e..6a488245db1e 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -3805,8 +3805,15 @@ static inline void netif_tx_wake_all_queues(struct net_device *dev)
 
 static __always_inline void netif_tx_stop_queue(struct netdev_queue *dev_queue)
 {
-	/* Paired with READ_ONCE() from dev_watchdog() */
-	WRITE_ONCE(dev_queue->trans_start, jiffies);
+	/* Only write trans_start on the initial state transition
+	 * (when the DRV_XOFF bit changes from 0 to 1).
+	 * This prevents recursive driver watchdog polls from continuously
+	 * resetting the timeout timeline and blinding dev_watchdog().
+	 *
+	 * Paired with READ_ONCE() from dev_watchdog()
+	 */
+	if (!test_bit(__QUEUE_STATE_DRV_XOFF, &dev_queue->state))
+		WRITE_ONCE(dev_queue->trans_start, jiffies);
 
 	/* This barrier is paired with smp_mb() from dev_watchdog() */
 	smp_mb__before_atomic();

base-commit: aa40afa831ebfe45e40a23721711059f2ac584d4
-- 
2.52.0


  reply	other threads:[~2026-11-16 14:38 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 14:37 [PATCH net 0/2] " Curtis McBurney
2026-11-16 14:37 ` Curtis McBurney [this message]
2026-11-16 14:37 ` [PATCH net 2/2] net: net_test: add tests for netif_tx_stop_queue() trans_start Curtis McBurney

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261116143716.367451-2-cmcburney@telcobridges.com \
    --to=cmcburney@telcobridges.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=horms@kernel.org \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=stable@vger.kernel.org \
    --cc=toke@redhat.com \
    --cc=willemb@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®