mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Josef Schlehofer <pepe.schlehofer@gmail.com>
To: netdev@vger.kernel.org
Cc: marcin.s.wojtas@gmail.com, andrew+netdev@lunn.ch,
	davem@davemloft.net, edumazet@kernel.org, kuba@kernel.org,
	pabeni@redhat.com, maxime.chevallier@bootlin.com, w@1wt.eu,
	linux-kernel@vger.kernel.org
Subject: [PATCH net-next v4 6/6] net: mvneta: serve the TX queues in weighted round robin
Date: Wed, 18 Nov 2026 10:30:12 +0100	[thread overview]
Message-ID: <20261118093012.90121-7-pepe.schlehofer@gmail.com> (raw)
In-Reply-To: <20261118093012.90121-1-pepe.schlehofer@gmail.com>

Under load, some TX queues get no service at all. With four TCP
streams on a 100 Mbit/s link, up to three streams sent nothing for 20
seconds. The TX queue arbiter is left at its reset default, which
serves all queues in fixed priority, and the stack spreads the flows
over the queues.

Give the TX queues the same WRR weight, the smallest one that is
larger than the largest packet, and clear the fixed priority. The
Armada 370 and XP manuals list separate registers for the version 1
arbiter, which the driver runs unless a shaper is set, so write both
register sets. On the Armada 385 they are the same registers, as
writing one set and reading the other shows.

Keep the fixed priority while an mqprio offload is set up, as its
traffic classes may rely on it, and clear it when the offload is
removed. A rejected setup leaves it as it was, and after a rejected
replace it applies to all flows until the qdisc is deleted. A class
that fills the link starves the lower classes, and as the fixed
priority is set per queue, the queues of one class still starve each
other.

Observed on machines in real use. OpenWrt users reported it as a
multi-flow lockout on a Linksys WRT1200AC, and OpenWrt has worked
around it since 2017 by sending all TX traffic of its 32-bit boards
through queue 0. Reproduced on a Turris Omnia (Armada 385). With this
patch each busy TX queue gets the same share and no stream stalls,
4 x 23.5 Mbit/s on a 100 Mbit/s link and 4 x 232 Mbit/s on a 1 Gbit/s
link. With an mqprio offload a higher queue still has priority over a
lower one. Only the Armada 385 was tested. No kernel selftest covers
the TX queue arbiter.

Fixes: ee40a116ebf1 ("net: mvneta: fix improper tx queue usage in mvneta_tx()")
Link: https://github.com/openwrt/openwrt/issues/5411
Suggested-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Signed-off-by: Josef Schlehofer <pepe.schlehofer@gmail.com>
---
v4: The fixed priority and the weights are also written to the version
1 registers. A setup that the driver rejects no longer clears the fixed
priority. Discovery statement added.
---
 drivers/net/ethernet/marvell/mvneta.c | 56 ++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index 89ade2ed7ce4..35409695d3d7 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -118,6 +118,7 @@
 #define      MVNETA_HSGMII_SERDES_PROTO		 0x1107
 #define MVNETA_TYPE_PRIO                         0x24bc
 #define      MVNETA_FORCE_UNI                    BIT(21)
+#define MVETH_TXQ_FIXED_PRIO_REG                 0x24dc
 #define MVNETA_TXQ_CMD_1                         0x24e4
 #define MVNETA_TXQ_CMD                           0x2448
 #define      MVNETA_TXQ_DISABLE_SHIFT            8
@@ -196,6 +197,7 @@
 #define      MVNETA_RXQ_ENABLE_MASK              0x000000ff
 #define MVETH_TXQ_TOKEN_COUNT_REG(q)             (0x2700 + ((q) << 4))
 #define MVETH_TXQ_TOKEN_CFG_REG(q)               (0x2704 + ((q) << 4))
+#define MVETH_TXQ_WRR_ARB_REG(q)                 (0x2708 + ((q) << 4))
 #define MVNETA_GMAC_CTRL_0                       0x2c00
 #define      MVNETA_GMAC_MAX_RX_SIZE_SHIFT       2
 #define      MVNETA_GMAC_MAX_RX_SIZE_MASK        0x7ffc
@@ -253,6 +255,8 @@
 #define MVNETA_TXQ_CMD1_REG			 0x3e00
 #define      MVNETA_TXQ_CMD1_BW_LIM_SEL_V1	 BIT(3)
 #define      MVNETA_TXQ_CMD1_BW_LIM_EN		 BIT(0)
+#define MVNETA_TXQ_FIXED_PRIO_REG		 0x3e04
+#define      MVNETA_TXQ_FIXED_PRIO_ALL		 0xff
 #define MVNETA_REFILL_NUM_CLK_REG		 0x3e08
 #define      MVNETA_REFILL_MAX_NUM_CLK		 0x0000ffff
 #define MVNETA_TX_MTU                            0x3e0c
@@ -264,6 +268,8 @@
 #define      MVNETA_TXQ_BUCKET_REFILL_VALUE_MAX	 0x0007ffff
 #define MVNETA_TXQ_TOKEN_SIZE_REG(q)             (0x3e40 + ((q) << 2))
 #define      MVNETA_TXQ_TOKEN_SIZE_MAX           0x7fffffff
+#define MVNETA_TXQ_WRR_ARB_REG(q)		 (0x3e80 + ((q) << 2))
+#define      MVNETA_TXQ_WRR_WEIGHT_MASK		 0xff
 
 /* The values of the bucket refill base period and refill period are taken from
  * the reference manual, and adds up to a base resolution of 10Kbps. This allows
@@ -518,6 +524,8 @@ struct mvneta_port {
 	 */
 	spinlock_t lock;
 	bool is_stopped;
+	/* An mqprio offload keeps the TX queues in fixed priority */
+	bool txq_fixed_prio;
 
 	u32 cause_rx_tx;
 	struct napi_struct napi;
@@ -1632,6 +1640,39 @@ static void mvneta_txq_max_tx_size_set(struct mvneta_port *pp, int max_tx_size)
 	}
 }
 
+static void mvneta_txq_wrr_weight(struct mvneta_port *pp, u32 reg, u32 weight)
+{
+	u32 val = mvreg_read(pp, reg);
+
+	val &= ~MVNETA_TXQ_WRR_WEIGHT_MASK;
+	mvreg_write(pp, reg, val | weight);
+}
+
+/* With the default fixed priority, a busy queue starves the others.
+ * The Armada 370 and XP manuals list separate registers for the
+ * version 1 arbiter, which runs without a shaper, so set both.
+ */
+static void mvneta_txq_wrr_set(struct mvneta_port *pp, int max_tx_size)
+{
+	/* In units of 256 bytes, must exceed the largest packet */
+	u32 weight = max_tx_size / 256 + 1;
+	int queue;
+
+	for (queue = 0; queue < txq_number; queue++) {
+		mvneta_txq_wrr_weight(pp, MVNETA_TXQ_WRR_ARB_REG(queue), weight);
+		mvneta_txq_wrr_weight(pp, MVETH_TXQ_WRR_ARB_REG(queue), weight);
+	}
+}
+
+/* Only the traffic classes of an mqprio offload keep the fixed priority */
+static void mvneta_txq_fixed_prio_set(struct mvneta_port *pp)
+{
+	u32 val = pp->txq_fixed_prio ? MVNETA_TXQ_FIXED_PRIO_ALL : 0;
+
+	mvreg_write(pp, MVNETA_TXQ_FIXED_PRIO_REG, val);
+	mvreg_write(pp, MVETH_TXQ_FIXED_PRIO_REG, val);
+}
+
 /* Set unicast address */
 static void mvneta_set_ucast_addr(struct mvneta_port *pp, u8 last_nibble,
 				  int queue)
@@ -3765,6 +3806,8 @@ static void mvneta_start_dev(struct mvneta_port *pp)
 
 	mvneta_max_rx_size_set(pp, pp->pkt_size);
 	mvneta_txq_max_tx_size_set(pp, pp->pkt_size);
+	mvneta_txq_wrr_set(pp, pp->pkt_size);
+	mvneta_txq_fixed_prio_set(pp);
 	mvneta_restore_per_queue_rate_limit(pp);
 
 	/* start the Rx/Tx activity */
@@ -5345,9 +5388,20 @@ static int mvneta_setup_mqprio(struct net_device *dev,
 static int mvneta_setup_tc(struct net_device *dev, enum tc_setup_type type,
 			   void *type_data)
 {
+	struct mvneta_port *pp = netdev_priv(dev);
+	struct tc_mqprio_qopt_offload *mqprio;
+	int ret;
+
 	switch (type) {
 	case TC_SETUP_QDISC_MQPRIO:
-		return mvneta_setup_mqprio(dev, type_data);
+		mqprio = type_data;
+		ret = mvneta_setup_mqprio(dev, mqprio);
+		if (ret)
+			return ret;
+
+		pp->txq_fixed_prio = mqprio->qopt.num_tc;
+		mvneta_txq_fixed_prio_set(pp);
+		return 0;
 	default:
 		return -EOPNOTSUPP;
 	}
-- 
2.54.0 (Apple Git-157)


      parent reply	other threads:[~2026-11-18  9:30 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18  9:30 [PATCH net-next v4 0/6] net: mvneta: fix the TX queue arbitration and the mqprio offload Josef Schlehofer
2026-11-18  9:30 ` [PATCH net-next v4 1/6] net: mvneta: lift the per-queue TX rate limits Josef Schlehofer
2026-11-18  9:30 ` [PATCH net-next v4 2/6] net: mvneta: fix the removal of the mqprio offload Josef Schlehofer
2026-11-18  9:30 ` [PATCH net-next v4 3/6] net: mvneta: check the TX queue range of an " Josef Schlehofer
2026-11-18  9:30 ` [PATCH net-next v4 4/6] net: mvneta: keep the TX rate limits across open and MTU change Josef Schlehofer
2026-11-18  9:30 ` [PATCH net-next v4 5/6] net: mvneta: validate an mqprio setup before applying it Josef Schlehofer
2026-11-18  9:30 ` Josef Schlehofer [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118093012.90121-7-pepe.schlehofer@gmail.com \
    --to=pepe.schlehofer@gmail.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=marcin.s.wojtas@gmail.com \
    --cc=maxime.chevallier@bootlin.com \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=w@1wt.eu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®