mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Théo Lebrun" <theo.lebrun@bootlin.com>
To: "Vineeth Karumanchi" <vineeth.karumanchi@amd.com>,
	"Conor Dooley" <conor.dooley@microchip.com>,
	"Andrew Lunn" <andrew+netdev@lunn.ch>,
	"David S. Miller" <davem@davemloft.net>,
	"Eric Dumazet" <edumazet@kernel.org>,
	"Jakub Kicinski" <kuba@kernel.org>,
	"Paolo Abeni" <pabeni@redhat.com>,
	"Russell King" <linux@armlinux.org.uk>
Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
	"Nicolai Buchwitz" <nb@tipi-net.de>,
	"Paolo Valerio" <pvalerio@redhat.com>,
	"Vladimir Kondratiev" <vladimir.kondratiev@mobileye.com>,
	"Gregory CLEMENT" <gregory.clement@bootlin.com>,
	"Benoît Monin" <benoit.monin@bootlin.com>,
	"Tawfik Bayouk" <tawfik.bayouk@mobileye.com>,
	"Thomas Petazzoni" <thomas.petazzoni@bootlin.com>
Subject: Re: [PATCH net-next v7 1/3] net: macb: cache ENST config across link down/up
Date: Thu, 19 Nov 2026 16:40:35 +0100	[thread overview]
Message-ID: <DMZAUI64FXCL.13646UPPGKGTC@bootlin.com> (raw)
In-Reply-To: <fe08db37-4415-4b4b-b226-cc1a0ec4f029@amd.com>

Hello Vineeth,

On Thu Nov 19, 2026 at 1:36 PM CET, Vineeth Karumanchi wrote:
> Hi Théo,
>
> On 11/19/2026 2:23 AM, Théo Lebrun wrote:
>> Currently, ENST is not explicitly restored across interface close/open.
>> If HW keeps its register state then ENST is kept, otherwise it is lost;
>> but we do not write the ENST schedule again on open.
>> 
>> This causes trouble if the config is not compatible with the new open
>> and link-up. The two issues that can occur are a link speed change (as
>> all timings are configured in a unit relative to the link speed) or a
>> change in the # of queues.
>> 
>
> Thanks for the patch.
>
> We are working on the same problem for CBS, covering the four scenarios
> listed in:
> https://lore.kernel.org/netdev/a04aa4d5-335b-47c1-b969-e89b18c9b142@amd.com/
>
> Since this patch reworks how ENST state is kept and restored, it would
> be good to introduce a generic top-level tc structure here, which ENST,
> CBS and any later TSN offloads can share, instead of an ENST-specific one.

Sorry I forgot replying on that thread. It is still marked as unread in
my mailbox!

Let me quote it here and reply:



On Thu Oct 8, 2026 at 5:27 PM CEST, Karumanchi, Vineeth wrote:
> On 9/17/2026 3:02 PM, Théo Lebrun wrote:
>> So we write cbs_control/0x04BC only on ndo_setup_tc(TC_SETUP_QDISC_CBS)
>> callback. As Sashiko pointed how I think that causes an issue with HW
>> resets & suspend:
>>  - what if configured then link down then link up?
>>  - what if configured before link up?
>>
>> Also when not using CBS we hope tx_sched_ctrl/0x0580 is at its reset
>> value (0b00, fixed priority) for all queues. I would expect an
>> unconditional writel tx_sched_ctrl/0x0580 in macb_init_hw(). This
>> solves the down/up issue, the speed change and also ensures we don't
>> inherit CBS or DWRR from the boot stages (super unlikely though).
>>
>> Also, it might be simpler to always use tx_sched_ctrl/0x0580 rather than
>> sometimes tx_sched_ctrl/0x0580 and sometimes cbs_control/0x04BC, which
>> are aliases.
>>
>> What do you think?
>
> Sorry for the delayed response.
>
> I tested the current implementation in the following four scenarios:
>
> 1. Configuring CBS while the link is down: CBS cannot be programmed
> because the link speed is unknown.
>
> 2. Link down followed by link up: The hardware retains the CBS register
> values, and CBS is effective again after the link comes up at the same
> speed.

Here you talk about link-down followed by link-up.

But about a close then open: this behavior might be board specific.
Runtime PM does suspend and this might shut off the IP on some
platforms and not others. That would in turn mean the IP loses its
state.

There is no easy way of knowing if that occurs on any platform that uses
MACB. It sounds probable though.

> 3. Suspend/resume: The CBS register contents are lost after resume.
> However, the CBS child qdisc remains installed in the kernel and is
> still shown by tc as offloaded, leaving the software configuration
> inconsistent with the hardware state.

Again, this is platform specific. Some boards + suspend combo keep the
SoC and its peripherals alive enough for them to keep their state.

A board with little runtime PM doing suspend-to-idle probably won't lose
CBS register state. A board doing proper suspend-to-RAM will surely
lose its state.

> 4. Link-speed change: The CBS register values are retained and shaping
> remains enabled. However, those values were calculated for the previous
> speed, so the resulting shaping bandwidth is incorrect.
>
> I see two possible approaches:
>
> - Retain the requested CBS parameters in the driver and restore the
> hardware configuration, following the saved-state approach used by IGB.
> For MACB, we would also recalculate the hardware idleslope on link-up
> using the negotiated speed.
> If the saved request cannot be supported at the new speed, retain the
> request, leave CBS disabled for that queue, and warn the user that the
> configuration is invalid for the new link speed.
>
> - Explicitly reset the scheduling configuration in macb_init_hw() and
> require users to remove and recreate the CBS qdisc. This establishes a
> known hardware state, but leaves the installed qdisc inconsistent until
> users reconfigure it. It would also need handling for speed changes that
> do not invoke macb_init_hw().
>
> Please let me know your thoughts on these approaches.

Option 2 is not super doable. There is no API for the driver to
say to the network stack and userspace "I'm dropping your qdisc, please
re-enable one if you care".

So ideally we would do option 1, but dealing with link speed changes
sounds complex. I'm super afraid of introducing bugs with code that
recomputes the timings on the fly. I'm sure I will create rounding bugs
in the edge cases.

That's why I made the cached state be in number of words and store the
link speed alongside. This means we cannot introduce such bugs AND is
less code overall AND is the common case.

It feels like easier and smaller incremental changes.



Now continuing onto the original quote:

>> The latter is the main reason we do this change before implementing
>> ethtool -L support.
>> 
>> If on link-up, the cached ENST config is invalid (link speed changed),
>> then we drop the cached config, log a warning and explicitly override
>> all ENST timings previously configured in the registers. If it is
>> valid, we explicitly re-apply it.
>> 
>> Take this opportunity to replace a kmalloc in
>> macb_taprio_setup_replace() with a simple and tiny (104b) stack value.
>> 
>> Drop the old hack of checking for pm_runtime_suspended(), which has
>> little value because we could race with the runtime_suspend() call.
>> 
>> Synchronise our config using the existing bp->mac_cfg_lock mutex, taken
>> by macb_taprio_setup_replace() and macb_taprio_destroy(). To avoid an
>> ABBA deadlock between bp->mac_cfg_lock and phydev->lock taken by
>> phylink_ethtool_ksettings_get(), cache the link speed as bp->speed_mbps.
>> 
>> Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
>> ---
>>  drivers/net/ethernet/cadence/macb.h      |  47 +++++---
>>  drivers/net/ethernet/cadence/macb_main.c | 187 +++++++++++++++----------------
>>  2 files changed, 124 insertions(+), 110 deletions(-)
>> 
>> diff --git a/drivers/net/ethernet/cadence/macb.h b/drivers/net/ethernet/cadence/macb.h
>> index b7b4b6a38298..00258866623f 100644
>> --- a/drivers/net/ethernet/cadence/macb.h
>> +++ b/drivers/net/ethernet/cadence/macb.h
>> @@ -1327,6 +1327,35 @@ struct macb_context {
>>  	struct macb_txq		txq[MACB_MAX_QUEUES];
>>  };
>>  
>> +/**
>> + * struct macb_queue_enst_config - Configuration for Enhanced Scheduled Traffic
>> + * @start_time_mask:  Bitmask representing the start time for the queue
>> + * @on_time_bytes:    "on" time nsec expressed in bytes
>> + * @off_time_bytes:   "off" time nsec expressed in bytes
>> + *
>> + * This structure holds the configuration parameters for an ENST queue,
>> + * used to control time-based transmission scheduling in the MACB driver.
>> + */
>> +struct macb_queue_enst_config {
>> +	u32 start_time_mask;
>> +	u32 on_time_bytes;
>> +	u32 off_time_bytes;
>> +};
>> +
>> +/**
>> + * struct macb_enst - Cached Enhanced Scheduled Traffic configuration
>> + * @queues:     per-queue timing (zeroed for queues without schedule)
>> + * @speed_mbps: link speed the timings were computed for, 0 if no schedule
>> + * @queue_mask: queues enabled by the current schedule, 0 if no schedule
>> + *
>> + * The timings are expressed in bytes, so they depend on the link speed.
>> + */
>> +struct macb_enst {
>> +	struct macb_queue_enst_config	queues[MACB_MAX_QUEUES];
>> +	u32				speed_mbps;
>> +	u32				queue_mask;
>> +};
>> +
>
> Could this be a per-queue structure shared by all tc offloads, so that
> new features only add their own part?
> For example:
>
> struct macb_tc_queue {
>         struct {
>                 u32 idleslope;          /* kbps */
>         } cbs;
>         struct {
>                 u32 start_time;         /* ENST_START_TIME register value */
>                 u32 on_time_ns;         /* gate open interval */
>                 u64 off_time_ns;        /* gate closed interval */
>         } enst;
> };

Yes sure!

Either

   // as you suggest
   struct macb_tc_queue tc_queues[MACB_MAX_QUEUES];

Or:

   struct macb_enst_queue enst_queues[MACB_MAX_QUEUES];
   struct macb_cbs_queue cbs_queues[MACB_MAX_QUEUES];
   // ... etc ...

Both would work.

>
>>  struct macb {
>>  	void __iomem		*regs;
>>  	bool			native_io;
>> @@ -1427,11 +1456,14 @@ struct macb {
>>  
>>  	/* Redundant to netif_carrier_ok(), but set under bp->mac_cfg_lock. */
>>  	bool			link_up;
>> +	u32			speed_mbps;
>>  
>>  	u32	rx_intr_mask;
>>  
>>  	struct macb_pm_data pm_data;
>>  	const struct macb_usrio_config *usrio;
>> +
>> +	struct macb_enst enst;
>>  };
>>  
>>  #ifdef CONFIG_MACB_USE_HWSTAMP
>> @@ -1532,19 +1564,4 @@ struct macb_platform_data {
>>  	struct clk	*hclk;
>>  };
>>  
>> -/**
>> - * struct macb_queue_enst_config - Configuration for Enhanced Scheduled Traffic
>> - * @start_time_mask:  Bitmask representing the start time for the queue
>> - * @on_time_bytes:    "on" time nsec expressed in bytes
>> - * @off_time_bytes:   "off" time nsec expressed in bytes
>> - *
>> - * This structure holds the configuration parameters for an ENST queue,
>> - * used to control time-based transmission scheduling in the MACB driver.
>> - */
>> -struct macb_queue_enst_config {
>> -	u32 start_time_mask;
>> -	u32 on_time_bytes;
>> -	u32 off_time_bytes;
>> -};
>> -
>>  #endif /* _MACB_H */
>> diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c
>> index 9f6111661b40..1e0666692116 100644
>> --- a/drivers/net/ethernet/cadence/macb_main.c
>> +++ b/drivers/net/ethernet/cadence/macb_main.c
>> @@ -847,6 +847,7 @@ static void macb_mac_link_down(struct phylink_config *config, unsigned int mode,
>>  	mutex_lock(&bp->mac_cfg_lock);
>>  
>>  	bp->link_up = false;
>> +	bp->speed_mbps = 0;
>>  
>>  	if (!(bp->caps & MACB_CAPS_MACB_IS_EMAC))
>>  		for (q = 0, queue = bp->queues; q < bp->num_queues; ++q, ++queue)
>> @@ -955,6 +956,43 @@ static void gem_shuffle_tx_rings(struct macb *bp)
>>  		gem_shuffle_tx_one_ring(queue);
>>  }
>>  
>> +static void macb_apply_enst_config(struct macb *bp)
>> +{
>> +	struct macb_queue_enst_config *config;
>> +	struct macb_queue *queue;
>> +	u32 queue_mask;
>> +	size_t q;
>> +
>> +	/* Short-circuit to avoid writing to invalid register offsets */
>> +	if (!(bp->caps & MACB_CAPS_QBV))
>> +		return;
>> +
>> +	guard(spinlock_irqsave)(&bp->lock);
>> +
>> +	/* Disable ENST for all queues.
>> +	 * Reconfiguring timings must be done with ENST disabled.
>> +	 */
>> +	queue_mask = BIT_U32(bp->num_queues) - 1;
>> +	gem_writel(bp, ENST_CONTROL,
>> +		   queue_mask << GEM_ENST_DISABLE_QUEUE_OFFSET);
>> +
>> +	for (q = 0, queue = bp->queues; q < bp->num_queues; ++q, ++queue) {
>> +		/* Configure queue timing registers.
>> +		 *
>> +		 * Those fields must be zero if the queue has ENST disabled:
>> +		 * notice we don't bother reading bp->enst.queue_mask & BIT(q).
>> +		 */
>> +		config = &bp->enst.queues[q];
>> +		queue_writel(queue, ENST_START_TIME, config->start_time_mask);
>> +		queue_writel(queue, ENST_ON_TIME, config->on_time_bytes);
>> +		queue_writel(queue, ENST_OFF_TIME, config->off_time_bytes);
>> +	}
>> +
>> +	/* Enable ENST for all configured queues in one write */
>> +	if (bp->enst.queue_mask)
>> +		gem_writel(bp, ENST_CONTROL, bp->enst.queue_mask);
>> +}
>> +
>>  static void macb_mac_link_up(struct phylink_config *config,
>>  			     struct phy_device *phydev,
>>  			     unsigned int mode, phy_interface_t interface,
>> @@ -972,6 +1010,7 @@ static void macb_mac_link_up(struct phylink_config *config,
>>  	spin_lock_irqsave(&bp->lock, flags);
>>  
>>  	bp->link_up = true;
>> +	bp->speed_mbps = speed > 0 ? speed : 0;
>>  
>>  	ctrl = macb_or_gem_readl(bp, NCFGR);
>>  
>> @@ -1009,6 +1048,15 @@ static void macb_mac_link_up(struct phylink_config *config,
>>  
>>  	spin_unlock_irqrestore(&bp->lock, flags);
>>  
>> +	if (bp->enst.speed_mbps && speed != bp->enst.speed_mbps) {
>
> Why drop the schedule on a speed change? The taprio qdisc stays
> installed, so the user's configuration is still valid;
> Could we instead recompute the ON/OFF times for the new speed on
> link-up, and only disable (with a warning) the queues whose intervals no
> longer fit the hardware range at that speed?

See explanation above. It felt like a good local maximum: simple code
that probably doesn't introduce rounding bugs and solves it for most
usecases.

In any case, we will sometimes have to drop the qdisc and warn about it.

>
>> +		netdev_warn(bp->netdev,
>> +			    "TAPRIO dropped because link speed changed\n");
>> +
>> +		memset(&bp->enst, 0, sizeof(bp->enst));
>> +	}
>> +
>> +	macb_apply_enst_config(bp);
>> +
>
> Could this be a single macb_apply_tc_config() (or similar) that
> reprograms every enabled tc offload? CBS needs exactly the same
> handling, since idleslope is also relative to the link rate.

Yes sure!

Do you want to take over and handle this? You seem to be interested in
those TC features.  You can either copy the patch as-is or modify it
under your own authorship, I don't mind.

Or if you prefer, I keep it here and iterate. But if I keep it, I'll
continue with ENST and then at a later stage we'll add CBS & co onto
it. This patch looks like a good starting point to add more onto it.

I'm mostly focused on getting a set_channels implementation (to then
have XDP/XSK) and I fixed ENST along the way.

Thanks,

--
Théo Lebrun, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


  reply	other threads:[~2026-11-19 15:40 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 20:53 [PATCH net-next v7 0/3] net: macb: implement ethtool set channels count operation Théo Lebrun
2026-11-18 20:53 ` [PATCH net-next v7 1/3] net: macb: cache ENST config across link down/up Théo Lebrun
2026-11-19 11:49   ` Nicolai Buchwitz
2026-11-19 12:36   ` Vineeth Karumanchi
2026-11-19 15:40     ` Théo Lebrun [this message]
2026-11-19 16:10       ` Théo Lebrun
2026-11-18 20:53 ` [PATCH net-next v7 2/3] net: macb: implement ethtool_ops.get|set_channels() Théo Lebrun
2026-11-19 11:54   ` Nicolai Buchwitz
2026-11-19 16:19     ` Théo Lebrun
2026-11-18 20:53 ` [PATCH net-next v7 3/3] net: macb: distribute evenly Tx SRAM segments Théo Lebrun
2026-11-19 11:45   ` Nicolai Buchwitz

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMZAUI64FXCL.13646UPPGKGTC@bootlin.com \
    --to=theo.lebrun@bootlin.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=benoit.monin@bootlin.com \
    --cc=conor.dooley@microchip.com \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=gregory.clement@bootlin.com \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux@armlinux.org.uk \
    --cc=nb@tipi-net.de \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=pvalerio@redhat.com \
    --cc=tawfik.bayouk@mobileye.com \
    --cc=thomas.petazzoni@bootlin.com \
    --cc=vineeth.karumanchi@amd.com \
    --cc=vladimir.kondratiev@mobileye.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®