Re: [PATCH net v3] net: mana: Add support to process throttled EQEs

[Sahil Chandna <sahilchandna@linux.microsoft.com>]

On 23-10-2026 01:12, Haiyang Zhang wrote:
> 
> 
>> -----Original Message-----
>> From: Sahil Chandna <sahilchandna@linux.microsoft.com>
>> Sent: Thursday, October 22, 2026 10:54 AM
>> To: Haiyang Zhang <haiyangz@microsoft.com>; wei.liu@kernel.org; Dexuan Cui
>> <DECUI@microsoft.com>; andrew+netdev@lunn.ch; davem@davemloft.net;
>> edumazet@kernel.org; kuba@kernel.org; pabeni@redhat.com; Konstantin
>> Taranov <kotaranov@microsoft.com>; horms@kernel.org;
>> ernis@linux.microsoft.com; gargaditya@linux.microsoft.com;
>> mawasthi@linux.microsoft.com; leitao@debian.org; linux-
>> hyperv@vger.kernel.org; netdev@vger.kernel.org; linux-
>> kernel@vger.kernel.org; linux-rdma@vger.kernel.org
>> Cc: stable@vger.kernel.org
>> Subject: [PATCH net v3] net: mana: Add support to process throttled EQEs
>>
>> When an event queue nears full, the hardware coalesces the per-CQ
>> GDMA_EQE_COMPLETION notifications into a single throttle EQE (type 4)
>> that carries no cq_id. The driver does not handle this event type, so
>> it falls through to the default case and is dropped. Any CQ whose
>> completion notification was replaced by the throttle EQE can remain
>> unserviced, causing its queue to stall.
>> The issue was observed internally and following Tx stall logs were
>> seen:
>>
>> 2026-10-03T05:29:01 hv_netvsc eth0: VF slot 0 removed
>> 2026-10-03T05:29:23 mana 7870:00:00.0 ens1: NETDEV WATCHDOG: CPU: 45:
>>                     transmit queue 1 timed out 15133 ms
>> 2026-10-03T05:31:05 mana 7870:00:00.0: HWC: Request timed out: 60000 ms
>> 2026-10-03T05:31:05 mana 7870:00:00.0: Gf stats wk handler: gf stats
>> query timed out.
>> 2026-10-03T05:31:21 ens1: NETDEV WATCHDOG: CPU: 20: transmit queue 0
>> timed out 15859 ms
>> 2026-10-03T05:31:37 ens1: NETDEV WATCHDOG: CPU: 15: transmit queue 0
>> timed out 31731 ms
>> 2026-10-03T05:31:53 ens1: NETDEV WATCHDOG: CPU: 21: transmit queue 0
>> timed out 47603 ms
>> 2026-10-03T05:32:08 ens1: NETDEV WATCHDOG: CPU: 42: transmit queue 0
>> timed out 62451 ms
>>
>> Add support to process the CQs which belong to a throttled EQE of type 4
>> advertise the capability bit, also add a counter to store the throttled
>> EQEs count.
>>
>> Fixes: ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network
>> Adapter (MANA)")
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Sahil Chandna <sahilchandna@linux.microsoft.com>
>> ---
>> Changes in v3:
>> - Resolve race for cq in mana_gd_destroy_cq and irq handler using cq.
>> - Rebase to net branch
>> Link to v2:
>> https://lore.ker/
>> nel.org%2Fall%2F20261014134304.4177657-1-
>> sahilchandna%40linux.microsoft.com%2F&data=05%7C02%7Chaiyangz%40microsoft.
>> com%7C534992a49c0340631e2f08df304e3c73%7C72f988bf86f141af91ab2d7cd011db47%
>> 7C1%7C0%7C639282784768349454%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRyd
>> WUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C
>> 0%7C%7C%7C&sdata=w2NpUCx%2FCNtqeFQSR0fIMOYnbHLP7XsFl90sL%2F%2F7Y0g%3D&rese
>> rved=0
>>
>> Changes in v2:
>> - Use WRITE_ONCE() when clearing max_num_cqs and add RCU
>>   grace period wait during HWC teardown.
>> - Use synchronize_net() during CQ destroy only when the CQ was
>>   attached to its parent EQ.
>> - Use WRITE_ONCE() and READ_ONCE() for the throttle-event counter.
>> - Add Fixes tag and net instead of net-next.
>> - Fix commit message to describe capability bit and counter.
>> Link to v1:
>> https://lore.ker/
>> nel.org%2Fall%2F20261008083315.306076-1-
>> sahilchandna%40linux.microsoft.com%2F&data=05%7C02%7Chaiyangz%40microsoft.
>> com%7C534992a49c0340631e2f08df304e3c73%7C72f988bf86f141af91ab2d7cd011db47%
>> 7C1%7C0%7C639282784768392174%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRyd
>> WUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C
>> 0%7C%7C%7C&sdata=oj6Ei41iOntNjgoAWNaLSZdvptpbZSkkZMOCa6c2PDw%3D&reserved=0
>>
>>  .../net/ethernet/microsoft/mana/gdma_main.c   | 62 +++++++++++++++++--
>>  .../net/ethernet/microsoft/mana/hw_channel.c  |  6 +-
>>  .../ethernet/microsoft/mana/mana_ethtool.c    |  9 +++
>>  include/net/mana/gdma.h                       | 12 +++-
>>  include/net/mana/mana.h                       |  1 +
>>  5 files changed, 81 insertions(+), 9 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c
>> b/drivers/net/ethernet/microsoft/mana/gdma_main.c
>> index 8e9bfc1d6a2a..d74ed00bb783 100644
>> --- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
>> +++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
>> @@ -904,6 +904,27 @@ ssize_t mana_gd_read_ring(struct gdma_queue *q, char
>> __user *buf,
>>       return copied;
>>  }
>>
>> +static void mana_gd_schedule_eq_cqs(struct gdma_queue *eq)
>> +{
>> +     struct gdma_context *gc = eq->gdma_dev->gdma_context;
>> +     struct gdma_queue *cq;
>> +     u8 i;
>> +
>> +     for (i = 0; i < GDMA_EQ_MAX_CHILD_CQ; i++) {
>> +             cq = rcu_dereference(eq->eq.child_cq[i]);
>> +
>> +             if (!cq)
>> +                     continue;
>> +
>> +             if (cq->id >= READ_ONCE(gc->max_num_cqs) ||
>> +                 !gc->cq_table || gc->cq_table[cq->id] != cq)
>> +                     continue;
>> +
>> +             if (cq->cq.callback)
>> +                     cq->cq.callback(cq->cq.context, cq);
>> +     }
>> +}
>> +
>>  static void mana_gd_process_eqe(struct gdma_queue *eq)
>>  {
>>       u32 head = eq->head % (eq->queue_size / GDMA_EQE_SIZE);
>> @@ -934,6 +955,11 @@ static void mana_gd_process_eqe(struct gdma_queue
>> *eq)
>>
>>               break;
>>
>> +     case GDMA_EQE_THROTTLE:
>> +             WRITE_ONCE(eq->eq.throttle_count, eq->eq.throttle_count + 1);
> Please use u64_stats_update_begin / end() for stats, which is the standard
> way -- see other stats code in this driver.
> 
Ack, will update it in v4.>> +             mana_gd_schedule_eq_cqs(eq);
>> +             break;
>> +
>>       case GDMA_EQE_TEST_EVENT:
>>               gc->test_event_eq_id = eq->id;
>>               complete(&gc->eq_test_event);
> [...]
> 
> 
>> @@ -250,6 +252,7 @@ static void mana_get_ethtool_stats(struct net_device
>> *ndev,
>>       u64 short_pkt_fmt;
>>       u64 csum_partial;
>>       u64 mana_map_err;
>> +     u64 throttle_events;
>>       int q, i = 0, j;
>>
>>       if (!apc->port_is_up)
>> @@ -261,6 +264,12 @@ static void mana_get_ethtool_stats(struct net_device
>> *ndev,
>>        */
>>       mana_query_phy_stats(apc);
>>
>> +     throttle_events = 0;
>> +     for (q = 0; q < num_queues; q++)
>> +             throttle_events += READ_ONCE(apc->eqs[q].eq-
>>> eq.throttle_count);
> 
> Since you already have per queue throttle_count, please output them for
> each EQ.
> 
> Thanks,
> - Haiyang
Ack, i will drop accumulated throttle count and report per EQ in v4.

The patch needs rebase again for net branch, i will rebase in v4.