[PATCH v3] irqchip/gic-v3-its: enable dynamic MSI-X allocation

[PATCH v3] irqchip/gic-v3-its: enable dynamic MSI-X allocation

[Jinqian Yang]

On ARM64 platforms with GICv3 ITS, VFIO PCI passthrough currently
cannot dynamically allocate MSI-X vectors after MSI-X has been
enabled. When QEMU needs to extend the vector range, it must
disable MSI-X, free all interrupts, then re-enable with a larger
allocation. This creates an interrupt loss window for already-active
vectors.

Consider HNS3 with RoCE: NIC and RDMA share one PCI device and
ITS DeviceID, with MSI-X vectors partitioned as NIC (lower range)
then RoCE (starting at base_vector = num_nic_msi). In VFIO
passthrough, loading hns_roce after hns3 forces QEMU to tear down
all interrupts before re-allocating the larger range. During this
process, NIC interrupts may be lost. Testing confirmed that this
occasionally occurs, causing the network port reset to fail. This
appears to be unavoidable, as it's a standard approach adopted by
all network card vendors.

On Hisilicon HIP09 (ARM64, GICv3/GICv4.1) with latest upstream kernel
and QEMU 8.2. VFIO passthrough of HNS3 NIC to VM: load both hns3 and
hns_roce_hw_v2 drivers and trigger FLR, this bug will occur occasionally.
After enabling dynamic MSIX allocation, this bug no longer occurs.

When set this flag, for the ITT, adding a new vector only inserts a new
entry, existing entries are untouched, so already-active vectors are
unaffected.

The ITT are sized to hwsize at MSI-X domain creation time. so there
is always sufficient space for any vector index that may be dynamiclly
allocated later. So the ITT never needs to be resized at runtime. The
dynamic allocation path(msi_domain_alloc_irq_at -> populate_alloc_info)
simply copies the pre-prepared alloc_data without re-invoking
msi_prepare(). Then picking a free EventID from the already-allocated
ITT and a free LPI form the already-reserved bitmap --- no existing
vector is affected.

Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
---
Changes in v2:
  - Updated the commit message to add test information.
Changes in v3:
  - Updated the commit message to add analysis.
  - Added Reviewed-by tags from Marc.

v1: https://lore.kernel.org/linux-arm-kernel/20260624025345.458387-1-yangjinqian1@huawei.com/
v2: https://lore.kernel.org/linux-arm-kernel/20260711022015.3049867-1-yangjinqian1@huawei.com/
---
 drivers/irqchip/irq-gic-its-msi-parent.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-gic-its-msi-parent.c b/drivers/irqchip/irq-gic-its-msi-parent.c
index b9257103a999..b2b9d2068bb1 100644
--- a/drivers/irqchip/irq-gic-its-msi-parent.c
+++ b/drivers/irqchip/irq-gic-its-msi-parent.c
@@ -18,7 +18,8 @@
 
 #define ITS_MSI_FLAGS_SUPPORTED (MSI_GENERIC_FLAGS_MASK |	\
 				 MSI_FLAG_PCI_MSIX      |	\
-				 MSI_FLAG_MULTI_PCI_MSI)
+				 MSI_FLAG_MULTI_PCI_MSI |	\
+				 MSI_FLAG_PCI_MSIX_ALLOC_DYN)
 
 static int its_translate_frame_address(struct fwnode_handle *msi_node, phys_addr_t *pa)
 {
-- 
2.33.0

[tip: irq/drivers] irqchip/gic-v3-its: Enable dynamic MSI-X allocation

[tip-bot2 for Jinqian Yang]

The following commit has been merged into the irq/drivers branch of tip:

Commit-ID:     0d6a7c417983b36f0aa2a61bc650e8df04dd6ba8
Gitweb:        https://git.kernel.org/tip/0d6a7c417983b36f0aa2a61bc650e8df04dd6ba8
Author:        Jinqian Yang <yangjinqian1@huawei.com>
AuthorDate:    Wed, 15 Jul 2026 11:54:37 +08:00
Committer:     Thomas Gleixner <tglx@kernel.org>
CommitterDate: Wed, 15 Jul 2026 09:10:45 +02:00

irqchip/gic-v3-its: Enable dynamic MSI-X allocation

On ARM64 platforms with GICv3 ITS, VFIO PCI passthrough currently cannot
dynamically allocate MSI-X vectors after MSI-X has been enabled. When QEMU
needs to extend the vector range, it must disable MSI-X, free all
interrupts, then re-enable with a larger allocation. This creates an
interrupt loss window for already-active vectors.

Consider HNS3 with RoCE: NIC and RDMA share one PCI device and ITS
DeviceID, with MSI-X vectors partitioned as NIC (lower range) then RoCE
(starting at base_vector = num_nic_msi). In VFIO passthrough, loading
hns_roce after hns3 forces QEMU to tear down all interrupts before
re-allocating the larger range. During this process, NIC interrupts may be
lost. Testing confirmed that this occasionally occurs, causing the network
port reset to fail. This appears to be unavoidable, as it's a standard
approach adopted by all network card vendors.

On Hisilicon HIP09 (ARM64, GICv3/GICv4.1) with latest upstream kernel
and QEMU 8.2. VFIO passthrough of HNS3 NIC to VM: load both hns3 and
hns_roce_hw_v2 drivers and trigger FLR, this bug will occur occasionally.
After enabling dynamic MSIX allocation, this bug no longer occurs.

When dynamic MSI-X allocatiojn support is enabled, adding a new vector only
inserts a new entry, existing entries are untouched, so already-active
vectors are unaffected.

The ITTs are sized to hardware size at MSI-X domain creation time. so there
is always sufficient space for any vector index that may be dynamiclly
allocated later. So the ITT never needs to be resized at runtime. The
dynamic allocation path uses the already initialized ITT and allocates a
free entry, which means no existing vector is affected.

Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260715035437.700601-1-yangjinqian1@huawei.com
---
 drivers/irqchip/irq-gic-its-msi-parent.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-gic-its-msi-parent.c b/drivers/irqchip/irq-gic-its-msi-parent.c
index b925710..b2b9d20 100644
--- a/drivers/irqchip/irq-gic-its-msi-parent.c
+++ b/drivers/irqchip/irq-gic-its-msi-parent.c
@@ -18,7 +18,8 @@
 
 #define ITS_MSI_FLAGS_SUPPORTED (MSI_GENERIC_FLAGS_MASK |	\
 				 MSI_FLAG_PCI_MSIX      |	\
-				 MSI_FLAG_MULTI_PCI_MSI)
+				 MSI_FLAG_MULTI_PCI_MSI |	\
+				 MSI_FLAG_PCI_MSIX_ALLOC_DYN)
 
 static int its_translate_frame_address(struct fwnode_handle *msi_node, phys_addr_t *pa)
 {

Re: [PATCH] irqchip/gic-v3-its: Enable dynamic MSI-X allocation

[Vlad Poenaru]

Hi,

Commit 0d6a7c417983 ("irqchip/gic-v3-its: Enable dynamic MSI-X
allocation"), merged for v7.3-rc1, breaks bnxt_en on our loving Grace
machines. Reverting it fixes the problem on current mainline.

Hardware:
  - NVIDIA Grace (Neoverse V2), ACPI, GICv3 + ITS
  - Broadcom BCM57508 (BCM957508-N1200FJ), FW 230.0.165.3, bnxt_en
  - The same platform with a Mellanox NIC (mlx5) is not affected

The trigger is a ring resize: a boot-time udev .link file with
RxBufferSize=max, i.e. "ethtool -G eth0 rx 4095 tx 4095". The close
path never gets the HWRM_RING_FREE completions, then TX times out and
the machine hangs:

  [   58.221] bnxt_en 0002:01:00.0 eth0: Resp cmpl intr err msg: 0x51
  [   58.221] bnxt_en 0002:01:00.0 eth0: hwrm_ring_free type 1 failed. rc:fffffff0 err:0
  ... (repeats every ~3.7s for every ring)
  [  215.539] bnxt_en 0002:01:00.0 eth0: NETDEV WATCHDOG: CPU: 69: transmit queue 18 timed out 5118 ms
  [  215.539] bnxt_en 0002:01:00.0 eth0: TX timeout detected, starting reset task!

Bisecting v7.2..v7.3-rc3 on this machine points at the irq-drivers
merge (8b5d31123f41), and within it:

  good: 9023960b74ab ("irqchip: Remove redundant dev_err()/dev_err_probe()")
  bad:  0d6a7c417983 ("irqchip/gic-v3-its: Enable dynamic MSI-X allocation")

On current mainline (f47d223627f8) the same machine fails at ~55s;
with the commit reverted it runs cleanly.

So far it only reproduces with our debug config (KASAN, lockdep,
DEBUG_PAGEALLOC, DMA_API_DEBUG, CPUMASK_OFFSTACK, ...), not with our
production config. I haven't narrowed down which option matters.

My guess is that with pci_msix_can_alloc_dyn() now true behind the ITS,
bnxt switches to dynamic MSI-X allocation and something on that path
loses the completion interrupts, but I haven't root-caused it.

Happy to test patches or collect more data.

Reported-by: Andrew Preece <apreece@meta.com>

#regzbot introduced: 0d6a7c417983b

Thanks,
Vlad