* [RFC PATCH] PCI: Sort resources by size as secondary key
@ 2026-06-18 7:25 Ding Hui
2026-06-22 9:24 ` Ilpo Järvinen
0 siblings, 1 reply; 4+ messages in thread
From: Ding Hui @ 2026-06-18 7:25 UTC (permalink / raw)
To: bhelgaas, linux-pci, linux-kernel; +Cc: ilpo.jarvinen, Ding Hui
We encountered an issue on BCM57414 NIC where function 1 failed to
enable SR-IOV after remove & rescan. Investigation revealed this is
caused by BAR allocation failure during rescan.
Simplified topology:
+-[0000:30]-+- ...
| +-02.0-[31]--+-00.0 Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7]
| | \-00.1 Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7]
iomem layout after init bootup:
22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]
22fffec00000-22ffff3fffff : 0000:31:00.1 [align=1M size=8M BAR 9 (VF BAR 2)]
22ffff400000-22ffffbfffff : 0000:31:00.0 [align=1M size=8M BAR 9 (VF BAR 2)]
22ffffc00000-22ffffcfffff : 0000:31:00.1 [align=1M size=1M BAR 2]
22ffffd00000-22ffffdfffff : 0000:31:00.0 [align=1M size=1M BAR 2]
22ffffe00000-22ffffe0ffff : 0000:31:00.1 [align=64K size=64K BAR 0]
22ffffe10000-22ffffe1ffff : 0000:31:00.0 [align=64K size=64K BAR 0]
22ffffe20000-22ffffe3ffff : 0000:31:00.1 [align=16K size=128K BAR 11(VF BAR 4)]
22ffffe40000-22ffffe5ffff : 0000:31:00.1 [align=16K size=128K BAR 7 (VF BAR 0)]
22ffffe60000-22ffffe7ffff : 0000:31:00.0 [align=16K size=128K BAR 11(VF BAR 4)]
22ffffe80000-22ffffe9ffff : 0000:31:00.0 [align=16K size=128K BAR 7 (VF BAR 0)]
22ffffea0000-22ffffea1fff : 0000:31:00.1 [align=8K size=8K BAR 4]
22ffffea2000-22ffffea3fff : 0000:31:00.0 [align=8K size=8K BAR 4]
iomem layout after remove function 1 by
echo "1" > /sys/bus/pci/devices/0000:31:00.1/remove
22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]
22ffff400000-22ffffbfffff : 0000:31:00.0 [align=1M size=8M BAR 9 (VF BAR 2)]
22ffffd00000-22ffffdfffff : 0000:31:00.0 [align=1M size=1M BAR 2]
22ffffe10000-22ffffe1ffff : 0000:31:00.0 [align=64K size=64K BAR 0]
22ffffe60000-22ffffe7ffff : 0000:31:00.0 [align=16K size=128K BAR 11(VF BAR 4)]
22ffffe80000-22ffffe9ffff : 0000:31:00.0 [align=16K size=128K BAR 7 (VF BAR 0)]
22ffffea2000-22ffffea3fff : 0000:31:00.0 [align=8K size=8K BAR 4]
Rescan logs triggered by
echo "1" > /sys/bus/pci/devices/0000:30:02.0/rescan
[ 90.585067] pci 0000:31:00.1: [14e4:16d7] type 00 class 0x020000 PCIe Endpoint
[ 90.585107] pci 0000:31:00.1: BAR 0 [mem 0x22ffffe00000-0x22ffffe0ffff 64bit pref]
[ 90.585113] pci 0000:31:00.1: BAR 2 [mem 0x22ffffc00000-0x22ffffcfffff 64bit pref]
[ 90.585116] pci 0000:31:00.1: BAR 4 [mem 0x22ffffea0000-0x22ffffea1fff 64bit pref]
[ 90.585119] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]
[ 90.585216] pci 0000:31:00.1: PME# supported from D0 D3hot D3cold
[ 90.585253] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe43fff 64bit pref]
[ 90.585255] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe5ffff 64bit pref]: contains BAR 0 for 8 VFs
[ 90.585258] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref]
[ 90.585260] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22ffff3fffff 64bit pref]: contains BAR 2 for 8 VFs
[ 90.585263] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe23fff 64bit pref]
[ 90.585265] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe3ffff 64bit pref]: contains BAR 4 for 8 VFs
[ 90.585534] pci 0000:31:00.1: Adding to iommu group 11
[ 90.585575] pci 0000:31:00.1: BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref]: assigned
[ 90.585585] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: can't assign; no space
[ 90.585587] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: failed to assign
[ 90.585589] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]: assigned
[ 90.585591] pci 0000:31:00.1: BAR 0 [mem 0x22fffed00000-0x22fffed0ffff 64bit pref]: assigned
[ 90.585599] pci 0000:31:00.1: VF BAR 0 [mem 0x22fffed10000-0x22fffed2ffff 64bit pref]: assigned
[ 90.585603] pci 0000:31:00.1: VF BAR 4 [mem 0x22fffed30000-0x22fffed4ffff 64bit pref]: assigned
[ 90.585606] pci 0000:31:00.1: BAR 4 [mem 0x22fffed50000-0x22fffed51fff 64bit pref]: assigned
Enable sriov failed logs triggered by
echo 2 > /sys/bus/pci/devices/0000:31:00.1/sriov_numvfs
[ 1666.918432] bnxt_en 0000:31:00.1: not enough MMIO resources for SR-IOV
[ 1666.918442] bnxt_en 0000:31:00.1 eth5: pci_enable_sriov failed : -12
The resource allocation process during rescan is as follows:
dev_rescan_store
pci_rescan_bus
pci_assign_unassigned_bus_resources
__pci_bus_assign_resources
pbus_assign_resources_sorted
pdev_sort_resources
__assign_resources_sorted
assign_requested_resources_sorted
pci_assign_resource
We noticed that current sort algorithm is only by alignment.
The BAR 2 (align=1M size=1M) is located before BAR 9 (VF BAR 2
align=1M size=8M), so the 8M cannot be satisfied.
If we keep alignment as primary sorting key, but use size as secondary
key, all resource can be satisfied when remove & rescan.
Does this approach only solve current specific case as a workaround,
or does it also benefit general PCI resource allocation?
I think it may help reduce allocation failures due to fragmentation
theoretically, but I'm not sure.
Appreciate any comment and suggestion, thanks.
Signed-off-by: Ding Hui <dinghui@sangfor.com.cn>
---
drivers/pci/setup-bus.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 4cf120ebe5ad..63f224f0c6be 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -367,7 +367,8 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head)
align = pci_resource_alignment(dev_res->dev,
dev_res->res);
- if (r_align > align) {
+ if (r_align > align ||
+ (r_align == align && resource_size(r) > resource_size(dev_res->res))) {
n = &dev_res->list;
break;
}
--
2.17.1
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [RFC PATCH] PCI: Sort resources by size as secondary key 2026-06-18 7:25 [RFC PATCH] PCI: Sort resources by size as secondary key Ding Hui @ 2026-06-22 9:24 ` Ilpo Järvinen 2026-06-22 12:54 ` Ding Hui 0 siblings, 1 reply; 4+ messages in thread From: Ilpo Järvinen @ 2026-06-22 9:24 UTC (permalink / raw) To: Ding Hui; +Cc: bhelgaas, linux-pci, LKML On Thu, 18 Jun 2026, Ding Hui wrote: > We encountered an issue on BCM57414 NIC where function 1 failed to Don't or "We" (or "I") in changelog sentences. Use imperative tone. Here you can start just with: BCM57414 NIC function 1 fails to ... > enable SR-IOV after remove & rescan. Investigation revealed this is > caused by BAR allocation failure during rescan. > > Simplified topology: > > +-[0000:30]-+- ... > | +-02.0-[31]--+-00.0 Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] > | | \-00.1 Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] > > iomem layout after init bootup: > > 22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M] > 22fffec00000-22ffff3fffff : 0000:31:00.1 [align=1M size=8M BAR 9 (VF BAR 2)] > 22ffff400000-22ffffbfffff : 0000:31:00.0 [align=1M size=8M BAR 9 (VF BAR 2)] > 22ffffc00000-22ffffcfffff : 0000:31:00.1 [align=1M size=1M BAR 2] > 22ffffd00000-22ffffdfffff : 0000:31:00.0 [align=1M size=1M BAR 2] > 22ffffe00000-22ffffe0ffff : 0000:31:00.1 [align=64K size=64K BAR 0] > 22ffffe10000-22ffffe1ffff : 0000:31:00.0 [align=64K size=64K BAR 0] > 22ffffe20000-22ffffe3ffff : 0000:31:00.1 [align=16K size=128K BAR 11(VF BAR 4)] > 22ffffe40000-22ffffe5ffff : 0000:31:00.1 [align=16K size=128K BAR 7 (VF BAR 0)] > 22ffffe60000-22ffffe7ffff : 0000:31:00.0 [align=16K size=128K BAR 11(VF BAR 4)] > 22ffffe80000-22ffffe9ffff : 0000:31:00.0 [align=16K size=128K BAR 7 (VF BAR 0)] > 22ffffea0000-22ffffea1fff : 0000:31:00.1 [align=8K size=8K BAR 4] > 22ffffea2000-22ffffea3fff : 0000:31:00.0 [align=8K size=8K BAR 4] > > iomem layout after remove function 1 by > echo "1" > /sys/bus/pci/devices/0000:31:00.1/remove > > 22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M] > 22ffff400000-22ffffbfffff : 0000:31:00.0 [align=1M size=8M BAR 9 (VF BAR 2)] > 22ffffd00000-22ffffdfffff : 0000:31:00.0 [align=1M size=1M BAR 2] > 22ffffe10000-22ffffe1ffff : 0000:31:00.0 [align=64K size=64K BAR 0] > 22ffffe60000-22ffffe7ffff : 0000:31:00.0 [align=16K size=128K BAR 11(VF BAR 4)] > 22ffffe80000-22ffffe9ffff : 0000:31:00.0 [align=16K size=128K BAR 7 (VF BAR 0)] > 22ffffea2000-22ffffea3fff : 0000:31:00.0 [align=8K size=8K BAR 4] > > Rescan logs triggered by > echo "1" > /sys/bus/pci/devices/0000:30:02.0/rescan > > [ 90.585067] pci 0000:31:00.1: [14e4:16d7] type 00 class 0x020000 PCIe Endpoint > [ 90.585107] pci 0000:31:00.1: BAR 0 [mem 0x22ffffe00000-0x22ffffe0ffff 64bit pref] > [ 90.585113] pci 0000:31:00.1: BAR 2 [mem 0x22ffffc00000-0x22ffffcfffff 64bit pref] > [ 90.585116] pci 0000:31:00.1: BAR 4 [mem 0x22ffffea0000-0x22ffffea1fff 64bit pref] > [ 90.585119] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref] > [ 90.585216] pci 0000:31:00.1: PME# supported from D0 D3hot D3cold > [ 90.585253] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe43fff 64bit pref] > [ 90.585255] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe5ffff 64bit pref]: contains BAR 0 for 8 VFs > [ 90.585258] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref] > [ 90.585260] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22ffff3fffff 64bit pref]: contains BAR 2 for 8 VFs > [ 90.585263] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe23fff 64bit pref] > [ 90.585265] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe3ffff 64bit pref]: contains BAR 4 for 8 VFs > [ 90.585534] pci 0000:31:00.1: Adding to iommu group 11 > [ 90.585575] pci 0000:31:00.1: BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref]: assigned > [ 90.585585] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: can't assign; no space > [ 90.585587] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: failed to assign > [ 90.585589] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]: assigned > [ 90.585591] pci 0000:31:00.1: BAR 0 [mem 0x22fffed00000-0x22fffed0ffff 64bit pref]: assigned > [ 90.585599] pci 0000:31:00.1: VF BAR 0 [mem 0x22fffed10000-0x22fffed2ffff 64bit pref]: assigned > [ 90.585603] pci 0000:31:00.1: VF BAR 4 [mem 0x22fffed30000-0x22fffed4ffff 64bit pref]: assigned > [ 90.585606] pci 0000:31:00.1: BAR 4 [mem 0x22fffed50000-0x22fffed51fff 64bit pref]: assigned Timestamps are irrelevant noise to the problem and should be removed. > > Enable sriov failed logs triggered by > echo 2 > /sys/bus/pci/devices/0000:31:00.1/sriov_numvfs > > [ 1666.918432] bnxt_en 0000:31:00.1: not enough MMIO resources for SR-IOV > [ 1666.918442] bnxt_en 0000:31:00.1 eth5: pci_enable_sriov failed : -12 > > The resource allocation process during rescan is as follows: > > dev_rescan_store > pci_rescan_bus > pci_assign_unassigned_bus_resources > __pci_bus_assign_resources > pbus_assign_resources_sorted > pdev_sort_resources > __assign_resources_sorted > assign_requested_resources_sorted > pci_assign_resource > > We noticed that current sort algorithm is only by alignment. > The BAR 2 (align=1M size=1M) is located before BAR 9 (VF BAR 2 > align=1M size=8M), so the 8M cannot be satisfied. I think you have a typo (located vs allocated)? The difference changes meaning significantly. (located implies before in address, allocated implies before in the order of made allocations). > If we keep alignment as primary sorting key, but use size as secondary > key, all resource can be satisfied when remove & rescan. > > Does this approach only solve current specific case as a workaround, > or does it also benefit general PCI resource allocation? > > I think it may help reduce allocation failures due to fragmentation > theoretically, but I'm not sure. I suppose trying the largest first does generally increase the chances of success in cases where the resource sizes are very heterogeneous like in your case, not just in this case. You should really rewrite the changelog text though. Try more to focus on how the other allocations from the sibling make it possible to fit some of the resource(s) only into a single place within the window. And therefore largest one should be requested first. I had to figure that bit myself as you didn't clearly state why it fails but only talked vaguely about the order of (al)location(s). There will still be some cases this greedy approach will not get right, such as align=2M,size=2M & align=1M,size=8M. This algorithm is not really designed for filling gaps in a window but the entire window from scratch, which is why it cannot handle all cases. > Appreciate any comment and suggestion, thanks. > > Signed-off-by: Ding Hui <dinghui@sangfor.com.cn> > --- > drivers/pci/setup-bus.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c > index 4cf120ebe5ad..63f224f0c6be 100644 > --- a/drivers/pci/setup-bus.c > +++ b/drivers/pci/setup-bus.c > @@ -367,7 +367,8 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) > align = pci_resource_alignment(dev_res->dev, > dev_res->res); > > - if (r_align > align) { > + if (r_align > align || > + (r_align == align && resource_size(r) > resource_size(dev_res->res))) { This is not the only place where the algorithm does sorting. (And I also know one restore place is lacking restoring ordering.) I was planning to move to rbtree for storing the resources that need to be in a certain order as doing it everywhere results in small variations which is error prone. So maybe it would be time to consider moving to that so we could do the sort order in one place. -- i. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC PATCH] PCI: Sort resources by size as secondary key 2026-06-22 9:24 ` Ilpo Järvinen @ 2026-06-22 12:54 ` Ding Hui 2026-06-22 15:09 ` Ilpo Järvinen 0 siblings, 1 reply; 4+ messages in thread From: Ding Hui @ 2026-06-22 12:54 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: bhelgaas, linux-pci, LKML On 2026/6/22 17:24, Ilpo Järvinen wrote: > On Thu, 18 Jun 2026, Ding Hui wrote: > >> We encountered an issue on BCM57414 NIC where function 1 failed to > > Don't or "We" (or "I") in changelog sentences. Use imperative tone. Here > you can start just with: > > BCM57414 NIC function 1 fails to ... > >> enable SR-IOV after remove & rescan. Investigation revealed this is >> caused by BAR allocation failure during rescan. >> >> Simplified topology: >> >> +-[0000:30]-+- ... >> | +-02.0-[31]--+-00.0 Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] >> | | \-00.1 Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] >> >> iomem layout after init bootup: >> >> 22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M] >> 22fffec00000-22ffff3fffff : 0000:31:00.1 [align=1M size=8M BAR 9 (VF BAR 2)] >> 22ffff400000-22ffffbfffff : 0000:31:00.0 [align=1M size=8M BAR 9 (VF BAR 2)] >> 22ffffc00000-22ffffcfffff : 0000:31:00.1 [align=1M size=1M BAR 2] >> 22ffffd00000-22ffffdfffff : 0000:31:00.0 [align=1M size=1M BAR 2] >> 22ffffe00000-22ffffe0ffff : 0000:31:00.1 [align=64K size=64K BAR 0] >> 22ffffe10000-22ffffe1ffff : 0000:31:00.0 [align=64K size=64K BAR 0] >> 22ffffe20000-22ffffe3ffff : 0000:31:00.1 [align=16K size=128K BAR 11(VF BAR 4)] >> 22ffffe40000-22ffffe5ffff : 0000:31:00.1 [align=16K size=128K BAR 7 (VF BAR 0)] >> 22ffffe60000-22ffffe7ffff : 0000:31:00.0 [align=16K size=128K BAR 11(VF BAR 4)] >> 22ffffe80000-22ffffe9ffff : 0000:31:00.0 [align=16K size=128K BAR 7 (VF BAR 0)] >> 22ffffea0000-22ffffea1fff : 0000:31:00.1 [align=8K size=8K BAR 4] >> 22ffffea2000-22ffffea3fff : 0000:31:00.0 [align=8K size=8K BAR 4] >> >> iomem layout after remove function 1 by >> echo "1" > /sys/bus/pci/devices/0000:31:00.1/remove >> >> 22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M] >> 22ffff400000-22ffffbfffff : 0000:31:00.0 [align=1M size=8M BAR 9 (VF BAR 2)] >> 22ffffd00000-22ffffdfffff : 0000:31:00.0 [align=1M size=1M BAR 2] >> 22ffffe10000-22ffffe1ffff : 0000:31:00.0 [align=64K size=64K BAR 0] >> 22ffffe60000-22ffffe7ffff : 0000:31:00.0 [align=16K size=128K BAR 11(VF BAR 4)] >> 22ffffe80000-22ffffe9ffff : 0000:31:00.0 [align=16K size=128K BAR 7 (VF BAR 0)] >> 22ffffea2000-22ffffea3fff : 0000:31:00.0 [align=8K size=8K BAR 4] >> >> Rescan logs triggered by >> echo "1" > /sys/bus/pci/devices/0000:30:02.0/rescan >> >> [ 90.585067] pci 0000:31:00.1: [14e4:16d7] type 00 class 0x020000 PCIe Endpoint >> [ 90.585107] pci 0000:31:00.1: BAR 0 [mem 0x22ffffe00000-0x22ffffe0ffff 64bit pref] >> [ 90.585113] pci 0000:31:00.1: BAR 2 [mem 0x22ffffc00000-0x22ffffcfffff 64bit pref] >> [ 90.585116] pci 0000:31:00.1: BAR 4 [mem 0x22ffffea0000-0x22ffffea1fff 64bit pref] >> [ 90.585119] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref] >> [ 90.585216] pci 0000:31:00.1: PME# supported from D0 D3hot D3cold >> [ 90.585253] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe43fff 64bit pref] >> [ 90.585255] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe5ffff 64bit pref]: contains BAR 0 for 8 VFs >> [ 90.585258] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref] >> [ 90.585260] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22ffff3fffff 64bit pref]: contains BAR 2 for 8 VFs >> [ 90.585263] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe23fff 64bit pref] >> [ 90.585265] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe3ffff 64bit pref]: contains BAR 4 for 8 VFs >> [ 90.585534] pci 0000:31:00.1: Adding to iommu group 11 >> [ 90.585575] pci 0000:31:00.1: BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref]: assigned >> [ 90.585585] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: can't assign; no space >> [ 90.585587] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: failed to assign >> [ 90.585589] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]: assigned >> [ 90.585591] pci 0000:31:00.1: BAR 0 [mem 0x22fffed00000-0x22fffed0ffff 64bit pref]: assigned >> [ 90.585599] pci 0000:31:00.1: VF BAR 0 [mem 0x22fffed10000-0x22fffed2ffff 64bit pref]: assigned >> [ 90.585603] pci 0000:31:00.1: VF BAR 4 [mem 0x22fffed30000-0x22fffed4ffff 64bit pref]: assigned >> [ 90.585606] pci 0000:31:00.1: BAR 4 [mem 0x22fffed50000-0x22fffed51fff 64bit pref]: assigned > > Timestamps are irrelevant noise to the problem and should be removed. > >> >> Enable sriov failed logs triggered by >> echo 2 > /sys/bus/pci/devices/0000:31:00.1/sriov_numvfs >> >> [ 1666.918432] bnxt_en 0000:31:00.1: not enough MMIO resources for SR-IOV >> [ 1666.918442] bnxt_en 0000:31:00.1 eth5: pci_enable_sriov failed : -12 >> >> The resource allocation process during rescan is as follows: >> >> dev_rescan_store >> pci_rescan_bus >> pci_assign_unassigned_bus_resources >> __pci_bus_assign_resources >> pbus_assign_resources_sorted >> pdev_sort_resources >> __assign_resources_sorted >> assign_requested_resources_sorted >> pci_assign_resource >> >> We noticed that current sort algorithm is only by alignment. >> The BAR 2 (align=1M size=1M) is located before BAR 9 (VF BAR 2 >> align=1M size=8M), so the 8M cannot be satisfied. > > I think you have a typo (located vs allocated)? The difference changes > meaning significantly. (located implies before in address, allocated > implies before in the order of made allocations). > My description could indeed be misleading. My intention was that BAR 2 is before BAR 9 in the sorted list, and therefore it is allocated before BAR 9. >> If we keep alignment as primary sorting key, but use size as secondary >> key, all resource can be satisfied when remove & rescan. >> >> Does this approach only solve current specific case as a workaround, >> or does it also benefit general PCI resource allocation? >> >> I think it may help reduce allocation failures due to fragmentation >> theoretically, but I'm not sure. > > I suppose trying the largest first does generally increase the chances of > success in cases where the resource sizes are very heterogeneous like in your > case, not just in this case. > Thanks for agreeing with this point. > You should really rewrite the changelog text though. Try more to focus on > how the other allocations from the sibling make it possible to fit some of > the resource(s) only into a single place within the window. And therefore > largest one should be requested first. I had to figure that bit myself as > you didn't clearly state why it fails but only talked vaguely about the > order of (al)location(s). > After .1 BAR 2 assigned, the iomem layout (deduce should be): 22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M] 22fffec00000-22fffecfffff : 0000:31:00.1 [align=1M size=1M BAR 2] [gap=7M] 22ffff400000-22ffffbfffff : 0000:31:00.0 [align=1M size=8M BAR 9 (VF BAR 2)] [gap=1M] 22ffffd00000-22ffffdfffff : 0000:31:00.0 [align=1M size=1M BAR 2] [gap=64K] 22ffffe10000-22ffffe1ffff : 0000:31:00.0 [align=64K size=64K BAR 0] [gap=256K] 22ffffe60000-22ffffe7ffff : 0000:31:00.0 [align=16K size=128K BAR 11(VF BAR 4)] 22ffffe80000-22ffffe9ffff : 0000:31:00.0 [align=16K size=128K BAR 7 (VF BAR 0)] [gap=8K] 22ffffea2000-22ffffea3fff : 0000:31:00.0 [align=8K size=8K BAR 4] [gap=368K] And then there is no suitable space available to satisfy both align=1M and size=8M (BAR 9), that lead to "0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: can't assign; no space". > There will still be some cases this greedy approach will not get right, > such as align=2M,size=2M & align=1M,size=8M. This algorithm is not really > designed for filling gaps in a window but the entire window from scratch, > which is why it cannot handle all cases. > Does the "remove & rescan" is still a corner case for kernel, especially that only removing a single function rather than the entire device for a multi-function device, even after you've fixed multiple issues in this scenario? Does the community resent the issues caused by this scenario? >> Appreciate any comment and suggestion, thanks. >> >> Signed-off-by: Ding Hui <dinghui@sangfor.com.cn> >> --- >> drivers/pci/setup-bus.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c >> index 4cf120ebe5ad..63f224f0c6be 100644 >> --- a/drivers/pci/setup-bus.c >> +++ b/drivers/pci/setup-bus.c >> @@ -367,7 +367,8 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) >> align = pci_resource_alignment(dev_res->dev, >> dev_res->res); >> >> - if (r_align > align) { >> + if (r_align > align || >> + (r_align == align && resource_size(r) > resource_size(dev_res->res))) { > > This is not the only place where the algorithm does sorting. > > (And I also know one restore place is lacking restoring ordering.) > Do you mean in __assign_resources_sorted(), retry normal assign after add_size assign failed? I noticed this function after being replied by Sashiko AI review. > I was planning to move to rbtree for storing the resources that need to be > in a certain order as doing it everywhere results in small variations > which is error prone. > > So maybe it would be time to consider moving to that so we could do the > sort order in one place. > Thank you for your guidance. -- Thanks, - Ding Hui ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC PATCH] PCI: Sort resources by size as secondary key 2026-06-22 12:54 ` Ding Hui @ 2026-06-22 15:09 ` Ilpo Järvinen 0 siblings, 0 replies; 4+ messages in thread From: Ilpo Järvinen @ 2026-06-22 15:09 UTC (permalink / raw) To: Ding Hui; +Cc: bhelgaas, linux-pci, LKML [-- Attachment #1: Type: text/plain, Size: 13001 bytes --] On Mon, 22 Jun 2026, Ding Hui wrote: > On 2026/6/22 17:24, Ilpo Järvinen wrote: > > On Thu, 18 Jun 2026, Ding Hui wrote: > > > > > We encountered an issue on BCM57414 NIC where function 1 failed to > > > > Don't or "We" (or "I") in changelog sentences. Use imperative tone. Here > > you can start just with: > > > > BCM57414 NIC function 1 fails to ... > > > > > enable SR-IOV after remove & rescan. Investigation revealed this is > > > caused by BAR allocation failure during rescan. > > > > > > Simplified topology: > > > > > > +-[0000:30]-+- ... > > > | +-02.0-[31]--+-00.0 Broadcom Inc. and subsidiaries BCM57414 > > > NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] > > > | | \-00.1 Broadcom Inc. and subsidiaries BCM57414 > > > NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] > > > > > > iomem layout after init bootup: > > > > > > 22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M] > > > 22fffec00000-22ffff3fffff : 0000:31:00.1 [align=1M size=8M BAR 9 > > > (VF BAR 2)] > > > 22ffff400000-22ffffbfffff : 0000:31:00.0 [align=1M size=8M BAR 9 > > > (VF BAR 2)] > > > 22ffffc00000-22ffffcfffff : 0000:31:00.1 [align=1M size=1M BAR 2] > > > 22ffffd00000-22ffffdfffff : 0000:31:00.0 [align=1M size=1M BAR 2] > > > 22ffffe00000-22ffffe0ffff : 0000:31:00.1 [align=64K size=64K BAR 0] > > > 22ffffe10000-22ffffe1ffff : 0000:31:00.0 [align=64K size=64K BAR 0] > > > 22ffffe20000-22ffffe3ffff : 0000:31:00.1 [align=16K size=128K BAR > > > 11(VF BAR 4)] > > > 22ffffe40000-22ffffe5ffff : 0000:31:00.1 [align=16K size=128K BAR 7 > > > (VF BAR 0)] > > > 22ffffe60000-22ffffe7ffff : 0000:31:00.0 [align=16K size=128K BAR > > > 11(VF BAR 4)] > > > 22ffffe80000-22ffffe9ffff : 0000:31:00.0 [align=16K size=128K BAR 7 > > > (VF BAR 0)] > > > 22ffffea0000-22ffffea1fff : 0000:31:00.1 [align=8K size=8K BAR 4] > > > 22ffffea2000-22ffffea3fff : 0000:31:00.0 [align=8K size=8K BAR 4] > > > > > > iomem layout after remove function 1 by > > > echo "1" > /sys/bus/pci/devices/0000:31:00.1/remove > > > > > > 22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M] > > > 22ffff400000-22ffffbfffff : 0000:31:00.0 [align=1M size=8M BAR 9 > > > (VF BAR 2)] > > > 22ffffd00000-22ffffdfffff : 0000:31:00.0 [align=1M size=1M BAR 2] > > > 22ffffe10000-22ffffe1ffff : 0000:31:00.0 [align=64K size=64K BAR 0] > > > 22ffffe60000-22ffffe7ffff : 0000:31:00.0 [align=16K size=128K BAR > > > 11(VF BAR 4)] > > > 22ffffe80000-22ffffe9ffff : 0000:31:00.0 [align=16K size=128K BAR 7 > > > (VF BAR 0)] > > > 22ffffea2000-22ffffea3fff : 0000:31:00.0 [align=8K size=8K BAR 4] > > > > > > Rescan logs triggered by > > > echo "1" > /sys/bus/pci/devices/0000:30:02.0/rescan > > > > > > [ 90.585067] pci 0000:31:00.1: [14e4:16d7] type 00 class 0x020000 PCIe > > > Endpoint > > > [ 90.585107] pci 0000:31:00.1: BAR 0 [mem 0x22ffffe00000-0x22ffffe0ffff > > > 64bit pref] > > > [ 90.585113] pci 0000:31:00.1: BAR 2 [mem 0x22ffffc00000-0x22ffffcfffff > > > 64bit pref] > > > [ 90.585116] pci 0000:31:00.1: BAR 4 [mem 0x22ffffea0000-0x22ffffea1fff > > > 64bit pref] > > > [ 90.585119] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref] > > > [ 90.585216] pci 0000:31:00.1: PME# supported from D0 D3hot D3cold > > > [ 90.585253] pci 0000:31:00.1: VF BAR 0 [mem > > > 0x22ffffe40000-0x22ffffe43fff 64bit pref] > > > [ 90.585255] pci 0000:31:00.1: VF BAR 0 [mem > > > 0x22ffffe40000-0x22ffffe5ffff 64bit pref]: contains BAR 0 for 8 VFs > > > [ 90.585258] pci 0000:31:00.1: VF BAR 2 [mem > > > 0x22fffec00000-0x22fffecfffff 64bit pref] > > > [ 90.585260] pci 0000:31:00.1: VF BAR 2 [mem > > > 0x22fffec00000-0x22ffff3fffff 64bit pref]: contains BAR 2 for 8 VFs > > > [ 90.585263] pci 0000:31:00.1: VF BAR 4 [mem > > > 0x22ffffe20000-0x22ffffe23fff 64bit pref] > > > [ 90.585265] pci 0000:31:00.1: VF BAR 4 [mem > > > 0x22ffffe20000-0x22ffffe3ffff 64bit pref]: contains BAR 4 for 8 VFs > > > [ 90.585534] pci 0000:31:00.1: Adding to iommu group 11 > > > [ 90.585575] pci 0000:31:00.1: BAR 2 [mem 0x22fffec00000-0x22fffecfffff > > > 64bit pref]: assigned > > > [ 90.585585] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit > > > pref]: can't assign; no space > > > [ 90.585587] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit > > > pref]: failed to assign > > > [ 90.585589] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]: > > > assigned > > > [ 90.585591] pci 0000:31:00.1: BAR 0 [mem 0x22fffed00000-0x22fffed0ffff > > > 64bit pref]: assigned > > > [ 90.585599] pci 0000:31:00.1: VF BAR 0 [mem > > > 0x22fffed10000-0x22fffed2ffff 64bit pref]: assigned > > > [ 90.585603] pci 0000:31:00.1: VF BAR 4 [mem > > > 0x22fffed30000-0x22fffed4ffff 64bit pref]: assigned > > > [ 90.585606] pci 0000:31:00.1: BAR 4 [mem 0x22fffed50000-0x22fffed51fff > > > 64bit pref]: assigned > > > > Timestamps are irrelevant noise to the problem and should be removed. > > > > > > > > Enable sriov failed logs triggered by > > > echo 2 > /sys/bus/pci/devices/0000:31:00.1/sriov_numvfs > > > > > > [ 1666.918432] bnxt_en 0000:31:00.1: not enough MMIO resources for SR-IOV > > > [ 1666.918442] bnxt_en 0000:31:00.1 eth5: pci_enable_sriov failed : -12 > > > > > > The resource allocation process during rescan is as follows: > > > > > > dev_rescan_store > > > pci_rescan_bus > > > pci_assign_unassigned_bus_resources > > > __pci_bus_assign_resources > > > pbus_assign_resources_sorted > > > pdev_sort_resources > > > __assign_resources_sorted > > > assign_requested_resources_sorted > > > pci_assign_resource > > > > > > We noticed that current sort algorithm is only by alignment. > > > The BAR 2 (align=1M size=1M) is located before BAR 9 (VF BAR 2 > > > align=1M size=8M), so the 8M cannot be satisfied. > > > > I think you have a typo (located vs allocated)? The difference changes > > meaning significantly. (located implies before in address, allocated > > implies before in the order of made allocations). > > > > My description could indeed be misleading. My intention was that BAR 2 is > before BAR 9 in the sorted list, and therefore it is allocated before BAR 9. Yes. First I misunderstood you but realized later what was your meaning after I managed to decipher what is the story your logs told. I'd prefer the changelog is written such that logs only prove things happened the way you description they did. That is, even if all lines would be deleted, the person looking at your patch should understand what's going wrong. > > > If we keep alignment as primary sorting key, but use size as secondary > > > key, all resource can be satisfied when remove & rescan. > > > > > > Does this approach only solve current specific case as a workaround, > > > or does it also benefit general PCI resource allocation? > > > > > > I think it may help reduce allocation failures due to fragmentation > > > theoretically, but I'm not sure. > > > > I suppose trying the largest first does generally increase the chances of > > success in cases where the resource sizes are very heterogeneous like in > > your > > case, not just in this case. > > > > Thanks for agreeing with this point. > > > You should really rewrite the changelog text though. Try more to focus on > > how the other allocations from the sibling make it possible to fit some of > > the resource(s) only into a single place within the window. And therefore > > largest one should be requested first. I had to figure that bit myself as > > you didn't clearly state why it fails but only talked vaguely about the > > order of (al)location(s). > > > > After .1 BAR 2 assigned, the iomem layout (deduce should be): > > 22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M] > 22fffec00000-22fffecfffff : 0000:31:00.1 [align=1M size=1M BAR 2] > [gap=7M] > 22ffff400000-22ffffbfffff : 0000:31:00.0 [align=1M size=8M BAR 9 (VF > BAR 2)] > [gap=1M] > 22ffffd00000-22ffffdfffff : 0000:31:00.0 [align=1M size=1M BAR 2] > [gap=64K] > 22ffffe10000-22ffffe1ffff : 0000:31:00.0 [align=64K size=64K BAR 0] > [gap=256K] > 22ffffe60000-22ffffe7ffff : 0000:31:00.0 [align=16K size=128K BAR 11(VF > BAR 4)] > 22ffffe80000-22ffffe9ffff : 0000:31:00.0 [align=16K size=128K BAR 7 (VF > BAR 0)] > [gap=8K] > 22ffffea2000-22ffffea3fff : 0000:31:00.0 [align=8K size=8K BAR 4] > [gap=368K] > > And then there is no suitable space available to satisfy both align=1M and > size=8M (BAR 9), > that lead to "0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: can't > assign; no space". I figured that out myself by deciphering your logs (which has quite high cost from reviewer point of view). It would have been much easier to tell expliciltly that resource tree ends up into this intermediate state while doing the assignments. That is, explicitly state before the log snippets there are 1M and 8M empty gaps in the bridge window, and the greedy approach places size=1M,align=1M one first into 8M gap, leaving no space for the size=8M,align=1M resource. > > There will still be some cases this greedy approach will not get right, > > such as align=2M,size=2M & align=1M,size=8M. This algorithm is not really > > designed for filling gaps in a window but the entire window from scratch, > > which is why it cannot handle all cases. > > > > Does the "remove & rescan" is still a corner case for kernel, especially that > only removing a single function rather than the entire device for a > multi-function > device, even after you've fixed multiple issues in this scenario? I didn't say it's a corner case but just pointed up there will always be some cases with which a greedy approach will end up in tears. How common they are, I don't know (but suspect they're rare). I'd prefer remove + rescan for the same device to work at least as good as before remove. To go beyond that, tracking things is hard with the current algorith. The complexity comes from the disjoint nature of sizing and assignment. We cannot easily retry sizing after learning there's an assignment failure because we lack of persisting structure where to store the fitting information (struct pci_dev_resource that would last across resource fitting rounds/passes). Those challenges make it a bit harder to come up better fitting strategies. > Does the community resent the issues caused by this scenario? I suspect those how encounter such issues feel powerless to make any change to it, even if some would resent their failing cases. Also, changing one thing easily breaks another scenario risking revert and return to status quo. This algorithm is not exactly easy to approach to and contains very much decades old code (largely unexplained, of course). > > > Appreciate any comment and suggestion, thanks. > > > > > > Signed-off-by: Ding Hui <dinghui@sangfor.com.cn> > > > --- > > > drivers/pci/setup-bus.c | 3 ++- > > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > > > diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c > > > index 4cf120ebe5ad..63f224f0c6be 100644 > > > --- a/drivers/pci/setup-bus.c > > > +++ b/drivers/pci/setup-bus.c > > > @@ -367,7 +367,8 @@ static void pdev_sort_resources(struct pci_dev *dev, > > > struct list_head *head) > > > align = pci_resource_alignment(dev_res->dev, > > > dev_res->res); > > > - if (r_align > align) { > > > + if (r_align > align || > > > + (r_align == align && resource_size(r) > > > > resource_size(dev_res->res))) { > > > > This is not the only place where the algorithm does sorting. > > > > (And I also know one restore place is lacking restoring ordering.) > > > > Do you mean in __assign_resources_sorted(), retry normal assign after add_size > assign failed? Yes. > I noticed this function after being replied by Sashiko AI review. > > > I was planning to move to rbtree for storing the resources that need to be > > in a certain order as doing it everywhere results in small variations > > which is error prone. > > > > So maybe it would be time to consider moving to that so we could do the > > sort order in one place. > > > > Thank you for your guidance. -- i. ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-06-22 15:09 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-06-18 7:25 [RFC PATCH] PCI: Sort resources by size as secondary key Ding Hui 2026-06-22 9:24 ` Ilpo Järvinen 2026-06-22 12:54 ` Ding Hui 2026-06-22 15:09 ` Ilpo Järvinen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®