Re: [PATCH v3 2/2] PCI: Allow 64-bit non-prefetchable BARs in prefetchable windows

["Ard Biesheuvel" <ardb@kernel.org>]

Hi Ilpo,

On Thu, 8 Oct 2026, at 19:52, Ilpo Järvinen wrote:
> On Wed, 30 Sep 2026, Ard Biesheuvel wrote:
>
>> From: Ard Biesheuvel <ardb@kernel.org>
>> 
>> The non-prefetchable memory window of a PCI-to-PCI bridge can only
>> decode 32-bit addresses, and so non-prefetchable BARs of devices below a
>> bridge can only be allocated from the part of the host bridge memory
>> space below 4 GB. This is the case even for 64-bit BARs, which could
>> easily be placed above 4 GB if there was a bridge window to put them in,
>> and on many platforms, 32-bit addressable MMIO space is scarce.
>> 
>> The PCIe spec addresses this in the implementation note "Additional
>> Guidance on the Prefetchable Bit in Memory Space BARs" (PCIe r7.0, sec
>> 7.5.1.2.1): on PCIe, setting the Prefetchable bit of a BAR still permits
>> correct operation even if the range has read side effects or cannot
>> tolerate write merging, as long as the entire path from the host to the
>> device is PCIe, given that PCIe Memory Reads always carry an explicit
>> length, and PCIe Switches never prefetch or merge writes. The same
>> reasoning applies when it is the OS that places a non-prefetchable BAR
>> in a prefetchable bridge window: PCIe Root Ports and Switch Ports
>> forward requests that hit either window in exactly the same way. Hence,
>> the prefetchable window of a PCIe Root Port or Switch Port, which may be
>> 64-bit, can serve as a 64-bit window for non-prefetchable BARs too. [0]
>> 
>> So add pci_bus_placement_flags(), which returns the flags of a resource
>> on a given bus with IORESOURCE_PREFETCH set if it is a 64-bit
>> non-prefetchable memory resource, the bus is not a root bus, all bridges
>> between the bus and the root bus are PCIe Root Ports or PCIe Switch
>> Ports, and the host bridge has no prefetchable memory window, but does
>> have one that extends above 4 GB in PCI bus address space. [1]
>> 
>> Evaluate the conditions on the bridges and the host bridge only once
>> per bus, when it is added, and record the result in a new bus flag,
>> PCI_BUS_FLAGS_NO_NP_BARS_IN_P_WINDOWS: pci_register_host_bridge() sets
>> it on the root bus if the host bridge does not qualify, child buses
>> inherit it, and pci_alloc_child_bus() sets it on the secondary bus of
>> any bridge that is not a PCIe Root Port or Switch Port.
>> 
>> Use pci_bus_placement_flags() wherever the resource allocator decides
>> which bridge window a device resource (including an SR-IOV VF BAR)
>> belongs in:
>> 
>> - in pbus_select_window_for_type(), which is used when sizing bridge
>>   windows, and when releasing them to retry failed assignments or to
>>   resize a BAR;
>> - when allocating the resource in __pci_assign_resource(), by passing
>>   its result to __pci_bus_alloc_resource();
>> - when claiming a resource assigned by firmware, in
>>   pci_find_parent_resource();
>> - when deciding which assigned resources to release after a failed
>>   assignment, and which failures are relevant to a resized BAR.
>> 
>> As a result, an eligible 64-bit non-prefetchable BAR is handled exactly
>> like a 64-bit prefetchable BAR: it is placed in the prefetchable window
>> of the upstream bridge, which may be above 4 GB, and allocation falls
>> back to the non-prefetchable window if the prefetchable one has no
>> space. [2]
>> 
>> 32-bit BARs are not affected, and neither are bridge windows, as only
>> prefetchable bridge windows can be 64-bit. Devices below conventional
>> PCI or CardBus bridges, below PCIe to PCI/PCI-X bridges (in either
>> direction), or below host bridges that have a prefetchable window or no
>> window above 4 GB are not affected either, and neither are devices on a
>> root bus, as there is no prefetchable window for their BARs to go to.
>> 
>> [0] This reasoning does not extend to the host bridge, though: how it
>>     treats the windows that firmware describes as prefetchable is
>>     platform specific. For instance, the V3 Semiconductor V360EPC
>>     (pci-v3-semi) enables prefetching for its prefetchable window, the
>>     MPC52xx uses Memory Read Multiple for it, and Freescale PCI/PCIe
>>     host bridges (fsl_pci) enable relaxed ordering for it. However, if
>>     the host bridge has no prefetchable windows at all (as appears to be
>>     the case on many x86 PCs), all prefetchable bridge windows are
>>     carved out of its non-prefetchable windows, and so the host bridge
>>     does not treat them any differently.
>> 
>> [1] Placing non-prefetchable BARs in prefetchable bridge windows only
>>     helps if one of those non-prefetchable host bridge windows extends
>>     above 4 GB. Otherwise, the prefetchable bridge windows end up below
>>     4 GB as well, and BARs would merely move between two bridge windows
>>     that are carved out of the same 32-bit space.
>> 
>> [2] Note that this only concerns where a BAR is placed. How it is mapped
>>     is decided by the driver and by the attributes of the BAR itself
>>     (e.g., pci_iomap_wc() and the sysfs resource<N>_wc files only honour
>>     IORESOURCE_PREFETCH on the BAR), and this change does not modify the
>>     flags of any BAR.)
>> 
>> Assisted-by: LLM
>> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
>> ---
>> Tested on QEMU arm64 'virt' (DT, all resources assigned by Linux) with
>> a qemu-xhci below a Root Port, an NVMe below a Switch, a qemu-xhci
>> below a PCIe-to-PCI bridge and another qemu-xhci on the root bus: the
>> 64-bit non-prefetchable BARs of the first two (and of the PCIe-to-PCI
>> bridge itself) move from the 32-bit non-prefetchable windows into the
>> 64-bit prefetchable windows above 4 GB, while the others stay where
>> they were. When the 64-bit host bridge window is marked prefetchable
>> in the DT, or removed from it, all resources are assigned exactly as
>> without this patch. Both drivers work, also after hot removal and
>> rescan of the endpoints and of the Switch. Also tested on QEMU x86_64
>> q35 with SeaBIOS, which assigns all resources itself: the firmware
>> assignment is claimed as before, and after hot removal and rescan, the
>> eligible BARs are placed in the prefetchable windows.
>
> Hi again,
>
> I think I now mostly understand the remaining problems, both relate to 
> multipass resource assignment logic. First of all this seems required as 
> well:
>
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index b662cbeda51e..b5fdcb10d45a 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -2206,8 +2206,10 @@ static void pci_prepare_next_assign_round(struct 
> list_head *fail_head,
>  	list_for_each_entry(fail_res, fail_head, list) {
>  		struct pci_bus *bus = fail_res->dev->bus;
>  		struct resource *b_win;
> +		unsigned long flags;
> 
> -		b_win = pbus_select_window_for_type(bus, fail_res->flags);
> +		flags = pci_bus_placement_flags(bus, fail_res->flags);
> +		b_win = pbus_select_window_for_type(bus, flags);
>  		if (!b_win)
>  			continue;
>  		pci_bus_release_bridge_resources(bus, b_win, rel_type);
>
> In addition, pci_bus_release_bridge_resources() -> 
> pci_bridge_release_resources() -> release_child_resources() may release 
> additional resources, some of which may be assigned to a different bridge 
> window by FW than where kernel will try to put them in the next pass. If 
> there are no failed resources for that other bridge window in the fail 
> list, the other bridge window is not released and thus doesn't get sized 
> properly.
>
> Likely, the best course of action would be to wrap release_child_resources()
> with PCI level equivalent (it would improve printing as well as then the 
> resources are properly annotated with the device information).
>
> That would also likely make it possible to avoid tearing down resources 
> that are in use by some endpoint driver (which was recently reported to 
> occur, during hotplug IIRC).
>
> I hope this helps.
>

Thanks a lot. I will get back to this shortly, I hope.