(Re)assignment of PCI BARs

(Re)assignment of PCI BARs

[Beng Tan]

Hi,

I have an issue relating to allocation of PCI resources in the kernel.
My brief question is ... how do I get the kernel to reassign PCI
resources (if that's not already automatic) and/or is it possible to
explicitly reassign a single BAR of a PCI device?

However, my understanding may be incomplete and this question may not
make sense to those who know the kernel better. So, here's some
background.

-- Start background --

I've connected an external PCI Express card to my laptop, but I am
having PCI resource allocation issues.

I have a PE2L (see http://www.hwtools.net/Adapter/PE2L.html)
connecting my laptop's Express Card slot to a Gigabyte ATI Radeon HD
4550. I got the idea from a DIY Vidock thread (see
http://forum.notebookreview.com/showthread.php?t=397667), and the PE2L
is apparently the same as a ViDock2 (see
http://www.villagetronic.com/vidock2/index.html).

My laptop is a Dell 640m (with integrated Intel graphics), and I'm
trying to find a way to hook up 2 external monitors without having to
buy a new system. Hence the external graphics card idea.

-- End background --

The symptom I'm seeing is that the card has BAR 0 unassigned ...

  Region 0: Memory at <unassigned> (64-bit, prefetchable) [disabled]

and indeed, the kernel reports this through dmesg ...

[    6.147093] PCI: Failed to allocate mem resource
#0:10000000@e0000000 for 0000:0d:00.0

Full logs are available at:

http://208.100.55.9/coldboot_20090827/dmesg.txt
http://208.100.55.9/coldboot_20090827/iomem.txt
http://208.100.55.9/coldboot_20090827/lspci_vv.txt

Doesn't the kernel reassign PCI resources if the BIOS doesnt do a good
enough job? Or is that not the case in my old kernel (2.6.24)?

Is it possible to force a reassignment of the impacted BAR? For
example, I wrote a small module which wrote to the device's PCI
configuration registers to explcitly reassign BAR 0 to 0xc00000000
(and also bit bashed the connecting bridge to do the same) but ... I'm
not sure if I'm doing it correctly.

I believe the hardware is working fine. X was able to find out about a
monitor attached to the device, but that is all.

And, for reference, you can see someone else's lspci output of the
same card in the first post of
http://ubuntuforums.org/archive/index.php/t-1215318.html.

So, any thoughts for things I can try to diagnose this? Should I try
compiling and running a bleeding edge kernel?

Thank you. Any help is much appreciated.

Beng Tan

ThinkLeft

Re: (Re)assignment of PCI BARs

[Jesse Barnes]

On Thu, 27 Aug 2009 19:49:32 +0800
Beng Tan <bengtan1@gmail.com> wrote:
> The symptom I'm seeing is that the card has BAR 0 unassigned ...
> 
>   Region 0: Memory at <unassigned> (64-bit, prefetchable) [disabled]
> 
> and indeed, the kernel reports this through dmesg ...
> 
> [    6.147093] PCI: Failed to allocate mem resource
> #0:10000000@e0000000 for 0000:0d:00.0
> 
> Full logs are available at:
> 
> http://208.100.55.9/coldboot_20090827/dmesg.txt
> http://208.100.55.9/coldboot_20090827/iomem.txt
> http://208.100.55.9/coldboot_20090827/lspci_vv.txt
> 
> Doesn't the kernel reassign PCI resources if the BIOS doesnt do a good
> enough job? Or is that not the case in my old kernel (2.6.24)?

A newer kernel could help, there have been some resource related fixes
since 2.6.24.

> Is it possible to force a reassignment of the impacted BAR? For
> example, I wrote a small module which wrote to the device's PCI
> configuration registers to explcitly reassign BAR 0 to 0xc00000000
> (and also bit bashed the connecting bridge to do the same) but ... I'm
> not sure if I'm doing it correctly.

There are some PCI calls you can use to assign space, e.g.
pci_bus_alloc_resource, but really the kernel should try to get this
right.  In your case, it looks like the BIOS isn't giving the bus with
your card a large enough window (256M BAR on your card vs. a 2M windows
on the bus).

-- 
Jesse Barnes, Intel Open Source Technology Center

Re: (Re)assignment of PCI BARs

[Beng Tan]

Hi,

> A newer kernel could help, there have been some resource related fixes
> since 2.6.24.

Okay, I've compiled and tried the latest stable kernel (2.6.30.5)
using the instructions from
https://wiki.ubuntu.com/KernelTeam/GitKernelBuild. This kernel loads
fine, but I still have the issue.

Dumps available at

http://208.100.55.9/cold_boot_20090828-2.6.30.5-custom/dmesg.txt
http://208.100.55.9/cold_boot_20090828-2.6.30.5-custom/iomem.txt
http://208.100.55.9/cold_boot_20090828-2.6.30.5-custom/lspci_vv.txt

dmesg shows ...

[    0.214238] pci 0000:0d:00.0: reg 10 64bit mmio: [0x000000-0xfffffff]

indicating that my laptop's bios is in error and allocated BAR 0 to
the first 256M of memory. This is obviously not right. My guess is
linux should be able to fix this?

However, later on in dmesg, I get ...

[    0.292416] pnp 00:00: mem resource (0x0-0x9fbff) overlaps
0000:0d:00.0 BAR 0 (0x0-0xfffffff), disabling
[    0.292421] pnp 00:00: mem resource (0x9fc00-0x9ffff) overlaps
0000:0d:00.0 BAR 0 (0x0-0xfffffff), disabling
[    0.292425] pnp 00:00: mem resource (0xc0000-0xcffff) overlaps
0000:0d:00.0 BAR 0 (0x0-0xfffffff), disabling
[    0.292429] pnp 00:00: mem resource (0xe0000-0xfffff) overlaps
0000:0d:00.0 BAR 0 (0x0-0xfffffff), disabling
[    0.292433] pnp 00:00: mem resource (0x100000-0x7f6d33ff) overlaps
0000:0d:00.0 BAR 0 (0x0-0xfffffff), disabling

and

[    0.344334] pci 0000:0d:00.0: BAR 0: can't allocate mem resource
[0xe0000000-0xe01fffff]
[    0.344337] pci 0000:00:1c.3: PCI bridge, secondary bus 0000:0d
[    0.344341] pci 0000:00:1c.3:   IO window: 0xd000-0xdfff
[    0.344348] pci 0000:00:1c.3:   MEM window: 0xefb00000-0xefcfffff
[    0.344354] pci 0000:00:1c.3:   PREFETCH window:
0x000000e0000000-0x000000e01fffff

so linux has not reallocated BAR 0 successfully.

> In your case, it looks like the BIOS isn't giving the bus with
> your card a large enough window (256M BAR on your card vs. a 2M windows
> on the bus).

Yes, the intervening bridge only has a 2M window. Is Linux supposed to
be intelligent enough to expand the bridge's window to 256M?

Would someone more knowledgeable be able to say whether this looks like a bug?

Also, I notice from iomem that PCI allocations start from 0xd0000000
up until the end of memory. There isn't actually a free 256M chunk in
this range.

Any idea who or what decides to start allocating at 0xd0000000? Is it
possible to tell linux to start allocating at 0xc0000000 instead?
There's nothing there so the space is free.

Also, booting the kernel with pci=assign-busses didn't affect the
issue. It just changed the bus numbering around.

Re: (Re)assignment of PCI BARs

[Beng Tan]

Hi,

I've found a satisfactory fix for myself ... not quite sure if I've
got the full picture correct, but for those who are interested in
reading ...

Using the boot parameter pci=resource_alignment=26@00:1c.3; I was able
to get the kernel to reassign the intervening bridge. My iomem was
then (excerpt only) ...

88000000-9fffffff : PCI Bus 0000:0d
  88000000-8801ffff : 0000:0d:00.0
  90000000-9fffffff : 0000:0d:00.0
a0000000-a00fffff : PCI Bus 0000:0d
  a0000000-a000ffff : 0000:0d:00.0
  a0010000-a0013fff : 0000:0d:00.1
a0100000-a0100fff : Intel Flush Page
d0000000-dfffffff : 0000:00:02.0

Forcing a resource_alignment reallocated the bridge's windows and
freed up space for the graphics card, which is now like so:

0d:00.0 VGA compatible controller: ATI Technologies Inc Unknown device
9540 (prog-if 00 [VGA controller])
  Subsystem: Giga-byte Technology Unknown device 21ae
  Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
  Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
  Latency: 0, Cache Line Size: 64 bytes
  Interrupt: pin A routed to IRQ 19
  Region 0: Memory at 90000000 (64-bit, prefetchable) [size=256M]
  Region 2: Memory at a0000000 (64-bit, non-prefetchable) [size=64K]
  Region 4: I/O ports at de00 [size=256]
  Expansion ROM at 88000000 [disabled] [size=128K]

(I'm not sure if an align order of 26 is too much. A lesser number may
work just as well, but I haven't tested this)

>From here on, the card and everything else just worked, including X
(albeit with some xorg.conf tweaking).

So, all I have to do is go buy two monitors :( and I can have 3
external monitors on my old crappy laptop :)

For those who are also interested in hooking external PCIe cards to
their laptop's Expresscard port, the PE2L hardware does work. However,
you may encounter BIOS-induced PCI resource allocation issues, and
whether your distribution is recent enough to fix this is an open
question.

Thanks all for reading.

Beng