Re: MSI, fun for the whole family

[David Miller <davem@davemloft.net>]

From: Roland Dreier <rdreier@cisco.com>
Date: Thu, 24 Apr 2008 20:57:48 -0700

> Now, it is true that the kernel could do something crazy and collapse
> all these interrupt vectors into a single "IRQ" and then tell the
> interrupt handler which vector it was by passing some "metadata" in, but
> why not just give each MSI message it's own IRQ?

Actually, it doesn't make any sense to have more MSI, or "MSI queue"
interrupts than you have cpus.

Non-x86 PCI-E controller impelemntations that I am familiar with collect
MSI and MSI-X interrupts into "queues", these queues being non-empty
is what actually triggers an interrupt to the CPU.  And, there are
enough MSI queue instances such that you can direct each one to a
unique cpu.

The MSI queue interrupt simply scans the ring buffer of pending MSI
interrupts and dispatches them to the device.

You can handle PCI-E frabric error messages the same way, and in fact
that's what the controllers I am familiar with do.

A Linux implementation of support for this kind of setup can be seen
in arch/sparc64/kernel/pci_msi.c:sparc64_msiq_interrupt().  It's
very generic and doesn't care whether it's talking to real PCI
controller hardware or a hypervisor based interface.

Besides the obvious extra indirection overhead, our IRQ layer is very
much capable of supporting multi-level dispatch like this correctly.