Re: Questions on interrupt routing / balancing (on x86)

[Suresh Siddha <suresh.b.siddha@intel.com>]

On Fri, 2011-10-14 at 05:36 -0700, Jean Delvare wrote:
> * In physical flat mode, all interrupts are bound to CPU0 by default. As 
> I understand it, it stays that way until user-space (for example 
> irqbalance) adjusts the smp_affinity masks in /proc/irq. Why don't we 
> pick a different CPU for every interrupt by default? For example 
> irq_nr%cpu_max? This would seem a better default for performance, but 
> maybe not for power savings. Is it the reason why it isn't done?

It really depends on what apic mode is selected and how HW implements
etc.

Also that may not always be better default. For example, two heavily
loaded irq's like network/disk interrupt can end up on two hyperthreaded
siblings leading to poor performance. So irqbalance is the right place
to have all the smartness.

> 
> * In (non-physical) flat mode, my understanding is that a given 
> interrupt can be mapped to several CPUs (and this happens by default) 
> and live round-robin balancing can happen. I have seen systems where it 
> actually happens, with interrupt counters perfectly balanced on all 
> CPUs, but I have also seen systems where CPU0 gets all the interrupts 
> all the time. Why is it so? Where is the kernel code which decides if 
> round-robin balancing should happen? Or is this a hardware decision?

Its chipset decision. Some chipsets implement round-robin and some
don't.

> 
> * Would it be possible to have a kernel boot parameter to force (non-
> physical) flat mode even with more than 8 CPUs, in order to restore 
> round-robin balancing? I understand that this would limit interrupt 
> routing to CPUs 0-7, but other than this, would it work?

you can use logical cluster mode which can support upto 60 logical
cpu's. Some old linux kernels used to support this mode.

> 
> * Do I properly understand that MSI and MSI-X interrupts do NOT go 
> through the IO-APIC and are thus not affected by flat vs. physical flat 
> APIC routing mode? If so, what determines whether these interrupts get 
> round-robin balanced or not? Here too, I've seen systems where it 
> happens and others where it doesn't (looking at /proc/interrupts.)
> 

Even though MSI and MSI-X don't go through io-apic, they still use the
apic model that is selected by the kernel.

Anyways, newer platforms are supporting x2apic extensions that will
support more cpu's in logical/physical mode etc.

thanks,
suresh