Questions on interrupt routing / balancing (on x86)

Questions on interrupt routing / balancing (on x86)

[Jean Delvare]

Hi all,

Lately, I have been looking into how interrupts are routed / balanced on 
x86 systems. I've learned a lot of things along the road and feel much 
more familiar with the whole thing now, however there are still a few 
things I don't understand and for which I would appreciate explanations 
or pointers.

I do understand that the IO-APIC has two routing modes, namely flat and 
physical flat, the former being limited to 8 logical CPUs for technical 
reasons. I also found that physical flat routing would be selected even 
with only 8 CPUs installed if the host chipset is known to support more. 
I have also read about how MSI-X allows smarter balancing of interrupts 
across CPUs by allowing multiple interrupt queues per device. Now to the 
things I do not fully understand:

* In physical flat mode, all interrupts are bound to CPU0 by default. As 
I understand it, it stays that way until user-space (for example 
irqbalance) adjusts the smp_affinity masks in /proc/irq. Why don't we 
pick a different CPU for every interrupt by default? For example 
irq_nr%cpu_max? This would seem a better default for performance, but 
maybe not for power savings. Is it the reason why it isn't done?

* In (non-physical) flat mode, my understanding is that a given 
interrupt can be mapped to several CPUs (and this happens by default) 
and live round-robin balancing can happen. I have seen systems where it 
actually happens, with interrupt counters perfectly balanced on all 
CPUs, but I have also seen systems where CPU0 gets all the interrupts 
all the time. Why is it so? Where is the kernel code which decides if 
round-robin balancing should happen? Or is this a hardware decision?

* Would it be possible to have a kernel boot parameter to force (non-
physical) flat mode even with more than 8 CPUs, in order to restore 
round-robin balancing? I understand that this would limit interrupt 
routing to CPUs 0-7, but other than this, would it work?

* Do I properly understand that MSI and MSI-X interrupts do NOT go 
through the IO-APIC and are thus not affected by flat vs. physical flat 
APIC routing mode? If so, what determines whether these interrupts get 
round-robin balanced or not? Here too, I've seen systems where it 
happens and others where it doesn't (looking at /proc/interrupts.)

Thanks for any answer or pointer you can provide.

-- 
Jean Delvare
Suse L3

Re: Questions on interrupt routing / balancing (on x86)

[Suresh Siddha]

On Fri, 2011-10-14 at 05:36 -0700, Jean Delvare wrote:
> * In physical flat mode, all interrupts are bound to CPU0 by default. As 
> I understand it, it stays that way until user-space (for example 
> irqbalance) adjusts the smp_affinity masks in /proc/irq. Why don't we 
> pick a different CPU for every interrupt by default? For example 
> irq_nr%cpu_max? This would seem a better default for performance, but 
> maybe not for power savings. Is it the reason why it isn't done?

It really depends on what apic mode is selected and how HW implements
etc.

Also that may not always be better default. For example, two heavily
loaded irq's like network/disk interrupt can end up on two hyperthreaded
siblings leading to poor performance. So irqbalance is the right place
to have all the smartness.

> 
> * In (non-physical) flat mode, my understanding is that a given 
> interrupt can be mapped to several CPUs (and this happens by default) 
> and live round-robin balancing can happen. I have seen systems where it 
> actually happens, with interrupt counters perfectly balanced on all 
> CPUs, but I have also seen systems where CPU0 gets all the interrupts 
> all the time. Why is it so? Where is the kernel code which decides if 
> round-robin balancing should happen? Or is this a hardware decision?

Its chipset decision. Some chipsets implement round-robin and some
don't.

> 
> * Would it be possible to have a kernel boot parameter to force (non-
> physical) flat mode even with more than 8 CPUs, in order to restore 
> round-robin balancing? I understand that this would limit interrupt 
> routing to CPUs 0-7, but other than this, would it work?

you can use logical cluster mode which can support upto 60 logical
cpu's. Some old linux kernels used to support this mode.

> 
> * Do I properly understand that MSI and MSI-X interrupts do NOT go 
> through the IO-APIC and are thus not affected by flat vs. physical flat 
> APIC routing mode? If so, what determines whether these interrupts get 
> round-robin balanced or not? Here too, I've seen systems where it 
> happens and others where it doesn't (looking at /proc/interrupts.)
> 

Even though MSI and MSI-X don't go through io-apic, they still use the
apic model that is selected by the kernel.

Anyways, newer platforms are supporting x2apic extensions that will
support more cpu's in logical/physical mode etc.

thanks,
suresh

Re: Questions on interrupt routing / balancing (on x86)

[Jean Delvare]

Thanks Suresh for all the answers, very useful! :)

-- 
Jean Delvare
Suse L3