x86: interrupt routing question

x86: interrupt routing question

[Mike Galbraith]

Greetings,

I have an x3550 M3 box which shows different behavior than my Q6600
desktop box.  The below is the rt kernel, but it doesn't matter which
kernel I boot, all interrupts are on CPU0 unless I move them.  On Q6600
box, IRQs magically appear on every cpu in the affinity mask.  Change
the eth0 mask while flood pinging, numbers start/stop changing on the
fly.

On x3550 M3, setting eg IRQ 63's mask to '0xe' will move irq to CPU2,
but nothing that I have found will make the thing behave the same as
trusty old Q6600 box, which seems strange.  Is this some kind of BIOS
thingie, or is every kernel busted on this hardware?  Busted kernel
seems highly doubtful.

OT question.  I have HT turned off on the x3550 because I do rt testing
on it, and occasionally boot idle=poll.  Is it still the case that you
should not run idle=poll with HT enabled, or can you do that with HT
these days (E5620 CPU) without munching too many cycles?

            CPU0       CPU1       CPU2       CPU3       
   0:        132          0          0          0   IO-APIC-edge      timer
   1:          0          0          0          0   IO-APIC-edge      i8042
   3:          1          0          0          0   IO-APIC-edge    
   4:          1          0          0          0   IO-APIC-edge    
   8:         26          0          0          0   IO-APIC-edge      rtc0
   9:          0          0          0          0   IO-APIC-fasteoi   acpi
  14:          0          0          0          0   IO-APIC-edge      ata_piix
  15:          0          0          0          0   IO-APIC-edge      ata_piix
  16:       5450          0          0          0   IO-APIC-fasteoi   megasas
  17:         52          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb5
  18:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4, uhci_hcd:usb6
  19:      61939          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb7
  21:          0          0          0          0   IO-APIC-fasteoi   ata_piix
  54:          3          0          0          0   PCI-MSI-edge      ioat-msix
  55:          3          0          0          0   PCI-MSI-edge      ioat-msix
  56:          3          0          0          0   PCI-MSI-edge      ioat-msix
  57:          3          0          0          0   PCI-MSI-edge      ioat-msix
  58:          3          0          0          0   PCI-MSI-edge      ioat-msix
  59:          3          0          0          0   PCI-MSI-edge      ioat-msix
  60:          3          0          0          0   PCI-MSI-edge      ioat-msix
  61:          3          0          0          0   PCI-MSI-edge      ioat-msix
  62:        178          0          0          0   PCI-MSI-edge      eth1-0
  63:    6327232          0          0          0   PCI-MSI-edge      eth1-1
  64:         73          0          0          0   PCI-MSI-edge      eth1-2
  65:    6327230          0          0          0   PCI-MSI-edge      eth1-3
  66:          2          0          0          0   PCI-MSI-edge      eth1-4
 NMI:          0          0          0          0   Non-maskable interrupts
 LOC:    2096607    2366453     260343    2225391   Local timer interrupts
 SPU:          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0   Performance monitoring interrupts
 PND:          0          0          0          0   Performance pending work
 RES:       1981       2973       2405       2001   Rescheduling interrupts
 CAL:         49         43         74         72   Function call interrupts
 TLB:        638        784       1175        674   TLB shootdowns
 TRM:          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0   Machine check exceptions
 MCP:        603        603        603        606   Machine check polls
 ERR:          0
 MIS:          0

Re: x86: interrupt routing question

[Mike Galbraith]

For the next sod who gets curious about irq routing behavior difference,
and tries to ask google.

On Tue, 2011-07-12 at 09:27 +0200, Mike Galbraith wrote:
> Greetings,
> 
> I have an x3550 M3 box which shows different behavior than my Q6600
> desktop box.  The below is the rt kernel, but it doesn't matter which
> kernel I boot, all interrupts are on CPU0 unless I move them.  On Q6600
> box, IRQs magically appear on every cpu in the affinity mask.  Change
> the eth0 mask while flood pinging, numbers start/stop changing on the
> fly.
> 
> On x3550 M3, setting eg IRQ 63's mask to '0xe' will move irq to CPU2,
> but nothing that I have found will make the thing behave the same as
> trusty old Q6600 box, which seems strange.  Is this some kind of BIOS
> thingie, or is every kernel busted on this hardware?  Busted kernel
> seems highly doubtful.

The difference is that Q6600 desktop box uses flat routing, and x3550 M3
uses physical flat.  What the heck does that mean you (didn't) ask?
Dunno, that's a "RTFM for all the gory technical details" thing, but if
you rummage around in x86 source, you'll end up here..

arch/kernel/x86/apic/io_apic.c::setup_ioapic_irq()

        if (assign_irq_vector(irq, cfg, apic->target_cpus()))
                return;

        dest = apic->cpu_mask_to_apicid_and(cfg->domain, apic->target_cpus());

        apic_printk(APIC_VERBOSE,KERN_DEBUG
                    "IOAPIC[%d]: Set routing entry (%d-%d -> 0x%x -> "
                    "IRQ %d Mode:%i Active:%i Dest:%d)\n",
                    apic_id, mpc_ioapic_id(apic_id), pin, cfg->vector,
                    irq, trigger, polarity, dest);

..and see that in physical flat mode, assign_irq_vector() scribbles only
one bit to cfg->domain.

arch/kernel/x86/apic/io_apic.c:__assign_irq_vector()

            for_each_cpu_and(cpu, mask, cpu_online_mask) {
                int new_cpu;
                int vector, offset;

                apic->vector_allocation_domain(cpu, tmp_mask);
		...
                cpumask_copy(cfg->domain, tmp_mask);
                err = 0;
                break;
        }

apic.vector_allocation_domain for physical flat is..
 
arch/x86/kernel/apic/apic_flat_64.c::

static void physflat_vector_allocation_domain(int cpu, struct cpumask *retmask)
{
        cpumask_clear(retmask);
        cpumask_set_cpu(cpu, retmask);
}

..whereas for flat it's..

static void flat_vector_allocation_domain(int cpu, struct cpumask *retmask)
{
	...
        cpumask_clear(retmask);
        cpumask_bits(retmask)[0] = APIC_ALL_CPUS;
}

So, while __assign_irq_vector() is called with a mask of all CPUs for
both boxen, apic->vector_allocation_domain() ensures the mask finally
written to cfg->domain has only one CPU bit set in physical flat mode,
and all CPU bits set in flat mode.

	-Mike

P.S.  if you try that RTFM thing, strap a pillow to your forehead first

Re: x86: interrupt routing question

[Bjorn Helgaas]

On Wed, Sep 28, 2011 at 1:28 AM, Mike Galbraith <efault@gmx.de> wrote:
>
> For the next sod who gets curious about irq routing behavior difference,
> and tries to ask google.

Thanks for following up on this!

So, having read the fine manual, do you think this is a Linux bug, or
something that could be improved in Linux, either by changing the
behavior or making the dmesg more informative?  It certainly cost you
a lot of time, and it'd be nice if we could save the next person :)

> On Tue, 2011-07-12 at 09:27 +0200, Mike Galbraith wrote:
> > Greetings,
> >
> > I have an x3550 M3 box which shows different behavior than my Q6600
> > desktop box.  The below is the rt kernel, but it doesn't matter which
> > kernel I boot, all interrupts are on CPU0 unless I move them.  On Q6600
> > box, IRQs magically appear on every cpu in the affinity mask.  Change
> > the eth0 mask while flood pinging, numbers start/stop changing on the
> > fly.
> >
> > On x3550 M3, setting eg IRQ 63's mask to '0xe' will move irq to CPU2,
> > but nothing that I have found will make the thing behave the same as
> > trusty old Q6600 box, which seems strange.  Is this some kind of BIOS
> > thingie, or is every kernel busted on this hardware?  Busted kernel
> > seems highly doubtful.
>
> The difference is that Q6600 desktop box uses flat routing, and x3550 M3
> uses physical flat.  What the heck does that mean you (didn't) ask?
> Dunno, that's a "RTFM for all the gory technical details" thing, but if
> you rummage around in x86 source, you'll end up here..
>
> arch/kernel/x86/apic/io_apic.c::setup_ioapic_irq()
>
>        if (assign_irq_vector(irq, cfg, apic->target_cpus()))
>                return;
>
>        dest = apic->cpu_mask_to_apicid_and(cfg->domain, apic->target_cpus());
>
>        apic_printk(APIC_VERBOSE,KERN_DEBUG
>                    "IOAPIC[%d]: Set routing entry (%d-%d -> 0x%x -> "
>                    "IRQ %d Mode:%i Active:%i Dest:%d)\n",
>                    apic_id, mpc_ioapic_id(apic_id), pin, cfg->vector,
>                    irq, trigger, polarity, dest);
>
> ..and see that in physical flat mode, assign_irq_vector() scribbles only
> one bit to cfg->domain.
>
> arch/kernel/x86/apic/io_apic.c:__assign_irq_vector()
>
>            for_each_cpu_and(cpu, mask, cpu_online_mask) {
>                int new_cpu;
>                int vector, offset;
>
>                apic->vector_allocation_domain(cpu, tmp_mask);
>                ...
>                cpumask_copy(cfg->domain, tmp_mask);
>                err = 0;
>                break;
>        }
>
> apic.vector_allocation_domain for physical flat is..
>
> arch/x86/kernel/apic/apic_flat_64.c::
>
> static void physflat_vector_allocation_domain(int cpu, struct cpumask *retmask)
> {
>        cpumask_clear(retmask);
>        cpumask_set_cpu(cpu, retmask);
> }
>
> ..whereas for flat it's..
>
> static void flat_vector_allocation_domain(int cpu, struct cpumask *retmask)
> {
>        ...
>        cpumask_clear(retmask);
>        cpumask_bits(retmask)[0] = APIC_ALL_CPUS;
> }
>
> So, while __assign_irq_vector() is called with a mask of all CPUs for
> both boxen, apic->vector_allocation_domain() ensures the mask finally
> written to cfg->domain has only one CPU bit set in physical flat mode,
> and all CPU bits set in flat mode.
>
>        -Mike
>
> P.S.  if you try that RTFM thing, strap a pillow to your forehead first
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: x86: interrupt routing question

[Mike Galbraith]

On Wed, 2011-09-28 at 09:26 -0600, Bjorn Helgaas wrote:
> On Wed, Sep 28, 2011 at 1:28 AM, Mike Galbraith <efault@gmx.de> wrote:
> >
> > For the next sod who gets curious about irq routing behavior difference,
> > and tries to ask google.
> 
> Thanks for following up on this!
> 
> So, having read the fine manual, do you think this is a Linux bug, or
> something that could be improved in Linux, either by changing the
> behavior or making the dmesg more informative?  It certainly cost you
> a lot of time, and it'd be nice if we could save the next person :)

It didn't take long at all after I stopped trying to ask google and
acpidump, and went to the source.  I don't think it's a bug, but I only
_started_ to RTFM, so whether it could be improved is a question I can't
begin to answer.. and kinda hope to never be able to answer :)

	-Mike