A note on APIC bus latency

A note on APIC bus latency

[Jonathan Lundell]

We recently ran into some issues caused by APIC bus latency. I was 
reminded of that by the recent discussion of NAPI and related 
interrupt-performance matters.

Intel processors that predate Pentium 4 but use an APIC transmit APIC 
messages over a serial APIC bus, typically at 16.7 MHz. (Pentium 4 
uses the system bus for APIC messages.)

A message exchange (IO-APIC sends an interrupt message; CPU sends 
back an EOI message) requires from 35 to 48 APIC bus clocks, or 2-3 
microseconds. That gets to be pretty significant compared to packet 
times, especially at Gbit speeds, but even at 100 MHz, and is the 
time required to burst a thousand bytes or more at faster PCI rates.

It's also likely to be significant for inter-processor interrupts, 
though I don't know what the implications are here.
-- 
/Jonathan Lundell.

Re: A note on APIC bus latency

[Alan Cox]

> A message exchange (IO-APIC sends an interrupt message; CPU sends 
> back an EOI message) requires from 35 to 48 APIC bus clocks, or 2-3 
> microseconds. That gets to be pretty significant compared to packet 
> times, especially at Gbit speeds, but even at 100 MHz, and is the 
> time required to burst a thousand bytes or more at faster PCI rates.
> 
> It's also likely to be significant for inter-processor interrupts, 
> though I don't know what the implications are here.

The big implication so far has been some extremely horrible to debug 
irq handling bugs where drivers such as the i810 audio assumed that the
disable of an irq on the pci device was immediate once then pci write
and a pci read to force posting completed. 

There are impacts on things like TLB shootdowns where the latency impacts
an SMP crosscall. I'm not sure how bad the impact is on the bigger numa
boxes as I notice Martin uses multiple sends for that

Alan