Re: [patch] x86_64, irq: check remote IRR bit before migrating level triggered irq

[ebiederm@xmission.com (Eric W. Biederman)]

"Siddha, Suresh B" <suresh.b.siddha@intel.com> writes:

> on x86_64 kernel, level triggered irq migration gets initiated in the context
> of that interrupt(after executing the irq handler) and following steps are
> followed to do the irq migration.
>
> 1. mask IOAPIC RTE entry;     // write to IOAPIC RTE
> 2. EOI;                       // processor EOI write
> 3. reprogram IOAPIC RTE entry // write to IOAPIC RTE with new destination and
>                               // and interrupt vector due to per cpu vector
>                               // allocation.
> 4. unmask IOAPIC RTE entry;   // write to IOAPIC RTE
>
> Because of the per cpu vector allocation in x86_64 kernels, when the irq
> migrates to a different cpu, new vector(corresponding to the new cpu) will
> get allocated.
>
> An EOI write to local APIC has a side effect of generating an EOI write 
> for level trigger interrupts (normally this is a broadcast to all IOAPICs). 
> The EOI broadcast generated as a side effect of EOI write to processor may 
> be delayed while the other IOAPIC writes (step 3 and 4) can go through.
>
> Normally, the EOI generated by local APIC for level trigger interrupt
> contains vector number. The IOAPIC will take this vector number and
> search the IOAPIC RTE entries for an entry with matching vector number and
> clear the remote IRR bit (indicate EOI). However, if the vector number is
> changed (as in step 3) the IOAPIC will not find the RTE entry when the EOI
> is received later. This will cause the remote IRR to get stuck causing the
> interrupt hang (no more interrupt from this RTE).
>
> Current x86_64 kernel assumes that remote IRR bit is cleared by the time
> IOAPIC RTE is reprogrammed. Fix this assumption by checking for remote IRR
> bit and if it still set, delay the irq migration to the next interrupt
> arrival event(hopefully, next time remote IRR bit will get cleared
> before the IOAPIC RTE is reprogrammed).
>
> Initial analysis and patch from Nanhai.

So why does any of this matter? 

My memory says that the ioapic state for sending irqs gets reset when we
unmask the irq.

If not I expect we can use the mask and edge, unmask and level
work-around from arch/i386/ioapic.c.  That looks like it would
be less code, less error prone and easier to implement then the
current work around.

Eric