Re: 2.6.19-rc1 genirq causes either boot hang or "do_IRQ: cannot handle IRQ -1"

[ebiederm@xmission.com (Eric W. Biederman)]

Linus Torvalds <torvalds@osdl.org> writes:

> On Sat, 7 Oct 2006, Arjan van de Ven wrote:
>> 
>> it seems the right mix at this time is to have the software select the
>> package, and the hardware pick the core within the package. 
>
> I think that sounds like a fairly good approach.
>
> Software obviously can make the "rough" selections, it's the fine-grained 
> ones that are harder (and might need to be done at a frequency that just 
> makes it impractical).
>
> So yes, having software say "We want to steer this particular interrupt to 
> this L3 cache domain" sounds eminently sane.
>
> Having software specify which L1 cache domain it wants to pollute is 
> likely just crazy micro-management.

The current interrupt delivery abstraction in the kernel is a
set of cpus an interrupt can be delivered to.  Which seem sufficient
to the cause of aiming at a cache domain.  Frequently the lower
levels of interrupt delivery map this to a single cpu because of
hardware limitations but in certain cases we can honor a multiple cpu
request.

I believe the scheduler has knowledge about different locality domains
for NUMA and everything else.  So what is wanting on our side is some
architecture? work to do the broad steering by default.

Our current policies on x86_64 are much less enlightened by default.
If we have < 8 cpus and CONFIG_CPU_HOTPLUG is not defined we let
the hardware pick the cpu.  Otherwise we send the interrupt to the
first cpu in the set.  Which means the first cpu.  Beyond that
everything is left up to the user space irqbalanced.

My patches were about keeping us from artificially merging multiple
irq sources into the same linux irq when we ran short of vectors
so we have a chance to aim and observe all irq sources as individuals.

Now it is possible to do all of this fine policy work that has been
discussed in this thread.  But since I don't see that problem yet I'm
probably not the man for that job.

The truly challenging corollary to my work and this discussion is
handling the up coming network adapters that can start demuxing large
network pipes with a different irq for each cache domain in the
system, but the details of how distinct irq sources you need from the
hardware are left up to the software to decide at run time.

Eric