Re: [PATCH 2/2] x86: implement multiple queues for smp function call IPIs

[Ingo Molnar <mingo@elte.hu>]

* Jeremy Fitzhardinge <jeremy@goop.org> wrote:

>> So this _has_ to be approached defensively. It _should_ work, and i'm 
>> all in favor of utilizing hardware resources more fully, but it's an 
>> entirely new mode of operation for the hardware. I think a Kconfig 
>> option (which defaults to off), and a boot option to disable it would 
>> be nice, so that we can introduce this gently, at least initially. 
>> Then when we see that it's 100% trouble-free we can flip around the 
>> default.
>
> As Andi pointed out, this is more or less functionally identical to 
> the code I ripped out of tlb_64.c, so this mode of operation has had 
> lots of exposure on the 64-bit side.  Because the number of queues is 
> a CONFIG variable, it would be relatively easy to make it a real 
> config option, and/or use different numbers for 32 and 64-bit.  
> Choosing 1 as the number of queues will make it behave exactly as the 
> current code does.
>
> I'm not really familiar with all the ins and outs of apic bugs.  
> What's the issue you're concerned about?

Yes on the 64-bit side we've had NUM_INVALIDATE_TLB_VECTORS (==8) for a 
long time, but note that 64-bit is obviously for more modern CPUs. What 
i'm mindful about (i'm not _that_ worried) are fragile APICs and unknown 
erratas.

The other issue is that the concurrency pattern changes somewhat and 
becomes more agressive. The existing 64-bit special-purpose TLB flush 
code uses in essence synchronous waiting for that specific IPI that 
belongs to that CPU, it sends the IPI then waits for the acknowledgement 
by polling the flush mask:

        send_IPI_mask(cpumask, INVALIDATE_TLB_VECTOR_START + sender);

        while (!cpus_empty(f->flush_cpumask))
                cpu_relax();

while with generic IPIs we could have asynchronous IPIs as well.

So with the TLB flush code there's never any true "multiple outstanding 
IPIs" scenario for the same IPI, for 8 CPUs and fewer. (which is the 
predominant majority of existing hardware)

> It also occurred to me that it might be more interesting to 
> parameterise the queues - and the mapping of cpus->queues - in a more 
> topology-aware way than simply NQUEUES=NCPUS/x, queue=cpuid % NQUEUES.  
> But I haven't given it much thought.

i'd suggest we keep it at the current simple modulo rule.

	Ingo