Re: [RFC]block: add a new flag to make request complete on submitted cpu

[Shaohua Li <shaohua.li@intel.com>]

On Mon, Dec 21, 2009 at 08:12:28PM +0800, Jens Axboe wrote:
> On Mon, Dec 21 2009, Jens Axboe wrote:
> > On Mon, Dec 21 2009, Shaohua Li wrote:
> > > We already have a QUEUE_FLAG_SAME_COMP, which makes request complete
> > > on the first cpu of a mc/ht, but this isn't sufficient. In a system
> > > with fast block devices (intel SSD), it turns out the first cpu is
> > > bottlenect. Add a flag to make request complete on cpu where request
> > > is submitted. The flag implies QUEUE_FLAG_SAME_COMP. By default, it is off.
> > 
> > It was a lazy trick to avoid doing any round robin work in there.
> > 
> > > My test machine has two CPUs and 4 intel SSD. Without the new flag,
> > > the io thoughput is about 400m/s; with it, the thoughput is about 500m/s.
> > 
> > So I think we should just fix it, I still think the group logic makes
> > sense. But instead of always going for the first one, let it complete
> > locally if part of the group, if not send to specific submitter CPU.
> 
> Is this enough? It renames cpu to local_cpu and ccpu to target_cpu to
> make things clearer to read, the real change is that we allow local
> completion if the cpu matches OR the group matches.
No. Interrupt is fired on one CPU, so the change is a nop. All requests
are handled by the CPU directed the interrupt.
AHCI supports multiple MSI but not support per-vector mask, which makes
it's hard to add multiple MSI support.

Thanks,
Shaohua