RE: per-cpu blk_plug_list

["Chen, Kenneth W" <kenneth.w.chen@intel.com>]

Andrew Morton wrote:
>>
>> We were surprised to find global lock in I/O path still exists on 2.6
>> kernel.
>
> Well I'm surprised that nobody has complained about it yet ;)

We are now, very loudly :-)


> How on earth do you know that when direct-io calls
blk_run_queues_cpu(),
> it is still running on the cpu which initially plugged the queue(s)?
>
> And your patch might have made the much-more-frequently-called
> blk_run_queues() more expensive.
>
> There are minor issues with lack of preemption safety and not using
the
> percpu data infrastructure, but they can wait.

Absolutely, these are the details need to be worked out.  And we are
working on that at the moment.


> You haven't told us how many disks are in use?  At 100k IO's per
second it
> would be 500 to 1000 disks, yes?

Right again!!, we are using 16 fiber-channel controllers connect to just
over 1000 disks.

> I assume you tried it, but does this help?
> 
> diff -puN drivers/block/ll_rw_blk.c~a drivers/block/ll_rw_blk.c
> --- 25/drivers/block/ll_rw_blk.c~a	Mon Mar  1 13:52:37 2004
> +++ 25-akpm/drivers/block/ll_rw_blk.c	Mon Mar  1 13:52:45 2004
> @@ -1251,6 +1251,9 @@ void blk_run_queues(void)
> {
> 	LIST_HEAD(local_plug_list);
>  
> +	if (list_empty(&blk_plug_list))
> +		return;
> +
> 	spin_lock_irq(&blk_plug_lock);
> 
> 	/*

Yeah, that's first thing we tried, didn't do a dent to the lock at all.