Re: [RFC] [PATCH] 2.5.35 patch for making DIO async--performance numbers

Re: [RFC] [PATCH] 2.5.35 patch for making DIO async--performance numbers

[Mingming Cao]

Hi Ben & Andrew,

I run sync raw I/O tests and Narasimha's async I/O tests on a 8 way PIII to
measure the I/O performance before/after the dio async patch.  All the
tests (sync and async) did 4000 * 256K I/O on 40 disks.

Basically, sync RAW read/write performance has no affect with the dio async
patch. Async I/O seems to be slower than the sync I/O.  Async RAW I/O got
better performance when the queue length for io_submit() is set to be 4.

I measured the time per test.  vmstat infos are also listed below.

sync RAW read w/o async dio patch:
-----------------------------------------------
- time
real  1m54.165s
user  0m0.153s
sys   0m24.369s

- vmstat
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy
id
 0 39  0      0 3795352      0  37752   0   0 384371     0 3980  6361   0
5  95
 0 39  0      0 3795344      0  37756   0   0 384832     7 3974  6356   0
5  95
 0 39  0      0 3795340      0  37756   0   0 384090     0 3982  6349   0
5  95
 1 38  0      0 3795336      0  37756   0   0 384320     0 3971  6335   0
5  95

sync RAW write w/o async dio patch
-----------------------------------------------
- time

real  2m19.465s
user  0m0.138s
sys   0m19.837s

- vmstat
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy
id
 1 38  0      0 3805916      0  37516   0   0     0 302963 3383  4882   0
3  97
 1 38  0      0 3805904      0  37516   0   0     0 302413 3376  4880   0
3  97
 0 39  0      0 3805892      0  37516   0   0     0 302912 3373  4877   0
3  97
 0 39  0      0 3805920      0  37516   0   0     0 303512 3390  4886   0
3  97

sync RAW read w/ async dio patch
----------------------------------------------
- time
real  1m53.405s
user  0m0.152s
sys   0m23.402sp

- vmstat 5
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy
id
 0 39  0      0 3795576      0  37592   0   0 384115     0 3929  3476   0
5  95
 0 39  0      0 3795568      0  37592   0   0 383834     0 3931  3464   0
5  95
 0 39  0      0 3795564      0  37592   0   0 384474     0 3941  3485   0
5  95
 0 39  0      0 3795560      0  37592   0   0 384013     0 3931  3488   0
5  95

sync RAW write w/ async dio patch
----------------------------------------------
- time
real  2m19.295s
user  0m0.147s
sys   0m17.754s

- vmstat 5
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy
id
 0 39  0      0 3805224      0  37660   0   0     0 303168 3360  2492   0
4  96
 0 39  0      0 3805212      0  37664   0   0     0 301427 3348  2486   0
4  96
 0 39  0      0 3805208      0  37664   0   0     0 304435 3371  2490   0
3  97
 0 39  0      0 3805204      0  37664   0   0     0 303072 3360  2476   0
3  97

async RAW read (queue lenth =1)
----------------------
- time
real  2m8.860s
user  0m0.472s
sys   0m24.020s

- vmstat 5
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy
id
 1  0  0      0 3779912      0  39660   0   0 347520     0 3663  3057   0
5  95
 0  0  0      0 3779664      0  39660   0   0 346854     0 3664  3062   0
5  95
 1  0  0      0 3779400      0  39660   0   0 348045     0 3663  3046   0
5  95
 0  0  0      0 3779124      0  39660   0   0 346278     0 3662  3038   0
5  95

asyn RAW write (queue lenth =1)
---------------------
- time

real  2m17.140s
user  0m0.410s
sys   0m22.317s

- vmstat 5
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy
id
 1  0  0      0 3789096      0  39692   0   0     0 304704 3371  2599   0
4  96
 1  0  0      0 3788884      0  39692   0   0     0 306918 3390  2602   0
4  96
 0  0  0      0 3788644      0  39692   0   0     0 304662 3376  2584   0
4  96
 0  0  0      0 3788408      0  39692   0   0     0 304334 3371  2573   0
4  96

async RAW read (queue lenth =4)
----------------------
- time

real  2m6.453s
user  0m0.344s
sys   0m23.802s

- vmstat 5
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy
id
 0  0  0      0 3715372      0  39988   0   0 354528     0 3690  3200   0
5  95
 0  0  0      0 3715288      0  39988   0   0 352032     0 3692  3198   0
5  95
 0  0  0      0 3715188      0  39988   0   0 356096     0 3670  3184   0
5  95
 0  0  0      0 3715104      0  39988   0   0 350816     0 3664  3165   0
5  95

async RAW write (queue lenth =4)
----------------------
- time

real  2m15.452s
user  0m0.318s
sys   0m21.715s

- vmstat 5
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy
id
 0  0  1      0 3754736      0  40048   0   0     0 306547 3367  2747   0
4  96
 0  0  0      0 3754564      0  40048   0   0     0 306970 3360  2758   0
4  96
 3  0  0      0 3754408      0  40048   0   0     0 307994 3365  2765   0
4  96
 0  0  0      0 3754100      0  40048   0   0     0 307162 3367  2763   0
4  96


Mingming Cao

Re: [RFC] [PATCH] 2.5.35 patch for making DIO async--performance numbers

[Andrew Morton]

Mingming Cao wrote:
> 
> Hi Ben & Andrew,
> 
> I run sync raw I/O tests and Narasimha's async I/O tests on a 8 way PIII to
> measure the I/O performance before/after the dio async patch.  All the
> tests (sync and async) did 4000 * 256K I/O on 40 disks.
> 
> Basically, sync RAW read/write performance has no affect with the dio async
> patch. Async I/O seems to be slower than the sync I/O.  Async RAW I/O got
> better performance when the queue length for io_submit() is set to be 4.
> 
> I measured the time per test.  vmstat infos are also listed below.
> 

Thanks.  Note that the old code (which seems to be a tiny bit faster,
and used less CPU as well) has a significantly higher context switch
rate.  At a guess I'd say that it is more efficient at getting userspace
up and running in response to IO completion.

I'd say it's only likely to affect these huge linear IOs.  Once you get
into real workloads which are seeking and merging then a bit of latency
here or there would just be soaked up by other system activity.

Ah.  The current direct-io.c uses wake_up_process(), not waitqueues.
So the aio version has to wear the waitqueue cost.  If you're using the
-mm patch I'd suggest that you convert aio.c to prepare_to_wait/finish_wait.
The waitqueue/wakeup costs on your 8-ways seem to be very high.

Re: [RFC] [PATCH] 2.5.35 patch for making DIO async--performance numbers

[Badari Pulavarty]

Andrew,

> 
> Thanks.  Note that the old code (which seems to be a tiny bit faster,
> and used less CPU as well) has a significantly higher context switch
> rate.  At a guess I'd say that it is more efficient at getting userspace
> up and running in response to IO completion.
> 

I my patch, I removed bio_list. So, I do all the processing of "bio"
in end_io() function, instead of postpone it to waiter. Do you think
this matters ? 


> I'd say it's only likely to affect these huge linear IOs.  Once you get
> into real workloads which are seeking and merging then a bit of latency
> here or there would just be soaked up by other system activity.
> 
> Ah.  The current direct-io.c uses wake_up_process(), not waitqueues.
> So the aio version has to wear the waitqueue cost.  If you're using the
> -mm patch I'd suggest that you convert aio.c to prepare_to_wait/finish_wait.
> The waitqueue/wakeup costs on your 8-ways seem to be very high.

Ok !! I still use wake_up_process() for the sync case.
I will try to use waitqueues and see.

Thanks,
Badari

Re: [RFC] [PATCH] 2.5.35 patch for making DIO async--performance numbers

[Andrew Morton]

Badari Pulavarty wrote:
> 
> Andrew,
> 
> >
> > Thanks.  Note that the old code (which seems to be a tiny bit faster,
> > and used less CPU as well) has a significantly higher context switch
> > rate.  At a guess I'd say that it is more efficient at getting userspace
> > up and running in response to IO completion.
> >
> 
> I my patch, I removed bio_list. So, I do all the processing of "bio"
> in end_io() function, instead of postpone it to waiter. Do you think
> this matters ?

Ah.  Yes, it matters.

Running the completion in process context is nicer from an interrupt latency
point of view.  But the completion code also runs set_page_dirty(), which
takes locks which are not interrupt-safe.  Running set_page_dirty() from
interrupt context can deadlock.

So if it's convenient, yes, let's do the completion in process context.
If not convenient then we'll need to find some way of running
set_page_dirty() outside the interrupt handler.

The set_page_dirty() is there to cover the case of direct-io into a
mmapped region of another file.  We need to tell the VM that the page
has been changed, because the CPU's ptes don't know that.  And we do
have to run set_page_dirty() after the read IO has completed.

The other thing we've lost is the BIO-pruning and recycling effect: the
current direct-io code will reap BIOs while it is actually submitting
them, so the peak consumption is kept under control.  Plus there are
cache-warmness issues.  But without having a process there to do all this,
we obviously have to lose some things.

Maybe, that's not very important in real life.

> > I'd say it's only likely to affect these huge linear IOs.  Once you get
> > into real workloads which are seeking and merging then a bit of latency
> > here or there would just be soaked up by other system activity.
> >
> > Ah.  The current direct-io.c uses wake_up_process(), not waitqueues.
> > So the aio version has to wear the waitqueue cost.  If you're using the
> > -mm patch I'd suggest that you convert aio.c to prepare_to_wait/finish_wait.
> > The waitqueue/wakeup costs on your 8-ways seem to be very high.
> 
> Ok !! I still use wake_up_process() for the sync case.
> I will try to use waitqueues and see.
> 

Well, you're using waitqueues now - please try the fancy new ones.
As ever, profiles will tell us what's going on.

Re: [RFC] [PATCH] 2.5.35 patch for making DIO async--performance numbers

[Badari Pulavarty]

> 
> Badari Pulavarty wrote:
> > 
> > Andrew,
> > 
> > >
> > > Thanks.  Note that the old code (which seems to be a tiny bit faster,
> > > and used less CPU as well) has a significantly higher context switch
> > > rate.  At a guess I'd say that it is more efficient at getting userspace
> > > up and running in response to IO completion.
> > >
> > 
> > I my patch, I removed bio_list. So, I do all the processing of "bio"
> > in end_io() function, instead of postpone it to waiter. Do you think
> > this matters ?
> 
> Ah.  Yes, it matters.
> 
> Running the completion in process context is nicer from an interrupt latency
> point of view.  But the completion code also runs set_page_dirty(), which
> takes locks which are not interrupt-safe.  Running set_page_dirty() from
> interrupt context can deadlock.
> 
> So if it's convenient, yes, let's do the completion in process context.
> If not convenient then we'll need to find some way of running
> set_page_dirty() outside the interrupt handler.
> 
> The set_page_dirty() is there to cover the case of direct-io into a
> mmapped region of another file.  We need to tell the VM that the page
> has been changed, because the CPU's ptes don't know that.  And we do
> have to run set_page_dirty() after the read IO has completed.

OK !! We have to find a way to do set_page_dirty() safely, I guess.

> 
> The other thing we've lost is the BIO-pruning and recycling effect: the
> current direct-io code will reap BIOs while it is actually submitting
> them, so the peak consumption is kept under control.  Plus there are
> cache-warmness issues.  But without having a process there to do all this,
> we obviously have to lose some things.
> 

I don't follow you. In original code, we only reap the BIO's on which IO 
is complete. How is it controlling peak consumption ? 

Now, I give back the BIO as and when IO is complete. So it should be better
for consumption. Isn't it ?

- Badari

Re: [RFC] [PATCH] 2.5.35 patch for making DIO async--performance numbers

[Andrew Morton]

Badari Pulavarty wrote:
> 
> ...
> > The other thing we've lost is the BIO-pruning and recycling effect: the
> > current direct-io code will reap BIOs while it is actually submitting
> > them, so the peak consumption is kept under control.  Plus there are
> > cache-warmness issues.  But without having a process there to do all this,
> > we obviously have to lose some things.
> >
> 
> I don't follow you. In original code, we only reap the BIO's on which IO
> is complete. How is it controlling peak consumption ?

Ah.  The current direct-io code will, while submitting those BIOs,
occasionally go and see if any have completed already.  If they have
then they are processed and returned to the global BIO pool.

But you're returning them from within end_io, so no prob.