Re: Slow file transfer speeds with CFQ IO scheduler in some cases

[Jeff Layton <jlayton@redhat.com>]

On Wed, 12 Nov 2008 13:20:35 +0100
Jens Axboe <jens.axboe@oracle.com> wrote:

> On Tue, Nov 11 2008, Jeff Layton wrote:
> > On Tue, 11 Nov 2008 16:41:04 -0500
> > Jeff Layton <jlayton@redhat.com> wrote:
> > 
> > > On Tue, 11 Nov 2008 14:36:07 -0500
> > > Jeff Moyer <jmoyer@redhat.com> wrote:
> > > 
> > > > Jens Axboe <jens.axboe@oracle.com> writes:
> > > > 
> > > > > OK, that looks better. Can I talk you into just trying this little
> > > > > patch, just to see what kind of performance that yields? Remove the cfq
> > > > > patch first. I would have patched nfsd only, but this is just a quick'n
> > > > > dirty.
> > > > 
> > > > I went ahead and gave it a shot.  The updated CFQ patch with no I/O
> > > > context sharing does about 40MB/s reading a 1GB file.  Backing that
> > > > patch out, and then adding the patch to share io_context's between
> > > > kthreads yields 45MB/s.
> > > > 
> > > 
> > > Here's a quick and dirty patch to make all of the nfsd's have the same
> > > io_context. Comments appreciated -- I'm not that familiar with the IO
> > > scheduling code. If this looks good, I'll clean it up, add some
> > > comments and formally send it to Bruce.
> > > 
> > 
> > No sooner than I send it out than I find a bug. We need to eventually
> > put the io_context reference we get. This should be more correct:
> 
> That sort of thing happens a lot, I can definitely sympathize with you
> there :-)
> 
> > ----------------[snip]-------------------
> > 
> > From d0ee67045a12c677883f77791c6f260588c7b41f Mon Sep 17 00:00:00 2001
> > From: Jeff Layton <jlayton@redhat.com>
> > Date: Tue, 11 Nov 2008 16:54:16 -0500
> > Subject: [PATCH] knfsd: make all nfsd threads share an io_context
> > 
> > This apparently makes the I/O scheduler treat the threads as a group
> > which helps throughput when sequential I/O is multiplexed over several
> > nfsd's.
> 
> That's a lot more nifty than my stupid CLONE_IO flag addition. Both are
> only good for test purposes though.
> 
> It's a bit difficult to make this really mergeable. I don't know
> anything about how nfsd manages its thread pool, but something more
> appropriate would be an io context per client mount. That's still not
> perfect as you could easily have more than one process doing simultanous
> IO on the client side, but it's a lot better.
> 

Ahh Good point...I wasn't thinking about it the right way. I guess what
we really need is some way to tell that a series of I/O requests
originated from the same client thread. NFS isn't really conducive to
this...

We might be able to do something like that with NFSv4 though. Maybe an
io_context per state owner or something. That's probably not going to
be trivial to implement however...

-- 
Jeff Layton <jlayton@redhat.com>