Re: [PATCH 2/2] blk-plug: don't flush nested plug lists

[Jeff Moyer <jmoyer@redhat.com>]

Jens Axboe <axboe@kernel.dk> writes:

> And agree with Ming, this can be cleaned up substantially. I'd also
> like to see some test results from the other end of the spectrum. Your
> posted cased is clearly based case (we missed tons of merging, now we
> don't), I'd like to see a normal case and a worst case result as well
> so we have an idea of what this would do to latencies.

I re-ran my tests (just sequential reads) to verify where the
performance benefit comes from.  Here are the results:

device|  vanilla   |    patch1      |   dio-noplug  |  noflush-nested
------+------------+----------------+---------------+-----------------
rssda | 701,684    |    1,168,527   |   1,342,177   |   1,297,612
      |     100%   |         +66%   |        +91%   |        +85%
vdb0  | 358,264    |      902,913   |     906,850   |     922,327
      |     100%   |        +152%   |       +153%   |       +157%

Patch1 refers to the first patch in this series, which fixes the merge
logic for single-queue blk-mq devices.  Each column after that includes
that first patch.  In dio-noplug, I removed the blk_plug from the
direct-io code path (so there is no nesting at all).  This is a control,
since it is what I expect the outcome of the noflush-nested column to
actually be.  Then, the noflush-nested column leaves the blk_plug in
place in the dio code, but includes the patch that prevents nested
blk_plug's from being flushed.  All numbers are the average of 5 runs.
With the exception of the vanilla run on rssda (the first run was
faster, causing the average to go up), the standard deviation is very
small.

As you can see, for the micron device (rssda) we get quite a bit of an
improvement from just fixing the plugging, but we could do quite a bit
better.  For the virtio-blk device, the follow-on patches don't
contribute very much to the overall performance.

So, I'm happy to push in just patch 1 at this time.  It is a bugfix,
after all, and could even be marked for stable.  Does that sound
reasonable?

Later, if time permits, I can look into refining patch 2, though I don't
have any firm plans to do that right now.

Thanks,
Jeff