From: Jens Axboe <axboe@kernel.dk>
To: Linus Walleij <linus.walleij@linaro.org>
Cc: Ulf Hansson <ulf.hansson@linaro.org>,
Paolo Valente <paolo.valente@linaro.org>,
Christoph Hellwig <hch@infradead.org>,
Arnd Bergmann <arnd@arndb.de>,
Bart Van Assche <bart.vanassche@sandisk.com>,
Jan Kara <jack@suse.cz>, Tejun Heo <tj@kernel.org>,
linux-block@vger.kernel.org,
Linux-Kernal <linux-kernel@vger.kernel.org>,
Mark Brown <broonie@kernel.org>, Hannes Reinecke <hare@suse.de>,
Grant Likely <grant.likely@secretlab.ca>,
James Bottomley <James.Bottomley@hansenpartnership.com>,
Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Subject: Re: [PATCH 00/14] introduce the BFQ-v0 I/O scheduler as an extra scheduler
Date: Fri, 28 Oct 2016 08:22:06 -0600 [thread overview]
Message-ID: <adfd57be-1e12-e900-6a78-3e9607017a10@kernel.dk> (raw)
In-Reply-To: <CACRpkda0fmPdM2WZ_gtW=T7xvMfP_4ErtNZseExN2AggzZbcDQ@mail.gmail.com>
On 10/28/2016 03:32 AM, Linus Walleij wrote:
> On Fri, Oct 28, 2016 at 12:27 AM, Linus Walleij
> <linus.walleij@linaro.org> wrote:
>> On Thu, Oct 27, 2016 at 11:08 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>
>>> blk-mq has evolved to support a variety of devices, there's nothing
>>> special about mmc that can't work well within that framework.
>>
>> There is. Read mmc_queue_thread() in drivers/mmc/card/queue.c
>
> So I'm not just complaining by the way, I'm trying to fix this. Also
> Bartlomiej from Samsung has done some stabs at switching MMC/SD
> to blk-mq. I just rebased my latest stab at a naïve switch to blk-mq
> to v4.9-rc2 with these results.
>
> The patch to enable MQ looks like this:
> https://git.kernel.org/cgit/linux/kernel/git/linusw/linux-stericsson.git/commit/?h=mmc-mq&id=8f79b527e2e854071d8da019451da68d4753f71d
>
> I run these tests directly after boot with cold caches. The results
> are consistent: I ran the same commands 10 times in a row.
>
>
> BEFORE switching to BLK-MQ (clean v4.9-rc2):
>
> time dd if=/dev/mmcblk0 of=/dev/null bs=1M count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.0GB) copied, 47.781464 seconds, 21.4MB/s
> real 0m 47.79s
> user 0m 0.02s
> sys 0m 9.35s
>
> mount /dev/mmcblk0p1 /mnt/
> cd /mnt/
> time find . > /dev/null
> real 0m 3.60s
> user 0m 0.25s
> sys 0m 1.58s
>
> mount /dev/mmcblk0p1 /mnt/
> iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
> (kBytes/second)
> random random
> kB reclen write rewrite read reread read write
> 20480 4 2112 2157 6052 6060 6025 40
> 20480 8 4820 5074 9163 9121 9125 81
> 20480 16 5755 5242 12317 12320 12280 165
> 20480 32 6176 6261 14981 14987 14962 336
> 20480 64 6547 5875 16826 16828 16810 692
> 20480 128 6762 6828 17899 17896 17896 1408
> 20480 256 6802 6871 16960 17513 18373 3048
> 20480 512 7220 7252 18675 18746 18741 7228
> 20480 1024 7222 7304 18436 17858 18246 7322
> 20480 2048 7316 7398 18744 18751 18526 7419
> 20480 4096 7520 7636 20774 20995 20703 7609
> 20480 8192 7519 7704 21850 21489 21467 7663
> 20480 16384 7395 7782 22399 22210 22215 7781
>
>
> AFTER switching to BLK-MQ:
>
> time dd if=/dev/mmcblk0 of=/dev/null bs=1M count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.0GB) copied, 60.551117 seconds, 16.9MB/s
> real 1m 0.56s
> user 0m 0.02s
> sys 0m 9.81s
>
> mount /dev/mmcblk0p1 /mnt/
> cd /mnt/
> time find . > /dev/null
> real 0m 4.42s
> user 0m 0.24s
> sys 0m 1.81s
>
> mount /dev/mmcblk0p1 /mnt/
> iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
> (kBytes/second)
> random random
> kB reclen write rewrite read reread read write
> 20480 4 2086 2201 6024 6036 6006 40
> 20480 8 4812 5036 8014 9121 9090 82
> 20480 16 5432 5633 12267 9776 12212 168
> 20480 32 6180 6233 14870 14891 14852 340
> 20480 64 6382 5454 16744 16771 16746 702
> 20480 128 6761 6776 17816 17846 17836 1394
> 20480 256 6828 6842 17789 17895 17094 3084
> 20480 512 7158 7222 17957 17681 17698 7232
> 20480 1024 7215 7274 18642 17679 18031 7300
> 20480 2048 7229 7269 17943 18642 17732 7358
> 20480 4096 7212 7360 18272 18157 18889 7371
> 20480 8192 7008 7271 18632 18707 18225 7282
> 20480 16384 6889 7211 18243 18429 18018 7246
>
>
> A simple dd readtest of 1 GB is always consistently 10+
> seconds slower with MQ. find in the rootfs is a second slower.
> iozone results are consistently lower throughput or the same.
>
> This is without using Bartlomiej's clever hack to pretend we have
> 2 elements in the HW queue though. His early tests indicate that
> it doesn't help much: the performance regression we see is due to
> lack of block scheduling.
A simple dd test, I don't see how that can be slower due to lack of
scheduling. There's nothing to schedule there, just issue them in order?
So that would probably be where I would start looking. A blktrace of the
in-kernel code and the blk-mq enabled code would perhaps be
enlightening. I don't think it's worth looking at the more complex test
cases until the dd test case is at least as fast as the non-mq version.
Was that with CFQ, btw, or what scheduler did it run?
It'd be nice to NOT have to rely on that fake QD=2 setup, since it will
mess with the IO scheduling as well.
> I try to find a way forward with this, and also massage the MMC/SD
> code to be more MQ friendly to begin with (like only pick requests
> when we get a request notification and stop pulling NULL requests
> off the queue) but it's really a messy piece of code.
Yeah, it does look pretty messy... I'd be happy to help out with that,
and particularly in figuring out why the direct conversion is slower for
a basic 'dd' test case.
--
Jens Axboe
next prev parent reply other threads:[~2016-10-28 14:22 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-26 9:27 Paolo Valente
2016-10-26 9:27 ` [PATCH 01/14] block, bfq: " Paolo Valente
2016-10-26 9:27 ` [PATCH 02/14] block, bfq: add full hierarchical scheduling and cgroups support Paolo Valente
2016-10-26 9:27 ` [PATCH 03/14] block, bfq: improve throughput boosting Paolo Valente
2016-10-26 9:27 ` [PATCH 04/14] block, bfq: modify the peak-rate estimator Paolo Valente
2016-10-26 9:27 ` [PATCH 05/14] block, bfq: add more fairness with writes and slow processes Paolo Valente
2016-10-26 9:27 ` [PATCH 06/14] block, bfq: improve responsiveness Paolo Valente
2016-10-26 9:28 ` [PATCH 07/14] block, bfq: reduce I/O latency for soft real-time applications Paolo Valente
2016-10-26 9:28 ` [PATCH 08/14] block, bfq: preserve a low latency also with NCQ-capable drives Paolo Valente
2016-10-26 9:28 ` [PATCH 09/14] block, bfq: reduce latency during request-pool saturation Paolo Valente
2016-10-26 10:19 ` [PATCH 00/14] introduce the BFQ-v0 I/O scheduler as an extra scheduler Christoph Hellwig
2016-10-26 11:34 ` Jan Kara
2016-10-26 15:05 ` Bart Van Assche
2016-10-26 15:13 ` Arnd Bergmann
2016-10-26 15:29 ` Christoph Hellwig
2016-10-26 15:32 ` Jens Axboe
2016-10-26 16:04 ` Paolo Valente
2016-10-26 16:12 ` Jens Axboe
2016-10-27 9:26 ` Jan Kara
2016-10-27 14:34 ` Grozdan
2016-10-27 15:55 ` Heinz Diehl
2016-10-27 16:28 ` Jens Axboe
2016-10-27 16:26 ` Jens Axboe
2016-10-28 7:59 ` Jan Kara
2016-10-28 14:10 ` Jens Axboe
2016-10-27 17:32 ` Ulf Hansson
2016-10-27 17:43 ` Jens Axboe
2016-10-27 18:13 ` Ulf Hansson
2016-10-27 18:21 ` Jens Axboe
2016-10-27 19:34 ` Ulf Hansson
2016-10-27 21:08 ` Jens Axboe
2016-10-27 22:27 ` Linus Walleij
2016-10-28 9:32 ` Linus Walleij
2016-10-28 14:22 ` Jens Axboe [this message]
2016-10-28 20:38 ` Linus Walleij
2016-10-28 15:29 ` Christoph Hellwig
2016-10-28 21:09 ` Linus Walleij
2016-10-28 15:30 ` Jens Axboe
2016-10-28 15:58 ` Bartlomiej Zolnierkiewicz
2016-10-28 16:05 ` Arnd Bergmann
2016-10-28 17:17 ` Mark Brown
2016-10-28 14:07 ` Jens Axboe
2016-10-28 6:36 ` Ulf Hansson
2016-10-28 14:17 ` Jens Axboe
2016-10-28 17:12 ` Mark Brown
2016-10-27 19:41 ` Mark Brown
2016-10-27 19:45 ` Christoph Hellwig
2016-10-27 22:01 ` Mark Brown
2016-10-28 12:07 ` Arnd Bergmann
2016-10-28 12:17 ` Richard Weinberger
2016-10-29 5:38 ` Paolo Valente
2016-10-29 13:12 ` Bart Van Assche
2016-10-29 14:12 ` Jens Axboe
2016-10-30 3:06 ` Paolo Valente
2016-10-26 12:37 ` Paolo Valente
2016-10-29 17:08 Manuel Krause
2016-10-30 17:48 Manuel Krause
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=adfd57be-1e12-e900-6a78-3e9607017a10@kernel.dk \
--to=axboe@kernel.dk \
--cc=James.Bottomley@hansenpartnership.com \
--cc=arnd@arndb.de \
--cc=b.zolnierkie@samsung.com \
--cc=bart.vanassche@sandisk.com \
--cc=broonie@kernel.org \
--cc=grant.likely@secretlab.ca \
--cc=hare@suse.de \
--cc=hch@infradead.org \
--cc=jack@suse.cz \
--cc=linus.walleij@linaro.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=paolo.valente@linaro.org \
--cc=tj@kernel.org \
--cc=ulf.hansson@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome