[PATCH] blk-mq: run the hw queue synchronously for unplugged scheduler inserts

[PATCH] blk-mq: run the hw queue synchronously for unplugged scheduler inserts

[Qiliang Yuan]

When the submitter holds no plug, blk_mq_submit_bio() inserts a request
into the I/O scheduler and runs the hardware queue asynchronously. Now
that __submit_bio() no longer opens a plug around every bio, this path
is taken by all unplugged I/O to devices with an elevator, such as
io_uring submitting at most two SQEs per io_uring_enter() or synchronous
O_DIRECT reads and writes.

Each of these requests queues hctx->run_work on kblockd. kblockd is a
per-CPU WQ_HIGHPRI workqueue and blk_mq_hctx_next_cpu() returns
WORK_CPU_UNBOUND for single-queue devices, so the work runs on the
submitting CPU and the kworker preempts the submitter once per I/O.
With fio (io_uring, 4k randread, iodepth=16, one job) the submitter
context switches about once per I/O. A single-queue null_blk device
with mq-deadline drops from 289K to 193K IOPS, and a single-queue ublk
device, which defaults to mq-deadline, drops from 265K to 113K IOPS.

Run the hardware queue from the submitting context for scheduler
inserts, as flushing the per-bio plug did. Keep the asynchronous run for
the hctx->dispatch_busy case.

fio, io_uring, 4k randread, 16 vCPU KVM guest:

                                                  before     after
  null_blk, 1 queue, mq-deadline, qd16              193K      289K
  null_blk, 1 queue, none, qd16                     354K      356K
  null_blk, 4 queues, none, qd1                     356K      358K
  ublk (ublksrv null), 1 queue, mq-deadline, qd16   113K      265K
  ublk (ublksrv null), 4 x 1-queue devices, qd16    290K      852K

Fixes: 9cbbac29d752 ("block: Remove redundant plug in __submit_bio()")
Cc: stable@vger.kernel.org
Signed-off-by: Qiliang Yuan <odys.yuan@gmail.com>
---
 block/blk-mq.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index a26a11c73ee3e..54eff3cf45fae 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -3200,8 +3200,16 @@ void blk_mq_submit_bio(struct bio *bio)
 	}
 
 	hctx = rq->mq_hctx;
-	if ((rq->rq_flags & RQF_USE_SCHED) ||
-	    (hctx->dispatch_busy && (q->nr_hw_queues == 1 || !is_sync))) {
+	if (rq->rq_flags & RQF_USE_SCHED) {
+		/*
+		 * Run the queue from the submitting context, as flushing a plug
+		 * does. Punting every unplugged request to kblockd lets the
+		 * kworker preempt the submitter once per I/O.
+		 */
+		blk_mq_insert_request(rq, 0);
+		blk_mq_run_hw_queue(hctx, false);
+	} else if (hctx->dispatch_busy &&
+		   (q->nr_hw_queues == 1 || !is_sync)) {
 		blk_mq_insert_request(rq, 0);
 		blk_mq_run_hw_queue(hctx, true);
 	} else {

---
base-commit: 551c722f40809618230001baccf219193e22fc5a
change-id: 20260930-bug-blk-mq-noplug-sched-dispatch-4f19360b0bf7

Best regards,
-- 
Qiliang Yuan <odys.yuan@gmail.com>

Re: [PATCH] blk-mq: run the hw queue synchronously for unplugged scheduler inserts

[Qiliang Yuan]

Hi Jens, Ming,

Gentle ping on this one. It fixes a v7.2 regression from 9cbbac29d752
("block: Remove redundant plug in __submit_bio()"): unplugged I/O to a
device with an I/O scheduler is now run from kblockd, and the kworker
preempts the submitter once per I/O. A single-queue null_blk device with
mq-deadline drops from 289K to 193K IOPS, and a single-queue ublk device
from 265K to 113K.

Every single-queue device that defaults to mq-deadline is affected, so it
would be good to get this fixed in 7.3. Any comments are welcome.

Thanks,
Qiliang

Re: [PATCH] blk-mq: run the hw queue synchronously for unplugged scheduler inserts

[Ming Lei]

On Sat, Oct 10, 2026 at 10:35:47AM +0800, Qiliang Yuan wrote:
> Hi Jens, Ming,
> 
> Gentle ping on this one. It fixes a v7.2 regression from 9cbbac29d752
> ("block: Remove redundant plug in __submit_bio()"): unplugged I/O to a

It is hard to say it is one regression cause 9cbbac29d752 is doing correct
thing, and single request isn't supposed to dispatch via plug list, and it
only recovers to previous behavior before 060406c61c7c 
("block: add plug while submitting IO").

> device with an I/O scheduler is now run from kblockd, and the kworker
> preempts the submitter once per I/O. A single-queue null_blk device with
> mq-deadline drops from 289K to 193K IOPS, and a single-queue ublk device
> from 265K to 113K.

For both SQ null & ublk, you can switch to none for better performance.

> 
> Every single-queue device that defaults to mq-deadline is affected, so it
> would be good to get this fixed in 7.3. Any comments are welcome.

Not sure you can observe big difference for other real SQ device.


Thanks,
Ming

Re: [PATCH] blk-mq: run the hw queue synchronously for unplugged scheduler inserts

[Qiliang Yuan]

Hi Ming,

On Fri, Oct 09, 2026 at 10:05:59PM -0500, Ming Lei wrote:
> It is hard to say it is one regression cause 9cbbac29d752 is doing correct
> thing, and single request isn't supposed to dispatch via plug list, and it
> only recovers to previous behavior before 060406c61c7c
> ("block: add plug while submitting IO").

Agreed that 9cbbac29d752 itself is right, and this patch doesn't bring
the plug back. What differs from the time before 060406c61c7c is how
the async run gets scheduled: on v6.9 with PREEMPT_NONE the submitter
context switched 0.23 times per I/O, so one run of kblockd still
dispatched several requests, while on v7.3 with PREEMPT_LAZY the
kworker preempts the submitter once per I/O.

> Not sure you can observe big difference for other real SQ device.

Here is a virtio-blk disk with a single queue, which gets mq-deadline
by default, in a 16 vCPU KVM guest with the backing file in host
memory. fio, io_uring, 4k, median of 3 runs:

                        v7.3-rc5    patched
  randread, qd16           89.3K     162.8K
  randwrite, qd16          86.3K     156.7K
  randread, qd1            14.3K      14.9K
  psync randread           14.3K      14.9K
  ctx switches/IO, qd16     0.99       0.04

With none instead of mq-deadline the disk does 163K-177K random reads
at qd16 on both kernels, so the patch brings mq-deadline close to none
and leaves none alone.

> For both SQ null & ublk, you can switch to none for better performance.

Right, and ublk now defaults to none. But mq-deadline is still the
default for every other single-queue device, and few users change it.

Thanks,
Qiliang

Re: [PATCH] blk-mq: run the hw queue synchronously for unplugged scheduler inserts

[Kaitao Cheng]

在 2026/9/30 15:38, Qiliang Yuan 写道:
> When the submitter holds no plug, blk_mq_submit_bio() inserts a request
> into the I/O scheduler and runs the hardware queue asynchronously. Now
> that __submit_bio() no longer opens a plug around every bio, this path
> is taken by all unplugged I/O to devices with an elevator, such as
> io_uring submitting at most two SQEs per io_uring_enter() or synchronous
> O_DIRECT reads and writes.
> 
> Each of these requests queues hctx->run_work on kblockd. kblockd is a
> per-CPU WQ_HIGHPRI workqueue and blk_mq_hctx_next_cpu() returns
> WORK_CPU_UNBOUND for single-queue devices, so the work runs on the
> submitting CPU and the kworker preempts the submitter once per I/O.
> With fio (io_uring, 4k randread, iodepth=16, one job) the submitter
> context switches about once per I/O. A single-queue null_blk device
> with mq-deadline drops from 289K to 193K IOPS, and a single-queue ublk
> device, which defaults to mq-deadline, drops from 265K to 113K IOPS.
> 
> Run the hardware queue from the submitting context for scheduler
> inserts, as flushing the per-bio plug did. Keep the asynchronous run for
> the hctx->dispatch_busy case.
> 
> fio, io_uring, 4k randread, 16 vCPU KVM guest:
> 
>                                                   before     after
>   null_blk, 1 queue, mq-deadline, qd16              193K      289K
>   null_blk, 1 queue, none, qd16                     354K      356K
>   null_blk, 4 queues, none, qd1                     356K      358K
>   ublk (ublksrv null), 1 queue, mq-deadline, qd16   113K      265K
>   ublk (ublksrv null), 4 x 1-queue devices, qd16    290K      852K
> 
> Fixes: 9cbbac29d752 ("block: Remove redundant plug in __submit_bio()")

Commit 9cbbac29d752 does need to be reevaluated.

Our customer ran fio tests with --bs=4K --rw=randwrite --iodepth=1 --numjobs=1
and confirmed that adding bio plugging to __submit_bio() improves performance.

Without bio plugging in __submit_bio():
fio:
__submit_bio                              (17.16%)
  → blk_mq_submit_bio                     (16.80%)
    → blk_mq_run_hw_queue                 (9.97%)
      → blk_mq_delay_run_hw_queue         (9.41%)
        → kblockd_mod_delayed_work_on     (9.13%)
          → mod_delayed_work_on           (9.11%)

kworker:
process_one_work                          (12.48%)
  → blk_mq_run_work_fn                    (11.70%)
    → blk_mq_sched_dispatch_requests      (11.00%)
      → __blk_mq_sched_dispatch_requests  (10.91%)
        → __blk_mq_do_dispatch_sched      (10.26%)
          → blk_mq_dispatch_rq_list       (7.84%)
            → nvme_queue_rq               (5.85%)

With bio plugging in __submit_bio():
__submit_bio                                (26.94%)
  → __blk_flush_plug                        (21.36%)
    → blk_mq_flush_plug_list                (21.24%)
      → blk_mq_flush_plug_list.part.0       (18.38%)
        → blk_mq_run_hw_queue               (15.34%)
          → blk_mq_sched_dispatch_requests  (14.55%)
            → __blk_mq_sched_dispatch_requests (11.27%)
              → blk_mq_dispatch_rq_list      (7.41%)
                → nvme_queue_rq              (5.72%)

Adding bio plugging increased IOPS from 60,000 to 74,000 and reduced latency
from 12.4 µs to 9.5 µs.

Commit 9cbbac29d752 states that bio plugging increases CPU usage. I suspect
that plugging may allow I/O to be dispatched faster, concentrating CPU
activity into a shorter period. This could increase instantaneous CPU
utilization while reducing the total test duration.

> Cc: stable@vger.kernel.org
> Signed-off-by: Qiliang Yuan <odys.yuan@gmail.com>
> ---
>  block/blk-mq.c | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index a26a11c73ee3e..54eff3cf45fae 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -3200,8 +3200,16 @@ void blk_mq_submit_bio(struct bio *bio)
>  	}
>  
>  	hctx = rq->mq_hctx;
> -	if ((rq->rq_flags & RQF_USE_SCHED) ||
> -	    (hctx->dispatch_busy && (q->nr_hw_queues == 1 || !is_sync))) {
> +	if (rq->rq_flags & RQF_USE_SCHED) {
> +		/*
> +		 * Run the queue from the submitting context, as flushing a plug
> +		 * does. Punting every unplugged request to kblockd lets the
> +		 * kworker preempt the submitter once per I/O.
> +		 */
> +		blk_mq_insert_request(rq, 0);
> +		blk_mq_run_hw_queue(hctx, false);
> +	} else if (hctx->dispatch_busy &&
> +		   (q->nr_hw_queues == 1 || !is_sync)) {
>  		blk_mq_insert_request(rq, 0);
>  		blk_mq_run_hw_queue(hctx, true);
>  	} else {
> 
> ---
> base-commit: 551c722f40809618230001baccf219193e22fc5a
> change-id: 20260930-bug-blk-mq-noplug-sched-dispatch-4f19360b0bf7
> 
> Best regards,

-- 
Thanks
Kaitao Cheng