[RFC PATCH 0/1] block: charge passthrough requests to the submitter's cgroup

[RFC PATCH 0/1] block: charge passthrough requests to the submitter's cgroup

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

While looking at cgroup IO control coverage, I stumbled into a hole
that is probably familiar to people who have worked on passthrough:
SG_IO, bsg, and the NVMe ioctl / uring command paths build requests
directly and dispatch them through blk_execute_rq{,_nowait}(),
never passing through submit_bio().  The mapped bio carries no blkcg
association, so the bytes never show up in cgroup io.stat, and none
of the rq_qos policies (iocost, iolatency, wbt) see these commands;
blk-throttle, hooking submit_bio_noacct() directly, is equally
blind.  On a scsi_debug device with iocost enabled and vrate pinned
to its 1% floor, a direct writer in a cgroup is throttled ~10x while
the same cgroup issuing sg_dd writes runs at full device speed with
io.stat staying at zero.

This RFC is a prototype asking for feedback on one possible
direction, not a proposal I am attached to: is charging passthrough
at dispatch time the right layer and the right semantics at all?

The patch associates the mapped bio with the submitter's blkcg in
blk_execute_rq{,_nowait}(), runs the regular bio accounting
(blk_cgroup_bio_start()) and the rq_qos throttle path with it, and
gates by opcode (READ/WRITE/DRV_IN/DRV_OUT, mapping the latter two
to READ/WRITE for io.stat classification) and to queues that already
have a gendisk so probing stays exempt.  On linux-next, with
scsi_debug + sg_dd and qemu emulated nvme + an NVME_IOCTL_IO64_CMD
loop, the transferred bytes are fully accounted in the issuing
cgroup, and with vrate clamped the pacing bites: the same 128MB
SG_IO run takes 0.025s unthrottled and 31.7s at the 1% floor, with
the issuing task sleeping on the iocost waitqueue like any bio
submitter.

Questions I am unsure about:

 1. Is always-on the right default, or should this hide behind the
    queue/iostats_passthrough opt-in from the passthrough iostats
    series, keeping default behavior unchanged?
 2. The throttle may sleep; all callers I found (ioctl / uring_cmd
    submit, target and error handling kthreads) are sleepable, and
    kthread-issued charges land on root and turn into no-ops - but
    I may have missed paths, and I'd rather ask than assume.
 3. What should happen on nvme multipath failover, where the bio is
    re-routed to another queue but carries the original queue's
    blkg?  Untested; guidance welcome.
 4. wbt also gains passthrough throttling through the same rq_qos
    walk - probably benign since wbt targets buffered writeback,
    but it is a behavior change I did not set out to make.
 5. blk-throttle does not benefit (it is not an rq_qos policy, and
    its queue-and-resubmit model does not fit bios already bound to
    a dispatched request); would a synchronous wait variant there
    be welcome as follow-up work, or is leaving it alone preferred?

Reproduction notes in case anyone tries: sg_dd coalesces into very
large single commands by default (use bpt=1), and a short burst from
a single cgroup rides the initial budget, so sustained transfers
are what show the pacing.


Tao Cui (1):
  block: charge passthrough requests to the submitter's cgroup

 block/blk-mq.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

-- 
2.43.0

[RFC PATCH 1/1] block: charge passthrough requests to the submitter's cgroup

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

Passthrough requests (SG_IO, bsg, nvme passthrough ioctls and uring
commands) are dispatched via blk_execute_rq{,_nowait}() without ever
passing through submit_bio(), so the bio mapped by blk_rq_map_user()
carries no blkcg association: the transferred bytes never show up in
cgroup io.stat, and every rq_qos policy on the queue (iocost,
iolatency, wbt) is bypassed, as is blk-throttle, which hooks
submit_bio_noacct() directly rather than going through rq_qos.

A quick demonstration on a scsi_debug device with iocost enabled and
vrate pinned to its 1% floor: a direct fio writer was throttled ~10x
while the same cgroup issuing sg_dd writes ran at full device speed
with zero io.stat accounting.

Associate the mapped bio with the submitter's blkcg at dispatch time
and run the regular bio accounting (blk_cgroup_bio_start()) and
rq_qos throttle paths with it.  DRV_IN/DRV_OUT commands are mapped to
READ/WRITE so io.stat classifies their bytes normally; request
completion already pairs with the throttle through bio_endio() ->
rq_qos_done_bio().

The charge is gated by opcode (READ/WRITE/DRV_IN/DRV_OUT) and to
queues that already have a gendisk: commands issued during device
probing (SCSI INQUIRY etc.) have no gendisk yet and stay exempt,
following the same probe-exemption reasoning as the passthrough
iostats support.

RFC notes:
 - validated on linux-next with scsi_debug + sg_dd (SG_IO) and qemu
   emulated nvme + a NVME_IOCTL_IO64_CMD loop: in both cases the
   transferred bytes are fully accounted (wbytes/wios) in the issuing
   cgroup, and the issuing task is observed waiting on the iocost
   waitqueue with vrate clamped;
 - iolatency gains the same coverage for free, and so does wbt:
   passthrough writes now pass through wbt_wait() like bio-path
   writes, a behavior change worth calling out even though wbt
   targets buffered writeback and direct passthrough rarely hits it;
   blk-throttle does not: it hooks
   submit_bio_noacct() directly and is not an rq_qos policy, so
   io.max stays unenforced for passthrough (measured).  Its
   queue-and-resubmit throttling model would also need a synchronous
   variant for request-bound bios;
 - nvme uring commands share the same blk_execute_rq_nowait()
   dispatch as the validated ioctl path;
 - the charge runs in the submitter's context and may sleep in
   rq_qos throttling; all data-op callers found run in sleepable
   context, but this deserves reviewer attention.

Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 block/blk-mq.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index a26a11c73ee3..2dd59f6df321 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -10,6 +10,7 @@
 #include <linux/backing-dev.h>
 #include <linux/bio.h>
 #include <linux/blkdev.h>
+#include "blk-cgroup.h"
 #include <linux/blk-integrity.h>
 #include <linux/kmemleak.h>
 #include <linux/mm.h>
@@ -1400,6 +1401,73 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq)
 	plug->rq_count++;
 }
 
+/*
+ * Passthrough bios are mapped directly onto requests via
+ * blk_rq_map_user() and never pass through submit_bio(), so they carry
+ * no blkcg association and are invisible to cgroup io.stat and to every
+ * rq_qos policy (iocost, blk-throttle, iolatency).  Charge the ones that
+ * carry data to the submitter's blkcg at dispatch time and run the
+ * regular bio accounting and rq_qos throttle paths with the associated
+ * bio.
+ *
+ * Gated by opcode (READ/WRITE/DRV_IN/DRV_OUT, the latter two mapped
+ * to READ/WRITE for io.stat classification) and to queues that
+ * already have a gendisk: commands issued during device probing (SCSI
+ * INQUIRY and friends) have no gendisk yet and stay exempt.  The
+ * request bios may carry a stale ->bi_blkg from the mempool; the
+ * association helper drops the old reference and re-associates.
+ */
+static void blk_mq_pt_charge(struct request *rq)
+{
+	struct bio *bio = rq->bio;
+	enum req_op op = req_op(rq);
+
+	if (!bio || !rq->q->disk)
+		return;
+
+	switch (op) {
+	case REQ_OP_READ:
+	case REQ_OP_WRITE:
+	case REQ_OP_DRV_IN:
+	case REQ_OP_DRV_OUT:
+		break;
+	default:
+		return;
+	}
+	if (op == REQ_OP_DRV_IN)
+		op = REQ_OP_READ;
+	else if (op == REQ_OP_DRV_OUT)
+		op = REQ_OP_WRITE;
+
+	if (!bio->bi_bdev)
+		bio->bi_bdev = rq->q->disk->part0;
+	bio->bi_opf &= ~REQ_OP_MASK;
+	bio->bi_opf |= op;
+#ifdef CONFIG_BLK_CGROUP
+	/*
+	 * Issued from kthreads the css is root and the charge is a
+	 * no-op through the root exemptions; data-op issuers that
+	 * matter run in the submitter's task context.
+	 */
+	{
+		struct cgroup_subsys_state *css;
+
+		rcu_read_lock();
+		css = task_css(current, io_cgrp_id);
+		bio_associate_blkg_from_css(bio, css);
+		rcu_read_unlock();
+	}
+#endif
+	blk_cgroup_bio_start(bio);
+
+	/*
+	 * The rq_qos throttle path may sleep on the waitqueues like any
+	 * bio submitter; all callers found (ioctl / uring_cmd submit,
+	 * target and error handling kthreads) run in sleepable context.
+	 */
+	rq_qos_throttle(rq->q, bio);
+}
+
 /**
  * blk_execute_rq_nowait - insert a request to I/O scheduler for execution
  * @rq:		request to insert
@@ -1412,6 +1480,7 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq)
  * Note:
  *    This function will invoke @done directly if the queue is dead.
  */
+
 void blk_execute_rq_nowait(struct request *rq, bool at_head)
 {
 	struct blk_mq_hw_ctx *hctx = rq->mq_hctx;
@@ -1419,6 +1488,7 @@ void blk_execute_rq_nowait(struct request *rq, bool at_head)
 	WARN_ON(irqs_disabled());
 	WARN_ON(!blk_rq_is_passthrough(rq));
 
+	blk_mq_pt_charge(rq);
 	blk_account_io_start(rq);
 
 	if (current->plug && !at_head) {
@@ -1484,6 +1554,8 @@ blk_status_t blk_execute_rq(struct request *rq, bool at_head)
 	WARN_ON(irqs_disabled());
 	WARN_ON(!blk_rq_is_passthrough(rq));
 
+	blk_mq_pt_charge(rq);
+
 	rq->end_io_data = &wait;
 	rq->end_io = blk_end_sync_rq;
 
-- 
2.43.0

Re: [RFC PATCH 0/1] block: charge passthrough requests to the submitter's cgroup

[Tao Cui]

Hi,

在 2026/9/21 15:06, Tao Cui 写道:
> From: Tao Cui <cuitao@kylinos.cn>
> 
> While looking at cgroup IO control coverage, I stumbled into a hole
> that is probably familiar to people who have worked on passthrough:
> SG_IO, bsg, and the NVMe ioctl / uring command paths build requests
> directly and dispatch them through blk_execute_rq{,_nowait}(),
> never passing through submit_bio().  The mapped bio carries no blkcg
> association, so the bytes never show up in cgroup io.stat, and none
> of the rq_qos policies (iocost, iolatency, wbt) see these commands;
> blk-throttle, hooking submit_bio_noacct() directly, is equally
> blind.  On a scsi_debug device with iocost enabled and vrate pinned
> to its 1% floor, a direct writer in a cgroup is throttled ~10x while
> the same cgroup issuing sg_dd writes runs at full device speed with
> io.stat staying at zero.
> 
> This RFC is a prototype asking for feedback on one possible
> direction, not a proposal I am attached to: is charging passthrough
> at dispatch time the right layer and the right semantics at all?
> 
> The patch associates the mapped bio with the submitter's blkcg in
> blk_execute_rq{,_nowait}(), runs the regular bio accounting
> (blk_cgroup_bio_start()) and the rq_qos throttle path with it, and
> gates by opcode (READ/WRITE/DRV_IN/DRV_OUT, mapping the latter two
> to READ/WRITE for io.stat classification) and to queues that already
> have a gendisk so probing stays exempt.  On linux-next, with
> scsi_debug + sg_dd and qemu emulated nvme + an NVME_IOCTL_IO64_CMD
> loop, the transferred bytes are fully accounted in the issuing
> cgroup, and with vrate clamped the pacing bites: the same 128MB
> SG_IO run takes 0.025s unthrottled and 31.7s at the 1% floor, with
> the issuing task sleeping on the iocost waitqueue like any bio
> submitter.
> 
> Questions I am unsure about:
> 
>  1. Is always-on the right default, or should this hide behind the
>     queue/iostats_passthrough opt-in from the passthrough iostats
>     series, keeping default behavior unchanged?
>  2. The throttle may sleep; all callers I found (ioctl / uring_cmd
>     submit, target and error handling kthreads) are sleepable, and
>     kthread-issued charges land on root and turn into no-ops - but
>     I may have missed paths, and I'd rather ask than assume.
>  3. What should happen on nvme multipath failover, where the bio is
>     re-routed to another queue but carries the original queue's
>     blkg?  Untested; guidance welcome.
>  4. wbt also gains passthrough throttling through the same rq_qos
>     walk - probably benign since wbt targets buffered writeback,
>     but it is a behavior change I did not set out to make.
>  5. blk-throttle does not benefit (it is not an rq_qos policy, and
>     its queue-and-resubmit model does not fit bios already bound to
>     a dispatched request); would a synchronous wait variant there
>     be welcome as follow-up work, or is leaving it alone preferred?
> 
> Reproduction notes in case anyone tries: sg_dd coalesces into very
> large single commands by default (use bpt=1), and a short burst from
> a single cgroup rides the initial budget, so sustained transfers
> are what show the pacing.
> 

The cover letter summarized the motivation briefly, so let me add some
background on how this came up and what the prototype is trying to
address.

How this was found
------------------

This came from stress testing blk-iocost on linux-next with mixed IO
workloads and cgroup churn. With the known bio-path accounting gaps
(flush, zone append) already carrying posted fixes, the remaining
missing cases had one property in common: they build requests directly
and enter through blk_execute_rq(), bypassing submit_bio() entirely.

The invariant being violated is simple to state: any IO that consumes
device bandwidth but bypasses submit_bio() currently bypasses blkcg IO
accounting. Passthrough commands, including SG_IO, bsg, nvme ioctls and
uring commands, currently fall into this category: they have no blkcg
association, do not show up in io.stat, and do not participate in
blkcg-aware rq_qos policies such as iocost and iolatency.

The observable difference
-------------------------

On the same scsi_debug device, with the same cgroup and iocost enabled
with vrate pinned at its 1% floor:

  bio writer (fio)                 throttled ~10x
  SG_IO writer (sg_dd)             full device speed,
                                   io.stat remains zero

The NVMe ioctl path shows the same behavior: passthrough provides a
path that bypasses the existing accounting boundary.

With the prototype patch
------------------------

The mapped bio is associated with the submitting blkcg in
blk_execute_rq{,_nowait}(), and goes through the normal bio accounting
and rq_qos paths.

With the same setup:

  - a 128MB SG_IO write (bs=1M, bpt=1) is accounted to the issuing
    cgroup's io.stat, with wbytes/wios matching the transfer;
  - the same workload takes 0.025s without throttling and 31.7s with
    vrate at the 1% floor, with the task sleeping in the iocost throttle
    path.

Two reproduction notes: sg_dd combines transfers into large commands by
default, so bpt=1 is needed for scsi_debug. Also, short bursts can be
covered by the initial budget; sustained transfers make the difference
visible.

Relation to passthrough iostats
-------------------------------

This RFC builds on Keith's passthrough iostats work. That series made
disk-level passthrough accounting possible and provided block-device
association for passthrough bios where it is available.

This RFC is about the next layer: whether passthrough IO should also
participate in blkcg accounting and enforcement.

The questions from the cover letter still stand. In particular, I am
not sure whether attaching blkcg/accounting at this layer is the right
abstraction, or whether this should remain an opt-in behavior like
passthrough iostats.

Thanks,
Tao

> 
> Tao Cui (1):
>   block: charge passthrough requests to the submitter's cgroup
> 
>  block/blk-mq.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 72 insertions(+)
> 

Re: [RFC PATCH 1/1] block: charge passthrough requests to the submitter's cgroup

[Tejun Heo]

On Mon, Sep 21, 2026 at 03:06:47PM +0800, Tao Cui wrote:
> From: Tao Cui <cuitao@kylinos.cn>
> 
> Passthrough requests (SG_IO, bsg, nvme passthrough ioctls and uring
> commands) are dispatched via blk_execute_rq{,_nowait}() without ever
> passing through submit_bio(), so the bio mapped by blk_rq_map_user()
> carries no blkcg association: the transferred bytes never show up in
> cgroup io.stat, and every rq_qos policy on the queue (iocost,
> iolatency, wbt) is bypassed, as is blk-throttle, which hooks
> submit_bio_noacct() directly rather than going through rq_qos.
> 
> A quick demonstration on a scsi_debug device with iocost enabled and
> vrate pinned to its 1% floor: a direct fio writer was throttled ~10x
> while the same cgroup issuing sg_dd writes ran at full device speed
> with zero io.stat accounting.
> 
> Associate the mapped bio with the submitter's blkcg at dispatch time
> and run the regular bio accounting (blk_cgroup_bio_start()) and
> rq_qos throttle paths with it.  DRV_IN/DRV_OUT commands are mapped to
> READ/WRITE so io.stat classifies their bytes normally; request
> completion already pairs with the throttle through bio_endio() ->
> rq_qos_done_bio().
> 
> The charge is gated by opcode (READ/WRITE/DRV_IN/DRV_OUT) and to
> queues that already have a gendisk: commands issued during device
> probing (SCSI INQUIRY etc.) have no gendisk yet and stay exempt,
> following the same probe-exemption reasoning as the passthrough
> iostats support.

Do you have an actual use case where this matters?

Thanks.

-- 
tejun