* [RFC PATCH v8 1/4] blk-iocost: add BPF struct_ops cost model support
2026-09-30 7:51 [RFC PATCH v8 0/4] blk-iocost: add BPF struct_ops cost model support Tao Cui
@ 2026-09-30 7:51 ` Tao Cui
2026-10-01 0:20 ` Tejun Heo
2026-09-30 7:51 ` [RFC PATCH v8 2/4] selftests/bpf: add iocost cost model test Tao Cui
` (2 subsequent siblings)
3 siblings, 1 reply; 10+ messages in thread
From: Tao Cui @ 2026-09-30 7:51 UTC (permalink / raw)
To: tj, josef, axboe
Cc: cgroups, linux-block, linux-kernel, bpf, andrii, eddyz87, ast,
daniel, linux-kselftest, cui.tao, cuitao, ameryhung,
alexei.starovoitov
Add the iocost_model_ops struct_ops. Attachment follows the
hid_bpf_ops model: the struct_ops instance is per-device, the target
device is set in the dev member from userspace before load, .reg
attaches the model to that device and switches it away from the
builtin linear model, and .unreg detaches it and restores the builtin
model. The struct_ops core owns the program lifetime, so there is no
name registry and no bound-state bookkeeping.
calc_cost() receives the bio itself and reads whatever it needs
(operation flags, size, sector, the issuing cgroup through
bi_blkg->blkcg); the merge indicator stays in the separate flags
argument as it is not a property of the bio. It is called from the
bio charging path, so the model owns pricing for every IO on the
device. The completion-time request sizing uses the transfer cost
coefficients carried in the struct_ops (vtime per page for reads and
writes) while the model is in use, so the builtin latency tracking
and vrate adjustment follow the model's pricing; letting a model take
over the QoS side is left for a later extension.
The cgroup callbacks are bound to the iocg policy init/free paths,
one (cgroup, device) pair per invocation, matching the builtin
cursor's lifetime, instead of the blkcg css lifecycle, which also
drops the mutex from the cgroup online/offline paths.
calc_cost() runs under RCU read lock; sleepable programs are
rejected in .check_member.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
block/Kconfig | 10 +
block/Makefile | 1 +
block/blk-core.c | 17 ++
block/blk-iocost-bpf.c | 220 ++++++++++++++++++++++
block/blk-iocost.c | 368 ++++++++++++++++++++++++++++++++++++-
include/linux/blk-iocost.h | 110 +++++++++++
6 files changed, 718 insertions(+), 8 deletions(-)
create mode 100644 block/blk-iocost-bpf.c
create mode 100644 include/linux/blk-iocost.h
diff --git a/block/Kconfig b/block/Kconfig
index 70e4a66d941f..1cafc1bd0dda 100644
--- a/block/Kconfig
+++ b/block/Kconfig
@@ -231,4 +231,14 @@ config BLK_ERROR_INJECTION
source "block/Kconfig.iosched"
+config BLK_CGROUP_IOCOST_BPF
+ bool "Enable BPF pluggable cost model support for the cost IO controller"
+ depends on BLK_CGROUP_IOCOST && BPF_SYSCALL && BPF_JIT && DEBUG_INFO_BTF
+ help
+ Enabling this option registers the "iocost_model_ops" BPF
+ struct_ops type, which allows a BPF program to fully replace
+ the builtin linear cost model on the device it is attached
+ to. The struct_ops is attached per device, following the
+ hid_bpf_ops model.
+
endif # BLOCK
diff --git a/block/Makefile b/block/Makefile
index e7bd320e3d69..ee5cebeea006 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -39,3 +39,4 @@ obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += blk-crypto.o blk-crypto-profile.o \
blk-crypto-sysfs.o
obj-$(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK) += blk-crypto-fallback.o
obj-$(CONFIG_BLOCK_HOLDER_DEPRECATED) += holder.o
+obj-$(CONFIG_BLK_CGROUP_IOCOST_BPF) += blk-iocost-bpf.o
diff --git a/block/blk-core.c b/block/blk-core.c
index 13dc70e8f55d..2c789b0d86ff 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -536,6 +536,23 @@ bool blk_get_queue(struct request_queue *q)
}
EXPORT_SYMBOL(blk_get_queue);
+/**
+ * blk_get_queue_rcu - get a queue reference regardless of the dying flag
+ * @q: the request_queue to reference
+ *
+ * Unlike blk_get_queue(), this succeeds on a dying queue, so a caller
+ * which holds the queue only through RCU (e.g. a detach path which
+ * read the pointer locklessly) can still take a reference and touch
+ * the queue under its own lifetime. The caller must hold
+ * rcu_read_lock() so the memory is valid. Fails only when the
+ * refcount already dropped to zero.
+ */
+bool blk_get_queue_rcu(struct request_queue *q)
+{
+ return refcount_inc_not_zero(&q->refs);
+}
+EXPORT_SYMBOL(blk_get_queue_rcu);
+
#ifdef CONFIG_FAIL_MAKE_REQUEST
static DECLARE_FAULT_ATTR(fail_make_request);
diff --git a/block/blk-iocost-bpf.c b/block/blk-iocost-bpf.c
new file mode 100644
index 000000000000..32a5107522b5
--- /dev/null
+++ b/block/blk-iocost-bpf.c
@@ -0,0 +1,220 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * blk-iocost: BPF struct_ops plumbing for pluggable cost models.
+ *
+ * Registers the "iocost_model_ops" struct_ops type. Attachment is
+ * per-device and follows the hid_bpf_ops model: the target device is
+ * set in the ops from userspace before load, .reg attaches the model
+ * to that device, creating the ioc if needed and switching to the
+ * model under the same queue freeze and quiesce as io.cost.model
+ * writes, and .unreg detaches it; enabling and disabling the
+ * controller stays with io.cost.qos. The struct_ops core owns the
+ * program lifetime. There is no name registry and no separate
+ * bound-state bookkeeping.
+ */
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf.h>
+#include <linux/blk-iocost.h>
+#include <linux/blk-mq.h>
+#include <linux/mutex.h>
+
+struct block_device *blkdev_get_no_open(dev_t dev, bool autoload);
+void blkdev_put_no_open(struct block_device *bdev);
+bool blk_get_queue_rcu(struct request_queue *q);
+void blk_put_queue(struct request_queue *q);
+
+static int bpf_iocost_model_init(struct btf *btf)
+{
+ s32 type_id;
+
+ type_id = btf_find_by_name_kind(btf, "iocost_model_ops", BTF_KIND_STRUCT);
+ if (type_id < 0)
+ return -EINVAL;
+ return 0;
+}
+
+static bool bpf_iocost_is_valid_access(int off, int size,
+ enum bpf_access_type type,
+ const struct bpf_prog *prog,
+ struct bpf_insn_access_aux *info)
+{
+ return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
+}
+
+/*
+ * No iocost-specific helpers; bpf_base_func_proto already covers the
+ * cgroup storage helpers under CONFIG_CGROUPS.
+ */
+static const struct bpf_func_proto *
+bpf_iocost_get_func_proto(enum bpf_func_id func_id,
+ const struct bpf_prog *prog)
+{
+ return bpf_base_func_proto(func_id, prog);
+}
+
+static int bpf_iocost_check_member(const struct btf_type *t,
+ const struct btf_member *member,
+ const struct bpf_prog *prog)
+{
+ /* calc_cost() is called with RCU read lock held */
+ if (prog->sleepable)
+ return -EINVAL;
+ return 0;
+}
+
+static int bpf_iocost_init_member(const struct btf_type *t,
+ const struct btf_member *member,
+ void *kdata, const void *udata)
+{
+ struct iocost_model_ops *ops = kdata;
+ const struct iocost_model_ops *uops = udata;
+ u32 moff = __btf_member_bit_offset(t, member) / 8;
+
+ switch (moff) {
+ case offsetof(struct iocost_model_ops, bdev):
+ /*
+ * kernel-private: the bdev reference held while
+ * attached; reject a userspace value
+ */
+ if (uops->bdev)
+ return -EINVAL;
+ ops->bdev = NULL;
+ return 1;
+ case offsetof(struct iocost_model_ops, q):
+ /* kernel-private: the queue of the attached device */
+ if (uops->q)
+ return -EINVAL;
+ ops->q = NULL;
+ return 1;
+ case offsetof(struct iocost_model_ops, dev):
+ /*
+ * copy it and return 1 to indicate that the member is
+ * handled here, or the verifier rejects the map if the
+ * userspace value is nonzero
+ */
+ ops->dev = uops->dev;
+ return 1;
+ case offsetof(struct iocost_model_ops, read_vtime_per_page):
+ ops->read_vtime_per_page = uops->read_vtime_per_page;
+ return 1;
+ case offsetof(struct iocost_model_ops, write_vtime_per_page):
+ ops->write_vtime_per_page = uops->write_vtime_per_page;
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * kvalue is zeroed at map allocation and function members are only
+ * written when the BPF side provides a prog, so a model which did
+ * not implement calc_cost leaves it NULL. The dispatch would call
+ * it on every bio, so reject it here.
+ */
+static int bpf_iocost_validate(void *kdata)
+{
+ struct iocost_model_ops *ops = kdata;
+
+ return ops->calc_cost ? 0 : -EINVAL;
+}
+
+static int bpf_iocost_reg(void *kdata, struct bpf_link *link)
+{
+ struct iocost_model_ops *ops = kdata;
+
+ if (!ops->dev)
+ return -EINVAL;
+
+ return ioc_bpf_attach(ops);
+}
+
+static void bpf_iocost_unreg(void *kdata, struct bpf_link *link)
+{
+ struct iocost_model_ops *ops = kdata;
+ struct block_device *bdev;
+ struct request_queue *q;
+
+ /*
+ * ops->q may already have been cleared by the removal ejection;
+ * take a queue reference under RCU before entering the queue,
+ * as the queue may be dying and its memory is only guaranteed
+ * under rcu_read_lock()
+ */
+ rcu_read_lock();
+ q = rcu_dereference(ops->q);
+ if (!q || !blk_get_queue_rcu(q)) {
+ rcu_read_unlock();
+ return;
+ }
+ rcu_read_unlock();
+
+ /*
+ * take rq_qos_mutex before touching the queue further: the
+ * removal ejection clears ops->q under it, inside rq_qos_exit()
+ * and before blk_mq_exit_queue() releases the hardware queues,
+ * so holding it and seeing a non-NULL ops->q guarantees the
+ * queue is still safe to freeze. The freeze and quiesce are
+ * done inside ioc_bpf_detach(), under the mutex.
+ */
+ mutex_lock(&q->rq_qos_mutex);
+ if (!ops->q) {
+ /* the removal ejection won the race; nothing to detach */
+ mutex_unlock(&q->rq_qos_mutex);
+ blk_put_queue(q);
+ return;
+ }
+ bdev = ops->bdev;
+ ioc_bpf_detach(ops);
+ mutex_unlock(&q->rq_qos_mutex);
+ blk_put_queue(q);
+
+ /* drop the attach reference when we did the detach */
+ if (bdev)
+ blkdev_put_no_open(bdev);
+}
+
+static const struct bpf_verifier_ops bpf_iocost_verifier_ops = {
+ .get_func_proto = bpf_iocost_get_func_proto,
+ .is_valid_access = bpf_iocost_is_valid_access,
+};
+
+static u64 bpf_iocost_calc_cost_stub(struct bio *bio, u64 flags)
+{
+ return 0;
+}
+
+static void bpf_iocost_iocg_init_stub(struct blkcg *blkcg,
+ struct request_queue *q)
+{ }
+static void bpf_iocost_iocg_free_stub(struct blkcg *blkcg,
+ struct request_queue *q)
+{ }
+
+static struct iocost_model_ops __bpf_ops_iocost_model_ops = {
+ .calc_cost = bpf_iocost_calc_cost_stub,
+ .iocg_init = bpf_iocost_iocg_init_stub,
+ .iocg_free = bpf_iocost_iocg_free_stub,
+};
+
+static struct bpf_struct_ops bpf_iocost_model_ops = {
+ .verifier_ops = &bpf_iocost_verifier_ops,
+ .init = bpf_iocost_model_init,
+ .check_member = bpf_iocost_check_member,
+ .init_member = bpf_iocost_init_member,
+ .validate = bpf_iocost_validate,
+ .reg = bpf_iocost_reg,
+ .unreg = bpf_iocost_unreg,
+ .name = "iocost_model_ops",
+ .cfi_stubs = &__bpf_ops_iocost_model_ops,
+ .owner = THIS_MODULE,
+};
+
+static int __init bpf_iocost_init(void)
+{
+ return register_bpf_struct_ops(&bpf_iocost_model_ops, iocost_model_ops);
+}
+late_initcall(bpf_iocost_init);
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 2745bffcd5ee..bc83827cced9 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -177,6 +177,7 @@
#include <linux/timer.h>
#include <linux/time64.h>
#include <linux/parser.h>
+#include <linux/blk-iocost.h>
#include <linux/sched/signal.h>
#include <asm/local.h>
#include <asm/local64.h>
@@ -445,6 +446,13 @@ struct ioc {
int autop_idx;
bool user_qos_params:1;
bool user_cost_model:1;
+
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ /* the struct_ops attached to this device, NULL = none */
+ const struct iocost_model_ops __rcu *attached;
+ /* the cost model in use, NULL = builtin linear model */
+ const struct iocost_model_ops __rcu *model;
+#endif
};
struct iocg_pcpu_stat {
@@ -780,6 +788,45 @@ static void ioc_refresh_period_us(struct ioc *ioc)
ioc_refresh_margins(ioc);
}
+/*
+ * The BPF model in use, or NULL when the builtin linear model prices
+ * this device. CONFIG_BLK_CGROUP_IOCOST_BPF=n compiles to a constant
+ * NULL so callers need no #ifdefs.
+ */
+static const struct iocost_model_ops *ioc_model_in_use(struct ioc *ioc)
+{
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ return rcu_dereference(ioc->model);
+#else
+ return NULL;
+#endif
+}
+
+/*
+ * The struct_ops attached to this device, or NULL. While attachment
+ * and model selection are independent, the cgroup callbacks follow
+ * the attachment, not the selection.
+ */
+static const struct iocost_model_ops *ioc_attached_or_null(struct ioc *ioc)
+{
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ return rcu_dereference(ioc->attached);
+#else
+ return NULL;
+#endif
+}
+
+static const struct iocost_model_ops *
+ioc_model_in_use_locked(struct ioc *ioc)
+{
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ return rcu_dereference_protected(ioc->model,
+ lockdep_is_held(&ioc->lock));
+#else
+ return NULL;
+#endif
+}
+
/*
* ioc->rqos.disk isn't initialized when this function is called from
* the init path.
@@ -803,8 +850,13 @@ static int ioc_autop_idx(struct ioc *ioc, struct gendisk *disk)
if (idx < AUTOP_SSD_DFL)
return AUTOP_SSD_DFL;
- /* if user is overriding anything, maintain what was there */
- if (ioc->user_qos_params || ioc->user_cost_model)
+ /*
+ * if user is overriding anything, or a BPF model is in use,
+ * maintain what was there: the builtin coefficients are inert
+ * then, so stepping the profile is pointless
+ */
+ if (ioc->user_qos_params || ioc->user_cost_model ||
+ ioc_model_in_use_locked(ioc))
return idx;
/* step up/down based on the vrate */
@@ -2571,8 +2623,19 @@ static void calc_vtime_cost_builtin(struct bio *bio, struct ioc_gq *iocg,
static u64 calc_vtime_cost(struct bio *bio, struct ioc_gq *iocg, bool is_merge)
{
+ const struct iocost_model_ops *model;
u64 cost;
+ rcu_read_lock();
+ model = ioc_model_in_use(iocg->ioc);
+ if (model) {
+ cost = model->calc_cost(bio,
+ is_merge ? IOCOST_COST_F_MERGE : 0);
+ rcu_read_unlock();
+ return min(cost, VTIME_PER_SEC);
+ }
+ rcu_read_unlock();
+
calc_vtime_cost_builtin(bio, iocg, is_merge, &cost);
return cost;
}
@@ -2594,10 +2657,32 @@ static void calc_size_vtime_cost_builtin(struct request *rq, struct ioc *ioc,
}
}
+/*
+ * Called from the request completion path, where no ioc->lock is
+ * held; the model pointer is read under RCU, matching the bio-side
+ * calc_vtime_cost().
+ */
static u64 calc_size_vtime_cost(struct request *rq, struct ioc *ioc)
{
+ const struct iocost_model_ops *model;
u64 cost;
+ rcu_read_lock();
+ model = ioc_model_in_use(ioc);
+ if (model && (req_op(rq) == REQ_OP_READ ||
+ req_op(rq) == REQ_OP_WRITE)) {
+ unsigned int pages =
+ blk_rq_stats_sectors(rq) >> IOC_SECT_TO_PAGE_SHIFT;
+ u64 coeff = req_op(rq) == REQ_OP_READ ?
+ model->read_vtime_per_page :
+ model->write_vtime_per_page;
+
+ cost = pages * coeff;
+ rcu_read_unlock();
+ return cost;
+ }
+ rcu_read_unlock();
+
calc_size_vtime_cost_builtin(rq, ioc, &cost);
return cost;
}
@@ -2900,6 +2985,24 @@ static void ioc_rqos_exit(struct rq_qos *rqos)
timer_shutdown_sync(&ioc->timer);
free_percpu(ioc->pcpu_stat);
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ {
+ struct iocost_model_ops *ops;
+
+ spin_lock_irq(&ioc->lock);
+ ops = (struct iocost_model_ops *)rcu_dereference_protected(
+ ioc->attached, lockdep_is_held(&ioc->lock));
+ rcu_assign_pointer(ioc->attached, NULL);
+ rcu_assign_pointer(ioc->model, NULL);
+ spin_unlock_irq(&ioc->lock);
+
+ /* eject the model completely on device removal */
+ if (ops) {
+ WRITE_ONCE(ops->q, NULL);
+ blkdev_put_no_open(ops->bdev);
+ }
+ }
+#endif
kfree(ioc);
}
@@ -3022,6 +3125,7 @@ static void ioc_pd_init(struct blkg_policy_data *pd)
struct ioc_now now;
struct blkcg_gq *tblkg;
unsigned long flags;
+ const struct iocost_model_ops *model;
ioc_now(ioc, &now);
@@ -3048,6 +3152,18 @@ static void ioc_pd_init(struct blkg_policy_data *pd)
spin_lock_irqsave(&ioc->lock, flags);
weight_updated(iocg, &now);
spin_unlock_irqrestore(&ioc->lock, flags);
+
+ /*
+ * the attached model is RCU-protected: a concurrent detach
+ * publishes NULL and the struct_ops image survives it by a
+ * grace period, so the callback is safe inside the read-side
+ * critical section
+ */
+ rcu_read_lock();
+ model = ioc_attached_or_null(ioc);
+ if (model && model->iocg_init)
+ model->iocg_init(blkg->blkcg, ioc->rqos.disk->queue);
+ rcu_read_unlock();
}
static void iocg_release(struct rcu_head *rcu)
@@ -3066,8 +3182,15 @@ static void ioc_pd_free(struct blkg_policy_data *pd)
struct blkcg_gq *blkg = pd_to_blkg(pd);
struct ioc *ioc = iocg->ioc;
unsigned long flags;
+ const struct iocost_model_ops *model;
if (ioc) {
+ rcu_read_lock();
+ model = ioc_attached_or_null(ioc);
+ if (model && model->iocg_free)
+ model->iocg_free(blkg->blkcg, ioc->rqos.disk->queue);
+ rcu_read_unlock();
+
spin_lock_irqsave(&ioc->lock, flags);
if (!list_empty(&iocg->active_list)) {
@@ -3433,17 +3556,21 @@ static u64 ioc_cost_model_prfill(struct seq_file *sf,
const char *dname = blkg_dev_name(pd->blkg);
struct ioc *ioc = pd_to_iocg(pd)->ioc;
u64 *u = ioc->params.i_lcoefs;
+ const struct iocost_model_ops *model;
if (!dname)
return 0;
spin_lock_irq(&ioc->lock);
- seq_printf(sf, "%s ctrl=%s model=linear "
+ model = ioc_model_in_use_locked(ioc);
+ seq_printf(sf, "%s ctrl=%s model=%s "
"rbps=%llu rseqiops=%llu rrandiops=%llu "
"wbps=%llu wseqiops=%llu wrandiops=%llu\n",
dname, ioc->user_cost_model ? "user" : "auto",
- u[I_LCOEF_RBPS], u[I_LCOEF_RSEQIOPS], u[I_LCOEF_RRANDIOPS],
- u[I_LCOEF_WBPS], u[I_LCOEF_WSEQIOPS], u[I_LCOEF_WRANDIOPS]);
+ model ? "bpf" : "linear",
+ u[I_LCOEF_RBPS], u[I_LCOEF_RSEQIOPS],
+ u[I_LCOEF_RRANDIOPS], u[I_LCOEF_WBPS],
+ u[I_LCOEF_WSEQIOPS], u[I_LCOEF_WRANDIOPS]);
spin_unlock_irq(&ioc->lock);
return 0;
}
@@ -3457,6 +3584,204 @@ static int ioc_cost_model_show(struct seq_file *sf, void *v)
return 0;
}
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+/*
+ * Deliver iocg_init()/iocg_free() to the cgroups which already have a
+ * blkg on the queue, the same q->blkg_list walk the blkcg policy
+ * teardown uses. The queue is frozen and quiesced and blkcg_mutex
+ * serializes against blkg creation and destruction. Cgroups without
+ * a blkg on the device yet are not missed: their blkg is created
+ * later and ioc_pd_init()/ioc_pd_free() deliver the callbacks then.
+ */
+static void ioc_bpf_walk_iocgs(struct ioc *ioc,
+ const struct iocost_model_ops *ops, bool init)
+{
+ struct request_queue *q = ioc->rqos.disk->queue;
+ struct blkcg_gq *blkg;
+
+ lockdep_assert_held(&q->blkcg_mutex);
+
+ rcu_read_lock();
+ list_for_each_entry(blkg, &q->blkg_list, q_node) {
+ if (!blkg_to_iocg(blkg))
+ continue;
+ if (init) {
+ if (ops->iocg_init)
+ ops->iocg_init(blkg->blkcg, q);
+ } else {
+ if (ops->iocg_free)
+ ops->iocg_free(blkg->blkcg, q);
+ }
+ }
+ rcu_read_unlock();
+}
+
+int ioc_bpf_attach(struct iocost_model_ops *ops)
+{
+ struct block_device *bdev;
+ struct request_queue *q;
+ struct gendisk *disk;
+ struct ioc *ioc;
+ unsigned int memflags;
+ int ret;
+
+ /* prevent multiple attach of the same struct_ops */
+ if (ops->q)
+ return -EINVAL;
+
+ bdev = blkdev_get_no_open(new_decode_dev(ops->dev), false);
+ if (!bdev)
+ return -ENODEV;
+ q = bdev->bd_queue;
+ disk = bdev->bd_disk;
+
+ if (bdev_is_partition(bdev)) {
+ ret = -EINVAL;
+ goto put;
+ }
+ if (!queue_is_mq(q)) {
+ ret = -EOPNOTSUPP;
+ goto put;
+ }
+
+ /*
+ * check liveness and create the ioc under rq_qos_mutex, like
+ * blkg_conf_open_bdev() does; enabling stays with io.cost.qos
+ *
+ * the queue reference is held across the unlocked window below:
+ * the bdev reference does not pin the queue, bdev only holds a
+ * raw bd_queue pointer, and concurrent device removal may eject
+ * the model and free the ioc while we are off the mutex, so
+ * without our own reference the second mutex_lock() would touch
+ * a freed queue
+ */
+ mutex_lock(&q->rq_qos_mutex);
+ if (!disk_live(disk) || !blk_get_queue(q)) {
+ mutex_unlock(&q->rq_qos_mutex);
+ ret = -ENODEV;
+ goto put;
+ }
+ ioc = q_to_ioc(q);
+ if (!ioc) {
+ ret = blk_iocost_init(disk);
+ mutex_unlock(&q->rq_qos_mutex);
+ if (ret)
+ goto put_q;
+ ioc = q_to_ioc(q);
+ } else {
+ mutex_unlock(&q->rq_qos_mutex);
+ }
+
+ /*
+ * ops->bdev is dropped by the removal ejection or by .unreg,
+ * whichever detaches the model first, similar to hid_bpf's
+ * per-ops device reference without the struct file and without
+ * pinning the driver module
+ */
+
+ /*
+ * switch the model under the same freeze and quiesce as the
+ * io.cost.model writes; the freeze does not pin the ioc, so
+ * rq_qos_mutex has to be held across the switch - ioc_rqos_exit()
+ * frees the ioc under it
+ */
+ mutex_lock(&q->rq_qos_mutex);
+ ioc = q_to_ioc(q);
+ if (!ioc) {
+ mutex_unlock(&q->rq_qos_mutex);
+ ret = -ENODEV;
+ goto put_q;
+ }
+ memflags = blk_mq_freeze_queue(q);
+ blk_mq_quiesce_queue(q);
+
+ /*
+ * hold blkcg_mutex across the publish and the iocg_init() walk:
+ * ioc_pd_init() runs under it too, so no cgroup can receive
+ * iocg_init() from both the walk and its own pd_init
+ */
+ mutex_lock(&q->blkcg_mutex);
+ spin_lock_irq(&ioc->lock);
+ if (rcu_dereference_protected(ioc->attached,
+ lockdep_is_held(&ioc->lock))) {
+ spin_unlock_irq(&ioc->lock);
+ mutex_unlock(&q->blkcg_mutex);
+ blk_mq_unquiesce_queue(q);
+ blk_mq_unfreeze_queue(q, memflags);
+ mutex_unlock(&q->rq_qos_mutex);
+ ret = -EBUSY;
+ goto put_q;
+ }
+ rcu_assign_pointer(ioc->attached, ops);
+ rcu_assign_pointer(ioc->model, ops);
+ spin_unlock_irq(&ioc->lock);
+
+ ops->bdev = bdev;
+ WRITE_ONCE(ops->q, q);
+
+ /* pair iocg_init() with the cgroups which already exist */
+ ioc_bpf_walk_iocgs(ioc, ops, true);
+ mutex_unlock(&q->blkcg_mutex);
+
+ blk_mq_unquiesce_queue(q);
+ blk_mq_unfreeze_queue(q, memflags);
+ mutex_unlock(&q->rq_qos_mutex);
+ blk_put_queue(q);
+ return 0;
+
+put_q:
+ blk_put_queue(q);
+put:
+ blkdev_put_no_open(bdev);
+ return ret;
+}
+
+/*
+ * Detach a model: switch back to the builtin model when the attached
+ * model is in use, clear the attachment, and deliver iocg_free() to
+ * the cgroups which still exist, under the same freeze and quiesce as
+ * the attach. The caller has already checked ops->q.
+ */
+void ioc_bpf_detach(struct iocost_model_ops *ops)
+{
+ struct request_queue *q = ops->q;
+ struct ioc *ioc;
+ unsigned int memflags;
+
+ if (!q)
+ return;
+ ioc = q_to_ioc(q);
+ if (!ioc)
+ return;
+
+ memflags = blk_mq_freeze_queue(q);
+ blk_mq_quiesce_queue(q);
+
+ /*
+ * hold blkcg_mutex across the clearing and the iocg_free() walk:
+ * ioc_pd_free() runs under it too, so no cgroup can receive
+ * iocg_free() from both the walk and its own pd_free
+ */
+ mutex_lock(&q->blkcg_mutex);
+ spin_lock_irq(&ioc->lock);
+ if (rcu_dereference_protected(ioc->model,
+ lockdep_is_held(&ioc->lock)) == ops)
+ rcu_assign_pointer(ioc->model, NULL);
+ rcu_assign_pointer(ioc->attached, NULL);
+ spin_unlock_irq(&ioc->lock);
+
+ /* pair iocg_free() with the cgroups which still exist */
+ ioc_bpf_walk_iocgs(ioc, ops, false);
+ mutex_unlock(&q->blkcg_mutex);
+
+ WRITE_ONCE(ops->q, NULL);
+
+ blk_mq_unquiesce_queue(q);
+ blk_mq_unfreeze_queue(q, memflags);
+}
+
+#endif
+
static const match_table_t cost_ctrl_tokens = {
{ COST_CTRL, "ctrl=%s" },
{ COST_MODEL, "model=%s" },
@@ -3484,6 +3809,10 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
bool user;
char *body, *p;
int ret;
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ const struct iocost_model_ops *new_model = NULL;
+ bool model_write = false;
+#endif
blkg_conf_init(&ctx, input);
@@ -3536,9 +3865,28 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
continue;
case COST_MODEL:
match_strlcpy(buf, &args[0], sizeof(buf));
- if (strcmp(buf, "linear"))
- goto unlock;
- continue;
+ if (!strcmp(buf, "linear")) {
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ /* staged and committed below, so a parse
+ * error later in the same write leaves
+ * the model selection untouched */
+ new_model = NULL;
+ model_write = true;
+#endif
+ continue;
+ }
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ if (!strcmp(buf, "bpf")) {
+ new_model = rcu_dereference_protected(
+ ioc->attached,
+ lockdep_is_held(&ioc->lock));
+ if (!new_model)
+ goto unlock;
+ model_write = true;
+ continue;
+ }
+#endif
+ goto unlock;
}
tok = match_token(p, i_lcoef_tokens, args);
@@ -3556,6 +3904,10 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
} else {
ioc->user_cost_model = false;
}
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ if (model_write)
+ rcu_assign_pointer(ioc->model, new_model);
+#endif
ioc_refresh_params(ioc, true);
ret = 0;
diff --git a/include/linux/blk-iocost.h b/include/linux/blk-iocost.h
new file mode 100644
index 000000000000..03a0c7dab71f
--- /dev/null
+++ b/include/linux/blk-iocost.h
@@ -0,0 +1,110 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_BLK_IOCOST_H
+#define _LINUX_BLK_IOCOST_H
+
+#include <linux/types.h>
+#include <linux/blk_types.h>
+#include <linux/blkdev.h>
+
+struct bio;
+struct blkcg;
+struct request_queue;
+
+/*
+ * Pluggable cost model interface for blk-iocost.
+ *
+ * A BPF struct_ops implementation is attached to one device, identified
+ * by the dev member set from userspace before load, following the
+ * hid_bpf_ops model: the struct_ops core owns the lifetime of the
+ * program. Attaching creates the ioc if needed, like an io.cost.model
+ * write does, switches the device to the BPF model under the same
+ * queue freeze and quiesce, and delivers iocg_init() to the cgroups
+ * which already exist on the device. Enabling and disabling the
+ * controller stays with io.cost.qos. While a model is attached,
+ * io.cost.model selects between it and the builtin model: "model=bpf"
+ * switches to the attached model, "model=linear" switches back to the
+ * builtin model, and neither detaches the struct_ops; only detaching
+ * removes the model. The builtin linear coefficients are kept while
+ * the BPF model is in use and take effect again when switched back.
+ *
+ * While the BPF model is the one in use, it owns pricing for every
+ * charged IO on the device: it prices all operations, including
+ * flushes, from the bio charging path.
+ *
+ * calc_cost() is called from the IO submission path with RCU read lock
+ * held and must not sleep. It receives the bio itself so the model
+ * can read whatever it needs (operation flags, size, sector, the
+ * issuing cgroup through bio->bi_blkg). It returns the cost of the
+ * IO in vtime units, where 1 second of device time equals
+ * VTIME_PER_SEC (2^37, available to BPF programs through vmlinux.h).
+ * The returned value is clamped to 1 second of device time per IO.
+ *
+ * The struct_ops also carries the transfer cost coefficients, vtime
+ * per page for reads and writes: while the BPF model is in use, the
+ * builtin latency tracking and vrate adjustment use these instead of
+ * the builtin linear coefficients for the completion-time request
+ * sizing, so the whole controller follows the model's pricing. Letting
+ * a model take over the QoS side entirely (latency tracking, vrate
+ * control) is left for a later extension.
+ *
+ * The cgroup callbacks are bound to the iocg policy lifetime, one
+ * (cgroup, device) pair per invocation: iocg_init() is delivered on
+ * attach to every cgroup which already exists on the device and to
+ * each one appearing afterwards; iocg_free() is delivered on detach to
+ * every cgroup still existing then, and at policy deactivation time
+ * for the rest, so init and free always pair up. Both callbacks run
+ * inside an RCU read-side critical section and must not sleep.
+ */
+
+/*
+ * iocost-specific call metadata for calc_cost()'s model_flags
+ * argument; the merge indicator is not a property of the bio.
+ * An enum so the value is exported through BTF and BPF models can
+ * use it from vmlinux.h.
+ */
+enum {
+ IOCOST_COST_F_MERGE = 1 << 0, /* called from merge path */
+};
+
+struct iocost_model_ops {
+ /*
+ * target device (major:minor), set from userspace before load
+ * through the struct_ops map's initial value, in the userspace
+ * dev_t encoding new_decode_dev() accepts
+ */
+ dev_t dev;
+
+ /* vtime per page, used by the builtin sizing and vrate logic */
+ u64 read_vtime_per_page;
+ u64 write_vtime_per_page;
+
+ u64 (*calc_cost)(struct bio *bio, u64 model_flags);
+ void (*iocg_init)(struct blkcg *blkcg, struct request_queue *q);
+ void (*iocg_free)(struct blkcg *blkcg, struct request_queue *q);
+
+ /* private: */
+
+ /*
+ * bdev reference held while attached; dropped by the removal
+ * ejection or .unreg, whichever detaches the model first
+ */
+ struct block_device *bdev;
+ /* queue of the attached device, NULL = not attached */
+ struct request_queue *q;
+};
+
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+
+int ioc_bpf_attach(struct iocost_model_ops *ops);
+void ioc_bpf_detach(struct iocost_model_ops *ops);
+
+#else /* CONFIG_BLK_CGROUP_IOCOST_BPF */
+
+static inline int ioc_bpf_attach(struct iocost_model_ops *ops)
+{
+ return -EOPNOTSUPP;
+}
+static inline void ioc_bpf_detach(struct iocost_model_ops *ops) { }
+
+#endif /* CONFIG_BLK_CGROUP_IOCOST_BPF */
+#endif /* _LINUX_BLK_IOCOST_H */
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [RFC PATCH v8 1/4] blk-iocost: add BPF struct_ops cost model support
2026-09-30 7:51 ` [RFC PATCH v8 1/4] " Tao Cui
@ 2026-10-01 0:20 ` Tejun Heo
0 siblings, 0 replies; 10+ messages in thread
From: Tejun Heo @ 2026-10-01 0:20 UTC (permalink / raw)
To: Tao Cui
Cc: josef, axboe, cgroups, linux-block, linux-kernel, bpf, andrii,
eddyz87, ast, daniel, linux-kselftest, Tao Cui, ameryhung,
alexei.starovoitov
Hello, Tao.
The following is a Claude-generated review.
On Wed, 30 Sep 2026 15:51:51 +0800, Tao Cui wrote:
> Add the iocost_model_ops struct_ops. Attachment follows the
> hid_bpf_ops model: the struct_ops instance is per-device, the target
> device is set in the dev member from userspace before load, .reg
> attaches the model to that device and switches it away from the
> builtin linear model, and .unreg detaches it and restores the builtin
> model. The struct_ops core owns the program lifetime, so there is no
> name registry and no bound-state bookkeeping.
...
> The cgroup callbacks are bound to the iocg policy init/free paths,
> one (cgroup, device) pair per invocation, matching the builtin
> cursor's lifetime, instead of the blkcg css lifecycle, which also
> drops the mutex from the cgroup online/offline paths.
The description is written against earlier versions. There is no name
registry, css lifecycle binding or mutex in the tree, so a reader of the
commit can't follow it. Can you open with why a pluggable model is wanted
and then describe what the patch does: attach creates the ioc and
switches the model under the same freeze as io.cost.model writes,
model=bpf and model=linear select between the attached and builtin
models, device removal ejects the model, and so on? Also, the model
prices every charged IO, not every IO. Root cgroup IOs and IOs while the
controller is disabled never reach it.
> +bool blk_get_queue_rcu(struct request_queue *q)
> +{
> + return refcount_inc_not_zero(&q->refs);
> +}
> +EXPORT_SYMBOL(blk_get_queue_rcu);
...
> +struct block_device *blkdev_get_no_open(dev_t dev, bool autoload);
> +void blkdev_put_no_open(struct block_device *bdev);
> +bool blk_get_queue_rcu(struct request_queue *q);
> +void blk_put_queue(struct request_queue *q);
blk_get_queue_rcu() has no prototype in any header, so every build warns
on it. Can you declare it in block/blk.h, include "blk.h" from
blk-iocost-bpf.c and drop these local prototypes? blkdev_get_no_open()
and blkdev_put_no_open() are already in blk.h and blk_put_queue() in
blkdev.h. The export isn't needed either as the only user is built-in.
> + case offsetof(struct iocost_model_ops, bdev):
...
> + case offsetof(struct iocost_model_ops, q):
The struct_ops core rejects non-zero non-function members that
init_member doesn't claim and kvalue starts zeroed, so these two cases
can go. Same for the name lookup in .init, the core has already found the
struct by then.
> + rcu_read_lock();
> + q = rcu_dereference(ops->q);
ops->q isn't __rcu annotated, so sparse will complain here. Either
annotate it and use RCU_INIT_POINTER() for the stores, or READ_ONCE() it.
The RCU section protects the queue, not the ops pointer.
> +#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
> + {
> + struct iocost_model_ops *ops;
> +
> + spin_lock_irq(&ioc->lock);
> + ops = (struct iocost_model_ops *)rcu_dereference_protected(
> + ioc->attached, lockdep_is_held(&ioc->lock));
> + rcu_assign_pointer(ioc->attached, NULL);
> + rcu_assign_pointer(ioc->model, NULL);
> + spin_unlock_irq(&ioc->lock);
> +
> + /* eject the model completely on device removal */
> + if (ops) {
> + WRITE_ONCE(ops->q, NULL);
> + blkdev_put_no_open(ops->bdev);
> + }
> + }
> +#endif
Once ops->q is NULL, .unreg returns without taking rq_qos_mutex, the link
can go away and the struct_ops map which holds ops can be freed after a
grace period. This path isn't in an RCU read section, so reading
ops->bdev after the store is a use-after-free window. Read bdev first and
make the NULL store the last access.
This is also a block scoping a variable. Can you make it an
ioc_bpf_eject() helper next to attach and detach, with an empty stub for
!CONFIG_BLK_CGROUP_IOCOST_BPF? That removes the #ifdef and the const
cast. The remaining #ifdefs in ioc_cost_model_write() would go away too
if the two pointers in struct ioc were unconditional.
> + * Deliver iocg_init()/iocg_free() to the cgroups which already have a
> + * blkg on the queue, the same q->blkg_list walk the blkcg policy
> + * teardown uses. The queue is frozen and quiesced and blkcg_mutex
> + * serializes against blkg creation and destruction. Cgroups without
> + * a blkg on the device yet are not missed: their blkg is created
> + * later and ioc_pd_init()/ioc_pd_free() deliver the callbacks then.
> + */
blkcg_mutex doesn't serialize blkg creation. The IO path creates blkgs
from bio_associate_blkg() through blkg_lookup_create(), before
bio_queue_enter(), so the freeze doesn't stop it either, and
blkg_create() runs pd_init and then list_add() to q->blkg_list under
queue_lock only. Against the attach:
1. ioc->attached is published, blkg_create() runs ioc_pd_init() which
delivers iocg_init(), then list_add(), then the walk delivers
iocg_init() again.
2. blkg_create() runs ioc_pd_init() while attached is NULL, the walk runs
before list_add(), and the blkg never gets iocg_init() but gets
iocg_free() from ioc_pd_free() or the detach walk.
Detach has the mirror cases. The walk is also the only q->blkg_list
walker without queue_lock, and a plain list_for_each_entry() against a
concurrent list_add() isn't safe. The teardown walk this mirrors holds
blkcg_mutex and queue_lock. Can you publish and clear ioc->attached and
walk under queue_lock as well? ioc_pd_init() already calls iocg_init()
under queue_lock, so nothing changes for the BPF side.
While at it, the walk goes newest first, so children usually get
iocg_init() before their parents. blkcg_activate_policy() walks in
reverse for that reason.
> + /* prevent multiple attach of the same struct_ops */
> + if (ops->q)
> + return -EINVAL;
On the link path the map stays READY after .unreg, so a map can be
attached again through a new link. If the device goes away, the ejection
clears ops->q while the old link is still open, a device with the same
dev_t comes back and a new link attaches the same map, closing the old
link then detaches the new attachment. Recording the owning link in the
ops and having .unreg detach only when it matches would close that.
> + /*
> + * check liveness and create the ioc under rq_qos_mutex, like
> + * blkg_conf_open_bdev() does; enabling stays with io.cost.qos
> + *
> + * the queue reference is held across the unlocked window below:
> + * the bdev reference does not pin the queue, bdev only holds a
> + * raw bd_queue pointer, and concurrent device removal may eject
> + * the model and free the ioc while we are off the mutex, so
> + * without our own reference the second mutex_lock() would touch
> + * a freed queue
> + */
> + mutex_lock(&q->rq_qos_mutex);
> + if (!disk_live(disk) || !blk_get_queue(q)) {
...
> + ioc = q_to_ioc(q);
> + if (!ioc) {
> + ret = blk_iocost_init(disk);
> + mutex_unlock(&q->rq_qos_mutex);
Why drop rq_qos_mutex here? ioc_cost_model_write() holds it from
blkg_conf_open_bdev() across blk_iocost_init() and its own freeze and
quiesce, so attach can hold it from the disk_live() check through the
unfreeze. Then the ioc can't be freed under us and the queue reference,
the second q_to_ioc() check and this comment go away. The comment is also
wrong: a whole-disk no-open bdev reference holds the disk device, and
disk_release() is what drops the queue reference.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC PATCH v8 2/4] selftests/bpf: add iocost cost model test
2026-09-30 7:51 [RFC PATCH v8 0/4] blk-iocost: add BPF struct_ops cost model support Tao Cui
2026-09-30 7:51 ` [RFC PATCH v8 1/4] " Tao Cui
@ 2026-09-30 7:51 ` Tao Cui
2026-10-01 0:20 ` Tejun Heo
2026-09-30 7:51 ` [RFC PATCH v8 3/4] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary Tao Cui
2026-09-30 7:51 ` [RFC PATCH v8 4/4] docs: cgroup-v2: document the iocost BPF cost model attachment Tao Cui
3 siblings, 1 reply; 10+ messages in thread
From: Tao Cui @ 2026-09-30 7:51 UTC (permalink / raw)
To: tj, josef, axboe
Cc: cgroups, linux-block, linux-kernel, bpf, andrii, eddyz87, ast,
daniel, linux-kselftest, cui.tao, cuitao, ameryhung,
alexei.starovoitov
Add an example cost model implementing the full builtin linear HDD
formula at double cost, and a test which attaches it to one device:
the dev member of the struct_ops is written through the map's
initial value before load, as hid_bpf tests do with hid_id, and
loading the struct_ops attaches the model to the device. The test
verifies the ctrl=bpf readback while attached, that a second model
on the same device fails with -EBUSY, and that detaching restores
the builtin model.
Under the same workload the doubled model charges 1.99x the
builtin model (measured 2882us -> 5722us per IO, completed IO count
halved). The example also sets the transfer cost coefficients so
the builtin sizing follows the doubled pricing.
Per-cgroup stream state uses a CGRP_STORAGE map keyed by the cgroup
of the issuing bio, so the model inherits the cgroup lifetime and
frees the state with the cgroup, so a
cgroup that comes back starts fresh. opf carries the full bio->bi_opf
including REQ_* flag bits, so the operation must be extracted with a
mask, not compared for equality.
CONFIG_BLK_CGROUP_IOCOST and CONFIG_BLK_CGROUP_IOCOST_BPF are added
to the selftest kernel config: without them vmlinux.h does not
contain iocost_model_ops and the skeletons fail to build; the
runtime skip cannot avoid a build dependency.
Requires root, cgroup v2 and a device given as major:minor in
$IOCOST_TEST_DEV.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
tools/testing/selftests/bpf/config | 2 +
.../selftests/bpf/prog_tests/iocost_model.c | 245 ++++++++++++++++++
.../selftests/bpf/progs/iocost_model.c | 140 ++++++++++
tools/testing/selftests/bpf/progs/iocost_ms.c | 159 ++++++++++++
4 files changed, 546 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/iocost_model.c
create mode 100644 tools/testing/selftests/bpf/progs/iocost_model.c
create mode 100644 tools/testing/selftests/bpf/progs/iocost_ms.c
diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index 2b883b388f90..c5634ee91e69 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -140,3 +140,5 @@ CONFIG_SMC_HS_CTRL_BPF=y
CONFIG_DIBS=y
CONFIG_DIBS_LO=y
CONFIG_PM_WAKELOCKS=y
+CONFIG_BLK_CGROUP_IOCOST=y
+CONFIG_BLK_CGROUP_IOCOST_BPF=y
diff --git a/tools/testing/selftests/bpf/prog_tests/iocost_model.c b/tools/testing/selftests/bpf/prog_tests/iocost_model.c
new file mode 100644
index 000000000000..5b7b1282fcb7
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/iocost_model.c
@@ -0,0 +1,245 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <fcntl.h>
+#include <sys/sysmacros.h>
+#include <unistd.h>
+#include "iocost_model.skel.h"
+#include "iocost_ms.skel.h"
+
+/*
+ * Read back the io.cost.model line of dev and copy the model= value
+ * into @model. Returns 0 on success.
+ */
+static int readback_model(const char *dev, char *model, size_t model_sz)
+{
+ char line[256], word[256], *m, *end;
+ FILE *fp;
+ int found = 0;
+
+ fp = fopen("/sys/fs/cgroup/io.cost.model", "r");
+ if (!fp)
+ return -1;
+ while (fgets(line, sizeof(line), fp)) {
+ if (sscanf(line, "%255s", word) == 1 && !strcmp(word, dev)) {
+ found = 1;
+ break;
+ }
+ }
+ fclose(fp);
+ if (!found)
+ return -1;
+
+ m = strstr(line, "model=");
+ if (!m)
+ return -1;
+ m += strlen("model=");
+ end = m;
+ while (*end && *end != ' ')
+ end++;
+ snprintf(model, model_sz, "%.*s", (int)(end - m), m);
+ return 0;
+}
+
+/*
+ * Write a line to io.cost.model for dev and return the errno of the
+ * write (0 on success).
+ */
+static int write_cost_model(const char *dev, const char *what)
+{
+ char buf[128], line[256], word[256];
+ FILE *fp;
+ int err = 0;
+
+ snprintf(buf, sizeof(buf), "%s %s", dev, what);
+ fp = fopen("/sys/fs/cgroup/io.cost.model", "w");
+ if (!fp)
+ return -EACCES;
+ if (fwrite(buf, 1, strlen(buf), fp) != strlen(buf))
+ err = ferror(fp) ? errno : EIO;
+ if (fclose(fp) && !err)
+ err = errno;
+ return err;
+}
+
+/*
+ * Attach the example model to one device, given as major:minor in
+ * $IOCOST_TEST_DEV: the dev member is written through the struct_ops
+ * map's initial value before load, as hid_bpf_ops does with hid_id,
+ * and attaching the struct_ops attaches the model to the device.
+ * Detaching the struct_ops restores the builtin model.
+ *
+ * Requires root, cgroup v2 and a device with iocost support.
+ */
+void serial_test_iocost_model(void)
+{
+ struct iocost_model *skel, *second;
+ unsigned int maj, min;
+ __u32 *ops_dev, *sdev;
+ int err;
+ char model[32], *dev;
+
+ dev = getenv("IOCOST_TEST_DEV");
+ if (!dev || geteuid() != 0 || sscanf(dev, "%u:%u", &maj, &min) != 2) {
+ printf("%s:SKIP:needs root and IOCOST_TEST_DEV=<maj>:<min>\n",
+ __func__);
+ test__skip();
+ return;
+ }
+
+ skel = iocost_model__open();
+ if (!ASSERT_OK_PTR(skel, "skel_open"))
+ return;
+
+ /* dev is the first member of struct iocost_model_ops */
+ ops_dev = bpf_map__initial_value(skel->maps.iocost_2x, NULL);
+ if (!ASSERT_OK_PTR(ops_dev, "initial_value")) {
+ iocost_model__destroy(skel);
+ return;
+ }
+ *ops_dev = (__u32)makedev(maj, min);
+
+ err = iocost_model__load(skel);
+ if (!ASSERT_OK(err, "skel_load")) {
+ iocost_model__destroy(skel);
+ return;
+ }
+
+ err = iocost_model__attach(skel);
+ if (err == -EOPNOTSUPP) {
+ printf("%s: device is not blk-mq or iocost unavailable, skipping\n",
+ __func__);
+ iocost_model__destroy(skel);
+ return;
+ }
+ if (ASSERT_OK(err, "attach")) {
+ /*
+ * attached: the read path reports model=bpf until the
+ * struct_ops is detached; ctrl keeps describing the
+ * builtin coefficients
+ */
+ err = readback_model(dev, model, sizeof(model));
+ if (ASSERT_OK(err, "readback"))
+ ASSERT_EQ(strcmp(model, "bpf"), 0, "model_bpf");
+
+ /* a second model on the same device fails with -EBUSY */
+ second = iocost_model__open();
+ if (ASSERT_OK_PTR(second, "second_open")) {
+ sdev = bpf_map__initial_value(
+ second->maps.iocost_2x, NULL);
+ if (!ASSERT_OK_PTR(sdev, "second_initial_value"))
+ goto out_destroy;
+ *sdev = (__u32)makedev(maj, min);
+ err = iocost_model__load(second);
+ if (ASSERT_OK(err, "second_load")) {
+ struct bpf_link *l2;
+
+ /*
+ * the kernel rejects attaching a second
+ * model to the device with EBUSY
+ */
+ l2 = bpf_map__attach_struct_ops(
+ second->maps.iocost_2x);
+ if (!ASSERT_ERR_PTR(l2, "second_ebusy"))
+ bpf_link__destroy(l2);
+ else
+ ASSERT_EQ(libbpf_get_error(l2), -EBUSY,
+ "second_ebusy_errno");
+ }
+out_destroy:
+ iocost_model__destroy(second);
+ }
+
+ /*
+ * attachment and model selection are independent:
+ * ctrl=bpf is never accepted, model=linear switches
+ * back to the builtin without detaching, model=bpf
+ * switches to the attached model again
+ */
+ err = write_cost_model(dev, "ctrl=bpf");
+ ASSERT_ERR(err, "ctrl_bpf_rejected");
+
+ err = write_cost_model(dev, "model=linear");
+ ASSERT_OK(err, "model_linear_write");
+ err = readback_model(dev, model, sizeof(model));
+ if (ASSERT_OK(err, "readback_linear"))
+ ASSERT_EQ(strcmp(model, "linear"), 0, "model_linear");
+
+ /* still attached: switching back to the model works */
+ err = write_cost_model(dev, "model=bpf");
+ ASSERT_OK(err, "model_bpf_write");
+ err = readback_model(dev, model, sizeof(model));
+ if (ASSERT_OK(err, "readback_bpf"))
+ ASSERT_EQ(strcmp(model, "bpf"), 0, "model_bpf");
+
+ iocost_model__detach(skel);
+
+ /* after detach, model=bpf fails: nothing is attached */
+ err = write_cost_model(dev, "model=bpf");
+ ASSERT_ERR(err, "model_bpf_after_detach");
+
+ err = readback_model(dev, model, sizeof(model));
+ if (ASSERT_OK(err, "readback_after_detach"))
+ ASSERT_EQ(strcmp(model, "linear"), 0, "model_linear");
+ }
+
+ iocost_model__destroy(skel);
+}
+/*
+ * Same check for the multi-stream example model. Only one model can
+ * be attached to a device at a time; both tests attach and detach, so
+ * they are serial and independent.
+ */
+void serial_test_iocost_model_streams(void)
+{
+ struct iocost_ms *skel;
+ unsigned int maj, min;
+ __u32 *ops_dev;
+ int err;
+ char model[32], *dev;
+
+ dev = getenv("IOCOST_TEST_DEV");
+ if (!dev || geteuid() != 0 || sscanf(dev, "%u:%u", &maj, &min) != 2) {
+ printf("%s:SKIP:needs root and IOCOST_TEST_DEV=<maj>:<min>\n",
+ __func__);
+ test__skip();
+ return;
+ }
+
+ skel = iocost_ms__open();
+ if (!ASSERT_OK_PTR(skel, "skel_open"))
+ return;
+
+ ops_dev = bpf_map__initial_value(skel->maps.iocost_ms, NULL);
+ if (!ASSERT_OK_PTR(ops_dev, "initial_value")) {
+ iocost_ms__destroy(skel);
+ return;
+ }
+ *ops_dev = (__u32)makedev(maj, min);
+
+ err = iocost_ms__load(skel);
+ if (!ASSERT_OK(err, "skel_load")) {
+ iocost_ms__destroy(skel);
+ return;
+ }
+
+ err = iocost_ms__attach(skel);
+ if (err == -EOPNOTSUPP) {
+ printf("%s: device is not blk-mq or iocost unavailable, skipping\n",
+ __func__);
+ iocost_ms__destroy(skel);
+ return;
+ }
+ if (ASSERT_OK(err, "attach")) {
+ err = readback_model(dev, model, sizeof(model));
+ if (ASSERT_OK(err, "readback"))
+ ASSERT_EQ(strcmp(model, "bpf"), 0, "model_bpf");
+
+ iocost_ms__detach(skel);
+
+ err = readback_model(dev, model, sizeof(model));
+ if (ASSERT_OK(err, "readback_after_detach"))
+ ASSERT_EQ(strcmp(model, "linear"), 0, "model_linear");
+ }
+
+ iocost_ms__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/iocost_model.c b/tools/testing/selftests/bpf/progs/iocost_model.c
new file mode 100644
index 000000000000..e8f222c50b39
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/iocost_model.c
@@ -0,0 +1,140 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Example iocost cost model: the builtin linear HDD formula with all
+ * costs doubled, for one device given by the dev member of the
+ * struct_ops.
+ *
+ * The constants mirror what calc_lcoefs() derives from the AUTOP_HDD
+ * defaults (rbps=174019176 rseqiops=41708 rrandiops=370, w-side
+ * analog) in vtime units where 1s == 2^37. On a rotational device
+ * still on ctrl=auto, a device with this model attached charges
+ * twice the builtin model under the same read/write workload
+ * (the builtin prices flushes as zero; this model prices them
+ * as one-page writes), so the
+ * doubled cost is a direct check that accounting goes through the
+ * BPF path. On a non-rotational device, or one with user-pinned
+ * coefficients, the ratio to the builtin model is arbitrary.
+ *
+ * The cursor handling mirrors the builtin: a zero cursor means "no
+ * previous IO", and the cursor is advanced for bios the builtin
+ * prices (READ/WRITE with a non-zero size), merged ones included, so
+ * flushes and discards leave it alone and long merged streams do not
+ * drift past the 16MB seek threshold.
+ *
+ * The model implements the full linear formula itself, including
+ * flushes: there is no fallback to the builtin model, a dataless
+ * WRITE|REQ_PREFLUSH is priced as a one-page write.
+ */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+/*
+ * VTIME_PER_SEC, IOC_PAGE_SIZE/SHIFT, IOC_SECT_TO_PAGE_SHIFT and
+ * IOCOST_COST_F_MERGE come from vmlinux.h (BTF enum constants)
+ */
+#define LCOEF_RANDIO_PAGES 4096 /* 16MB seek threshold */
+#define IOCOST_REQ_OP_MASK 0xff /* REQ_OP_MASK, not in BTF */
+
+/*
+ * DIV64_U64_ROUND_UP / DIV_ROUND_UP_ULL equivalents, folded at
+ * compile time
+ */
+#define RU(x, y) ((x) / (y) + (((x) % (y)) ? 1 : 0))
+
+#define RBPS 174019176ULL
+#define RSEQIOPS 41708ULL
+#define RRANDIOPS 370ULL
+#define WBPS 178075866ULL
+#define WSEQIOPS 42705ULL
+#define WRANDIOPS 378ULL
+
+#define RPAGE (RU(VTIME_PER_SEC, RU(RBPS, IOC_PAGE_SIZE)))
+#define RSEQIO (RU(VTIME_PER_SEC, RSEQIOPS) - RPAGE)
+#define RRANDIO (RU(VTIME_PER_SEC, RRANDIOPS) - RPAGE)
+#define WPAGE (RU(VTIME_PER_SEC, RU(WBPS, IOC_PAGE_SIZE)))
+#define WSEQIO (RU(VTIME_PER_SEC, WSEQIOPS) - WPAGE)
+#define WRANDIO (RU(VTIME_PER_SEC, WRANDIOPS) - WPAGE)
+
+/*
+ * per-cgroup cursor storage: keyed by the cgroup and freed with it,
+ * so a cgroup that comes back starts with no cursor
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_CGRP_STORAGE);
+ __uint(map_flags, BPF_F_NO_PREALLOC);
+ __type(key, int);
+ __type(value, __u64);
+} cursor_store SEC(".maps");
+
+SEC("struct_ops")
+u64 BPF_PROG(iocost_2x_calc_cost, struct bio *bio, u64 model_flags)
+{
+ u64 opf = bio->bi_opf, nbytes = bio->bi_iter.bi_size;
+ u64 sector = bio->bi_iter.bi_sector;
+ struct blkcg *blkcg = bio->bi_blkg->blkcg;
+ u64 pages, seek_pages = 0, base, coef_page, randio, cost;
+ __u64 *cursor, cur;
+ int priced;
+
+ /* builtin truncates: max(sectors >> IOC_SECT_TO_PAGE_SHIFT, 1) */
+ pages = nbytes >> IOC_PAGE_SHIFT;
+ if (!pages)
+ pages = 1;
+
+ if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ) {
+ base = RSEQIO; coef_page = RPAGE; randio = RRANDIO;
+ } else if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE) {
+ base = WSEQIO; coef_page = WPAGE; randio = WRANDIO;
+ } else {
+ /*
+ * a fully owning model must price every op; unknown
+ * ops are priced per page at the write coefficient
+ */
+ base = 0; coef_page = WPAGE; randio = 0;
+ }
+
+ /*
+ * mirror the builtin cursor semantics: seek distance is only
+ * computed against a non-zero cursor, and the cursor is
+ * advanced for bios the builtin prices (READ/WRITE with a
+ * non-zero size), merged ones included, so flushes and
+ * discards leave it alone
+ */
+ priced = (opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ ||
+ (opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE;
+ cursor = bpf_cgrp_storage_get(&cursor_store,
+ blkcg->css.cgroup, NULL,
+ BPF_LOCAL_STORAGE_GET_F_CREATE);
+ if (!cursor) {
+ if (model_flags & IOCOST_COST_F_MERGE)
+ base = 0;
+ return 2 * (base + pages * coef_page);
+ }
+ cur = *cursor;
+ if (cur && priced) {
+ seek_pages = sector > cur ? sector - cur
+ : cur - sector;
+ seek_pages >>= IOC_SECT_TO_PAGE_SHIFT;
+ if (seek_pages > LCOEF_RANDIO_PAGES)
+ base = randio;
+ }
+ if (priced && nbytes)
+ *cursor = sector + (nbytes >> 9);
+
+ if (model_flags & IOCOST_COST_F_MERGE)
+ base = 0;
+
+ cost = 2 * (base + pages * coef_page);
+ return cost;
+}
+
+SEC(".struct_ops")
+struct iocost_model_ops iocost_2x = {
+ .read_vtime_per_page = 2 * RPAGE,
+ .write_vtime_per_page = 2 * WPAGE,
+ .calc_cost = (void *)iocost_2x_calc_cost,
+};
+
+char LICENSE[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/iocost_ms.c b/tools/testing/selftests/bpf/progs/iocost_ms.c
new file mode 100644
index 000000000000..50a670edfe39
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/iocost_ms.c
@@ -0,0 +1,159 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Example multi-stream sequentiality detection cost model.
+ *
+ * The builtin model keeps a single cursor per cgroup, so two
+ * interleaved sequential readers in one cgroup are all priced random
+ * (measured 89x overcharge, 12.9x throughput collapse), while random
+ * IO inside a hot window smaller than the 16MB seek threshold is
+ * priced sequential (measured 107x undercharge). This model replaces
+ * the single cursor with a per-cgroup table of stream slots: an IO is
+ * sequential iff its sector matches the expected next sector of any
+ * tracked stream. Interleaved streams keep their own slots, and
+ * windowed random IO rarely matches a moving expectation.
+ *
+ * Stream state lives in a CGRP_STORAGE map, so it is created and
+ * freed with the cgroup. The model implements the full builtin
+ * linear formula itself, including flush pricing.
+ */
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+/*
+ * VTIME_PER_SEC, IOC_PAGE_SIZE/SHIFT, IOC_SECT_TO_PAGE_SHIFT and
+ * IOCOST_COST_F_MERGE come from vmlinux.h (BTF enum constants)
+ */
+#define IOCOST_REQ_OP_MASK 0xff /* REQ_OP_MASK, not in BTF */
+
+/*
+ * DIV64_U64_ROUND_UP / DIV_ROUND_UP_ULL equivalents, folded at
+ * compile time
+ */
+#define RU(x, y) ((x) / (y) + (((x) % (y)) ? 1 : 0))
+
+#define RBPS 174019176ULL
+#define RSEQIOPS 41708ULL
+#define RRANDIOPS 370ULL
+#define WBPS 178075866ULL
+#define WSEQIOPS 42705ULL
+#define WRANDIOPS 378ULL
+
+#define RPAGE (RU(VTIME_PER_SEC, RU(RBPS, IOC_PAGE_SIZE)))
+#define RSEQIO (RU(VTIME_PER_SEC, RSEQIOPS) - RPAGE)
+#define RRANDIO (RU(VTIME_PER_SEC, RRANDIOPS) - RPAGE)
+#define WPAGE (RU(VTIME_PER_SEC, RU(WBPS, IOC_PAGE_SIZE)))
+#define WSEQIO (RU(VTIME_PER_SEC, WSEQIOPS) - WPAGE)
+#define WRANDIO (RU(VTIME_PER_SEC, WRANDIOPS) - WPAGE)
+
+#define NSLOTS 4
+
+struct streams {
+ __u64 expected[NSLOTS]; /* next expected sector, per stream */
+ __u64 stamp[NSLOTS]; /* LRU stamp, 0 = empty */
+};
+
+/*
+ * per-cgroup stream table: keyed by the cgroup, freed with it
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_CGRP_STORAGE);
+ __uint(map_flags, BPF_F_NO_PREALLOC);
+ __type(key, int);
+ __type(value, struct streams);
+} stream_tab SEC(".maps");
+
+SEC("struct_ops")
+u64 BPF_PROG(iocost_ms_calc_cost, struct bio *bio, u64 model_flags)
+{
+ u64 opf = bio->bi_opf, nbytes = bio->bi_iter.bi_size;
+ u64 sector = bio->bi_iter.bi_sector;
+ struct blkcg *blkcg = bio->bi_blkg->blkcg;
+ struct streams *s;
+ u64 pages, base, coef_page, randio, advance, now;
+ u32 i, victim = 0, found = 0xFFFFFFFF;
+
+ if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ) {
+ base = RSEQIO; coef_page = RPAGE; randio = RRANDIO;
+ } else if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE) {
+ base = WSEQIO; coef_page = WPAGE; randio = WRANDIO;
+ } else {
+ /*
+ * a fully owning model must price every op; unknown
+ * ops are priced as per-page writes
+ */
+ base = 0; coef_page = WPAGE; randio = 0;
+ }
+ advance = nbytes >> 9; /* sectors, truncated like the builtin */
+
+ /* only bios the builtin prices participate in stream tracking */
+ if (!(((opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ ||
+ (opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE) && nbytes)) {
+ pages = nbytes >> IOC_PAGE_SHIFT;
+ if (!pages)
+ pages = 1;
+ return base + pages * coef_page;
+ }
+
+ s = bpf_cgrp_storage_get(&stream_tab, blkcg->css.cgroup, NULL,
+ BPF_LOCAL_STORAGE_GET_F_CREATE);
+ if (!s) {
+ /* no storage: price per page, truncating like the builtin */
+ pages = nbytes >> IOC_PAGE_SHIFT;
+ if (!pages)
+ pages = 1;
+ return base + pages * coef_page;
+ }
+
+ /*
+ * Slot access is lockless, mirroring the builtin single-cursor
+ * update in ioc_rqos_throttle(): concurrent CPUs submitting for
+ * the same cgroup can race on slot updates; mispricing is
+ * bounded and acceptable for an example model.
+ */
+ now = bpf_ktime_get_ns();
+ for (i = 0; i < NSLOTS; i++) {
+ if (s->expected[i] == sector && s->stamp[i]) {
+ found = i;
+ break;
+ }
+ }
+ if (found != 0xFFFFFFFF) {
+ /* sequential: keep the seq base from the op branch */
+ s->expected[found] = sector + advance;
+ s->stamp[found] = now;
+ } else {
+ base = randio;
+ for (i = 1; i < NSLOTS; i++) {
+ if (s->stamp[i] < s->stamp[victim])
+ victim = i;
+ }
+ s->expected[victim] = sector + advance;
+ s->stamp[victim] = now;
+ }
+
+ /* builtin truncates: max(sectors >> IOC_SECT_TO_PAGE_SHIFT, 1) */
+ pages = nbytes >> IOC_PAGE_SHIFT;
+ if (!pages)
+ pages = 1;
+ if (model_flags & IOCOST_COST_F_MERGE) {
+ /*
+ * merged bios skip the base cost but still advance
+ * the stream position above, so a merge at the
+ * expected sector does not make the following new IO
+ * look random
+ */
+ base = 0;
+ }
+
+ return base + pages * coef_page;
+}
+
+SEC(".struct_ops")
+struct iocost_model_ops iocost_ms = {
+ .read_vtime_per_page = RPAGE,
+ .write_vtime_per_page = WPAGE,
+ .calc_cost = (void *)iocost_ms_calc_cost,
+};
+
+char LICENSE[] SEC("license") = "GPL";
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [RFC PATCH v8 2/4] selftests/bpf: add iocost cost model test
2026-09-30 7:51 ` [RFC PATCH v8 2/4] selftests/bpf: add iocost cost model test Tao Cui
@ 2026-10-01 0:20 ` Tejun Heo
0 siblings, 0 replies; 10+ messages in thread
From: Tejun Heo @ 2026-10-01 0:20 UTC (permalink / raw)
To: Tao Cui
Cc: josef, axboe, cgroups, linux-block, linux-kernel, bpf, andrii,
eddyz87, ast, daniel, linux-kselftest, Tao Cui, ameryhung,
alexei.starovoitov
Hello, Tao.
The following is a Claude-generated review.
On Wed, 30 Sep 2026 15:51:52 +0800, Tao Cui wrote:
> Add an example cost model implementing the full builtin linear HDD
> formula at double cost, and a test which attaches it to one device:
> the dev member of the struct_ops is written through the map's
> initial value before load, as hid_bpf tests do with hid_id, and
> loading the struct_ops attaches the model to the device. The test
> verifies the ctrl=bpf readback while attached, that a second model
> on the same device fails with -EBUSY, and that detaching restores
> the builtin model.
...
> cgroup that comes back starts fresh. opf carries the full bio->bi_opf
> including REQ_* flag bits, so the operation must be extracted with a
> mask, not compared for equality.
The description is out of date. The test reads back model=bpf and checks
that ctrl=bpf is rejected, calc_cost() takes the bio so there is no opf
argument, attaching rather than loading binds the model, and the
multi-stream model and its test aren't mentioned at all.
> + if (fwrite(buf, 1, strlen(buf), fp) != strlen(buf))
> + err = ferror(fp) ? errno : EIO;
> + if (fclose(fp) && !err)
> + err = errno;
> + return err;
...
> + err = write_cost_model(dev, "ctrl=bpf");
> + ASSERT_ERR(err, "ctrl_bpf_rejected");
write_cost_model() returns a positive errno while ASSERT_ERR() wants a
negative value, so this and model_bpf_after_detach fail with "unexpected
success: 22" on a correct kernel. Return -errno. Can you double check
that the posted runner passes against the posted kernel?
> + /* dev is the first member of struct iocost_model_ops */
> + ops_dev = bpf_map__initial_value(skel->maps.iocost_2x, NULL);
The skeleton exposes the struct_ops shadow type, so
skel->struct_ops.iocost_2x->dev = ... is type checked and drops the
layout assumption. Same for iocost_ms.
> +SEC(".struct_ops")
> +struct iocost_model_ops iocost_2x = {
With a plain struct_ops map, a test that dies between attach and detach
leaves the model attached until something deletes the map element. The
hid and sched_ext selftests use ".struct_ops.link" so that closing the fd
detaches. Can you use that here too?
> + cur = *cursor;
> + if (cur && priced) {
> + seek_pages = sector > cur ? sector - cur
> + : cur - sector;
A dataless flush is REQ_OP_WRITE|REQ_PREFLUSH at sector 0 with bi_size 0,
so priced is set here. Once a cgroup's cursor is past 16MB, every fsync
is judged a random write and charged 2 * (WRANDIO + WPAGE), about 5ms,
not the one-page write the header comment describes, and iocost_ms.c
prices the same bio with the sequential base. Can you gate the seek
judgement on a non-zero size?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC PATCH v8 3/4] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary
2026-09-30 7:51 [RFC PATCH v8 0/4] blk-iocost: add BPF struct_ops cost model support Tao Cui
2026-09-30 7:51 ` [RFC PATCH v8 1/4] " Tao Cui
2026-09-30 7:51 ` [RFC PATCH v8 2/4] selftests/bpf: add iocost cost model test Tao Cui
@ 2026-09-30 7:51 ` Tao Cui
2026-10-01 0:20 ` Tejun Heo
2026-09-30 7:51 ` [RFC PATCH v8 4/4] docs: cgroup-v2: document the iocost BPF cost model attachment Tao Cui
3 siblings, 1 reply; 10+ messages in thread
From: Tao Cui @ 2026-09-30 7:51 UTC (permalink / raw)
To: tj, josef, axboe
Cc: cgroups, linux-block, linux-kernel, bpf, andrii, eddyz87, ast,
daniel, linux-kselftest, cui.tao, cuitao, ameryhung,
alexei.starovoitov
Add iocost_ioc_tick, emitted once per period from the tail of
ioc_timer_fn() with the overall controller state: period_us, vrate,
busy_level, active iocg count, usage percentage and running state.
It fires every period the controller is running, including steady
states, plus one final tick before the controller goes idle, which
makes dormancy (e.g. a device saturated entirely by uncharged IO)
directly visible.
Unlike the existing iocost tracepoints, which are state-change
driven and silent in steady state, this allows a bound cost model's
behaviour to be evaluated without drgn.
Depending on the autop profile this is 2-100 events per second per
device; the added cost outside the tracepoint static key is one
increment per active cgroup per period.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
block/blk-iocost.c | 12 ++++++++++
include/trace/events/iocost.h | 44 +++++++++++++++++++++++++++++++++++
2 files changed, 56 insertions(+)
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index bc83827cced9..64744266cfd3 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -2290,6 +2290,7 @@ static void ioc_timer_fn(struct timer_list *timer)
struct ioc_now now;
LIST_HEAD(surpluses);
int nr_debtors, nr_shortages = 0, nr_lagging = 0;
+ int nr_active = 0;
u64 usage_us_sum = 0;
u32 ppm_rthr;
u32 ppm_wthr;
@@ -2326,6 +2327,8 @@ static void ioc_timer_fn(struct timer_list *timer)
u64 vdone, vtime, usage_us;
u32 hw_active, hw_inuse;
+ nr_active++;
+
/*
* Collect unused and wind vtime closer to vnow to prevent
* iocgs from accumulating a large amount of budget.
@@ -2487,6 +2490,15 @@ static void ioc_timer_fn(struct timer_list *timer)
ioc->busy_level = clamp(ioc->busy_level, -1000, 1000);
+ /*
+ * Everything the tick reports is final here: busy_level was just
+ * computed, running and cur_period haven't changed, nr_active and
+ * usage_us_sum are complete, and vrate and period_us still hold
+ * the values this period ran in. Emit before the refresh below
+ * so the event reads the completed period directly.
+ */
+ trace_iocost_ioc_tick(ioc, &now, nr_active, usage_us_sum);
+
ioc_adjust_base_vrate(ioc, rq_wait_pct, nr_lagging, nr_shortages,
prev_busy_level, missed_ppm);
diff --git a/include/trace/events/iocost.h b/include/trace/events/iocost.h
index e772b1bc60d6..79773e82d42c 100644
--- a/include/trace/events/iocost.h
+++ b/include/trace/events/iocost.h
@@ -178,6 +178,50 @@ TRACE_EVENT(iocost_ioc_vrate_adj,
)
);
+/*
+ * Periodic per-device summary, emitted once per period from the tail of
+ * ioc_timer_fn(). Unlike the state-change events above, this fires every
+ * period the controller is running, including steady states, and carries
+ * the overall controller state so basic monitoring doesn't require drgn.
+ */
+TRACE_EVENT(iocost_ioc_tick,
+
+ TP_PROTO(struct ioc *ioc, const struct ioc_now *now,
+ int nr_active, u64 usage_us_sum),
+
+ TP_ARGS(ioc, now, nr_active, usage_us_sum),
+
+ TP_STRUCT__entry (
+ __string(devname, ioc_name(ioc))
+ __field(u64, cur_period)
+ __field(u32, period_us)
+ __field(u64, vrate)
+ __field(int, busy_level)
+ __field(int, nr_active)
+ __field(u32, usage_pct)
+ __field(int, running)
+ ),
+
+ TP_fast_assign(
+ __assign_str(devname);
+ __entry->cur_period = atomic64_read(&ioc->cur_period);
+ __entry->period_us = ioc->period_us;
+ __entry->vrate = ioc->vtime_base_rate;
+ __entry->busy_level = ioc->busy_level;
+ __entry->nr_active = nr_active;
+ __entry->usage_pct = now->now > ioc->period_at ?
+ div64_u64(usage_us_sum * 100,
+ now->now - ioc->period_at) : 0;
+ __entry->running = ioc->running;
+ ),
+
+ TP_printk("[%s] period=%llu:%uus vrate=%llu busy=%d active=%d usage=%u%% running=%d",
+ __get_str(devname), __entry->cur_period, __entry->period_us,
+ __entry->vrate, __entry->busy_level, __entry->nr_active,
+ __entry->usage_pct, __entry->running
+ )
+);
+
TRACE_EVENT(iocost_iocg_forgive_debt,
TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [RFC PATCH v8 3/4] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary
2026-09-30 7:51 ` [RFC PATCH v8 3/4] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary Tao Cui
@ 2026-10-01 0:20 ` Tejun Heo
0 siblings, 0 replies; 10+ messages in thread
From: Tejun Heo @ 2026-10-01 0:20 UTC (permalink / raw)
To: Tao Cui
Cc: josef, axboe, cgroups, linux-block, linux-kernel, bpf, andrii,
eddyz87, ast, daniel, linux-kselftest, Tao Cui, ameryhung,
alexei.starovoitov
Hello, Tao.
The following is a Claude-generated review.
On Wed, 30 Sep 2026 15:51:53 +0800, Tao Cui wrote:
> Add iocost_ioc_tick, emitted once per period from the tail of
> ioc_timer_fn() with the overall controller state: period_us, vrate,
> busy_level, active iocg count, usage percentage and running state.
> It fires every period the controller is running, including steady
> states, plus one final tick before the controller goes idle, which
> makes dormancy (e.g. a device saturated entirely by uncharged IO)
> directly visible.
...
> + __entry->running = ioc->running;
The timer only runs while ioc->running is IOC_RUNNING and the switch to
IOC_IDLE happens after the trace call, so this field always reads running
except when racing ioc_rqos_exit(). The final tick before going idle
reads active=0 running=1, and dormancy is only visible through the ticks
stopping, not through the field. Can you either drop the field or report
list_empty(&ioc->active_iocgs), and adjust the description? The event is
also emitted before the vrate adjustment and the period transition, not
from the tail of the function.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC PATCH v8 4/4] docs: cgroup-v2: document the iocost BPF cost model attachment
2026-09-30 7:51 [RFC PATCH v8 0/4] blk-iocost: add BPF struct_ops cost model support Tao Cui
` (2 preceding siblings ...)
2026-09-30 7:51 ` [RFC PATCH v8 3/4] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary Tao Cui
@ 2026-09-30 7:51 ` Tao Cui
2026-09-30 8:45 ` bot+bpf-ci
2026-10-01 0:20 ` Tejun Heo
3 siblings, 2 replies; 10+ messages in thread
From: Tao Cui @ 2026-09-30 7:51 UTC (permalink / raw)
To: tj, josef, axboe
Cc: cgroups, linux-block, linux-kernel, bpf, andrii, eddyz87, ast,
daniel, linux-kselftest, cui.tao, cuitao, ameryhung,
alexei.starovoitov
Document the BPF cost model attachment in the io.cost.model section
of the cgroup v2 documentation: attaching an iocost_model_ops
struct_ops to a device by its major:minor, the model=bpf readback
while attached (ctrl keeps describing the coefficients), that
detaching restores the builtin model, and that ctrl= writes never
select a model.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
Documentation/admin-guide/cgroup-v2.rst | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 7fe950425216..5e780bdcf324 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -2117,8 +2117,27 @@ IO Interface Files
===== ================================
ctrl "auto" or "user"
model The cost model in use - "linear"
+ or "bpf" while the BPF model
+ is in use
===== ================================
+ When CONFIG_BLK_CGROUP_IOCOST_BPF is enabled, a BPF cost model
+ can be attached to a device by loading an "iocost_model_ops"
+ struct_ops with the whole disk's major:minor in its "dev" member
+ (a partition's major:minor is rejected). Attaching creates the
+ controller if needed, like an io.cost.model write does, and
+ switches the device's pricing to the model; enabling and
+ disabling the controller stays with io.cost.qos. While a model
+ is attached, "model" selects between it and the builtin model:
+ "model=bpf" switches to the attached model and "model=linear"
+ switches back to the builtin model, and neither detaches the
+ struct_ops; only detaching removes the model, after which
+ "model=bpf" fails. "ctrl" keeps describing the builtin
+ coefficients, which are kept while the BPF model is in use and
+ take effect again when switched back, and the automatic profile
+ stepping does not switch profiles while the BPF model is in
+ use.
+
When "ctrl" is "auto", the kernel may change all parameters
dynamically. When "ctrl" is set to "user" or any other
parameters are written to, "ctrl" become "user" and the
--
2.43.0
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [RFC PATCH v8 4/4] docs: cgroup-v2: document the iocost BPF cost model attachment
2026-09-30 7:51 ` [RFC PATCH v8 4/4] docs: cgroup-v2: document the iocost BPF cost model attachment Tao Cui
@ 2026-09-30 8:45 ` bot+bpf-ci
2026-10-01 0:20 ` Tejun Heo
1 sibling, 0 replies; 10+ messages in thread
From: bot+bpf-ci @ 2026-09-30 8:45 UTC (permalink / raw)
To: cui.tao, tj, josef, axboe
Cc: cgroups, linux-block, linux-kernel, bpf, andrii, eddyz87, ast,
daniel, linux-kselftest, cui.tao, cuitao, ameryhung,
alexei.starovoitov, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1967 bytes --]
> docs: cgroup-v2: document the iocost BPF cost model attachment
TL;DR: Commit message only: it claims the docs cover detach restoring the
builtin model and ctrl= never selecting a model, but the added text may not
say either; add a sentence or trim the message.
> docs: cgroup-v2: document the iocost BPF cost model attachment
>
> Document the BPF cost model attachment in the io.cost.model section
> of the cgroup v2 documentation: attaching an iocost_model_ops
> struct_ops to a device by its major:minor, the model=bpf readback
> while attached (ctrl keeps describing the coefficients), that
> detaching restores the builtin model, and that ctrl= writes never
> select a model.
>
> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
The commit message says the patch documents "that detaching restores the
builtin model, and that ctrl= writes never select a model". Does the added
text actually cover both of those points?
The new paragraph says:
only detaching removes the model, after which "model=bpf" fails.
"ctrl" keeps describing the builtin coefficients, ...
It never says that the device goes back to the builtin model on detach,
although ioc_bpf_detach() clears ioc->model and the readback then shows
"linear".
It also never says that a ctrl= write cannot select the BPF model. That
is what the code does:
block/blk-iocost.c:ioc_cost_model_write() {
...
accepts only ctrl=auto and ctrl=user, and any other value takes
'goto unlock' and returns -EINVAL
...
}
Could a sentence be added to the documentation saying that "ctrl" accepts
only "auto" or "user" and that detaching returns the device to the builtin
model? Otherwise, could those two clauses be dropped from the commit
message?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36689638400
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [RFC PATCH v8 4/4] docs: cgroup-v2: document the iocost BPF cost model attachment
2026-09-30 7:51 ` [RFC PATCH v8 4/4] docs: cgroup-v2: document the iocost BPF cost model attachment Tao Cui
2026-09-30 8:45 ` bot+bpf-ci
@ 2026-10-01 0:20 ` Tejun Heo
1 sibling, 0 replies; 10+ messages in thread
From: Tejun Heo @ 2026-10-01 0:20 UTC (permalink / raw)
To: Tao Cui
Cc: josef, axboe, cgroups, linux-block, linux-kernel, bpf, andrii,
eddyz87, ast, daniel, linux-kselftest, Tao Cui, ameryhung,
alexei.starovoitov
Hello, Tao.
The following is a Claude-generated review.
On Wed, 30 Sep 2026 15:51:54 +0800, Tao Cui wrote:
> + When CONFIG_BLK_CGROUP_IOCOST_BPF is enabled, a BPF cost model
> + can be attached to a device by loading an "iocost_model_ops"
> + struct_ops with the whole disk's major:minor in its "dev" member
...
> + switches back to the builtin model, and neither detaches the
> + struct_ops; only detaching removes the model, after which
> + "model=bpf" fails. "ctrl" keeps describing the builtin
Attaching, not loading, binds the model. The test loads a second model
fine and only its attach fails. Device removal also removes the model,
and the user still has to tear down the struct_ops link afterwards. Can
you mention that too?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 10+ messages in thread