* [RFC PATCH v6 1/5] blk-iocost: add BPF struct_ops cost model support
2026-09-18 5:49 [RFC PATCH v6 0/5] blk-iocost: BPF struct_ops cost model Tao Cui
@ 2026-09-18 5:49 ` Tao Cui
2026-09-18 5:49 ` [RFC PATCH v6 2/5] selftests/bpf: add iocost cost model test Tao Cui
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Tao Cui @ 2026-09-18 5:49 UTC (permalink / raw)
To: tj, josef, axboe, ameryhung
Cc: cgroups, linux-block, linux-kernel, bpf, andrii, eddyz87, ast,
daniel, linux-kselftest, cui.tao, cuitao
From: Tao Cui <cuitao@kylinos.cn>
Add the iocost_model_ops struct_ops: a bound BPF model fully replaces
the builtin linear model on a device. calc_cost() receives the full
bio->bi_opf (including REQ_PREFLUSH and REQ_FUA), the IO size, the
start sector (sector_t), the issuing blkcg and the iocost-specific call
metadata (the merge-path indicator), and is called from the bio
charging path, so a model owns pricing for every IO on the device.
The completion-time request sizing used for the latency met/missed
accounting still uses the builtin linear coefficients: blk-mq has
already cleared the request's bio by the time the controller sees the
completion, so there is no issuing cgroup to pass to the model at that
point; extending the model to the sizing path is left open
by this interface. The builtin cursor is not exposed: a model
is expected to track its own stream state.
The registration and binding model follows the TCP congestion
model registration pattern: registering a struct_ops makes the model
available by its name (char name[IOCOST_MODEL_NAME_LEN], validated at
init_member), while
io.cost.model binds one registered model to a device with
"model=<name>" and unbinds with "model=linear" or "ctrl=auto/user".
Unregistering a model removes it from the registry, but its name
keeps working for io.cost.model writes until the last device unbinds,
so a coefficient-only write on a device using an unregistered model
keeps it bound instead of failing with ENOENT. The struct_ops core holds a
map reference while the model is registered, so .reg/.unreg take no
extra reference; each device binding takes one with
bpf_struct_ops_get() and releases it when the device switches back to
the builtin model, which keeps the kdata of an unregistered but
still-bound model alive.
calc_cost() runs under RCU read lock; sleepable programs are rejected
in .check_member. blkcg_online()/blkcg_offline() callbacks mirroring
the blkcg css lifecycle let models manage per-cgroup state; a model
receives these notifications exactly while at least one device has it
bound, not while it is merely registered.
- a model which does not implement calc_cost is rejected at load:
kvalue is zeroed at map allocation and function members are only
written when the BPF side provides a prog, so the missing member
stays NULL and the dispatch would call it on every bio
- the builtin model keeps the reserved name "linear" (model=linear
unbinds), so a BPF model registering under that name is rejected
- io.cost.model writes preserve the bound model: bpf_model is seeded
from the currently bound model so a coefficient-only write keeps it
bound; ctrl=auto/user and model=linear are the explicit ways back
to the builtin model
- ctrl=bpf is accepted on write (it is what the read path prints),
so a saved configuration can be restored as-is
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
block/Kconfig | 9 ++
block/Makefile | 1 +
block/blk-cgroup.c | 4 +
block/blk-iocost-bpf.c | 321 +++++++++++++++++++++++++++++++++++++
block/blk-iocost.c | 196 +++++++++++++++++++---
include/linux/blk-iocost.h | 86 ++++++++++
6 files changed, 599 insertions(+), 18 deletions(-)
create mode 100644 block/blk-iocost-bpf.c
create mode 100644 include/linux/blk-iocost.h
diff --git a/block/Kconfig b/block/Kconfig
index 70e4a66d941ff..91e808f86d28d 100644
--- a/block/Kconfig
+++ b/block/Kconfig
@@ -231,4 +231,13 @@ config BLK_ERROR_INJECTION
source "block/Kconfig.iosched"
+config BLK_CGROUP_IOCOST_BPF
+ bool "Enable BPF pluggable cost model support for the cost IO controller"
+ depends on BLK_CGROUP_IOCOST && BPF_SYSCALL && BPF_JIT && DEBUG_INFO_BTF
+ help
+ Enabling this option registers the "iocost_model_ops" BPF
+ struct_ops type, which allows a BPF program to fully replace
+ the builtin linear cost model on a device it is bound to
+ through io.cost.model.
+
endif # BLOCK
diff --git a/block/Makefile b/block/Makefile
index e7bd320e3d697..ee5cebeea006f 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -39,3 +39,4 @@ obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += blk-crypto.o blk-crypto-profile.o \
blk-crypto-sysfs.o
obj-$(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK) += blk-crypto-fallback.o
obj-$(CONFIG_BLOCK_HOLDER_DEPRECATED) += holder.o
+obj-$(CONFIG_BLK_CGROUP_IOCOST_BPF) += blk-iocost-bpf.o
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index b56db1cc6778f..b08ceb94a281b 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -32,6 +32,7 @@
#include <linux/part_stat.h>
#include "blk.h"
#include "blk-cgroup.h"
+#include <linux/blk-iocost.h>
#include "blk-ioprio.h"
#include "blk-throttle.h"
@@ -1341,6 +1342,7 @@ void blkcg_unpin_online(struct cgroup_subsys_state *blkcg_css)
*/
static void blkcg_css_offline(struct cgroup_subsys_state *css)
{
+ iocost_notify_blkcg_offline(css_to_blkcg(css));
/* this prevents anyone from attaching or migrating to this blkcg */
wb_blkcg_offline(css);
@@ -1447,6 +1449,8 @@ static int blkcg_css_online(struct cgroup_subsys_state *css)
{
struct blkcg *parent = blkcg_parent(css_to_blkcg(css));
+ iocost_notify_blkcg_online(css_to_blkcg(css));
+
/*
* blkcg_pin_online() is used to delay blkcg offline so that blkgs
* don't go offline while cgwbs are still active on them. Pin the
diff --git a/block/blk-iocost-bpf.c b/block/blk-iocost-bpf.c
new file mode 100644
index 0000000000000..444ca52ed0b08
--- /dev/null
+++ b/block/blk-iocost-bpf.c
@@ -0,0 +1,321 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * blk-iocost: BPF struct_ops plumbing for pluggable cost models.
+ *
+ * Registers the "iocost_model_ops" struct_ops type and maintains the
+ * name registry of registered models. A registered model is bound to
+ * a device through io.cost.model; see include/linux/blk-iocost.h.
+ */
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf.h>
+#include <linux/blk-iocost.h>
+
+static DEFINE_MUTEX(iocost_bpf_reg_lock);
+static LIST_HEAD(iocost_bpf_models);
+static LIST_HEAD(iocost_bpf_lifecycle);
+
+/*
+ * One node per registered model. refcount: 1 while registered
+ * (dropped in .unreg) plus one per bound device (dropped in
+ * iocost_bpf_model_put(), which pairs its bpf_struct_ops_get(), so the
+ * kdata of an unregistered model stays alive while any device is still
+ * bound to it). The node sits on the lifecycle notify list exactly
+ * while at least one device has the model bound.
+ */
+struct iocost_bpf_model {
+ struct list_head list; /* name registry */
+ struct list_head lifecycle; /* lifecycle notify list */
+ const struct iocost_model_ops *ops;
+ refcount_t refs;
+};
+
+/*
+ * Take a name and acquire a binding reference on the model it selects.
+ * The registry is searched first; a model which is unregistered but
+ * still bound to a device remains selectable by name until the last
+ * device unbinds, so a coefficient-only write on such a device keeps
+ * the model bound instead of failing with ENOENT. The registry lock
+ * is held across the lookup and bpf_struct_ops_get() so the model
+ * cannot be unregistered in between. Returns the ops or an ERR_PTR.
+ */
+const struct iocost_model_ops *iocost_bpf_model_get(const char *name)
+{
+ struct iocost_bpf_model *m;
+ const struct iocost_model_ops *ops = ERR_PTR(-ENOENT);
+
+ mutex_lock(&iocost_bpf_reg_lock);
+ list_for_each_entry(m, &iocost_bpf_models, list) {
+ if (!strcmp(m->ops->name, name))
+ goto found;
+ }
+ list_for_each_entry(m, &iocost_bpf_lifecycle, lifecycle) {
+ if (!strcmp(m->ops->name, name))
+ goto found;
+ }
+ mutex_unlock(&iocost_bpf_reg_lock);
+ return ops;
+found:
+ if (bpf_struct_ops_get(m->ops)) {
+ refcount_inc(&m->refs);
+ /* first bind: join the notify list */
+ if (refcount_read(&m->refs) == 2 &&
+ list_empty(&m->lifecycle))
+ list_add(&m->lifecycle, &iocost_bpf_lifecycle);
+ ops = m->ops;
+ }
+ mutex_unlock(&iocost_bpf_reg_lock);
+ return ops;
+}
+
+static struct iocost_bpf_model *
+iocost_bpf_model_lookup(const struct iocost_model_ops *ops)
+{
+ struct iocost_bpf_model *m;
+
+ list_for_each_entry(m, &iocost_bpf_models, list) {
+ if (m->ops == ops)
+ return m;
+ }
+ return NULL;
+}
+
+/*
+ * Lifecycle notifications walk the lifecycle list, which keeps a node
+ * for as long as any device has the model bound, so an unregistered
+ * but still-bound model keeps receiving blkcg online/offline.
+ */
+void iocost_notify_blkcg_online(struct blkcg *blkcg)
+{
+ struct iocost_bpf_model *m;
+
+ guard(mutex)(&iocost_bpf_reg_lock);
+ list_for_each_entry(m, &iocost_bpf_lifecycle, lifecycle) {
+ if (m->ops->blkcg_online)
+ m->ops->blkcg_online(blkcg);
+ }
+}
+
+void iocost_notify_blkcg_offline(struct blkcg *blkcg)
+{
+ struct iocost_bpf_model *m;
+
+ guard(mutex)(&iocost_bpf_reg_lock);
+ list_for_each_entry(m, &iocost_bpf_lifecycle, lifecycle) {
+ if (m->ops->blkcg_offline)
+ m->ops->blkcg_offline(blkcg);
+ }
+}
+
+static int bpf_iocost_model_init(struct btf *btf)
+{
+ s32 type_id;
+
+ type_id = btf_find_by_name_kind(btf, "iocost_model_ops", BTF_KIND_STRUCT);
+ if (type_id < 0)
+ return -EINVAL;
+ return 0;
+}
+
+static bool bpf_iocost_is_valid_access(int off, int size,
+ enum bpf_access_type type,
+ const struct bpf_prog *prog,
+ struct bpf_insn_access_aux *info)
+{
+ return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
+}
+
+/*
+ * No iocost-specific helpers; bpf_base_func_proto already covers the
+ * cgroup storage helpers under CONFIG_CGROUPS.
+ */
+static const struct bpf_func_proto *
+bpf_iocost_get_func_proto(enum bpf_func_id func_id,
+ const struct bpf_prog *prog)
+{
+ return bpf_base_func_proto(func_id, prog);
+}
+
+static int bpf_iocost_check_member(const struct btf_type *t,
+ const struct btf_member *member,
+ const struct bpf_prog *prog)
+{
+ /* calc_cost() is called with RCU read lock held */
+ if (prog->sleepable)
+ return -EINVAL;
+ return 0;
+}
+
+static int bpf_iocost_init_member(const struct btf_type *t,
+ const struct btf_member *member,
+ void *kdata, const void *udata)
+{
+ struct iocost_model_ops *ops = kdata;
+ const struct iocost_model_ops *uops = udata;
+ u32 moff = __btf_member_bit_offset(t, member) / 8;
+
+ switch (moff) {
+ case offsetof(struct iocost_model_ops, name):
+ if (bpf_obj_name_cpy(ops->name, uops->name,
+ sizeof(ops->name)) <= 0)
+ return -EINVAL;
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * kvalue is zeroed at map allocation and function members are only
+ * written when the BPF side provides a prog, so a model which did
+ * not implement calc_cost leaves it NULL. The dispatch would call
+ * it on every bio, so reject it here.
+ */
+static int bpf_iocost_validate(void *kdata)
+{
+ struct iocost_model_ops *ops = kdata;
+
+ if (!ops->calc_cost)
+ return -EINVAL;
+ /* "linear" is the builtin model; model=linear unbinds */
+ if (!strcmp(ops->name, "linear"))
+ return -EINVAL;
+ return 0;
+}
+
+/*
+ * The struct_ops core holds a reference on the map while the model is
+ * registered, so kdata stays valid until .unreg returns; no extra
+ * reference is needed here.
+ */
+static int bpf_iocost_reg(void *kdata, struct bpf_link *link)
+{
+ struct iocost_model_ops *ops = kdata;
+ struct iocost_bpf_model *m;
+ int ret = 0;
+
+ m = kzalloc_obj(struct iocost_bpf_model, GFP_KERNEL);
+ if (!m)
+ return -ENOMEM;
+ refcount_set(&m->refs, 1);
+ INIT_LIST_HEAD(&m->lifecycle);
+
+ mutex_lock(&iocost_bpf_reg_lock);
+ {
+ struct iocost_bpf_model *other;
+
+ list_for_each_entry(other, &iocost_bpf_models, list) {
+ if (!strcmp(other->ops->name, ops->name)) {
+ ret = -EEXIST;
+ break;
+ }
+ }
+ }
+ if (!ret) {
+ m->ops = ops;
+ list_add(&m->list, &iocost_bpf_models);
+ }
+ mutex_unlock(&iocost_bpf_reg_lock);
+
+ if (ret)
+ kfree(m);
+ return ret;
+}
+
+/*
+ * Unregistering drops the registration reference. If a device is
+ * still bound, the binding references keep the node (and the kdata,
+ * through their bpf_struct_ops_get()) alive and it keeps receiving
+ * blkcg online/offline notifications; otherwise the node is freed.
+ */
+static void bpf_iocost_unreg(void *kdata, struct bpf_link *link)
+{
+ struct iocost_model_ops *ops = kdata;
+ struct iocost_bpf_model *m;
+
+ mutex_lock(&iocost_bpf_reg_lock);
+ m = iocost_bpf_model_lookup(ops);
+ if (m) {
+ /*
+ * keep the linkage queryable so model_put() can tell a
+ * still-registered node (one ref held by the registry)
+ * from one kept alive only by device bindings
+ */
+ list_del_init(&m->list);
+ if (refcount_dec_and_test(&m->refs))
+ kfree(m);
+ }
+ mutex_unlock(&iocost_bpf_reg_lock);
+}
+
+void iocost_bpf_model_put(const struct iocost_model_ops *ops)
+{
+ struct iocost_bpf_model *m;
+
+ mutex_lock(&iocost_bpf_reg_lock);
+ list_for_each_entry(m, &iocost_bpf_lifecycle, lifecycle) {
+ if (m->ops == ops)
+ break;
+ }
+ if (&m->lifecycle != &iocost_bpf_lifecycle) {
+ bool freed = refcount_dec_and_test(&m->refs);
+
+ /*
+ * leave the notify list with the last binding: refs is
+ * now zero (node freed) or the registry's single one
+ */
+ if (freed || (!list_empty(&m->list) &&
+ refcount_read(&m->refs) == 1))
+ list_del_init(&m->lifecycle);
+ mutex_unlock(&iocost_bpf_reg_lock);
+ bpf_struct_ops_put(ops);
+ if (freed)
+ kfree(m);
+ return;
+ }
+ mutex_unlock(&iocost_bpf_reg_lock);
+}
+
+static const struct bpf_verifier_ops bpf_iocost_verifier_ops = {
+ .get_func_proto = bpf_iocost_get_func_proto,
+ .is_valid_access = bpf_iocost_is_valid_access,
+};
+
+static u64 bpf_iocost_calc_cost_stub(u64 opf, u64 nbytes, u64 sector,
+ struct blkcg *blkcg, u64 flags)
+{
+ return 0;
+}
+
+static void bpf_iocost_blkcg_online_stub(struct blkcg *blkcg) { }
+static void bpf_iocost_blkcg_offline_stub(struct blkcg *blkcg) { }
+
+static struct iocost_model_ops __bpf_ops_iocost_model_ops = {
+ .calc_cost = bpf_iocost_calc_cost_stub,
+ .blkcg_online = bpf_iocost_blkcg_online_stub,
+ .blkcg_offline = bpf_iocost_blkcg_offline_stub,
+};
+
+static struct bpf_struct_ops bpf_iocost_model_ops = {
+ .verifier_ops = &bpf_iocost_verifier_ops,
+ .init = bpf_iocost_model_init,
+ .check_member = bpf_iocost_check_member,
+ .init_member = bpf_iocost_init_member,
+ .validate = bpf_iocost_validate,
+ .reg = bpf_iocost_reg,
+ .unreg = bpf_iocost_unreg,
+ .name = "iocost_model_ops",
+ .cfi_stubs = &__bpf_ops_iocost_model_ops,
+ .owner = THIS_MODULE,
+};
+
+static int __init bpf_iocost_init(void)
+{
+ return register_bpf_struct_ops(&bpf_iocost_model_ops, iocost_model_ops);
+}
+late_initcall(bpf_iocost_init);
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 2745bffcd5eef..307065c592950 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -177,6 +177,7 @@
#include <linux/timer.h>
#include <linux/time64.h>
#include <linux/parser.h>
+#include <linux/blk-iocost.h>
#include <linux/sched/signal.h>
#include <asm/local.h>
#include <asm/local64.h>
@@ -445,6 +446,11 @@ struct ioc {
int autop_idx;
bool user_qos_params:1;
bool user_cost_model:1;
+
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ /* bound BPF cost model, NULL = builtin linear model */
+ const struct iocost_model_ops __rcu *model;
+#endif
};
struct iocg_pcpu_stat {
@@ -2571,10 +2577,28 @@ static void calc_vtime_cost_builtin(struct bio *bio, struct ioc_gq *iocg,
static u64 calc_vtime_cost(struct bio *bio, struct ioc_gq *iocg, bool is_merge)
{
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ const struct iocost_model_ops *model;
u64 cost;
- calc_vtime_cost_builtin(bio, iocg, is_merge, &cost);
- return cost;
+ rcu_read_lock();
+ model = rcu_dereference(iocg->ioc->model);
+ if (model) {
+ cost = model->calc_cost(bio->bi_opf, bio->bi_iter.bi_size,
+ bio->bi_iter.bi_sector,
+ iocg_to_blkg(iocg)->blkcg,
+ is_merge ? IOCOST_COST_F_MERGE : 0);
+ rcu_read_unlock();
+ return min(cost, VTIME_PER_SEC);
+ }
+ rcu_read_unlock();
+#endif
+ {
+ u64 cost;
+
+ calc_vtime_cost_builtin(bio, iocg, is_merge, &cost);
+ return cost;
+ }
}
static void calc_size_vtime_cost_builtin(struct request *rq, struct ioc *ioc,
@@ -2596,10 +2620,28 @@ static void calc_size_vtime_cost_builtin(struct request *rq, struct ioc *ioc,
static u64 calc_size_vtime_cost(struct request *rq, struct ioc *ioc)
{
- u64 cost;
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ const struct iocost_model_ops *model;
- calc_size_vtime_cost_builtin(rq, ioc, &cost);
- return cost;
+ rcu_read_lock();
+ model = rcu_dereference(ioc->model);
+ if (model && rq->bio && rq->bio->bi_blkg) {
+ u64 cost;
+
+ cost = model->calc_cost(rq->cmd_flags, blk_rq_bytes(rq),
+ blk_rq_pos(rq),
+ rq->bio->bi_blkg->blkcg, 0);
+ rcu_read_unlock();
+ return min(cost, VTIME_PER_SEC);
+ }
+ rcu_read_unlock();
+#endif
+ {
+ u64 cost;
+
+ calc_size_vtime_cost_builtin(rq, ioc, &cost);
+ return cost;
+ }
}
enum over_budget_action {
@@ -2900,6 +2942,19 @@ static void ioc_rqos_exit(struct rq_qos *rqos)
timer_shutdown_sync(&ioc->timer);
free_percpu(ioc->pcpu_stat);
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ {
+ const struct iocost_model_ops *model;
+
+ spin_lock_irq(&ioc->lock);
+ model = rcu_dereference_protected(ioc->model,
+ lockdep_is_held(&ioc->lock));
+ rcu_assign_pointer(ioc->model, NULL);
+ spin_unlock_irq(&ioc->lock);
+ if (model)
+ iocost_bpf_model_put(model);
+ }
+#endif
kfree(ioc);
}
@@ -3438,12 +3493,30 @@ static u64 ioc_cost_model_prfill(struct seq_file *sf,
return 0;
spin_lock_irq(&ioc->lock);
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ {
+ const struct iocost_model_ops *model =
+ rcu_dereference_protected(ioc->model,
+ lockdep_is_held(&ioc->lock));
+
+ seq_printf(sf, "%s ctrl=%s model=%s "
+ "rbps=%llu rseqiops=%llu rrandiops=%llu "
+ "wbps=%llu wseqiops=%llu wrandiops=%llu\n",
+ dname, model ? "bpf" :
+ ioc->user_cost_model ? "user" : "auto",
+ model ? model->name : "linear",
+ u[I_LCOEF_RBPS], u[I_LCOEF_RSEQIOPS],
+ u[I_LCOEF_RRANDIOPS], u[I_LCOEF_WBPS],
+ u[I_LCOEF_WSEQIOPS], u[I_LCOEF_WRANDIOPS]);
+ }
+#else
seq_printf(sf, "%s ctrl=%s model=linear "
"rbps=%llu rseqiops=%llu rrandiops=%llu "
"wbps=%llu wseqiops=%llu wrandiops=%llu\n",
dname, ioc->user_cost_model ? "user" : "auto",
u[I_LCOEF_RBPS], u[I_LCOEF_RSEQIOPS], u[I_LCOEF_RRANDIOPS],
u[I_LCOEF_WBPS], u[I_LCOEF_WSEQIOPS], u[I_LCOEF_WRANDIOPS]);
+#endif
spin_unlock_irq(&ioc->lock);
return 0;
}
@@ -3457,6 +3530,48 @@ static int ioc_cost_model_show(struct seq_file *sf, void *v)
return 0;
}
+/*
+ * Resolve the model name and take a reference on the new model before
+ * anything is applied, so a bad name rejects the whole write. The
+ * registry lookup takes the registration mutex and must stay outside
+ * ioc->lock. The returned model is per-write state passed back into
+ * ioc_bpf_model_commit(), which performs the pointer swap under
+ * ioc->lock so concurrent writers cannot interleave a half-applied
+ * configuration, and the caller drops the old model's reference
+ * afterwards.
+ */
+static const struct iocost_model_ops *
+ioc_bpf_model_prepare(const char *name)
+{
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ if (!name[0])
+ return NULL;
+ return iocost_bpf_model_get(name);
+#else
+ return name[0] ? ERR_PTR(-ENOENT) : NULL;
+#endif
+}
+
+/*
+ * Swap in the model returned by ioc_bpf_model_prepare(). Called with
+ * ioc->lock held; dropping the old model's reference may sleep, so
+ * the caller does it after releasing the lock.
+ */
+static const struct iocost_model_ops *
+ioc_bpf_model_commit(struct ioc *ioc, const struct iocost_model_ops *new)
+{
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ const struct iocost_model_ops *old;
+
+ old = rcu_dereference_protected(ioc->model,
+ lockdep_is_held(&ioc->lock));
+ rcu_assign_pointer(ioc->model, new);
+ return old;
+#else
+ return NULL;
+#endif
+}
+
static const match_table_t cost_ctrl_tokens = {
{ COST_CTRL, "ctrl=%s" },
{ COST_MODEL, "model=%s" },
@@ -3482,6 +3597,7 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
struct ioc *ioc;
u64 u[NR_I_LCOEFS];
bool user;
+ char bpf_model[IOCOST_MODEL_NAME_LEN];
char *body, *p;
int ret;
@@ -3512,6 +3628,25 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
spin_lock_irq(&ioc->lock);
memcpy(u, ioc->params.i_lcoefs, sizeof(u));
user = ioc->user_cost_model;
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+ {
+ const struct iocost_model_ops *model =
+ rcu_dereference_protected(ioc->model,
+ lockdep_is_held(&ioc->lock));
+
+ /*
+ * seed from the bound model so a coefficient-only write
+ * keeps it bound; ctrl=auto/user and model=linear are
+ * the explicit ways back to the builtin model
+ */
+ if (model)
+ strscpy(bpf_model, model->name, sizeof(bpf_model));
+ else
+ bpf_model[0] = '\0';
+ }
+#else
+ bpf_model[0] = '\0';
+#endif
ret = -EINVAL;
@@ -3527,17 +3662,27 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
switch (match_token(p, cost_ctrl_tokens, args)) {
case COST_CTRL:
match_strlcpy(buf, &args[0], sizeof(buf));
- if (!strcmp(buf, "auto"))
+ if (!strcmp(buf, "auto")) {
user = false;
- else if (!strcmp(buf, "user"))
+ } else if (!strcmp(buf, "user")) {
user = true;
- else
+ } else if (!strcmp(buf, "bpf")) {
+ /* readback value; keep the bound model */
+ continue;
+ } else {
goto unlock;
+ }
+ /* ctrl=auto/user: explicit return to builtin */
+ bpf_model[0] = '\0';
continue;
case COST_MODEL:
match_strlcpy(buf, &args[0], sizeof(buf));
- if (strcmp(buf, "linear"))
- goto unlock;
+ if (!strcmp(buf, "linear")) {
+ /* back to the builtin linear model */
+ bpf_model[0] = '\0';
+ continue;
+ }
+ match_strlcpy(bpf_model, &args[0], sizeof(bpf_model));
continue;
}
@@ -3550,19 +3695,34 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
user = true;
}
- if (user) {
- memcpy(ioc->params.i_lcoefs, u, sizeof(u));
- ioc->user_cost_model = true;
- } else {
- ioc->user_cost_model = false;
- }
- ioc_refresh_params(ioc, true);
-
ret = 0;
unlock:
spin_unlock_irq(&ioc->lock);
+ /* resolve the name outside ioc->lock; see ioc_bpf_model_prepare() */
+ if (!ret) {
+ const struct iocost_model_ops *new, *old;
+
+ new = ioc_bpf_model_prepare(bpf_model);
+ if (IS_ERR(new)) {
+ ret = PTR_ERR(new);
+ } else {
+ spin_lock_irq(&ioc->lock);
+ if (user) {
+ memcpy(ioc->params.i_lcoefs, u, sizeof(u));
+ ioc->user_cost_model = true;
+ } else {
+ ioc->user_cost_model = false;
+ }
+ ioc_refresh_params(ioc, true);
+ old = ioc_bpf_model_commit(ioc, new);
+ spin_unlock_irq(&ioc->lock);
+ if (IS_ENABLED(CONFIG_BLK_CGROUP_IOCOST_BPF) && old)
+ iocost_bpf_model_put(old);
+ }
+ }
+
blk_mq_unquiesce_queue(q);
blk_mq_unfreeze_queue(q, memflags);
diff --git a/include/linux/blk-iocost.h b/include/linux/blk-iocost.h
new file mode 100644
index 0000000000000..e6dc6772ef5c9
--- /dev/null
+++ b/include/linux/blk-iocost.h
@@ -0,0 +1,86 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_BLK_IOCOST_H
+#define _LINUX_BLK_IOCOST_H
+
+#include <linux/types.h>
+#include <linux/blk_types.h>
+
+#define IOCOST_MODEL_NAME_LEN 16
+
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+
+struct blkcg;
+
+/*
+ * Pluggable cost model interface for blk-iocost.
+ *
+ * A BPF struct_ops implementation registered against "iocost_model_ops"
+ * fully replaces the builtin linear model on the devices it is bound to
+ * through io.cost.model. The model owns pricing for every IO on a bound
+ * device: it prices all operations, including flushes, and it is called
+ * from the bio charging path. The completion-time request sizing for
+ * the latency met/missed accounting still uses the builtin coefficients
+ * (the request's bio, and with it the issuing cgroup, is gone by then);
+ * extending the model there is left open by this interface.
+ *
+ * calc_cost() is called from the IO submission path with RCU read lock
+ * held and must not sleep. It returns the cost of the IO in vtime
+ * units, where 1 second of device time equals VTIME_PER_SEC (2^37,
+ * available to BPF programs through vmlinux.h). The returned value is
+ * clamped to 1 second of device time per IO.
+ *
+ * The model is passed the blkcg of the issuing cgroup so it can keep
+ * per-cgroup state. State keyed by the blkcg alone is shared across
+ * every device the model is bound to, unlike the builtin cursor which
+ * is per (cgroup, device). blkcg_online()/blkcg_offline() are optional
+ * callbacks mirroring the blkcg css lifecycle: they are delivered
+ * only while at least one device has the model bound, and state
+ * created on online (or lazily on first use) must be released on
+ * offline.
+ *
+ * The registration and binding model follows the TCP congestion
+ * control framework: registering a struct_ops makes the model available
+ * by its name, while io.cost.model binds one registered model to a
+ * device. Unregistering removes the name from the registry; devices
+ * already bound keep using it until switched back to the builtin
+ * model, and the name remains usable for io.cost.model writes (so a
+ * coefficient-only write keeps such a model bound) until the last
+ * device unbinds.
+ */
+
+/*
+ * iocost-specific call metadata for calc_cost()'s model_flags
+ * argument; everything else, including REQ_PREFLUSH/REQ_FUA, is
+ * already present in the opf argument. An enum so the value is
+ * exported through BTF and BPF models can use it from vmlinux.h.
+ */
+enum {
+ IOCOST_COST_F_MERGE = 1 << 0, /* called from merge path */
+};
+
+struct iocost_model_ops {
+ u64 (*calc_cost)(u64 opf, u64 nbytes, sector_t sector,
+ struct blkcg *blkcg, u64 model_flags);
+ void (*blkcg_online)(struct blkcg *blkcg);
+ void (*blkcg_offline)(struct blkcg *blkcg);
+
+ /* model name, used to select the model through io.cost.model */
+ char name[IOCOST_MODEL_NAME_LEN];
+};
+
+const struct iocost_model_ops *iocost_bpf_model_get(const char *name);
+void iocost_bpf_model_put(const struct iocost_model_ops *ops);
+void iocost_notify_blkcg_online(struct blkcg *blkcg);
+void iocost_notify_blkcg_offline(struct blkcg *blkcg);
+
+#else /* CONFIG_BLK_CGROUP_IOCOST_BPF */
+
+struct blkcg;
+struct iocost_model_ops;
+
+static inline void iocost_bpf_model_put(const struct iocost_model_ops *ops) { }
+static inline void iocost_notify_blkcg_online(struct blkcg *blkcg) { }
+static inline void iocost_notify_blkcg_offline(struct blkcg *blkcg) { }
+
+#endif /* CONFIG_BLK_CGROUP_IOCOST_BPF */
+#endif /* _LINUX_BLK_IOCOST_H */
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [RFC PATCH v6 2/5] selftests/bpf: add iocost cost model test
2026-09-18 5:49 [RFC PATCH v6 0/5] blk-iocost: BPF struct_ops cost model Tao Cui
2026-09-18 5:49 ` [RFC PATCH v6 1/5] blk-iocost: add BPF struct_ops cost model support Tao Cui
@ 2026-09-18 5:49 ` Tao Cui
2026-09-18 5:49 ` [RFC PATCH v6 3/5] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary Tao Cui
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Tao Cui @ 2026-09-18 5:49 UTC (permalink / raw)
To: tj, josef, axboe, ameryhung
Cc: cgroups, linux-block, linux-kernel, bpf, andrii, eddyz87, ast,
daniel, linux-kselftest, cui.tao, cuitao
From: Tao Cui <cuitao@kylinos.cn>
Add an example cost model implementing the full builtin linear HDD
formula at double cost, including flush pricing, and a runner which
registers it as a struct_ops and binds it to a device through
io.cost.model with "model=iocost_2x", verifying the readback and
restoring "model=linear" afterwards, checking the write error and
errno of every step including the restoration. Binding an unknown
model name is verified to fail with ENOENT. Under the same workload
the doubled model charges exactly twice the builtin model (verified
2882us -> 5722us per IO, completed IO count halved).
Per-cgroup stream state uses a CGRP_STORAGE map keyed by the cgroup
of the blkcg argument, so the model inherits the cgroup lifetime and
never leaks or reuses stale state. opf carries the full
bio->bi_opf including REQ_* flag bits, so the operation must be
extracted with a mask, not compared for equality.
All writes go through write(2) rather than stdio, since the kernel's
rejection happens in the write() syscall, not in the userspace buffer
copy. CONFIG_BLK_CGROUP_IOCOST and CONFIG_BLK_CGROUP_IOCOST_BPF are
added to the selftest kernel config: without them vmlinux.h does not
contain iocost_model_ops and the skeletons fail to build; the runtime
skip cannot avoid a build dependency.
The runner is skipped unless $IOCOST_TEST_DEV gives a major:minor of
a device with iocost enabled.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
tools/testing/selftests/bpf/config | 2 +
.../selftests/bpf/prog_tests/iocost_model.c | 166 ++++++++++++++++++
.../selftests/bpf/progs/iocost_model.c | 136 ++++++++++++++
3 files changed, 304 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/iocost_model.c
create mode 100644 tools/testing/selftests/bpf/progs/iocost_model.c
diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index 2f79688dcf7ce..67a630cb56148 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -138,3 +138,5 @@ CONFIG_SMC_HS_CTRL_BPF=y
CONFIG_DIBS=y
CONFIG_DIBS_LO=y
CONFIG_PM_WAKELOCKS=y
+CONFIG_BLK_CGROUP_IOCOST=y
+CONFIG_BLK_CGROUP_IOCOST_BPF=y
diff --git a/tools/testing/selftests/bpf/prog_tests/iocost_model.c b/tools/testing/selftests/bpf/prog_tests/iocost_model.c
new file mode 100644
index 0000000000000..85b5ea7482eed
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/iocost_model.c
@@ -0,0 +1,166 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <ctype.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include "iocost_model.skel.h"
+
+/*
+ * Write a line to io.cost.model with write(2) and return the errno of
+ * the failed write, or 0 on success. stdio is not used here on
+ * purpose: the kernel's rejection happens in the write() syscall,
+ * not in the userspace buffer copy, and every write, including
+ * the error paths of the callers below, is checked.
+ */
+static int write_cost_model(const char *buf)
+{
+ int fd, err = 0;
+ ssize_t n;
+
+ fd = open("/sys/fs/cgroup/io.cost.model", O_WRONLY);
+ if (fd < 0)
+ return -errno; /* negative: the file is not there */
+ n = write(fd, buf, strlen(buf));
+ if (n < 0)
+ err = errno;
+ close(fd);
+ return err;
+}
+
+/*
+ * Read back the io.cost.model line of dev and copy the model= value
+ * into @model. Returns 0 on success.
+ */
+static int readback_model(const char *dev, char *model, size_t model_sz)
+{
+ char line[256], word[256], *m, *end;
+ FILE *fp;
+ int found = 0;
+
+ fp = fopen("/sys/fs/cgroup/io.cost.model", "r");
+ if (!fp)
+ return -1;
+ while (fgets(line, sizeof(line), fp)) {
+ if (sscanf(line, "%255s", word) == 1 && !strcmp(word, dev)) {
+ found = 1;
+ break;
+ }
+ }
+ fclose(fp);
+ if (!found)
+ return -1;
+
+ m = strstr(line, "model=");
+ if (!m)
+ return -1;
+ m += strlen("model=");
+ end = m;
+ while (*end && !isspace(*end))
+ end++;
+ snprintf(model, model_sz, "%.*s", (int)(end - m), m);
+ return 0;
+}
+
+/*
+ * Bind the named model to a device with "model=<name>", verify the
+ * readback and restore the builtin model. Returns 0 on success.
+ */
+static int bind_model(const char *dev, const char *name)
+{
+ char buf[300], got[64];
+ int err;
+
+ snprintf(buf, sizeof(buf), "%s model=%s\n", dev, name);
+ err = write_cost_model(buf);
+ if (err)
+ return err > 0 ? -err : -EIO;
+ err = readback_model(dev, got, sizeof(got));
+ if (!err && !strcmp(got, name)) {
+ /*
+ * restore the builtin linear model on every path so
+ * the device is not left bound to a model the caller
+ * may unregister right after; the write is checked too
+ */
+ snprintf(buf, sizeof(buf), "%s model=linear\n", dev);
+ err = write_cost_model(buf);
+ return err > 0 ? -err : (err ? -EIO : 0);
+ }
+
+ /*
+ * bind failed or does not read back: still restore, but report
+ * the original failure rather than the restore result
+ */
+ snprintf(buf, sizeof(buf), "%s model=linear\n", dev);
+ write_cost_model(buf);
+ return err ? -EIO : -EINVAL;
+}
+
+/*
+ * True when the device line exists in io.cost.qos and iocost is
+ * enabled on it.
+ */
+static int dev_has_iocost(const char *dev)
+{
+ char line[256], word[256];
+ FILE *fp;
+ int found = 0;
+
+ fp = fopen("/sys/fs/cgroup/io.cost.qos", "r");
+ if (!fp)
+ return 0;
+ while (fgets(line, sizeof(line), fp)) {
+ if (sscanf(line, "%255s", word) == 1 && !strcmp(word, dev)) {
+ found = strstr(line, "enable=1") != NULL;
+ break;
+ }
+ }
+ fclose(fp);
+ return found;
+}
+
+/*
+ * Bind the 2x example model and verify the io.cost.model readback.
+ * IO accounting itself is not checked here; it needs a device doing
+ * real IO under iocost and is covered by the kernel-side validation
+ * described in the cover letter.
+ *
+ * Requires root, cgroup v2 and a device with iocost support. The
+ * device must be given as major:minor in $IOCOST_TEST_DEV, otherwise
+ * the test is skipped.
+ */
+void serial_test_iocost_model(void)
+{
+ struct iocost_model *skel;
+ char buf[300], *dev;
+ int err;
+
+ dev = getenv("IOCOST_TEST_DEV");
+ if (!dev || geteuid() != 0) {
+ test__skip();
+ return;
+ }
+ if (!dev_has_iocost(dev)) {
+ printf("skip: %s has no iocost enabled\n", dev);
+ test__skip();
+ return;
+ }
+
+ /*
+ * negative: binding an unknown model name must be rejected,
+ * so a typo cannot silently disable cost model updates
+ */
+ snprintf(buf, sizeof(buf), "%s model=no_such_model\n", dev);
+ err = write_cost_model(buf);
+ ASSERT_EQ(err > 0 ? err : 0, ENOENT, "unknown_model_rejected");
+
+ skel = iocost_model__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "skel_open_load"))
+ return;
+
+ /* attaching the struct_ops registers the model by name */
+ err = iocost_model__attach(skel);
+ if (ASSERT_OK(err, "attach"))
+ ASSERT_OK(bind_model(dev, "iocost_2x"), "bind_and_readback");
+
+ iocost_model__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/iocost_model.c b/tools/testing/selftests/bpf/progs/iocost_model.c
new file mode 100644
index 0000000000000..66fab4711c0b5
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/iocost_model.c
@@ -0,0 +1,136 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Example iocost cost model: the builtin linear HDD formula with all
+ * costs doubled.
+ *
+ * The constants mirror what calc_lcoefs() derives from the AUTOP_HDD
+ * defaults (rbps=174019176 rseqiops=41708 rrandiops=370, w-side
+ * analog) in vtime units where 1s == 2^37. On a rotational device
+ * still on ctrl=auto, a device bound to this model through
+ * io.cost.model charges exactly twice the builtin model under the
+ * same workload, so the doubled cost is a direct check that
+ * accounting goes through the BPF path. On a non-rotational device,
+ * or one with user-pinned coefficients, the ratio to the builtin
+ * model is arbitrary.
+ *
+ * The cursor handling mirrors the builtin: a zero cursor means "no
+ * previous IO", and the cursor is only advanced for READ and WRITE
+ * bios the builtin prices, so flushes and discards leave it alone.
+ *
+ * The model implements the full linear formula itself, including
+ * flushes: there is no fallback to the builtin model, a dataless
+ * WRITE|REQ_PREFLUSH is priced as a one-page write.
+ */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+/*
+ * VTIME_PER_SEC, IOC_PAGE_SIZE/SHIFT, IOC_SECT_TO_PAGE_SHIFT and
+ * IOCOST_COST_F_MERGE come from vmlinux.h (BTF enum constants)
+ */
+#define LCOEF_RANDIO_PAGES 4096 /* 16MB seek threshold */
+#define IOCOST_REQ_OP_MASK 0xff /* REQ_OP_MASK, not in BTF */
+
+/*
+ * DIV64_U64_ROUND_UP / DIV_ROUND_UP_ULL equivalents, folded at
+ * compile time
+ */
+#define RU(x, y) ((x) / (y) + (((x) % (y)) ? 1 : 0))
+
+#define RBPS 174019176ULL
+#define RSEQIOPS 41708ULL
+#define RRANDIOPS 370ULL
+#define WBPS 178075866ULL
+#define WSEQIOPS 42705ULL
+#define WRANDIOPS 378ULL
+
+#define RPAGE (RU(VTIME_PER_SEC, RU(RBPS, IOC_PAGE_SIZE)))
+#define RSEQIO (RU(VTIME_PER_SEC, RSEQIOPS) - RPAGE)
+#define RRANDIO (RU(VTIME_PER_SEC, RRANDIOPS) - RPAGE)
+#define WPAGE (RU(VTIME_PER_SEC, RU(WBPS, IOC_PAGE_SIZE)))
+#define WSEQIO (RU(VTIME_PER_SEC, WSEQIOPS) - WPAGE)
+#define WRANDIO (RU(VTIME_PER_SEC, WRANDIOPS) - WPAGE)
+
+/*
+ * per-cgroup cursor storage: keyed by the cgroup, freed with it, so
+ * the model never leaks or reuses stale per-cgroup state
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_CGRP_STORAGE);
+ __uint(map_flags, BPF_F_NO_PREALLOC);
+ __type(key, int);
+ __type(value, __u64);
+} cursor_store SEC(".maps");
+
+SEC("struct_ops")
+u64 BPF_PROG(iocost_2x_calc_cost, u64 opf, u64 nbytes, u64 sector,
+ struct blkcg *blkcg, u64 model_flags)
+{
+ u64 pages, seek_pages = 0, base, coef_page, randio, cost;
+
+ /* builtin truncates: max(sectors >> IOC_SECT_TO_PAGE_SHIFT, 1) */
+ pages = nbytes >> IOC_PAGE_SHIFT;
+ if (!pages)
+ pages = 1;
+
+ if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ) {
+ base = RSEQIO; coef_page = RPAGE; randio = RRANDIO;
+ } else if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE) {
+ base = WSEQIO; coef_page = WPAGE; randio = WRANDIO;
+ } else {
+ /*
+ * a fully owning model must price every op; unknown
+ * ops are priced as a single page write
+ */
+ base = 0; coef_page = WPAGE; randio = 0;
+ }
+
+ /*
+ * mirror the builtin cursor semantics: seek distance is only
+ * computed against a non-zero cursor, and the cursor is
+ * advanced for bios the builtin prices (READ/WRITE with a
+ * non-zero size), merged ones included, so flushes and
+ * discards leave it alone and long merged streams do not
+ * drift past the 16MB seek threshold
+ */
+ {
+ __u64 *cursor, cur;
+ int priced = (opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ ||
+ (opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE;
+
+ cursor = bpf_cgrp_storage_get(&cursor_store,
+ blkcg->css.cgroup, NULL,
+ BPF_LOCAL_STORAGE_GET_F_CREATE);
+ if (!cursor) {
+ if (model_flags & IOCOST_COST_F_MERGE)
+ base = 0;
+ return 2 * (base + pages * coef_page);
+ }
+ cur = *cursor;
+ if (cur && priced) {
+ seek_pages = sector > cur ? sector - cur
+ : cur - sector;
+ seek_pages >>= IOC_SECT_TO_PAGE_SHIFT;
+ if (seek_pages > LCOEF_RANDIO_PAGES)
+ base = randio;
+ }
+ if (priced && nbytes)
+ *cursor = sector + RU(nbytes, 512);
+ }
+
+ if (model_flags & IOCOST_COST_F_MERGE)
+ base = 0;
+
+ cost = 2 * (base + pages * coef_page);
+ return cost;
+}
+
+SEC(".struct_ops")
+struct iocost_model_ops iocost_2x = {
+ .calc_cost = (void *)iocost_2x_calc_cost,
+ .name = "iocost_2x",
+};
+
+char LICENSE[] SEC("license") = "GPL";
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [RFC PATCH v6 3/5] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary
2026-09-18 5:49 [RFC PATCH v6 0/5] blk-iocost: BPF struct_ops cost model Tao Cui
2026-09-18 5:49 ` [RFC PATCH v6 1/5] blk-iocost: add BPF struct_ops cost model support Tao Cui
2026-09-18 5:49 ` [RFC PATCH v6 2/5] selftests/bpf: add iocost cost model test Tao Cui
@ 2026-09-18 5:49 ` Tao Cui
2026-09-18 5:50 ` [RFC PATCH v6 4/5] selftests/bpf: add multi-stream sequentiality example model Tao Cui
2026-09-18 5:50 ` [RFC PATCH v6 5/5] docs: cgroup-v2: document io.cost model=<name> binding Tao Cui
4 siblings, 0 replies; 6+ messages in thread
From: Tao Cui @ 2026-09-18 5:49 UTC (permalink / raw)
To: tj, josef, axboe, ameryhung
Cc: cgroups, linux-block, linux-kernel, bpf, andrii, eddyz87, ast,
daniel, linux-kselftest, cui.tao, cuitao
From: Tao Cui <cuitao@kylinos.cn>
The existing iocost tracepoints are state-change driven: vrate_adj
fires only when the adjustment logic runs, inuse_* only on surplus
state transitions, activate/idle only on cgroup state changes. In a
steady state none of them fire. The only other way to observe the
controller (period length, vrate, busy level, active cgroup count,
device utilization) is iocost_monitor.py, which reads kernel memory
through drgn and is not usable in most production environments.
Add iocost_ioc_tick, emitted once per period from the tail of
ioc_timer_fn() with the overall controller state: period_us, vrate,
busy_level, active iocg count, usage percentage and running state.
It fires every period the controller runs, including steady states,
plus one final tick before the controller goes idle, which makes
dormancy (e.g. a device saturated entirely by uncharged IO) directly
visible.
Depending on the autop profile this is 2-100 events per second per
device; the added cost outside the tracepoint static key is one
increment per active cgroup per period.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
block/blk-iocost.c | 32 +++++++++++++++++++++++++
include/trace/events/iocost.h | 45 +++++++++++++++++++++++++++++++++++
2 files changed, 77 insertions(+)
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 307065c592950..9eb97f38a39cf 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -2244,6 +2244,10 @@ static void ioc_timer_fn(struct timer_list *timer)
struct ioc_now now;
LIST_HEAD(surpluses);
int nr_debtors, nr_shortages = 0, nr_lagging = 0;
+ int nr_active = 0;
+ u32 tick_period_us;
+ u64 tick_vrate, tick_dur;
+ int tick_busy, tick_running;
u64 usage_us_sum = 0;
u32 ppm_rthr;
u32 ppm_wthr;
@@ -2280,6 +2284,8 @@ static void ioc_timer_fn(struct timer_list *timer)
u64 vdone, vtime, usage_us;
u32 hw_active, hw_inuse;
+ nr_active++;
+
/*
* Collect unused and wind vtime closer to vnow to prevent
* iocgs from accumulating a large amount of budget.
@@ -2441,6 +2447,14 @@ static void ioc_timer_fn(struct timer_list *timer)
ioc->busy_level = clamp(ioc->busy_level, -1000, 1000);
+ /*
+ * vrate and period_us change right below; snapshot the values
+ * this period ran in so the tick below reports the period's own
+ * parameters instead of the next period's.
+ */
+ tick_period_us = ioc->period_us;
+ tick_vrate = ioc->vtime_base_rate;
+
ioc_adjust_base_vrate(ioc, rq_wait_pct, nr_lagging, nr_shortages,
prev_busy_level, missed_ppm);
@@ -2448,6 +2462,24 @@ static void ioc_timer_fn(struct timer_list *timer)
ioc_forgive_debts(ioc, usage_us_sum, nr_debtors, &now);
+ /*
+ * Snapshot the state this period ran in before the idle
+ * transition below wipes it, so the final tick reports the period's
+ * own busy level (e.g. the saturation that drove the controller
+ * idle) instead of the cleared one. usage is normalized by the
+ * measured period length, captured before ioc_start_period()
+ * overwrites period_at, the same way the donation loop does.
+ * Emitted before cur_period is bumped, so the tick carries the
+ * number of the period it describes.
+ */
+ tick_busy = ioc->busy_level;
+ tick_running = ioc->running;
+ tick_dur = now.now - ioc->period_at;
+
+ trace_iocost_ioc_tick(ioc, nr_active, usage_us_sum,
+ tick_period_us, tick_vrate,
+ tick_busy, tick_running, tick_dur);
+
/*
* This period is done. Move onto the next one. If nothing's
* going on with the device, stop the timer.
diff --git a/include/trace/events/iocost.h b/include/trace/events/iocost.h
index e772b1bc60d60..32f19861a78fd 100644
--- a/include/trace/events/iocost.h
+++ b/include/trace/events/iocost.h
@@ -178,6 +178,51 @@ TRACE_EVENT(iocost_ioc_vrate_adj,
)
);
+/*
+ * Periodic per-device summary, emitted once per period from the tail of
+ * ioc_timer_fn(). Unlike the state-change events above, this fires every
+ * period the controller is running, including steady states, and carries
+ * the overall controller state so basic monitoring doesn't require drgn.
+ */
+TRACE_EVENT(iocost_ioc_tick,
+
+ TP_PROTO(struct ioc *ioc, int nr_active, u64 usage_us_sum,
+ u32 tick_period_us, u64 tick_vrate,
+ int tick_busy, int tick_running, u64 tick_dur),
+
+ TP_ARGS(ioc, nr_active, usage_us_sum, tick_period_us, tick_vrate,
+ tick_busy, tick_running, tick_dur),
+
+ TP_STRUCT__entry (
+ __string(devname, ioc_name(ioc))
+ __field(u64, cur_period)
+ __field(u32, period_us)
+ __field(u64, vrate)
+ __field(int, busy_level)
+ __field(int, nr_active)
+ __field(u32, usage_pct)
+ __field(int, running)
+ ),
+
+ TP_fast_assign(
+ __assign_str(devname);
+ __entry->cur_period = atomic64_read(&ioc->cur_period);
+ __entry->period_us = tick_period_us;
+ __entry->vrate = tick_vrate;
+ __entry->busy_level = tick_busy;
+ __entry->nr_active = nr_active;
+ __entry->usage_pct = tick_dur ?
+ div64_u64(usage_us_sum * 100, tick_dur) : 0;
+ __entry->running = tick_running;
+ ),
+
+ TP_printk("[%s] period=%llu:%uus vrate=%llu busy=%d active=%d usage=%u%% running=%d",
+ __get_str(devname), __entry->cur_period, __entry->period_us,
+ __entry->vrate, __entry->busy_level, __entry->nr_active,
+ __entry->usage_pct, __entry->running
+ )
+);
+
TRACE_EVENT(iocost_iocg_forgive_debt,
TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [RFC PATCH v6 4/5] selftests/bpf: add multi-stream sequentiality example model
2026-09-18 5:49 [RFC PATCH v6 0/5] blk-iocost: BPF struct_ops cost model Tao Cui
` (2 preceding siblings ...)
2026-09-18 5:49 ` [RFC PATCH v6 3/5] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary Tao Cui
@ 2026-09-18 5:50 ` Tao Cui
2026-09-18 5:50 ` [RFC PATCH v6 5/5] docs: cgroup-v2: document io.cost model=<name> binding Tao Cui
4 siblings, 0 replies; 6+ messages in thread
From: Tao Cui @ 2026-09-18 5:50 UTC (permalink / raw)
To: tj, josef, axboe, ameryhung
Cc: cgroups, linux-block, linux-kernel, bpf, andrii, eddyz87, ast,
daniel, linux-kselftest, cui.tao, cuitao
From: Tao Cui <cuitao@kylinos.cn>
Add a second example cost model which replaces the builtin
single-cursor sequentiality heuristic with a per-cgroup table of
stream slots: an IO is sequential iff its sector matches the expected
next sector of any tracked stream. Interleaved sequential readers in
one cgroup keep their own slots instead of ping-ponging a single
cursor, and random IO inside a hot window rarely matches a moving
expectation. Merged bios skip the base cost but still advance the
matched stream position, so a merge at the expected sector does not
make the following new IO look random.
Stream state lives in a CGRP_STORAGE map keyed by the cgroup of the
blkcg argument, following the cgroup lifetime; there is no
fixed-size registry to exhaust.
Measured (QEMU, virtio-blk with the HDD profile, 4k IOs, w=1000):
two sequential readers in one cgroup are priced 1961us/op by the
builtin model (judged random) and 23us/op by this model (judged
sequential), the completed IO count rises from 4495 to 207505;
random IO inside an 8M window is priced 24us/op by builtin
(undercharge) and 2607us/op by this model; single-stream sequential
and whole-disk random pricing are unchanged.
Stream updates are lockless like the builtin cursor; a lost update
misclassifies a single IO. Slots are only advanced for bios the
builtin prices. The streams test lives here with the model it
loads. The page count truncates like the builtin.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
.../selftests/bpf/prog_tests/iocost_model.c | 34 ++++
tools/testing/selftests/bpf/progs/iocost_ms.c | 156 ++++++++++++++++++
2 files changed, 190 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/iocost_ms.c
diff --git a/tools/testing/selftests/bpf/prog_tests/iocost_model.c b/tools/testing/selftests/bpf/prog_tests/iocost_model.c
index 85b5ea7482eed..ac734c0e09cb0 100644
--- a/tools/testing/selftests/bpf/prog_tests/iocost_model.c
+++ b/tools/testing/selftests/bpf/prog_tests/iocost_model.c
@@ -4,6 +4,7 @@
#include <fcntl.h>
#include <unistd.h>
#include "iocost_model.skel.h"
+#include "iocost_ms.skel.h"
/*
* Write a line to io.cost.model with write(2) and return the errno of
@@ -164,3 +165,36 @@ void serial_test_iocost_model(void)
iocost_model__destroy(skel);
}
+
+/*
+ * Same check for the multi-stream example model. Only one model can
+ * be bound to a device at a time; both tests bind and restore, so
+ * they are serial and independent.
+ */
+void serial_test_iocost_model_streams(void)
+{
+ struct iocost_ms *skel;
+ char *dev;
+ int err;
+
+ dev = getenv("IOCOST_TEST_DEV");
+ if (!dev || geteuid() != 0) {
+ test__skip();
+ return;
+ }
+ if (!dev_has_iocost(dev)) {
+ printf("skip: %s has no iocost enabled\n", dev);
+ test__skip();
+ return;
+ }
+
+ skel = iocost_ms__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "skel_open_load"))
+ return;
+
+ err = iocost_ms__attach(skel);
+ if (ASSERT_OK(err, "attach"))
+ ASSERT_OK(bind_model(dev, "iocost_ms"), "bind_and_readback");
+
+ iocost_ms__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/iocost_ms.c b/tools/testing/selftests/bpf/progs/iocost_ms.c
new file mode 100644
index 0000000000000..70019db0fd790
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/iocost_ms.c
@@ -0,0 +1,156 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Example multi-stream sequentiality detection cost model.
+ *
+ * The builtin model keeps a single cursor per cgroup, so two
+ * interleaved sequential readers in one cgroup are all priced random
+ * (measured 89x overcharge, 12.9x throughput collapse), while random
+ * IO inside a hot window smaller than the 16MB seek threshold is
+ * priced sequential (measured 107x undercharge). This model replaces
+ * the single cursor with a per-cgroup table of stream slots: an IO is
+ * sequential iff its sector matches the expected next sector of any
+ * tracked stream. Interleaved streams keep their own slots, and
+ * windowed random IO rarely matches a moving expectation.
+ *
+ * Stream state lives in a CGRP_STORAGE map, so it is created and
+ * freed with the cgroup. The model implements the full builtin
+ * linear formula itself, including flush pricing.
+ */
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+/*
+ * VTIME_PER_SEC, IOC_PAGE_SIZE/SHIFT, IOC_SECT_TO_PAGE_SHIFT and
+ * IOCOST_COST_F_MERGE come from vmlinux.h (BTF enum constants)
+ */
+#define IOCOST_REQ_OP_MASK 0xff /* REQ_OP_MASK, not in BTF */
+
+/*
+ * DIV64_U64_ROUND_UP / DIV_ROUND_UP_ULL equivalents, folded at
+ * compile time
+ */
+#define RU(x, y) ((x) / (y) + (((x) % (y)) ? 1 : 0))
+
+#define RBPS 174019176ULL
+#define RSEQIOPS 41708ULL
+#define RRANDIOPS 370ULL
+#define WBPS 178075866ULL
+#define WSEQIOPS 42705ULL
+#define WRANDIOPS 378ULL
+
+#define RPAGE (RU(VTIME_PER_SEC, RU(RBPS, IOC_PAGE_SIZE)))
+#define RSEQIO (RU(VTIME_PER_SEC, RSEQIOPS) - RPAGE)
+#define RRANDIO (RU(VTIME_PER_SEC, RRANDIOPS) - RPAGE)
+#define WPAGE (RU(VTIME_PER_SEC, RU(WBPS, IOC_PAGE_SIZE)))
+#define WSEQIO (RU(VTIME_PER_SEC, WSEQIOPS) - WPAGE)
+#define WRANDIO (RU(VTIME_PER_SEC, WRANDIOPS) - WPAGE)
+
+#define NSLOTS 4
+
+struct streams {
+ __u64 expected[NSLOTS]; /* next expected sector, per stream */
+ __u64 stamp[NSLOTS]; /* LRU stamp, 0 = empty */
+};
+
+/*
+ * per-cgroup stream table: keyed by the cgroup, freed with it
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_CGRP_STORAGE);
+ __uint(map_flags, BPF_F_NO_PREALLOC);
+ __type(key, int);
+ __type(value, struct streams);
+} stream_tab SEC(".maps");
+
+SEC("struct_ops")
+u64 BPF_PROG(iocost_ms_calc_cost, u64 opf, u64 nbytes, u64 sector,
+ struct blkcg *blkcg, u64 model_flags)
+{
+ struct streams *s;
+ u64 pages, base, coef_page, randio, advance, now;
+ u32 i, victim = 0, found = 0xFFFFFFFF;
+
+ if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ) {
+ base = RSEQIO; coef_page = RPAGE; randio = RRANDIO;
+ } else if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE) {
+ base = WSEQIO; coef_page = WPAGE; randio = WRANDIO;
+ } else {
+ /*
+ * a fully owning model must price every op; unknown
+ * ops are priced as per-page writes
+ */
+ base = 0; coef_page = WPAGE; randio = 0;
+ }
+ advance = RU(nbytes, 512); /* sectors */
+
+ /* only bios the builtin prices participate in stream tracking */
+ if (!(((opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ ||
+ (opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE) && nbytes)) {
+ pages = nbytes >> IOC_PAGE_SHIFT;
+ if (!pages)
+ pages = 1;
+ return base + pages * coef_page;
+ }
+
+ s = bpf_cgrp_storage_get(&stream_tab, blkcg->css.cgroup, NULL,
+ BPF_LOCAL_STORAGE_GET_F_CREATE);
+ if (!s) {
+ /* no storage: price per page, truncating like the builtin */
+ pages = nbytes >> IOC_PAGE_SHIFT;
+ if (!pages)
+ pages = 1;
+ return base + pages * coef_page;
+ }
+
+ /*
+ * Slot access is lockless, mirroring the builtin single-cursor
+ * update in ioc_rqos_throttle(): concurrent CPUs submitting for
+ * the same cgroup can race on slot updates; mispricing is
+ * bounded and acceptable for an example model.
+ */
+ now = bpf_ktime_get_ns();
+ for (i = 0; i < NSLOTS; i++) {
+ if (s->expected[i] == sector && s->stamp[i]) {
+ found = i;
+ break;
+ }
+ }
+ if (found != 0xFFFFFFFF) {
+ /* sequential: keep the seq base from the op branch */
+ s->expected[found] = sector + advance;
+ s->stamp[found] = now;
+ } else {
+ base = randio;
+ for (i = 1; i < NSLOTS; i++) {
+ if (s->stamp[i] < s->stamp[victim])
+ victim = i;
+ }
+ s->expected[victim] = sector + advance;
+ s->stamp[victim] = now;
+ }
+
+ /* builtin truncates: max(sectors >> IOC_SECT_TO_PAGE_SHIFT, 1) */
+ pages = nbytes >> IOC_PAGE_SHIFT;
+ if (!pages)
+ pages = 1;
+ if (model_flags & IOCOST_COST_F_MERGE) {
+ /*
+ * merged bios skip the base cost but still advance
+ * the stream position above, so a merge at the
+ * expected sector does not make the following new IO
+ * look random
+ */
+ base = 0;
+ }
+
+ return base + pages * coef_page;
+}
+
+SEC(".struct_ops")
+struct iocost_model_ops iocost_ms = {
+ .calc_cost = (void *)iocost_ms_calc_cost,
+ .name = "iocost_ms",
+};
+
+char LICENSE[] SEC("license") = "GPL";
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [RFC PATCH v6 5/5] docs: cgroup-v2: document io.cost model=<name> binding
2026-09-18 5:49 [RFC PATCH v6 0/5] blk-iocost: BPF struct_ops cost model Tao Cui
` (3 preceding siblings ...)
2026-09-18 5:50 ` [RFC PATCH v6 4/5] selftests/bpf: add multi-stream sequentiality example model Tao Cui
@ 2026-09-18 5:50 ` Tao Cui
4 siblings, 0 replies; 6+ messages in thread
From: Tao Cui @ 2026-09-18 5:50 UTC (permalink / raw)
To: tj, josef, axboe, ameryhung
Cc: cgroups, linux-block, linux-kernel, bpf, andrii, eddyz87, ast,
daniel, linux-kselftest, cui.tao, cuitao
From: Tao Cui <cuitao@kylinos.cn>
Document the named-model binding of io.cost.model in the
io.cost.model section of the cgroup v2 documentation: the binding
and restore syntax, the unknown-name rejection, the full-replacement
semantics of a bound model and the unregister lifetime.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
Documentation/admin-guide/cgroup-v2.rst | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 8d2603751c51a..d9d82559e2bc6 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -2115,8 +2115,10 @@ IO Interface Files
are defined.
===== ================================
- ctrl "auto" or "user"
+ ctrl "auto", "user" or "bpf"
model The cost model in use - "linear"
+ or the name of the bound
+ model
===== ================================
When "ctrl" is "auto", the kernel may change all parameters
@@ -2124,6 +2126,23 @@ IO Interface Files
parameters are written to, "ctrl" become "user" and the
automatic changes are disabled.
+ When CONFIG_BLK_CGROUP_IOCOST_BPF is enabled, a bound model is
+ read back as "ctrl=bpf model=<name>", and writing "ctrl=bpf" is
+ accepted as a no-op so a saved configuration can be restored as-is.
+ "model" also accepts
+ the name of a registered iocost_model_ops BPF struct_ops model:
+ "model=<name>" binds the model to the device and the model fully
+ replaces the builtin linear pricing, for every operation including
+ flushes; "model=linear" (or "ctrl=auto/user") restores the builtin
+ model. Writing an unknown name fails with ENOENT and nothing is
+ applied. Unregistering a model removes its name from the
+ registry; devices already bound keep using it, and keep receiving
+ cgroup lifecycle notifications, until switched back to the builtin
+ model, and its name keeps working for writes (so a coefficient-only
+ write does not disturb the binding) until the last device unbinds.
+ See
+ include/linux/blk-iocost.h for the model interface.
+
When "model" is "linear", the following model parameters are
defined.
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread