[RFC PATCH v11 0/5] blk-iocost: BPF struct_ops cost model

[RFC PATCH v11 0/5] blk-iocost: BPF struct_ops cost model

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

This is v11 of the RFC.  A BPF program can now take over the cost
model of one device: attaching an iocost_model_ops struct_ops creates
the ioc if needed and switches the device to the model, io.cost.model
selects between the attached and the builtin model, and detaching or
device removal restores the builtin one.  The attachment, the cgroup
callback pairing and the removal paths are serialized with what blkg
creation and destruction actually use.

The series is based on next-20261027, which includes flush cost support
(LCOEF_FLUSH) in the builtin model: once a model is attached, the BPF
path takes over flushes like every other charged IO, and the builtin
flush coefficients stay for the linear model.  The definitions shared
between blk-iocost.c and the new BPF file moved to a private
block/blk-iocost.h, and the model cost path lives in inline helpers
which compile away completely when the BPF support is disabled.

Why a pluggable model at all
----------------------------

When iocost landed in 2019, its commit message already promised that
"a later patch will also allow using bpf progs for cost models", and
the code has carried the split for it ever since: calc_vtime_cost()
is a dispatcher whose only implementation is calc_vtime_cost_builtin().
Seven years later the builtin linear model is still the only one.
This series fills that slot: builtin algorithms remain the default
while new ones can be prototyped in BPF.

The measured problems
---------------------

The builtin model prices each IO with a binary sequential/random base
picked by a single per-cgroup cursor and a 16MB seek threshold, plus
a per-page cost.  On a virtio-blk device with the HDD autop profile,
a 4k IO costs ~24us when judged sequential and ~2.7ms when judged
random, a 112x spread, so a wrong judgement becomes a wrong price.
Three classes of mispricing, all measured:

 1. Heuristic rigidity.  Two legitimate sequential readers in one
    cgroup (a database with multiple tablespaces, a threaded backup)
    ping-pong the single cursor and are all priced random: a
    measured 89x overcharge collapses throughput under the same
    weight.  Random IO within a hot window smaller than the 16MB
    threshold is priced sequential: measured 107x undercharge, an
    accounting escape for hotspot workloads.  No setting of the six
    builtin parameters seems able to fix this: telling the streams
    apart requires per-IO state tracking, which looks like logic
    rather than coefficients.

 2. Device nonlinearity.  SLC-cache phases, SMR band placement and
    shared controllers (multiple NVMe namespaces multiplexing one
    device) make the real cost of an identical IO vary by an order
    of magnitude over time or across namespaces.  A static
    6-parameter linear model has no way to express that.

 3. Unpriced operations.  Flush and zone append fall through to a
    cost of zero and bypass throttling entirely on the pre-LCOEF_FLUSH
    model, and the same pattern extends to device quirks the builtin
    model was never taught.

Mispricing feeds directly into the control loop: vtime budgets,
surplus donation and the vrate feedback all consume the model's
output, so a wrong model can skew the whole controller.

How
---

Attachment is per device: the target device is set
in the dev member of the struct_ops from userspace before load; the
device is looked up with blkdev_get_no_open() and disk_live() is
checked under rq_qos_mutex, like blkg_conf_open_bdev().  Attaching
creates the ioc if needed, like an io.cost.model write does, and
switches the model with the queue frozen; the attach and detach
take the freeze before rq_qos_mutex, the order ioc_qos_write()
uses.  Enabling stays with io.cost.qos and wbt stays with it too.
The ops carries no device state: the attachment records the owning
link in the ioc and holds a map reference, so a stale link cannot
tear down a newer attachment, and a second attach of an attached
map fails with -EBUSY.  When the disk goes away, the model is
ejected completely.

Attachment is independent of the model selection and of the
controller state: while a model is attached, "model" selects between
it and the builtin model - "model=bpf" switches to the attached
model, "model=linear" switches back to the builtin model, and
neither detaches the struct_ops; only detaching removes the model,
after which "model=bpf" fails.  "ctrl" keeps describing the builtin
coefficients, which are kept while the BPF model is in use and take
effect again when switched back.  Enabling and disabling the
controller, and with it the wbt handover, remains with io.cost.qos.

A model in use owns pricing for every charged IO on the device: it
is called from the bio charging path and prices every operation
including flushes.  The completion-time request sizing which feeds
the latency met/missed accounting uses the transfer cost
coefficients the struct_ops carries (vtime per page for reads and
writes), so the builtin latency tracking and vrate adjustment
follow the model's pricing while it is in use; extending the
model to the QoS side itself is left for a later interface.

The cgroup callbacks are bound to the iocg policy lifetime and pair
up exactly: iocg_init() is delivered from ioc_pd_online(), which
blkg_create() only calls after the blkg has been inserted into the
radix tree and the queue's blkg list, and on attach to every cgroup
which is already online on the device; iocg_free() is delivered at
detach to the cgroups which still exist then and at pd_free() time
for the rest, gated per iocg on a delivered flag.  The attach and
detach walks go over q->blkg_list under q->blkcg_mutex and
q->queue_lock, the same locking blkg creation and destruction use.

    u64 calc_cost(struct bio *bio, u64 model_flags)

The model reads whatever it needs from the bio itself: the
operation flags (the operation must be extracted with a mask, and
the REQ_* flag bits, including PREFLUSH/FUA, are part of it), the
size, the start sector and the issuing cgroup through
bio->bi_blkg.  model_flags carries iocost-specific metadata which
is not a property of the bio, such as whether the cost calculation
is for a merged request; the return value is vtime, clamped to 1
second of device time per IO.  Per-cgroup state can be stored in
BPF_MAP_TYPE_CGRP_STORAGE keyed by the cgroup of bi_blkg; it
follows the cgroup lifetime.  Sleepable models are rejected at
verification, since calc_cost() runs under RCU read lock.

Patch overview:

 1/5: blk-iocost: move shared definitions to blk-iocost.h; the
      enums, structs and accessors private to blk-iocost.c move to
      a private header, with no functional change
 2/5: the BPF struct_ops cost model support: Kconfig, ops
      definition, per-device attachment, unified dispatch and
      verifier checks
 3/5: selftests with two example models (the full builtin linear
      HDD formula at double cost, and a multi-stream sequentiality
      model) plus a runner and the selftest kernel config entries
 4/5: add an iocost_ioc_tick tracepoint emitting the per-period
      controller state, so model quality can be evaluated without
      drgn (existing events are state-change driven and silent in
      steady state)
 5/5: document the attachment in cgroup-v2.rst

Does it work
------------

Mechanism, verified functionally on a virtio-blk device (HDD
profile, sequential-read workload from a 1%-weight cgroup, builtin
vs the 2x example model):

 - per-IO charge: 2882us -> 5722us, a factor of 1.985x; the
   completed IO count halves and total cost.usage is conserved,
   i.e. the model output drives both charging and budgeting
 - attachment: model=bpf readback while in use (ctrl keeps
   describing the builtin coefficients), EBUSY for a second model
   on the same device, a second device attaches an independent
   model, "model=linear"/"model=bpf" switch between the attached
   model and the builtin without detaching, detaching removes the
   model and "model=bpf" fails afterwards
 - edge cases: writing an unknown device fails and nothing is
   applied; the selftest runner checks the write error and errno
   of every step, including the restoration

Workload-shape verification (same setup, 4k IOs at weight 1000,
builtin vs the 2x example model):

 - flush-heavy workload (read/write/fsync alternating): priced
   1.99x the builtin, i.e. flushes no longer reset the cursor
   and misjudge the following IO as random
 - non-page-multiple IO (6 KiB): priced ~2x, matching the
   builtin's truncating page count
 - first IO from a high LBA (past 16 MiB): priced 2.01x, i.e.
   a fresh cgroup's zero cursor no longer misjudges the first IO
   as random

Payoff, demonstrated with the multi-stream example model (3/5) on
the same setup, 4k IOs at weight 1000, builtin vs the model:

 - two sequential readers in one cgroup: priced 1961us/op by builtin
   (both judged random by the single cursor) and 23us/op by the
   model (each stream keeps its own slot); the completed IO count
   rises by two orders of magnitude
 - random IO inside an 8M window: priced 24us/op by builtin
   (undercharge, an accounting escape) and 2607us/op by the model
 - single-stream sequential and whole-disk random pricing are
   unchanged, so the model fixes both directions of mispricing
   without introducing a new one

Non-interference, measured on enterprise NVMe: no measurable
overhead when the BPF model is not attached.

On next-20261027 (qemu x86_64 with null_blk) the selftests pass,
including the device-removal case, and the 2x example model doubles
the charged cost over the builtin, returning to ~1.1x after
"model=linear".

Changes in v9:
- the attach holds rq_qos_mutex from the disk_live() check through
  the unfreeze, like an io.cost.model write does, instead of
  releasing it between creating the ioc and switching the model;
  the queue reference, the second q_to_ioc() and the lockless
  traversal it needed are gone
- the attachment is published, cleared and the blkg list is walked
  under blkcg_mutex and queue_lock: blkg creation runs ioc_pd_init()
  and adds to q->blkg_list under queue_lock only, so the previous
  blkcg_mutex-only serialization could double-deliver iocg_init() or
  deliver iocg_free() without iocg_init().  The init walk now goes
  parents first, like blkcg_activate_policy()
- the device removal ejection reads ops->bdev before clearing ops->q:
  once q is NULL .unreg returns without a lock and the map can be
  freed, so touching ops afterwards was a use-after-free window.  The
  ejection moved to an ioc_bpf_eject() helper
- .unreg only detaches when the closing link is the one which owns
  the attachment, so a map re-attached to a new device through a
  newer link is not torn down by an old link; the v8 ops->q re-entry
  guard at .reg is gone, a second attach of an attached map now
  fails with -EBUSY like any other model would
- the ioc->attached and ioc->model pointers are unconditional and
  blk_get_queue_rcu() is declared in block/blk.h without an export,
  which drops the local prototypes and the remaining #ifdefs
- the unused struct_ops .init name lookup, the bdev/q cases in
  .init_member (the core already rejects nonzero non-function members
  it does not claim) and the unused write_cost_model() buffers are
  gone; the -EOPNOTSUPP selftest paths call test__skip()
- the selftest writes the dev member through the skeleton's typed
  struct_ops access instead of assuming it is the first member, and
  the maps are ".struct_ops.link" so closing the fd detaches; the
  example model's seek judgement is gated on a non-zero IO size so a
  dataless flush is priced sequentially, and write_cost_model()
  returns the negated errno ASSERT_ERR() expects
- the tick tracepoint's running field reports the active iocg list
  instead of ioc->running, which has not transitioned to IOC_IDLE at
  the emit point, so the final tick reads active=0 running=0
- the attach and detach take the queue freeze before rq_qos_mutex,
  like ioc_qos_write() does; taking the mutex around the freeze
  instead formed a lockdep cycle with the io.cost.qos write path,
  and .unreg's link recheck moved under the mutex inside
  ioc_bpf_detach()
- the io.cost.model documentation states that detaching restores the
  builtin model and that ctrl only accepts "auto" and "user" and
  never selects the model, says attaching rather than loading binds
  the model, and that removing the device removes the model as well,
  with the struct_ops link still having to be closed afterwards

Changes in v10:
- one map binds one device at a time: the attach rejects a second
  attach while the ops is still bound, which the v9 owning-link
  rework had dropped.  Without it, attaching the same map to a
  second device clobbered the single device slot, leaked the first
  bdev reference and left the first device pointing at the ops
- the detach takes the queue the caller holds a reference on as an
  argument instead of re-reading ops->q, fetches the ioc under
  rq_qos_mutex, and re-checks the owning link under the mutex, so
  closing an old link after an ejection and a re-attach no longer
  tears down the newer attachment
- iocg_free() is only delivered to a cgroup which received
  iocg_init(), tracked per iocg, so a blkg appearing between the
  attach walk and its own pd_init no longer gets an unpaired
  iocg_free(); a blkg whose radix_tree_insert() failed may still
  keep an iocg_init() without iocg_free(), which is now documented
  in the interface comment

Changes in v11:
- the shared definitions move to a private block/blk-iocost.h: the
  pluggable model interface has no users outside the two block
  files and no longer sits in include/linux; the model cost path
  lives in inline helpers and the ioc/iocg state is #ifdef'd, so a
  kernel without the BPF support compiles the whole thing away
- the ops carries no device state: the bdev/q members are gone, the
  attachment records the owning link in the ioc and holds a map
  reference for as long as it is set, and detaching re-checks the
  attachment under rq_qos_mutex and only freezes the queue when it
  still matches, so a removal ejection racing a detach cannot
  quiesce a queue whose tag set is already being freed
- reg validates completeness: calc_cost and both transfer
  coefficients are required, and zero coefficients are rejected
  before the model can be registered, with a selftest for it
- iocg_init()/iocg_free() pair exactly: init is delivered from
  ioc_pd_online(), which runs only after the radix tree insert, so
  the v10 caveat about a blkg keeping an init without a free after
  a failed radix_tree_insert() is gone; free is additionally gated
  on the per-iocg delivered flag
- the ioc_tick tracepoint drops the running field: the lock is held
  from the active count to the emission, so running would always
  equal active != 0
- the documentation uses "linear" or "bpf" for the model readback
  and the attachment paragraph is split into attaching, model
  selection and detaching
- selftests: added a device-removal test (attach, remove the
  null_blk device out from under the model, verify the ejection
  and the re-attach pairing) and pairing assertions
- ioc_bpf_attach() and ioc_bpf_detach() are goto-free: every error
  path unwinds inline and the blkcg_mutex is scoped to the walks,
  so the tail bdev put runs outside of it - dropping the last
  device reference can reach disk_release() -> blkg_destroy_all(),
  which takes q->blkcg_mutex
- the transfer coefficients are bounded at validate time (at most
  VTIME_PER_SEC >> IOC_SECT_TO_PAGE_SHIFT per page) and the
  completion-time sizing result is clamped to VTIME_PER_SEC, so a
  broken model cannot wrap the pricing arithmetic
- reg() only checks the device: validate() runs on the map-update
  path of both map flavors, so repeating the completeness check
  there was unreachable
- the interface comment states that calc_cost() may run for a
  (cgroup, device) pair before iocg_init() has been delivered for
  it, so models relying on per-pair state must tolerate that
  window
- the nr_active counting moved to the tracepoint patch which reads
  it, MAINTAINERS gains include/trace/events/iocost.h, and the
  documentation now states the -EBUSY exclusivity, that the
  builtin coefficients stay readable and writable but inert while
  the BPF model is in use, that attaching does not enable the
  controller, and that the profile selected from the device class
  still applies while the BPF model is in use

Link: https://lore.kernel.org/r/20260908100143.47598-1-cui.tao@linux.dev # v1
Link: https://lore.kernel.org/r/20260910125817.223354-1-cui.tao@linux.dev # v2
Link: https://lore.kernel.org/r/20260914073356.791518-1-cui.tao@linux.dev # v3
Link: https://lore.kernel.org/r/20260916072302.1068871-1-cui.tao@linux.dev # v4
Link: https://lore.kernel.org/r/20260918031751.1255420-1-cui.tao@linux.dev # v5
Link: https://lore.kernel.org/r/20260918055001.1273840-1-cui.tao@linux.dev # v6
Link: https://lore.kernel.org/r/20260924054549.2271705-1-cui.tao@linux.dev # v7
Link: https://lore.kernel.org/r/20260930075154.189958-1-cui.tao@linux.dev # v8
Link: https://lore.kernel.org/r/20261003013033.149288-1-cui.tao@linux.dev # v9
Link: https://lore.kernel.org/r/20261003124045.1289305-1-cui.tao@linux.dev # v10

[PATCH 1/5] blk-iocost: move shared definitions to blk-iocost.h

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

The BPF cost model code to be added in the following patch needs the
ioc/ioc_gq definitions and the queue accessors which are private to
blk-iocost.c.  Move the enums, the structs and the accessors into a
private blk-iocost.h so that everything can go where it belongs.
blkcg_policy_iocost becomes an extern declaration in the header so
that the inline accessors can be used from both translation units.

No functional change intended.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 MAINTAINERS        |   1 +
 block/blk-iocost.c | 415 +-----------------------------------------
 block/blk-iocost.h | 436 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 439 insertions(+), 413 deletions(-)
 create mode 100644 block/blk-iocost.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 365bec8eb7fb..b3b421ee262e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6827,6 +6827,7 @@ F:	Documentation/admin-guide/cgroup-v1/blkio-controller.rst
 F:	block/bfq-cgroup.c
 F:	block/blk-cgroup.c
 F:	block/blk-iocost.c
+F:	block/blk-iocost.h
 F:	block/blk-iolatency.c
 F:	block/blk-throttle.c
 F:	include/linux/blk-cgroup.h
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index a81488c755ca..9b610734871f 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -184,6 +184,7 @@
 #include "blk-stat.h"
 #include "blk-wbt.h"
 #include "blk-cgroup.h"
+#include "blk-iocost.h"
 
 #ifdef CONFIG_TRACEPOINTS
 
@@ -209,374 +210,6 @@ static char trace_iocg_path[TRACE_IOCG_PATH_LEN];
 #define TRACE_IOCG_PATH(type, iocg, ...)	do { } while (0)
 #endif	/* CONFIG_TRACE_POINTS */
 
-enum {
-	MILLION			= 1000000,
-
-	/* timer period is calculated from latency requirements, bound it */
-	MIN_PERIOD		= USEC_PER_MSEC,
-	MAX_PERIOD		= USEC_PER_SEC,
-
-	/*
-	 * iocg->vtime is targeted at 50% behind the device vtime, which
-	 * serves as its IO credit buffer.  Surplus weight adjustment is
-	 * immediately canceled if the vtime margin runs below 10%.
-	 */
-	MARGIN_MIN_PCT		= 10,
-	MARGIN_LOW_PCT		= 20,
-	MARGIN_TARGET_PCT	= 50,
-
-	INUSE_ADJ_STEP_PCT	= 25,
-
-	/* Have some play in timer operations */
-	TIMER_SLACK_PCT		= 1,
-
-	/* 1/64k is granular enough and can easily be handled w/ u32 */
-	WEIGHT_ONE		= 1 << 16,
-};
-
-enum {
-	/*
-	 * As vtime is used to calculate the cost of each IO, it needs to
-	 * be fairly high precision.  For example, it should be able to
-	 * represent the cost of a single page worth of discard with
-	 * suffificient accuracy.  At the same time, it should be able to
-	 * represent reasonably long enough durations to be useful and
-	 * convenient during operation.
-	 *
-	 * 1s worth of vtime is 2^37.  This gives us both sub-nanosecond
-	 * granularity and days of wrap-around time even at extreme vrates.
-	 */
-	VTIME_PER_SEC_SHIFT	= 37,
-	VTIME_PER_SEC		= 1LLU << VTIME_PER_SEC_SHIFT,
-	VTIME_PER_USEC		= VTIME_PER_SEC / USEC_PER_SEC,
-	VTIME_PER_NSEC		= VTIME_PER_SEC / NSEC_PER_SEC,
-
-	/* bound vrate adjustments within two orders of magnitude */
-	VRATE_MIN_PPM		= 10000,	/* 1% */
-	VRATE_MAX_PPM		= 100000000,	/* 10000% */
-
-	VRATE_MIN		= VTIME_PER_USEC * VRATE_MIN_PPM / MILLION,
-	VRATE_CLAMP_ADJ_PCT	= 4,
-
-	/* switch iff the conditions are met for longer than this */
-	AUTOP_CYCLE_NSEC	= 10LLU * NSEC_PER_SEC,
-};
-
-enum {
-	/* if IOs end up waiting for requests, issue less */
-	RQ_WAIT_BUSY_PCT	= 5,
-
-	/* unbusy hysterisis */
-	UNBUSY_THR_PCT		= 75,
-
-	/*
-	 * The effect of delay is indirect and non-linear and a huge amount of
-	 * future debt can accumulate abruptly while unthrottled. Linearly scale
-	 * up delay as debt is going up and then let it decay exponentially.
-	 * This gives us quick ramp ups while delay is accumulating and long
-	 * tails which can help reducing the frequency of debt explosions on
-	 * unthrottle. The parameters are experimentally determined.
-	 *
-	 * The delay mechanism provides adequate protection and behavior in many
-	 * cases. However, this is far from ideal and falls shorts on both
-	 * fronts. The debtors are often throttled too harshly costing a
-	 * significant level of fairness and possibly total work while the
-	 * protection against their impacts on the system can be choppy and
-	 * unreliable.
-	 *
-	 * The shortcoming primarily stems from the fact that, unlike for page
-	 * cache, the kernel doesn't have well-defined back-pressure propagation
-	 * mechanism and policies for anonymous memory. Fully addressing this
-	 * issue will likely require substantial improvements in the area.
-	 */
-	MIN_DELAY_THR_PCT	= 500,
-	MAX_DELAY_THR_PCT	= 25000,
-	MIN_DELAY		= 250,
-	MAX_DELAY		= 250 * USEC_PER_MSEC,
-
-	/* halve debts if avg usage over 100ms is under 50% */
-	DFGV_USAGE_PCT		= 50,
-	DFGV_PERIOD		= 100 * USEC_PER_MSEC,
-
-	/* don't let cmds which take a very long time pin lagging for too long */
-	MAX_LAGGING_PERIODS	= 10,
-
-	/*
-	 * Count IO size in 4k pages.  The 12bit shift helps keeping
-	 * size-proportional components of cost calculation in closer
-	 * numbers of digits to per-IO cost components.
-	 */
-	IOC_PAGE_SHIFT		= 12,
-	IOC_PAGE_SIZE		= 1 << IOC_PAGE_SHIFT,
-	IOC_SECT_TO_PAGE_SHIFT	= IOC_PAGE_SHIFT - SECTOR_SHIFT,
-
-	/* if apart further than 16M, consider randio for linear model */
-	LCOEF_RANDIO_PAGES	= 4096,
-};
-
-enum ioc_running {
-	IOC_IDLE,
-	IOC_RUNNING,
-	IOC_STOP,
-};
-
-/* io.cost.qos controls including per-dev enable of the whole controller */
-enum {
-	QOS_ENABLE,
-	QOS_CTRL,
-	NR_QOS_CTRL_PARAMS,
-};
-
-/* io.cost.qos params */
-enum {
-	QOS_RPPM,
-	QOS_RLAT,
-	QOS_WPPM,
-	QOS_WLAT,
-	QOS_MIN,
-	QOS_MAX,
-	NR_QOS_PARAMS,
-};
-
-/* io.cost.model controls */
-enum {
-	COST_CTRL,
-	COST_MODEL,
-	NR_COST_CTRL_PARAMS,
-};
-
-/* builtin linear cost model coefficients */
-enum {
-	I_LCOEF_RBPS,
-	I_LCOEF_RSEQIOPS,
-	I_LCOEF_RRANDIOPS,
-	I_LCOEF_WBPS,
-	I_LCOEF_WSEQIOPS,
-	I_LCOEF_WRANDIOPS,
-	I_LCOEF_FLUSHIOPS,
-	NR_I_LCOEFS,
-};
-
-enum {
-	LCOEF_RPAGE,
-	LCOEF_RSEQIO,
-	LCOEF_RRANDIO,
-	LCOEF_WPAGE,
-	LCOEF_WSEQIO,
-	LCOEF_WRANDIO,
-	LCOEF_FLUSH,
-	NR_LCOEFS,
-};
-
-enum {
-	AUTOP_INVALID,
-	AUTOP_HDD,
-	AUTOP_SSD_QD1,
-	AUTOP_SSD_DFL,
-	AUTOP_SSD_FAST,
-};
-
-struct ioc_params {
-	u32				qos[NR_QOS_PARAMS];
-	u64				i_lcoefs[NR_I_LCOEFS];
-	u64				lcoefs[NR_LCOEFS];
-	u32				too_fast_vrate_pct;
-	u32				too_slow_vrate_pct;
-};
-
-struct ioc_margins {
-	s64				min;
-	s64				low;
-	s64				target;
-};
-
-struct ioc_missed {
-	local_t				nr_met;
-	local_t				nr_missed;
-	u32				last_met;
-	u32				last_missed;
-};
-
-struct ioc_pcpu_stat {
-	struct ioc_missed		missed[2];
-
-	local64_t			rq_wait_ns;
-	u64				last_rq_wait_ns;
-};
-
-/* per device */
-struct ioc {
-	struct rq_qos			rqos;
-
-	bool				enabled;
-
-	struct ioc_params		params;
-	struct ioc_margins		margins;
-	u32				period_us;
-	u32				timer_slack_ns;
-	u64				vrate_min;
-	u64				vrate_max;
-
-	spinlock_t			lock;
-	struct timer_list		timer;
-	struct list_head		active_iocgs;	/* active cgroups */
-	struct ioc_pcpu_stat __percpu	*pcpu_stat;
-
-	enum ioc_running		running;
-	atomic64_t			vtime_rate;
-	u64				vtime_base_rate;
-	s64				vtime_err;
-
-	seqcount_spinlock_t		period_seqcount;
-	u64				period_at;	/* wallclock starttime */
-	u64				period_at_vtime; /* vtime starttime */
-
-	atomic64_t			cur_period;	/* inc'd each period */
-	int				busy_level;	/* saturation history */
-
-	bool				weights_updated;
-	atomic_t			hweight_gen;	/* for lazy hweights */
-
-	/* debt forgivness */
-	u64				dfgv_period_at;
-	u64				dfgv_period_rem;
-	u64				dfgv_usage_us_sum;
-
-	u64				autop_too_fast_at;
-	u64				autop_too_slow_at;
-	int				autop_idx;
-	bool				user_qos_params:1;
-	bool				user_cost_model:1;
-};
-
-struct iocg_pcpu_stat {
-	local64_t			abs_vusage;
-};
-
-struct iocg_stat {
-	u64				usage_us;
-	u64				wait_us;
-	u64				indebt_us;
-	u64				indelay_us;
-};
-
-/* per device-cgroup pair */
-struct ioc_gq {
-	struct blkg_policy_data		pd;
-	struct ioc			*ioc;
-
-	/*
-	 * A iocg can get its weight from two sources - an explicit
-	 * per-device-cgroup configuration or the default weight of the
-	 * cgroup.  `cfg_weight` is the explicit per-device-cgroup
-	 * configuration.  `weight` is the effective considering both
-	 * sources.
-	 *
-	 * When an idle cgroup becomes active its `active` goes from 0 to
-	 * `weight`.  `inuse` is the surplus adjusted active weight.
-	 * `active` and `inuse` are used to calculate `hweight_active` and
-	 * `hweight_inuse`.
-	 *
-	 * `last_inuse` remembers `inuse` while an iocg is idle to persist
-	 * surplus adjustments.
-	 *
-	 * `inuse` may be adjusted dynamically during period. `saved_*` are used
-	 * to determine and track adjustments.
-	 */
-	u32				cfg_weight;
-	u32				weight;
-	u32				active;
-	u32				inuse;
-
-	u32				last_inuse;
-	s64				saved_margin;
-
-	sector_t			cursor;		/* to detect randio */
-
-	/*
-	 * `vtime` is this iocg's vtime cursor which progresses as IOs are
-	 * issued.  If lagging behind device vtime, the delta represents
-	 * the currently available IO budget.  If running ahead, the
-	 * overage.
-	 *
-	 * `vtime_done` is the same but progressed on completion rather
-	 * than issue.  The delta behind `vtime` represents the cost of
-	 * currently in-flight IOs.
-	 */
-	atomic64_t			vtime;
-	atomic64_t			done_vtime;
-	u64				abs_vdebt;
-
-	/* current delay in effect and when it started */
-	u64				delay;
-	u64				delay_at;
-
-	/*
-	 * The period this iocg was last active in.  Used for deactivation
-	 * and invalidating `vtime`.
-	 */
-	atomic64_t			active_period;
-	struct list_head		active_list;
-
-	/* see __propagate_weights() and current_hweight() for details */
-	u64				child_active_sum;
-	u64				child_inuse_sum;
-	u64				child_adjusted_sum;
-	int				hweight_gen;
-	u32				hweight_active;
-	u32				hweight_inuse;
-	u32				hweight_donating;
-	u32				hweight_after_donation;
-
-	struct list_head		walk_list;
-	struct list_head		surplus_list;
-
-	struct wait_queue_head		waitq;
-	struct hrtimer			waitq_timer;
-
-	/* timestamp at the latest activation */
-	u64				activated_at;
-
-	/* statistics */
-	struct iocg_pcpu_stat __percpu	*pcpu_stat;
-	struct iocg_stat		stat;
-	struct iocg_stat		last_stat;
-	u64				last_stat_abs_vusage;
-	u64				usage_delta_us;
-	u64				wait_since;
-	u64				indebt_since;
-	u64				indelay_since;
-
-	/* this iocg's depth in the hierarchy and ancestors including self */
-	int				level;
-	struct ioc_gq			*ancestors[];
-};
-
-/* per cgroup */
-struct ioc_cgrp {
-	struct blkcg_policy_data	cpd;
-	unsigned int			dfl_weight;
-};
-
-struct ioc_now {
-	u64				now_ns;
-	u64				now;
-	u64				vnow;
-};
-
-struct iocg_wait {
-	struct wait_queue_entry		wait;
-	struct bio			*bio;
-	u64				abs_cost;
-	bool				committed;
-};
-
-struct iocg_wake_ctx {
-	struct ioc_gq			*iocg;
-	u32				hw_inuse;
-	s64				vbudget;
-};
-
 static const struct ioc_params autop[] = {
 	[AUTOP_HDD] = {
 		.qos				= {
@@ -656,49 +289,6 @@ static const u32 vrate_adj_pct[] =
 	  2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
 	  4, 4, 4, 4, 4, 4, 4, 4, 8, 8, 8, 8, 8, 8, 8, 8, 16 };
 
-static struct blkcg_policy blkcg_policy_iocost;
-
-/* accessors and helpers */
-static struct ioc *rqos_to_ioc(struct rq_qos *rqos)
-{
-	return container_of(rqos, struct ioc, rqos);
-}
-
-static struct ioc *q_to_ioc(struct request_queue *q)
-{
-	return rqos_to_ioc(rq_qos_id(q, RQ_QOS_COST));
-}
-
-static const char __maybe_unused *ioc_name(struct ioc *ioc)
-{
-	struct gendisk *disk = ioc->rqos.disk;
-
-	if (!disk)
-		return "<unknown>";
-	return disk->disk_name;
-}
-
-static struct ioc_gq *pd_to_iocg(struct blkg_policy_data *pd)
-{
-	return pd ? container_of(pd, struct ioc_gq, pd) : NULL;
-}
-
-static struct ioc_gq *blkg_to_iocg(struct blkcg_gq *blkg)
-{
-	return pd_to_iocg(blkg_to_pd(blkg, &blkcg_policy_iocost));
-}
-
-static struct blkcg_gq *iocg_to_blkg(struct ioc_gq *iocg)
-{
-	return pd_to_blkg(&iocg->pd);
-}
-
-static struct ioc_cgrp *blkcg_to_iocc(struct blkcg *blkcg)
-{
-	return container_of(blkcg_to_cpd(blkcg, &blkcg_policy_iocost),
-			    struct ioc_cgrp, cpd);
-}
-
 /*
  * Scale @abs_cost to the inverse of @hw_inuse.  The lower the hierarchical
  * weight, the more expensive each IO.  Must round up.
@@ -3150,7 +2740,6 @@ static u64 ioc_weight_prfill(struct seq_file *sf, struct blkg_policy_data *pd,
 	return 0;
 }
 
-
 static int ioc_weight_show(struct seq_file *sf, void *v)
 {
 	struct blkcg *blkcg = css_to_blkcg(seq_css(sf));
@@ -3616,7 +3205,7 @@ static struct cftype ioc_files[] = {
 	{}
 };
 
-static struct blkcg_policy blkcg_policy_iocost = {
+struct blkcg_policy blkcg_policy_iocost = {
 	.dfl_cftypes	= ioc_files,
 	.cpd_alloc_fn	= ioc_cpd_alloc,
 	.cpd_free_fn	= ioc_cpd_free,
diff --git a/block/blk-iocost.h b/block/blk-iocost.h
new file mode 100644
index 000000000000..e4e517ef6a0f
--- /dev/null
+++ b/block/blk-iocost.h
@@ -0,0 +1,436 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * blk-iocost private definitions, shared between blk-iocost.c and
+ * blk-iocost-bpf.c. Not for direct inclusion by other files.
+ */
+#ifndef _BLK_IOCOST_H
+#define _BLK_IOCOST_H
+
+#include <linux/types.h>
+#include <linux/blk_types.h>
+#include <linux/blkdev.h>
+#include "blk-cgroup.h"
+#include <linux/blk-mq.h>
+#include <linux/timer.h>
+#include <linux/hrtimer.h>
+#include <linux/seqlock.h>
+#include <linux/atomic.h>
+#include <linux/minmax.h>
+#include <linux/time64.h>
+#include <linux/wait.h>
+#include <asm/local.h>
+#include <asm/local64.h>
+#include "blk-rq-qos.h"
+
+enum {
+	MILLION			= 1000000,
+
+	/* timer period is calculated from latency requirements, bound it */
+	MIN_PERIOD		= USEC_PER_MSEC,
+	MAX_PERIOD		= USEC_PER_SEC,
+
+	/*
+	 * iocg->vtime is targeted at 50% behind the device vtime, which
+	 * serves as its IO credit buffer.  Surplus weight adjustment is
+	 * immediately canceled if the vtime margin runs below 10%.
+	 */
+	MARGIN_MIN_PCT		= 10,
+	MARGIN_LOW_PCT		= 20,
+	MARGIN_TARGET_PCT	= 50,
+
+	INUSE_ADJ_STEP_PCT	= 25,
+
+	/* Have some play in timer operations */
+	TIMER_SLACK_PCT		= 1,
+
+	/* 1/64k is granular enough and can easily be handled w/ u32 */
+	WEIGHT_ONE		= 1 << 16,
+};
+
+enum {
+	/*
+	 * As vtime is used to calculate the cost of each IO, it needs to
+	 * be fairly high precision.  For example, it should be able to
+	 * represent the cost of a single page worth of discard with
+	 * suffificient accuracy.  At the same time, it should be able to
+	 * represent reasonably long enough durations to be useful and
+	 * convenient during operation.
+	 *
+	 * 1s worth of vtime is 2^37.  This gives us both sub-nanosecond
+	 * granularity and days of wrap-around time even at extreme vrates.
+	 */
+	VTIME_PER_SEC_SHIFT	= 37,
+	VTIME_PER_SEC		= 1LLU << VTIME_PER_SEC_SHIFT,
+	VTIME_PER_USEC		= VTIME_PER_SEC / USEC_PER_SEC,
+	VTIME_PER_NSEC		= VTIME_PER_SEC / NSEC_PER_SEC,
+
+	/* bound vrate adjustments within two orders of magnitude */
+	VRATE_MIN_PPM		= 10000,	/* 1% */
+	VRATE_MAX_PPM		= 100000000,	/* 10000% */
+
+	VRATE_MIN		= VTIME_PER_USEC * VRATE_MIN_PPM / MILLION,
+	VRATE_CLAMP_ADJ_PCT	= 4,
+
+	/* switch iff the conditions are met for longer than this */
+	AUTOP_CYCLE_NSEC	= 10LLU * NSEC_PER_SEC,
+};
+
+enum {
+	/* if IOs end up waiting for requests, issue less */
+	RQ_WAIT_BUSY_PCT	= 5,
+
+	/* unbusy hysterisis */
+	UNBUSY_THR_PCT		= 75,
+
+	/*
+	 * The effect of delay is indirect and non-linear and a huge amount of
+	 * future debt can accumulate abruptly while unthrottled. Linearly scale
+	 * up delay as debt is going up and then let it decay exponentially.
+	 * This gives us quick ramp ups while delay is accumulating and long
+	 * tails which can help reducing the frequency of debt explosions on
+	 * unthrottle. The parameters are experimentally determined.
+	 *
+	 * The delay mechanism provides adequate protection and behavior in many
+	 * cases. However, this is far from ideal and falls shorts on both
+	 * fronts. The debtors are often throttled too harshly costing a
+	 * significant level of fairness and possibly total work while the
+	 * protection against their impacts on the system can be choppy and
+	 * unreliable.
+	 *
+	 * The shortcoming primarily stems from the fact that, unlike for page
+	 * cache, the kernel doesn't have well-defined back-pressure propagation
+	 * mechanism and policies for anonymous memory. Fully addressing this
+	 * issue will likely require substantial improvements in the area.
+	 */
+	MIN_DELAY_THR_PCT	= 500,
+	MAX_DELAY_THR_PCT	= 25000,
+	MIN_DELAY		= 250,
+	MAX_DELAY		= 250 * USEC_PER_MSEC,
+
+	/* halve debts if avg usage over 100ms is under 50% */
+	DFGV_USAGE_PCT		= 50,
+	DFGV_PERIOD		= 100 * USEC_PER_MSEC,
+
+	/* don't let cmds which take a very long time pin lagging for too long */
+	MAX_LAGGING_PERIODS	= 10,
+
+	/*
+	 * Count IO size in 4k pages.  The 12bit shift helps keeping
+	 * size-proportional components of cost calculation in closer
+	 * numbers of digits to per-IO cost components.
+	 */
+	IOC_PAGE_SHIFT		= 12,
+	IOC_PAGE_SIZE		= 1 << IOC_PAGE_SHIFT,
+	IOC_SECT_TO_PAGE_SHIFT	= IOC_PAGE_SHIFT - SECTOR_SHIFT,
+
+	/* if apart further than 16M, consider randio for linear model */
+	LCOEF_RANDIO_PAGES	= 4096,
+};
+
+enum ioc_running {
+	IOC_IDLE,
+	IOC_RUNNING,
+	IOC_STOP,
+};
+
+/* io.cost.qos controls including per-dev enable of the whole controller */
+enum {
+	QOS_ENABLE,
+	QOS_CTRL,
+	NR_QOS_CTRL_PARAMS,
+};
+
+/* io.cost.qos params */
+enum {
+	QOS_RPPM,
+	QOS_RLAT,
+	QOS_WPPM,
+	QOS_WLAT,
+	QOS_MIN,
+	QOS_MAX,
+	NR_QOS_PARAMS,
+};
+
+/* io.cost.model controls */
+enum {
+	COST_CTRL,
+	COST_MODEL,
+	NR_COST_CTRL_PARAMS,
+};
+
+/* builtin linear cost model coefficients */
+enum {
+	I_LCOEF_RBPS,
+	I_LCOEF_RSEQIOPS,
+	I_LCOEF_RRANDIOPS,
+	I_LCOEF_WBPS,
+	I_LCOEF_WSEQIOPS,
+	I_LCOEF_WRANDIOPS,
+	I_LCOEF_FLUSHIOPS,
+	NR_I_LCOEFS,
+};
+
+enum {
+	LCOEF_RPAGE,
+	LCOEF_RSEQIO,
+	LCOEF_RRANDIO,
+	LCOEF_WPAGE,
+	LCOEF_WSEQIO,
+	LCOEF_WRANDIO,
+	LCOEF_FLUSH,
+	NR_LCOEFS,
+};
+
+enum {
+	AUTOP_INVALID,
+	AUTOP_HDD,
+	AUTOP_SSD_QD1,
+	AUTOP_SSD_DFL,
+	AUTOP_SSD_FAST,
+};
+
+struct ioc_params {
+	u32				qos[NR_QOS_PARAMS];
+	u64				i_lcoefs[NR_I_LCOEFS];
+	u64				lcoefs[NR_LCOEFS];
+	u32				too_fast_vrate_pct;
+	u32				too_slow_vrate_pct;
+};
+
+struct ioc_margins {
+	s64				min;
+	s64				low;
+	s64				target;
+};
+
+struct ioc_missed {
+	local_t				nr_met;
+	local_t				nr_missed;
+	u32				last_met;
+	u32				last_missed;
+};
+
+struct ioc_pcpu_stat {
+	struct ioc_missed		missed[2];
+
+	local64_t			rq_wait_ns;
+	u64				last_rq_wait_ns;
+};
+
+/* per device */
+struct ioc {
+	struct rq_qos			rqos;
+
+	bool				enabled;
+
+	struct ioc_params		params;
+	struct ioc_margins		margins;
+	u32				period_us;
+	u32				timer_slack_ns;
+	u64				vrate_min;
+	u64				vrate_max;
+
+	spinlock_t			lock;
+	struct timer_list		timer;
+	struct list_head		active_iocgs;	/* active cgroups */
+	struct ioc_pcpu_stat __percpu	*pcpu_stat;
+
+	enum ioc_running		running;
+	atomic64_t			vtime_rate;
+	u64				vtime_base_rate;
+	s64				vtime_err;
+
+	seqcount_spinlock_t		period_seqcount;
+	u64				period_at;	/* wallclock starttime */
+	u64				period_at_vtime; /* vtime starttime */
+
+	atomic64_t			cur_period;	/* inc'd each period */
+	int				busy_level;	/* saturation history */
+
+	bool				weights_updated;
+	atomic_t			hweight_gen;	/* for lazy hweights */
+
+	/* debt forgivness */
+	u64				dfgv_period_at;
+	u64				dfgv_period_rem;
+	u64				dfgv_usage_us_sum;
+
+	u64				autop_too_fast_at;
+	u64				autop_too_slow_at;
+	int				autop_idx;
+	bool				user_qos_params:1;
+	bool				user_cost_model:1;
+
+};
+
+struct iocg_pcpu_stat {
+	local64_t			abs_vusage;
+};
+
+struct iocg_stat {
+	u64				usage_us;
+	u64				wait_us;
+	u64				indebt_us;
+	u64				indelay_us;
+};
+
+/* per device-cgroup pair */
+struct ioc_gq {
+	struct blkg_policy_data		pd;
+	struct ioc			*ioc;
+
+	/*
+	 * A iocg can get its weight from two sources - an explicit
+	 * per-device-cgroup configuration or the default weight of the
+	 * cgroup.  `cfg_weight` is the explicit per-device-cgroup
+	 * configuration.  `weight` is the effective considering both
+	 * sources.
+	 *
+	 * When an idle cgroup becomes active its `active` goes from 0 to
+	 * `weight`.  `inuse` is the surplus adjusted active weight.
+	 * `active` and `inuse` are used to calculate `hweight_active` and
+	 * `hweight_inuse`.
+	 *
+	 * `last_inuse` remembers `inuse` while an iocg is idle to persist
+	 * surplus adjustments.
+	 *
+	 * `inuse` may be adjusted dynamically during period. `saved_*` are used
+	 * to determine and track adjustments.
+	 */
+	u32				cfg_weight;
+	u32				weight;
+	u32				active;
+	u32				inuse;
+
+	u32				last_inuse;
+	s64				saved_margin;
+
+	sector_t			cursor;		/* to detect randio */
+
+	/*
+	 * `vtime` is this iocg's vtime cursor which progresses as IOs are
+	 * issued.  If lagging behind device vtime, the delta represents
+	 * the currently available IO budget.  If running ahead, the
+	 * overage.
+	 *
+	 * `vtime_done` is the same but progressed on completion rather
+	 * than issue.  The delta behind `vtime` represents the cost of
+	 * currently in-flight IOs.
+	 */
+	atomic64_t			vtime;
+	atomic64_t			done_vtime;
+	u64				abs_vdebt;
+
+	/* current delay in effect and when it started */
+	u64				delay;
+	u64				delay_at;
+
+	/*
+	 * The period this iocg was last active in.  Used for deactivation
+	 * and invalidating `vtime`.
+	 */
+	atomic64_t			active_period;
+	struct list_head		active_list;
+
+	/* see __propagate_weights() and current_hweight() for details */
+	u64				child_active_sum;
+	u64				child_inuse_sum;
+	u64				child_adjusted_sum;
+	int				hweight_gen;
+	u32				hweight_active;
+	u32				hweight_inuse;
+	u32				hweight_donating;
+	u32				hweight_after_donation;
+
+	struct list_head		walk_list;
+	struct list_head		surplus_list;
+
+	struct wait_queue_head		waitq;
+	struct hrtimer			waitq_timer;
+
+	/* timestamp at the latest activation */
+	u64				activated_at;
+
+	/* statistics */
+	struct iocg_pcpu_stat __percpu	*pcpu_stat;
+	struct iocg_stat		stat;
+	struct iocg_stat		last_stat;
+	u64				last_stat_abs_vusage;
+	u64				usage_delta_us;
+	u64				wait_since;
+	u64				indebt_since;
+	u64				indelay_since;
+
+	/* this iocg's depth in the hierarchy and ancestors including self */
+	int				level;
+	struct ioc_gq			*ancestors[];
+};
+
+/* per cgroup */
+struct ioc_cgrp {
+	struct blkcg_policy_data	cpd;
+	unsigned int			dfl_weight;
+};
+
+struct ioc_now {
+	u64				now_ns;
+	u64				now;
+	u64				vnow;
+};
+
+struct iocg_wait {
+	struct wait_queue_entry		wait;
+	struct bio			*bio;
+	u64				abs_cost;
+	bool				committed;
+};
+
+struct iocg_wake_ctx {
+	struct ioc_gq			*iocg;
+	u32				hw_inuse;
+	s64				vbudget;
+};
+
+extern struct blkcg_policy blkcg_policy_iocost;
+
+static inline struct ioc *rqos_to_ioc(struct rq_qos *rqos)
+{
+	return container_of(rqos, struct ioc, rqos);
+}
+
+static inline struct ioc *q_to_ioc(struct request_queue *q)
+{
+	return rqos_to_ioc(rq_qos_id(q, RQ_QOS_COST));
+}
+
+static inline const char __maybe_unused *ioc_name(struct ioc *ioc)
+{
+	struct gendisk *disk = ioc->rqos.disk;
+
+	if (!disk)
+		return "<unknown>";
+	return disk->disk_name;
+}
+
+static inline struct ioc_gq *pd_to_iocg(struct blkg_policy_data *pd)
+{
+	return pd ? container_of(pd, struct ioc_gq, pd) : NULL;
+}
+
+static inline struct ioc_gq *blkg_to_iocg(struct blkcg_gq *blkg)
+{
+	return pd_to_iocg(blkg_to_pd(blkg, &blkcg_policy_iocost));
+}
+
+static inline struct blkcg_gq *iocg_to_blkg(struct ioc_gq *iocg)
+{
+	return pd_to_blkg(&iocg->pd);
+}
+
+static inline struct ioc_cgrp *blkcg_to_iocc(struct blkcg *blkcg)
+{
+	return container_of(blkcg_to_cpd(blkcg, &blkcg_policy_iocost),
+			    struct ioc_cgrp, cpd);
+}
+
+#endif	/* _BLK_IOCOST_H */
-- 
2.43.0

[PATCH 2/5] blk-iocost: add BPF struct_ops cost model support

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

iocost prices IO with a linear model derived from a handful of
device parameters.  The real cost of an IO depends on device
internals the formula cannot capture and the right formula is
device specific.  This lets a BPF program take over pricing for a
device so that the model can be adapted to it without rebuilding
the kernel.

Attaching an iocost_model_ops struct_ops binds the model to one
device: the target is set in the dev member from userspace, .reg
creates the ioc if needed like an io.cost.model write and switches
the device to the model with the queue frozen before rq_qos_mutex
like an io.cost.qos write.  The attachment records the owning link
in the ioc and holds a reference on the map; detach verifies the
attachment under rq_qos_mutex and device removal ejects the model,
so a stale link cannot tear down a newer attachment.  io.cost.model
selects between the attached and builtin models without detaching
the struct_ops; only .unreg or device removal detaches.

While in use, the BPF model prices every charged IO: calc_cost()
is called from the submission path with the bio and returns the
cost in vtime units, clamped to one second of device time per IO.
The struct_ops also carries the transfer coefficients used by the
completion-time request sizing, so latency tracking and vrate
adjustment follow the model's pricing.  Models without calc_cost
or with zero or out-of-range transfer coefficients are rejected
at registration.

The cgroup callbacks pair with the iocg policy lifetime:
iocg_init() is delivered from ioc_pd_online() and, on attach, to
cgroups already online on the device; iocg_free() on detach and at
pd_free(), so each (cgroup, device) pair sees exactly one init and
one free.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 MAINTAINERS            |   1 +
 block/Kconfig          |   4 +
 block/Makefile         |   1 +
 block/blk-iocost-bpf.c | 464 +++++++++++++++++++++++++++++++++++++++++
 block/blk-iocost.c     | 124 ++++++++++-
 block/blk-iocost.h     | 181 +++++++++++++++-
 6 files changed, 763 insertions(+), 12 deletions(-)
 create mode 100644 block/blk-iocost-bpf.c

diff --git a/MAINTAINERS b/MAINTAINERS
index b3b421ee262e..598a5f65423d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6826,6 +6826,7 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git
 F:	Documentation/admin-guide/cgroup-v1/blkio-controller.rst
 F:	block/bfq-cgroup.c
 F:	block/blk-cgroup.c
+F:	block/blk-iocost-bpf.c
 F:	block/blk-iocost.c
 F:	block/blk-iocost.h
 F:	block/blk-iolatency.c
diff --git a/block/Kconfig b/block/Kconfig
index 70e4a66d941f..8eaa2a803b61 100644
--- a/block/Kconfig
+++ b/block/Kconfig
@@ -229,6 +229,10 @@ config BLK_ERROR_INJECTION
 
 	  See Documentation/block/error-injection.rst for details.
 
+config BLK_CGROUP_IOCOST_BPF
+	def_bool y
+	depends on BLK_CGROUP_IOCOST && BPF_SYSCALL && BPF_JIT && DEBUG_INFO_BTF
+
 source "block/Kconfig.iosched"
 
 endif # BLOCK
diff --git a/block/Makefile b/block/Makefile
index e7bd320e3d69..ee5cebeea006 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -39,3 +39,4 @@ obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= blk-crypto.o blk-crypto-profile.o \
 					   blk-crypto-sysfs.o
 obj-$(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK)	+= blk-crypto-fallback.o
 obj-$(CONFIG_BLOCK_HOLDER_DEPRECATED)	+= holder.o
+obj-$(CONFIG_BLK_CGROUP_IOCOST_BPF)	+= blk-iocost-bpf.o
diff --git a/block/blk-iocost-bpf.c b/block/blk-iocost-bpf.c
new file mode 100644
index 000000000000..8ff51c26fb25
--- /dev/null
+++ b/block/blk-iocost-bpf.c
@@ -0,0 +1,464 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * blk-iocost: BPF struct_ops plumbing for pluggable cost models.
+ *
+ * Copyright (C) 2026 KylinSoft Corporation.
+ *
+ * Registers the "iocost_model_ops" struct_ops type. Attachment is per-device:
+ * the target device is set in the ops from userspace before load, .reg
+ * attaches the model to that device, creating the ioc if needed and switching
+ * to the model with the queue frozen before rq_qos_mutex, like io.cost.qos
+ * writes, and .unreg detaches it; enabling and disabling the controller stays
+ * with io.cost.qos. The struct_ops core owns the program lifetime.
+ */
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf.h>
+#include "blk-iocost.h"
+#include <linux/blk-mq.h>
+#include <linux/mutex.h>
+#include "blk.h"
+
+/*
+/*
+ * Deliver iocg_init()/iocg_free() to the cgroups already online on the
+ * queue: init walks q->blkg_list in reverse so parents receive it first,
+ * like blkcg_activate_policy(); free walks it forward like the blkcg
+ * policy teardown. The queue is frozen and quiesced; queue_lock
+ * serializes against blkg creation and blkcg_mutex against the delayed
+ * list removal in blkg destruction; the callbacks are called with
+ * ioc->lock held, the same lock the attachment updates take.
+ */
+static void ioc_bpf_deliver_iocg_callbacks(struct ioc *ioc,
+					   const struct iocost_model_ops *ops,
+					   bool init)
+{
+	struct request_queue *q = ioc->rqos.disk->queue;
+	struct blkcg_gq *blkg;
+
+	lockdep_assert_held(&q->blkcg_mutex);
+	lockdep_assert_held(&q->queue_lock);
+	lockdep_assert_held(&ioc->lock);
+
+	if (init) {
+		/* parents first, like blkcg_activate_policy() */
+		list_for_each_entry_reverse(blkg, &q->blkg_list, q_node) {
+			struct ioc_gq *iocg = blkg_to_iocg(blkg);
+
+			/*
+			 * A blkg which never went online never reached the
+			 * radix tree, so it must not receive an init either;
+			 * the attach runs under the same queue_lock as
+			 * blkg_create(), so pd->online is stable here. Any
+			 * previous attachment had its delivered flags cleared
+			 * before it went away, so none can still be set here.
+			 */
+			if (!iocg || !iocg->pd.online)
+				continue;
+			if (ops->iocg_init) {
+				ops->iocg_init(blkg->blkcg, q);
+				iocg->bpf_iocg_inited = true;
+			}
+		}
+	} else {
+		list_for_each_entry(blkg, &q->blkg_list, q_node) {
+			struct ioc_gq *iocg = blkg_to_iocg(blkg);
+
+			if (!iocg || !iocg->bpf_iocg_inited)
+				continue;
+			if (ops->iocg_free)
+				ops->iocg_free(blkg->blkcg, q);
+			iocg->bpf_iocg_inited = false;
+		}
+	}
+}
+
+static int ioc_bpf_attach(struct iocost_model_ops *ops, struct bpf_link *link)
+{
+	struct block_device *bdev;
+	struct request_queue *q;
+	struct gendisk *disk;
+	struct ioc *ioc;
+	unsigned int memflags;
+	int ret;
+
+	bdev = blkdev_get_no_open(new_decode_dev(ops->dev), false);
+	if (!bdev)
+		return -ENODEV;
+	q = bdev->bd_queue;
+	disk = bdev->bd_disk;
+
+	if (bdev_is_partition(bdev)) {
+		blkdev_put_no_open(bdev);
+		return -EINVAL;
+	}
+	if (!queue_is_mq(q)) {
+		blkdev_put_no_open(bdev);
+		return -EOPNOTSUPP;
+	}
+
+	/*
+	 * The freeze is taken before rq_qos_mutex like ioc_qos_write() does;
+	 * holding the mutex from the disk_live() check through the model
+	 * switch keeps the ioc from being freed under us.
+	 */
+	memflags = blk_mq_freeze_queue(q);
+
+	mutex_lock(&q->rq_qos_mutex);
+
+	if (!disk_live(disk)) {
+		mutex_unlock(&q->rq_qos_mutex);
+		blk_mq_unfreeze_queue(q, memflags);
+		blkdev_put_no_open(bdev);
+		return -ENODEV;
+	}
+	ioc = q_to_ioc(q);
+	if (!ioc) {
+		ret = blk_iocost_init(disk);
+		if (ret) {
+			mutex_unlock(&q->rq_qos_mutex);
+			blk_mq_unfreeze_queue(q, memflags);
+			blkdev_put_no_open(bdev);
+			return ret;
+		}
+		ioc = q_to_ioc(q);
+	}
+
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Scope the mutex so the tail bdev put runs with blkcg_mutex
+	 * already released: dropping the last device reference can
+	 * reach disk_release() -> blkcg_exit_disk() ->
+	 * blkg_destroy_all(), which takes q->blkcg_mutex.
+	 */
+	{
+		guard(mutex)(&q->blkcg_mutex);
+		scoped_guard(spinlock_irqsave, &q->queue_lock) {
+			spin_lock(&ioc->lock);
+			ret = ioc_attached_model_locked(ioc) ? -EBUSY : 0;
+			if (!ret) {
+				/*
+				 * Take the map reference before the attachment
+				 * is published: the readers find the ops
+				 * through it. get() cannot fail while .reg
+				 * runs, as the core holds a map reference
+				 * then, but check it anyway; on failure leave
+				 * with no attachment set.
+				 */
+				if (!bpf_struct_ops_get(ops)) {
+					ret = -ENODEV;
+				} else {
+					rcu_assign_pointer(ioc->attached, ops);
+					rcu_assign_pointer(ioc->model, ops);
+					ioc->bpf_link = link;
+
+					/* pair iocg_init() with the cgroups online */
+					ioc_bpf_deliver_iocg_callbacks(ioc, ops,
+								       true);
+				}
+			}
+			spin_unlock(&ioc->lock);
+		}
+	}
+
+	blk_mq_unquiesce_queue(q);
+	blk_mq_unfreeze_queue(q, memflags);
+	mutex_unlock(&q->rq_qos_mutex);
+	blkdev_put_no_open(bdev);
+	return ret;
+}
+
+/*
+ * Detach a model: switch back to the builtin model when the attached model is
+ * in use, clear the attachment and deliver iocg_free() to the cgroups which
+ * still exist, under the same freeze and quiesce as the attach.
+ */
+static void ioc_bpf_detach(struct iocost_model_ops *ops, struct bpf_link *link)
+{
+	struct block_device *bdev;
+	struct request_queue *q;
+	struct ioc *ioc;
+	bool matched;
+	unsigned int memflags;
+
+	/*
+	 * A failed lookup means the device is gone; the removal ejection will
+	 * detach the model and drop the map reference when rq_qos_exit() runs,
+	 * so there is nothing to do here.
+	 */
+	bdev = blkdev_get_no_open(new_decode_dev(ops->dev), false);
+	if (!bdev)
+		return;
+	q = bdev->bd_queue;
+
+	/*
+	 * The attachment is checked under rq_qos_mutex first, so an unmatched
+	 * detach exits without freezing.
+	 */
+	mutex_lock(&q->rq_qos_mutex);
+	spin_lock_irq(&q->queue_lock);
+	ioc = q_to_ioc(q);
+	if (ioc) {
+		spin_lock(&ioc->lock);
+		matched = ioc_attached_model_locked(ioc) == ops &&
+			  ioc->bpf_link == link;
+		spin_unlock(&ioc->lock);
+	} else {
+		matched = false;
+	}
+	spin_unlock_irq(&q->queue_lock);
+	mutex_unlock(&q->rq_qos_mutex);
+	if (!matched) {
+		blkdev_put_no_open(bdev);
+		return;
+	}
+
+	/* the freeze is taken before rq_qos_mutex like ioc_qos_write() */
+	memflags = blk_mq_freeze_queue(q);
+
+	mutex_lock(&q->rq_qos_mutex);
+
+	/*
+	 * Re-check under the mutex before quiescing: the freeze itself does
+	 * not touch the tag set, and the removal ejection holds rq_qos_mutex
+	 * before the driver frees the tag set, so the check passing here means
+	 * the tag set is still alive.
+	 */
+	spin_lock_irq(&q->queue_lock);
+	ioc = q_to_ioc(q);
+	if (ioc) {
+		spin_lock(&ioc->lock);
+		matched = ioc_attached_model_locked(ioc) == ops &&
+			  ioc->bpf_link == link;
+		spin_unlock(&ioc->lock);
+	} else {
+		matched = false;
+	}
+	spin_unlock_irq(&q->queue_lock);
+	if (!matched) {
+		mutex_unlock(&q->rq_qos_mutex);
+		blk_mq_unfreeze_queue(q, memflags);
+		blkdev_put_no_open(bdev);
+		return;
+	}
+
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Scope the mutex so the tail bdev put runs with blkcg_mutex
+	 * already released: dropping the last device reference can
+	 * reach disk_release() -> blkcg_exit_disk() ->
+	 * blkg_destroy_all(), which takes q->blkcg_mutex.
+	 */
+	{
+		guard(mutex)(&q->blkcg_mutex);
+		scoped_guard(spinlock_irqsave, &q->queue_lock) {
+			spin_lock(&ioc->lock);
+			if (ioc_bpf_model_locked(ioc) == ops)
+				rcu_assign_pointer(ioc->model, NULL);
+			rcu_assign_pointer(ioc->attached, NULL);
+			ioc->bpf_link = NULL;
+
+			ioc_bpf_deliver_iocg_callbacks(ioc, ops, false);
+			spin_unlock(&ioc->lock);
+		}
+	}
+
+	blk_mq_unquiesce_queue(q);
+	blk_mq_unfreeze_queue(q, memflags);
+	mutex_unlock(&q->rq_qos_mutex);
+
+	blkdev_put_no_open(bdev);
+	bpf_struct_ops_put(ops);
+}
+
+/*
+ * Eject the model on device removal. The caller holds rq_qos_mutex and
+ * blkcg_deactivate_policy() has already run, so every existing blkg got
+ * iocg_free() from ioc_pd_free() and no walk is needed. The ioc is freed right
+ * after, so clearing the pointers and dropping the map reference is all that
+ * is left to do.
+ */
+void ioc_bpf_eject(struct ioc *ioc)
+{
+	const struct iocost_model_ops *ops;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ioc->lock, flags);
+	ops = ioc_attached_model_locked(ioc);
+	if (ops) {
+		rcu_assign_pointer(ioc->attached, NULL);
+		if (ioc_bpf_model_locked(ioc) == ops)
+			rcu_assign_pointer(ioc->model, NULL);
+		ioc->bpf_link = NULL;
+	}
+	spin_unlock_irqrestore(&ioc->lock, flags);
+
+	if (ops)
+		bpf_struct_ops_put(ops);
+}
+
+/* nothing to look up: the core finds the struct by name before calling */
+static int bpf_iocost_model_init(struct btf *btf)
+{
+	return 0;
+}
+
+static bool bpf_iocost_is_valid_access(int off, int size,
+				       enum bpf_access_type type,
+				       const struct bpf_prog *prog,
+				       struct bpf_insn_access_aux *info)
+{
+	return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
+}
+
+/*
+ * No iocost-specific helpers; bpf_base_func_proto already covers the
+ * cgroup storage helpers under CONFIG_CGROUPS.
+ */
+static const struct bpf_func_proto *
+bpf_iocost_get_func_proto(enum bpf_func_id func_id,
+			  const struct bpf_prog *prog)
+{
+	return bpf_base_func_proto(func_id, prog);
+}
+
+static int bpf_iocost_check_member(const struct btf_type *t,
+				   const struct btf_member *member,
+				   const struct bpf_prog *prog)
+{
+	/* every callback runs under RCU read lock or a spinlock */
+	if (prog->sleepable)
+		return -EINVAL;
+	return 0;
+}
+
+static int bpf_iocost_init_member(const struct btf_type *t,
+				  const struct btf_member *member,
+				  void *kdata, const void *udata)
+{
+	struct iocost_model_ops *ops = kdata;
+	const struct iocost_model_ops *uops = udata;
+	u32 moff = __btf_member_bit_offset(t, member) / 8;
+
+	switch (moff) {
+	/*
+	 * The struct_ops core rejects a nonzero userspace value for
+	 * non-function members this callback does not claim.
+	 */
+	case offsetof(struct iocost_model_ops, dev):
+		ops->dev = uops->dev;
+		return 1;
+	case offsetof(struct iocost_model_ops, read_vtime_per_page):
+		ops->read_vtime_per_page = uops->read_vtime_per_page;
+		return 1;
+	case offsetof(struct iocost_model_ops, write_vtime_per_page):
+		ops->write_vtime_per_page = uops->write_vtime_per_page;
+		return 1;
+	}
+
+	return 0;
+}
+
+/*
+ * A complete model prices every charged IO and carries both transfer
+ * coefficients; a missing calc_cost would be a NULL call from the
+ * submission path and missing coefficients would silently zero the
+ * completion-time sizing (see ioc_rqos_done()), so the model is
+ * rejected before it can be registered. iocg_init() and iocg_free()
+ * must come in pairs too: a model implementing only one of them would
+ * build per-cgroup state that nothing ever tears down.
+ */
+static bool iocost_model_complete(const struct iocost_model_ops *ops)
+{
+	return ops->calc_cost && ops->read_vtime_per_page &&
+	       ops->write_vtime_per_page &&
+	       !!ops->iocg_init == !!ops->iocg_free;
+}
+
+/*
+ * kvalue is zeroed at map allocation and function members are only written
+ * when the BPF side provides a prog, so a model which did not implement
+ * calc_cost leaves it NULL, and missing coefficients read as zero. This is
+ * reached from the map-update path of both map flavors, before the model can
+ * be registered.
+ */
+static int bpf_iocost_validate(void *kdata)
+{
+	struct iocost_model_ops *ops = kdata;
+
+	/*
+	 * Bound the transfer coefficients so that coefficient * pages cannot
+	 * wrap u64 for any possible request size.
+	 */
+	if (ops->read_vtime_per_page > (VTIME_PER_SEC >> IOC_SECT_TO_PAGE_SHIFT) ||
+	    ops->write_vtime_per_page > (VTIME_PER_SEC >> IOC_SECT_TO_PAGE_SHIFT))
+		return -EINVAL;
+
+	return iocost_model_complete(ops) ? 0 : -EINVAL;
+}
+
+static int bpf_iocost_reg(void *kdata, struct bpf_link *link)
+{
+	struct iocost_model_ops *ops = kdata;
+
+	/* validate() has run by now; only the device is left to check */
+	if (!ops->dev)
+		return -EINVAL;
+
+	return ioc_bpf_attach(ops, link);
+}
+
+static void bpf_iocost_unreg(void *kdata, struct bpf_link *link)
+{
+	struct iocost_model_ops *ops = kdata;
+
+	/* a gone device is handled inside ioc_bpf_detach() */
+	ioc_bpf_detach(ops, link);
+}
+
+static const struct bpf_verifier_ops bpf_iocost_verifier_ops = {
+	.get_func_proto = bpf_iocost_get_func_proto,
+	.is_valid_access = bpf_iocost_is_valid_access,
+};
+
+static u64 bpf_iocost_calc_cost_stub(struct bio *bio, u64 flags)
+{
+	return 0;
+}
+
+static void bpf_iocost_iocg_init_stub(struct blkcg *blkcg,
+				      struct request_queue *q)
+{ }
+static void bpf_iocost_iocg_free_stub(struct blkcg *blkcg,
+				      struct request_queue *q)
+{ }
+
+static struct iocost_model_ops __bpf_ops_iocost_model_ops = {
+	.calc_cost = bpf_iocost_calc_cost_stub,
+	.iocg_init = bpf_iocost_iocg_init_stub,
+	.iocg_free = bpf_iocost_iocg_free_stub,
+};
+
+static struct bpf_struct_ops bpf_iocost_model_ops = {
+	.verifier_ops = &bpf_iocost_verifier_ops,
+	.init = bpf_iocost_model_init,
+	.check_member = bpf_iocost_check_member,
+	.init_member = bpf_iocost_init_member,
+	.validate = bpf_iocost_validate,
+	.reg = bpf_iocost_reg,
+	.unreg = bpf_iocost_unreg,
+	.name = "iocost_model_ops",
+	.cfi_stubs = &__bpf_ops_iocost_model_ops,
+	.owner = THIS_MODULE,
+};
+
+static int __init bpf_iocost_init(void)
+{
+	return register_bpf_struct_ops(&bpf_iocost_model_ops, iocost_model_ops);
+}
+late_initcall(bpf_iocost_init);
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 9b610734871f..1e84232e6740 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -395,8 +395,14 @@ static int ioc_autop_idx(struct ioc *ioc, struct gendisk *disk)
 	if (idx < AUTOP_SSD_DFL)
 		return AUTOP_SSD_DFL;
 
-	/* if user is overriding anything, maintain what was there */
-	if (ioc->user_qos_params || ioc->user_cost_model)
+	/*
+	 * If user is overriding anything, or a BPF model is in use, maintain
+	 * what was there: with a BPF model the builtin cost coefficients are
+	 * inert, and stepping the profile would change the builtin parameters
+	 * that apply again after switching back, for no benefit in between.
+	 */
+	if (ioc->user_qos_params || ioc->user_cost_model ||
+	    ioc_bpf_model_locked(ioc))
 		return idx;
 
 	/* step up/down based on the vrate */
@@ -2178,7 +2184,9 @@ static u64 calc_vtime_cost(struct bio *bio, struct ioc_gq *iocg, bool is_merge)
 {
 	u64 cost;
 
-	calc_vtime_cost_builtin(bio, iocg, is_merge, &cost);
+	if (!ioc_model_calc_cost(iocg->ioc, bio, is_merge, &cost))
+		calc_vtime_cost_builtin(bio, iocg, is_merge, &cost);
+
 	return cost;
 }
 
@@ -2200,11 +2208,19 @@ static void calc_size_vtime_cost_builtin(struct request *rq, struct ioc *ioc,
 	}
 }
 
-static u64 calc_size_vtime_cost(struct request *rq, struct ioc *ioc)
+/*
+ * Called from the request completion path for READ and WRITE
+ * requests, where no ioc->lock is held; the model pointer is read
+ * under RCU, matching the bio-side calc_vtime_cost().
+ */
+static u64 calc_size_vtime_cost(struct request *rq, struct ioc *ioc,
+				bool is_read)
 {
 	u64 cost;
 
-	calc_size_vtime_cost_builtin(rq, ioc, &cost);
+	if (!ioc_model_size_cost(ioc, rq, is_read, &cost))
+		calc_size_vtime_cost_builtin(rq, ioc, &cost);
+
 	return cost;
 }
 
@@ -2474,7 +2490,8 @@ static void ioc_rqos_done(struct rq_qos *rqos, struct request *rq)
 
 	on_q_ns = blk_time_get_ns() - rq->alloc_time_ns;
 	rq_wait_ns = rq->start_time_ns - rq->alloc_time_ns;
-	size_nsec = div64_u64(calc_size_vtime_cost(rq, ioc), VTIME_PER_NSEC);
+	size_nsec = div64_u64(calc_size_vtime_cost(rq, ioc, pidx == QOS_RLAT),
+			       VTIME_PER_NSEC);
 
 	ccs = get_cpu_ptr(ioc->pcpu_stat);
 
@@ -2510,6 +2527,10 @@ static void ioc_rqos_exit(struct rq_qos *rqos)
 
 	timer_shutdown_sync(&ioc->timer);
 	free_percpu(ioc->pcpu_stat);
+
+	/* eject the model on device removal */
+	ioc_bpf_eject(ioc);
+
 	kfree(ioc);
 }
 
@@ -2522,7 +2543,7 @@ static const struct rq_qos_ops ioc_rqos_ops = {
 	.exit = ioc_rqos_exit,
 };
 
-static int blk_iocost_init(struct gendisk *disk)
+int blk_iocost_init(struct gendisk *disk)
 {
 	struct ioc *ioc;
 	int i, cpu, ret;
@@ -2660,6 +2681,35 @@ static void ioc_pd_init(struct blkg_policy_data *pd)
 	spin_unlock_irqrestore(&ioc->lock, flags);
 }
 
+/*
+ * iocg_init() delivery for the attached model. blkg_create() calls this only
+ * after the blkg has been inserted into the radix tree and the queue's blkg
+ * list, so a blkg whose insertion failed is never seen here and the init/free
+ * pairing cannot be broken by a model switch in between; the caller holds
+ * queue_lock, the same lock the attachment updates are serialized with.
+ */
+static void ioc_pd_online(struct blkg_policy_data *pd)
+{
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+	struct ioc_gq *iocg = pd_to_iocg(pd);
+	struct blkcg_gq *blkg = pd_to_blkg(pd);
+	struct ioc *ioc = iocg->ioc;
+	const struct iocost_model_ops *model;
+	unsigned long flags;
+
+	if (!ioc)
+		return;
+
+	spin_lock_irqsave(&ioc->lock, flags);
+	model = ioc_attached_model_locked(ioc);
+	if (model && model->iocg_init) {
+		model->iocg_init(blkg->blkcg, ioc->rqos.disk->queue);
+		iocg->bpf_iocg_inited = true;
+	}
+	spin_unlock_irqrestore(&ioc->lock, flags);
+#endif
+}
+
 static void iocg_release(struct rcu_head *rcu)
 {
 	struct blkg_policy_data *pd =
@@ -2680,6 +2730,25 @@ static void ioc_pd_free(struct blkg_policy_data *pd)
 	if (ioc) {
 		spin_lock_irqsave(&ioc->lock, flags);
 
+		/*
+		 * Gate the free on the delivered flag: a blkg which never went
+		 * online never received iocg_init(), and bios still holding
+		 * the blkg can be priced between its going offline and this
+		 * free, so iocg_free() follows pd_free() rather than
+		 * pd_offline.
+		 */
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+		if (iocg->bpf_iocg_inited) {
+			const struct iocost_model_ops *amodel =
+				ioc_attached_model_locked(ioc);
+
+			if (amodel && amodel->iocg_free)
+				amodel->iocg_free(blkg->blkcg,
+						  ioc->rqos.disk->queue);
+			iocg->bpf_iocg_inited = false;
+		}
+#endif
+
 		if (!list_empty(&iocg->active_list)) {
 			struct ioc_now now;
 
@@ -3042,16 +3111,19 @@ static u64 ioc_cost_model_prfill(struct seq_file *sf,
 	const char *dname = blkg_dev_name(pd->blkg);
 	struct ioc *ioc = pd_to_iocg(pd)->ioc;
 	u64 *u = ioc->params.i_lcoefs;
+	const struct iocost_model_ops *model;
 
 	if (!dname)
 		return 0;
 
 	spin_lock_irq(&ioc->lock);
-	seq_printf(sf, "%s ctrl=%s model=linear "
+	model = ioc_bpf_model_locked(ioc);
+	seq_printf(sf, "%s ctrl=%s model=%s "
 		   "rbps=%llu rseqiops=%llu rrandiops=%llu "
 		   "wbps=%llu wseqiops=%llu wrandiops=%llu "
 		   "flushiops=%llu\n",
 		   dname, ioc->user_cost_model ? "user" : "auto",
+		   model ? "bpf" : "linear",
 		   u[I_LCOEF_RBPS], u[I_LCOEF_RSEQIOPS], u[I_LCOEF_RRANDIOPS],
 		   u[I_LCOEF_WBPS], u[I_LCOEF_WSEQIOPS], u[I_LCOEF_WRANDIOPS],
 		   u[I_LCOEF_FLUSHIOPS]);
@@ -3096,6 +3168,10 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
 	bool user;
 	char *body, *p;
 	int ret;
+	const struct iocost_model_ops *new_model = NULL;
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+	bool model_write = false;
+#endif
 
 	blkg_conf_init(&ctx, input);
 
@@ -3148,9 +3224,30 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
 			continue;
 		case COST_MODEL:
 			match_strlcpy(buf, &args[0], sizeof(buf));
-			if (strcmp(buf, "linear"))
-				goto unlock;
-			continue;
+			if (!strcmp(buf, "linear")) {
+				/*
+				 * Staged and committed below, so a parse
+				 * error later in the same write leaves the
+				 * model selection untouched.
+				 */
+				new_model = NULL;
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+				model_write = true;
+#endif
+				continue;
+			}
+			if (!strcmp(buf, "bpf")) {
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+				new_model = rcu_dereference_protected(
+						ioc->attached,
+						lockdep_is_held(&ioc->lock));
+				model_write = true;
+#endif
+				if (!new_model)
+					goto unlock;
+				continue;
+			}
+			goto unlock;
 		}
 
 		tok = match_token(p, i_lcoef_tokens, args);
@@ -3168,6 +3265,10 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
 	} else {
 		ioc->user_cost_model = false;
 	}
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+	if (model_write)
+		rcu_assign_pointer(ioc->model, new_model);
+#endif
 	ioc_refresh_params(ioc, true);
 
 	ret = 0;
@@ -3211,6 +3312,7 @@ struct blkcg_policy blkcg_policy_iocost = {
 	.cpd_free_fn	= ioc_cpd_free,
 	.pd_alloc_fn	= ioc_pd_alloc,
 	.pd_init_fn	= ioc_pd_init,
+	.pd_online_fn	= ioc_pd_online,
 	.pd_free_fn	= ioc_pd_free,
 	.pd_stat_fn	= ioc_pd_stat,
 };
diff --git a/block/blk-iocost.h b/block/blk-iocost.h
index e4e517ef6a0f..6d62b4fd0c54 100644
--- a/block/blk-iocost.h
+++ b/block/blk-iocost.h
@@ -22,6 +22,79 @@
 #include <asm/local64.h>
 #include "blk-rq-qos.h"
 
+struct bpf_link;
+
+/*
+ * Pluggable cost model interface for blk-iocost.
+ *
+ * A BPF struct_ops implementation can take over the cost model of one device.
+ * The device is identified by the dev member, which is set from userspace
+ * before the program is loaded; the struct_ops core owns the program lifetime.
+ * Attaching creates the ioc if needed, like an io.cost.model write, and
+ * switches the device to the BPF model with the queue frozen before
+ * rq_qos_mutex, like an io.cost.qos write, and delivers iocg_init() to the
+ * cgroups which already exist on the device. Enabling and disabling the
+ * controller stays with io.cost.qos.
+ *
+ * While a model is attached, io.cost.model selects between it and the builtin
+ * model: "model=bpf" switches to the attached model and "model=linear" back to
+ * the builtin. Neither detaches the struct_ops; only detaching does. The
+ * builtin linear coefficients are kept while the BPF model is in use and take
+ * effect again when switched back.
+ *
+ * While in use, the BPF model prices every charged IO on the device, including
+ * flushes, from the bio charging path. calc_cost() is called from the IO
+ * submission path with RCU read lock held and must not sleep. It receives the
+ * bio itself so that the model can use what it needs (operation flags, size,
+ * sector, the issuing cgroup through bio->bi_blkg, which is guaranteed
+ * non-NULL) and returns the cost of the IO in vtime units, where 1 second
+ * of device time equals VTIME_PER_SEC (2^37,
+ * available to BPF programs through vmlinux.h). The return value is clamped to
+ * 1 second of device time per IO.
+ *
+ * The struct_ops also carries the transfer cost coefficients, vtime per page
+ * for reads and writes. While the BPF model is in use, they replace the
+ * builtin coefficients in the completion-time request sizing of READ and
+ * WRITE requests, with ZONE_APPEND counted as a write. Taking over the QoS
+ * side (latency tracking, vrate control) is left for later extensions.
+ *
+ * The cgroup callbacks are bound to the iocg policy lifetime, one (cgroup,
+ * device) pair per invocation. iocg_init() is delivered from ioc_pd_online()
+ * and on attach to the cgroups already online on the device; iocg_free() on
+ * detach to the cgroups which still exist then and at pd_free() time for the
+ * rest, so that init and free pair up exactly. iocg_free() is gated on a
+ * per-iocg delivered flag, as bios can still be priced between the blkg going
+ * offline and its free. Both callbacks are called with ioc->lock held and must
+ * not sleep; the delivery points nest inside queue_lock except for iocg_free()
+ * from ioc_pd_free() in blkg_free_workfn().
+ */
+
+/*
+ * Call metadata for calc_cost()'s model_flags; the merge indicator is not a
+ * property of the bio. An enum so that the value is exported through BTF and
+ * usable from vmlinux.h.
+ */
+enum {
+	IOCOST_COST_F_MERGE	= 1 << 0,	/* called from merge path */
+};
+
+struct iocost_model_ops {
+	/*
+	 * target device (major:minor), set from userspace before load
+	 * through the struct_ops map's initial value, in the userspace
+	 * dev_t encoding new_decode_dev() accepts
+	 */
+	dev_t dev;
+
+	/* vtime per page for the completion-time request sizing */
+	u64 read_vtime_per_page;
+	u64 write_vtime_per_page;
+
+	u64 (*calc_cost)(struct bio *bio, u64 model_flags);
+	void (*iocg_init)(struct blkcg *blkcg, struct request_queue *q);
+	void (*iocg_free)(struct blkcg *blkcg, struct request_queue *q);
+};
+
 enum {
 	MILLION			= 1000000,
 
@@ -260,7 +333,14 @@ struct ioc {
 	int				autop_idx;
 	bool				user_qos_params:1;
 	bool				user_cost_model:1;
-
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+	/* the struct_ops attached to this device, NULL = none */
+	const struct iocost_model_ops	__rcu *attached;
+	/* the link which owns the attachment, written with ioc->lock held */
+	struct bpf_link			*bpf_link;
+	/* the cost model in use, NULL = builtin linear model */
+	const struct iocost_model_ops	__rcu *model;
+#endif
 };
 
 struct iocg_pcpu_stat {
@@ -278,6 +358,10 @@ struct iocg_stat {
 struct ioc_gq {
 	struct blkg_policy_data		pd;
 	struct ioc			*ioc;
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+	/* whether the attached model's iocg_init() was delivered */
+	bool				bpf_iocg_inited;
+#endif
 
 	/*
 	 * A iocg can get its weight from two sources - an explicit
@@ -433,4 +517,99 @@ static inline struct ioc_cgrp *blkcg_to_iocc(struct blkcg *blkcg)
 			    struct ioc_cgrp, cpd);
 }
 
+/*
+ * The attached struct_ops and the model in use. The writers hold
+ * ioc->lock, the hot path readers are inside rcu_read_lock() and the
+ * map is not freed until after an RCU grace period, the slow readers
+ * hold ioc->lock.
+ */
+static inline const struct iocost_model_ops *
+ioc_attached_model_locked(struct ioc *ioc)
+{
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+	return rcu_dereference_protected(ioc->attached,
+					 lockdep_is_held(&ioc->lock));
+#else
+	return NULL;
+#endif
+}
+
+static inline const struct iocost_model_ops *
+ioc_bpf_model_locked(struct ioc *ioc)
+{
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+	return rcu_dereference_protected(ioc->model,
+					 lockdep_is_held(&ioc->lock));
+#else
+	return NULL;
+#endif
+}
+
+/*
+ * Price one charged IO with the model in use into *@cost and return
+ * true, or return false when no model is in use and the builtin model
+ * must price it. The model is read under rcu_read_lock(), which the
+ * helper takes and releases itself, and the returned cost is clamped
+ * to one second of device time. calc_cost() may be invoked for a
+ * (cgroup, device) pair before iocg_init() has been delivered for
+ * it; models relying on per-pair state must tolerate that window.
+ */
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+static inline bool ioc_model_calc_cost(struct ioc *ioc, struct bio *bio,
+				       bool is_merge, u64 *cost)
+{
+	const struct iocost_model_ops *model;
+
+	rcu_read_lock();
+	model = rcu_dereference(ioc->model);
+	if (model)
+		*cost = min(model->calc_cost(bio, is_merge ?
+			    IOCOST_COST_F_MERGE : 0), VTIME_PER_SEC);
+	rcu_read_unlock();
+	return model;
+}
+#else	/* CONFIG_BLK_CGROUP_IOCOST_BPF */
+static inline bool ioc_model_calc_cost(struct ioc *ioc, struct bio *bio,
+				       bool is_merge, u64 *cost)
+{
+	return false;
+}
+#endif	/* CONFIG_BLK_CGROUP_IOCOST_BPF */
+
+/*
+ * The same for the completion-time request sizing: the transfer cost
+ * is the model's vtime per page for the request direction, and the
+ * returned cost is clamped to one second of device time.
+ */
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+static inline bool ioc_model_size_cost(struct ioc *ioc, struct request *rq,
+				       bool is_read, u64 *cost)
+{
+	const struct iocost_model_ops *model;
+
+	rcu_read_lock();
+	model = rcu_dereference(ioc->model);
+	if (model)
+		*cost = min((is_read ? model->read_vtime_per_page :
+			     model->write_vtime_per_page) *
+			    (blk_rq_stats_sectors(rq) >> IOC_SECT_TO_PAGE_SHIFT),
+			    VTIME_PER_SEC);
+	rcu_read_unlock();
+	return model;
+}
+#else	/* CONFIG_BLK_CGROUP_IOCOST_BPF */
+static inline bool ioc_model_size_cost(struct ioc *ioc, struct request *rq,
+				       bool is_read, u64 *cost)
+{
+	return false;
+}
+#endif	/* CONFIG_BLK_CGROUP_IOCOST_BPF */
+
+int blk_iocost_init(struct gendisk *disk);
+
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+void ioc_bpf_eject(struct ioc *ioc);
+#else	/* CONFIG_BLK_CGROUP_IOCOST_BPF */
+static inline void ioc_bpf_eject(struct ioc *ioc) { }
+#endif	/* CONFIG_BLK_CGROUP_IOCOST_BPF */
 #endif	/* _BLK_IOCOST_H */
-- 
2.43.0

[PATCH 3/5] selftests/bpf: add iocost cost model test

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

Add two example iocost cost models and tests which attach them to a
null_blk instance the tests create through configfs, so everything
runs without external setup.

The first model implements the builtin linear HDD formula at double
cost, plus iocg_init()/iocg_free() counters for the pairing checks.
The second implements the multi-stream sequentiality detection: it
keeps a per-cgroup table of stream slots instead of the single
builtin cursor, so concurrent sequential streams in one cgroup are
priced sequentially.  Both set the target device through the
struct_ops member before load, and the maps are declared
".struct_ops.link" so that a test dying in between does not leave
the model attached.

The tests verify the model=bpf readback while attached, that
ctrl=bpf is rejected, that a second model on the same device fails
with -EBUSY, that model=linear and model=bpf switch between the
attached and builtin models without detaching, and that detaching
restores the builtin model.  With the controller enabled through
io.cost.qos, they also verify that IOs are priced through
calc_cost(), that the iocost_ioc_tick tracepoint is seen from BPF,
and that iocg_init()/iocg_free() pair up across attach, detach, a
cgroup created while attached, and device removal under the
attached model, with a recreated device attachable again.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 tools/testing/selftests/bpf/config            |   2 +
 .../selftests/bpf/prog_tests/iocost_model.c   | 806 ++++++++++++++++++
 .../selftests/bpf/progs/iocost_model.c        | 201 +++++
 tools/testing/selftests/bpf/progs/iocost_ms.c | 175 ++++
 4 files changed, 1184 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/iocost_model.c
 create mode 100644 tools/testing/selftests/bpf/progs/iocost_model.c
 create mode 100644 tools/testing/selftests/bpf/progs/iocost_ms.c

diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index 49dfe939467c..78ed59f83a16 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -141,3 +141,5 @@ CONFIG_SMC_HS_CTRL_BPF=y
 CONFIG_DIBS=y
 CONFIG_DIBS_LO=y
 CONFIG_PM_WAKELOCKS=y
+CONFIG_BLK_CGROUP_IOCOST=y
+CONFIG_BLK_DEV_NULL_BLK=y
diff --git a/tools/testing/selftests/bpf/prog_tests/iocost_model.c b/tools/testing/selftests/bpf/prog_tests/iocost_model.c
new file mode 100644
index 000000000000..5373d52d93a6
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/iocost_model.c
@@ -0,0 +1,806 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (C) 2026 KylinSoft Corporation. */
+#define _GNU_SOURCE
+#include <test_progs.h>
+#include <fcntl.h>
+#include <sched.h>
+#include <sys/statfs.h>
+#include <linux/magic.h>
+#include <sys/mount.h>
+#include <sys/stat.h>
+#include <sys/sysmacros.h>
+#include <unistd.h>
+#include "iocost_model.skel.h"
+#include "iocost_ms.skel.h"
+
+/*
+ * Write val to the configfs attribute at path, returning the errno of the
+ * write negated (0 on success).
+ */
+static int configfs_write(const char *path, const char *val)
+{
+	FILE *fp;
+	int err = 0;
+
+	fp = fopen(path, "w");
+	if (!fp)
+		return -errno;
+	if (fwrite(val, 1, strlen(val), fp) != strlen(val))
+		err = ferror(fp) ? -errno : -EIO;
+	if (fclose(fp) && !err)
+		err = -errno;
+	return err;
+}
+
+/*
+ * Read the first token of the file at path, for the dev attribute.
+ */
+static int read_file_token(const char *path, char *tok, size_t tok_sz)
+{
+	char fmt[16];
+	FILE *fp;
+
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+	snprintf(fmt, sizeof(fmt), "%%%zus", tok_sz - 1);
+	if (fscanf(fp, fmt, tok) != 1) {
+		fclose(fp);
+		return -EIO;
+	}
+	fclose(fp);
+	return 0;
+}
+
+/*
+ * Create a null_blk instance through configfs and copy its maj:min
+ * into @dev. The device appears only once power is written, and is
+ * removed by rmdir of the instance directory; the maj:min comes from
+ * the sysfs directory of the created disk, which is named after the
+ * instance.
+ */
+static int nullb_create(const char *name, char *dev, size_t dev_sz)
+{
+	char path[128];
+	char p2[128];
+	int err;
+
+	snprintf(path, sizeof(path), "/sys/kernel/config/nullb/%s", name);
+	if (mkdir(path, 0755)) {
+		if (errno != EEXIST)
+			return -errno;
+		/*
+		 * a leftover instance from an earlier crashed run may
+		 * still be powered: tear it down before reusing the
+		 * name
+		 */
+		snprintf(p2, sizeof(p2),
+			 "/sys/kernel/config/nullb/%s/power", name);
+		configfs_write(p2, "0");
+		if (rmdir(path) && errno != ENOENT)
+			return -errno;
+		if (mkdir(path, 0755) && errno != EEXIST)
+			return -errno;
+	}
+
+	snprintf(path, sizeof(path), "/sys/kernel/config/nullb/%s/size", name);
+	err = configfs_write(path, "128");
+	if (err)
+		return err;
+
+	snprintf(path, sizeof(path), "/sys/kernel/config/nullb/%s/power", name);
+	err = configfs_write(path, "1");
+	if (err)
+		return err;
+
+	snprintf(path, sizeof(path), "/sys/block/%s/dev", name);
+	err = read_file_token(path, dev, dev_sz);
+	if (err) {
+		/* do not leave a powered instance behind */
+		snprintf(path, sizeof(path),
+			 "/sys/kernel/config/nullb/%s/power", name);
+		configfs_write(path, "0");
+		snprintf(path, sizeof(path),
+			 "/sys/kernel/config/nullb/%s", name);
+		rmdir(path);
+	}
+	return err;
+}
+
+static int nullb_remove(const char *name)
+{
+	char path[128];
+
+	/* a powered instance cannot be removed */
+	snprintf(path, sizeof(path), "/sys/kernel/config/nullb/%s/power", name);
+	configfs_write(path, "0");
+	snprintf(path, sizeof(path), "/sys/kernel/config/nullb/%s", name);
+	if (rmdir(path))
+		return -errno;
+	return 0;
+}
+
+#define IOCOST_NULLB	"iocost_model_test"
+#define IOCOST_NULLB2	"iocost_model_test2"
+
+/*
+ * The tests run against their own null_blk instance created through configfs,
+ * so they need root, configfs and the null_blk driver.
+ */
+/*
+ * The BPF cost model needs cgroup v2 at /sys/fs/cgroup and
+ * CONFIG_BLK_CGROUP_IOCOST_BPF in the running kernel; both are environment
+ * properties, so their absence is a skip.
+ */
+static bool iocost_bpf_supported(void)
+{
+	FILE *f;
+	char line[128];
+	bool any = false, on = false;
+
+	f = popen("zcat /proc/config.gz 2>/dev/null", "r");
+	if (!f)
+		return false;
+	while (fgets(line, sizeof(line), f)) {
+		if (strstr(line, "CONFIG_"))
+			any = true;
+		if (strstr(line, "CONFIG_BLK_CGROUP_IOCOST_BPF=y"))
+			on = true;
+	}
+	pclose(f);
+	return on;
+}
+
+static bool iocost_test_env(void)
+{
+	struct statfs sf;
+
+	if (geteuid() != 0) {
+		printf("%s:SKIP:needs root\n", __func__);
+		test__skip();
+		return false;
+	}
+
+	if (statfs("/sys/fs/cgroup", &sf) ||
+	    sf.f_type != CGROUP2_SUPER_MAGIC) {
+		printf("%s:SKIP:needs cgroup v2 at /sys/fs/cgroup\n",
+		       __func__);
+		test__skip();
+		return false;
+	}
+
+	if (!iocost_bpf_supported()) {
+		printf("%s:SKIP:needs CONFIG_BLK_CGROUP_IOCOST_BPF\n",
+		       __func__);
+		test__skip();
+		return false;
+	}
+
+	/* isolate the configfs and cgroup mounts from the host */
+	if (unshare(CLONE_NEWNS)) {
+		printf("%s:SKIP:unshare(CLONE_NEWNS) failed: %s\n",
+		       __func__, strerror(errno));
+		test__skip();
+		return false;
+	}
+	mkdir("/sys/kernel/config", 0755);
+	(void)mount("none", "/sys/kernel/config", "configfs", 0, NULL);
+	SYS_NOFAIL("modprobe null_blk nr_devices=0");
+	if (access("/sys/kernel/config/nullb", W_OK)) {
+		printf("%s:SKIP:needs null_blk with configfs support\n",
+		       __func__);
+		test__skip();
+		return false;
+	}
+	return true;
+}
+
+/*
+ * Read back the io.cost.model line of dev and copy the model= value into
+ * @model. Returns 0 on success.
+ */
+static int readback_model(const char *dev, char *model, size_t model_sz)
+{
+	char line[256], word[256], *m, *end;
+	FILE *fp;
+	int found = 0;
+
+	fp = fopen("/sys/fs/cgroup/io.cost.model", "r");
+	if (!fp)
+		return -1;
+	while (fgets(line, sizeof(line), fp)) {
+		if (sscanf(line, "%255s", word) == 1 && !strcmp(word, dev)) {
+			found = 1;
+			break;
+		}
+	}
+	fclose(fp);
+	if (!found)
+		return -1;
+
+	m = strstr(line, "model=");
+	if (!m)
+		return -1;
+	m += strlen("model=");
+	end = m;
+	while (*end && *end != ' ')
+		end++;
+	snprintf(model, model_sz, "%.*s", (int)(end - m), m);
+	return 0;
+}
+
+/*
+ * Read the cost.usage accumulator of dev from the cgroup's io.stat. Returns 0
+ * on success, -ENODEV when the device has no line yet (no charge recorded).
+ */
+static int read_cost_usage(const char *cgroup, const char *dev,
+			   unsigned long long *usage)
+{
+	char line[512];
+	FILE *fp;
+	int found = 0;
+
+	fp = fopen(cgroup, "r");
+	if (!fp)
+		return -EIO;
+	*usage = 0;
+	while (fgets(line, sizeof(line), fp)) {
+		char *tok = strtok(line, " \n");
+
+		if (!tok || strcmp(tok, dev))
+			continue;
+		found = 1;
+		while ((tok = strtok(NULL, " \n"))) {
+			if (!strncmp(tok, "cost.usage=", 11)) {
+				*usage = strtoull(tok + 11, NULL, 0);
+				fclose(fp);
+				return 0;
+			}
+		}
+	}
+	fclose(fp);
+	return found ? -ENOENT : -ENODEV;
+}
+
+/* mkdir, recovering from a leftover of an aborted earlier run */
+static int cgroup_mkdir(const char *path)
+{
+	if (!mkdir(path, 0755))
+		return 0;
+	if (errno == EEXIST && !rmdir(path))
+		return mkdir(path, 0755) ? -errno : 0;
+	return -errno;
+}
+
+/*
+ * Write a line to io.cost.model for dev and return the errno of the write
+ * negated (0 on success), as ASSERT_ERR() expects.
+ */
+static int write_cost_model(const char *dev, const char *what)
+{
+	char buf[128];
+	FILE *fp;
+	int err = 0;
+
+	snprintf(buf, sizeof(buf), "%s %s", dev, what);
+	fp = fopen("/sys/fs/cgroup/io.cost.model", "w");
+	if (!fp)
+		return -EACCES;
+	if (fwrite(buf, 1, strlen(buf), fp) != strlen(buf))
+		err = ferror(fp) ? -errno : -EIO;
+	if (fclose(fp) && !err)
+		err = -errno;
+	return err;
+}
+
+/*
+ * Attach the example model to the null_blk instance the test creates: the dev
+ * member is written through the struct_ops map's initial value before load,
+ * and attaching the struct_ops attaches the model to the device. Detaching the
+ * struct_ops restores the builtin model.
+ *
+ * Requires root, cgroup v2 and null_blk with configfs support.
+ */
+void serial_test_iocost_model(void)
+{
+	struct iocost_model *skel, *second, *bad;
+	unsigned int maj, min;
+	char pidbuf[16], buf[4096], path[64], qbuf[160], dev2[32];
+	int devfd, i, err, lfd;
+	char model[32], dev[32];
+	unsigned long long u1, u2, inits1;
+
+	if (!iocost_test_env())
+		return;
+	if (!ASSERT_OK(nullb_create(IOCOST_NULLB, dev, sizeof(dev)),
+		       "nullb_create"))
+		return;
+	if (sscanf(dev, "%u:%u", &maj, &min) != 2) {
+		ASSERT_TRUE(false, "dev_parse");
+		nullb_remove(IOCOST_NULLB);
+		return;
+	}
+
+	skel = iocost_model__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open")) {
+		nullb_remove(IOCOST_NULLB);
+		return;
+	}
+
+	/*
+	 * a model with a zero coefficient must fail to attach instead of
+	 * silently zeroing the completion sizing (validate rejects it on
+	 * the map update the attach performs; the link form of the
+	 * struct_ops only reaches it there)
+	 */
+	bad = iocost_model__open();
+	if (!ASSERT_OK_PTR(bad, "bad_open")) {
+		iocost_model__destroy(skel);
+		nullb_remove(IOCOST_NULLB);
+		return;
+	}
+	bad->struct_ops.iocost_2x->dev = (__u32)makedev(maj, min);
+	bad->struct_ops.iocost_2x->write_vtime_per_page = 0;
+	if (ASSERT_OK(iocost_model__load(bad), "bad_load"))
+		ASSERT_EQ(iocost_model__attach(bad), -EINVAL,
+			  "attach_rejects_zero_coeff");
+	iocost_model__destroy(bad);
+
+	/* same for the read-side transfer coefficient */
+	bad = iocost_model__open();
+	if (!ASSERT_OK_PTR(bad, "bad_read_open")) {
+		iocost_model__destroy(skel);
+		nullb_remove(IOCOST_NULLB);
+		return;
+	}
+	bad->struct_ops.iocost_2x->dev = (__u32)makedev(maj, min);
+	bad->struct_ops.iocost_2x->read_vtime_per_page = 0;
+	if (ASSERT_OK(iocost_model__load(bad), "bad_read_load"))
+		ASSERT_EQ(iocost_model__attach(bad), -EINVAL,
+			  "attach_rejects_zero_read");
+	iocost_model__destroy(bad);
+
+	skel->struct_ops.iocost_2x->dev = (__u32)makedev(maj, min);
+	/*
+	 * create a cgroup for the pairing checks; its blkg comes to life
+	 * when the controller is enabled and the cgroup configured below,
+	 * exercising the iocg_init()/iocg_free() pairing end to end
+	 */
+	/*
+	 * Enable the io controller for the pairing cgroup: without it
+	 * the cgroup has no io.weight/io.stat and the iocg never
+	 * activates, so the model is consulted but nothing is charged.
+	 */
+	ASSERT_OK(configfs_write("/sys/fs/cgroup/cgroup.subtree_control",
+				 "+io"), "subtree_io");
+	if (!ASSERT_OK(cgroup_mkdir("/sys/fs/cgroup/iocost_pairing"),
+		       "mkdir_cg")) {
+		iocost_model__destroy(skel);
+		nullb_remove(IOCOST_NULLB);
+		return;
+	}
+	snprintf(pidbuf, sizeof(pidbuf), "%d", getpid());
+	ASSERT_OK(configfs_write("/sys/fs/cgroup/iocost_pairing/cgroup.procs",
+				  pidbuf), "cgroup_procs");
+	snprintf(path, sizeof(path), "/dev/%s", IOCOST_NULLB);
+	devfd = open(path, O_RDONLY);
+	if (!ASSERT_OK(devfd < 0 ? -errno : 0, "open_dev")) {
+		configfs_write("/sys/fs/cgroup/cgroup.procs", pidbuf);
+		rmdir("/sys/fs/cgroup/iocost_pairing");
+		iocost_model__destroy(skel);
+		nullb_remove(IOCOST_NULLB);
+		return;
+	}
+	for (i = 0; i < 32; i++)
+		(void)read(devfd, buf, sizeof(buf));
+	close(devfd);
+	configfs_write("/sys/fs/cgroup/cgroup.procs", pidbuf);
+
+
+	err = iocost_model__load(skel);
+	if (!ASSERT_OK(err, "skel_load")) {
+		iocost_model__destroy(skel);
+		rmdir("/sys/fs/cgroup/iocost_pairing");
+		nullb_remove(IOCOST_NULLB);
+		return;
+	}
+
+	err = iocost_model__attach(skel);
+	if (err == -EOPNOTSUPP || err == -ENODEV) {
+		printf("%s:SKIP:device is not blk-mq, has no iocost or does not exist\n",
+		       __func__);
+		iocost_model__destroy(skel);
+		rmdir("/sys/fs/cgroup/iocost_pairing");
+		nullb_remove(IOCOST_NULLB);
+		test__skip();
+		return;
+	}
+	if (ASSERT_OK(err, "attach")) {
+		/*
+		 * attached: the read path reports model=bpf until the
+		 * struct_ops is detached; ctrl keeps describing the
+		 * builtin coefficients
+		 */
+		err = readback_model(dev, model, sizeof(model));
+		if (ASSERT_OK(err, "readback"))
+			ASSERT_STREQ(model, "bpf", "model_bpf");
+
+		/*
+		 * the blkg created above was already on the blkg list
+		 * (online) when the attach walk ran, so the walk
+		 * delivered iocg_init() to it and to every other cgroup
+		 * which has a blkg on this device (the count depends on
+		 * the environment); none of them can have received
+		 * iocg_free() yet
+		 */
+		ASSERT_GT(skel->bss->iocg_inits, 0, "pair_init");
+		ASSERT_EQ(skel->bss->iocg_frees, 0, "pair_no_free");
+
+		/* a second model on the same device fails with -EBUSY */
+		second = iocost_model__open();
+		if (ASSERT_OK_PTR(second, "second_open")) {
+			second->struct_ops.iocost_2x->dev =
+				(__u32)makedev(maj, min);
+			err = iocost_model__load(second);
+			if (ASSERT_OK(err, "second_load")) {
+				struct bpf_link *l2;
+
+				/*
+				 * the kernel rejects attaching a second
+				 * model to the device with EBUSY
+				 */
+				l2 = bpf_map__attach_struct_ops(
+						second->maps.iocost_2x);
+				if (!ASSERT_ERR_PTR(l2, "second_ebusy"))
+					bpf_link__destroy(l2);
+				else
+					ASSERT_EQ(libbpf_get_error(l2), -EBUSY,
+						  "second_ebusy_errno");
+			}
+			iocost_model__destroy(second);
+		}
+
+		/*
+		 * pricing coverage: root cgroup IOs bypass the model, so
+		 * move into the pairing cgroup, enable the controller
+		 * and verify IOs actually go through calc_cost() and get
+		 * charged
+		 */
+		ASSERT_OK(configfs_write(
+				"/sys/fs/cgroup/iocost_pairing/cgroup.procs",
+				pidbuf), "procs_pairing_priced");
+		snprintf(qbuf, sizeof(qbuf), "%s 1000", dev);
+		ASSERT_OK(configfs_write(
+				"/sys/fs/cgroup/iocost_pairing/io.weight",
+				qbuf), "weight_pairing");
+		snprintf(qbuf, sizeof(qbuf),
+			 "%s enable=1 ctrl=user min=100.00 max=100.00", dev);
+		ASSERT_OK(configfs_write("/sys/fs/cgroup/io.cost.qos", qbuf),
+			  "qos_enable");
+
+		devfd = open(path, O_RDONLY);
+		if (!ASSERT_OK(devfd < 0 ? -errno : 0, "open_dev_priced")) {
+			iocost_model__destroy(skel);
+			configfs_write("/sys/fs/cgroup/cgroup.procs", pidbuf);
+			snprintf(qbuf, sizeof(qbuf), "%s enable=0", dev);
+			(void)configfs_write("/sys/fs/cgroup/io.cost.qos", qbuf);
+			if (rmdir("/sys/fs/cgroup/iocost_pairing") && errno != ENOENT)
+				printf("%s: rmdir cgroup failed: %s\n", __func__,
+				       strerror(errno));
+			nullb_remove(IOCOST_NULLB);
+			return;
+		}
+		ASSERT_OK(read_cost_usage("/sys/fs/cgroup/iocost_pairing/io.stat",
+					  dev, &u1), "cost_usage_pre");
+		for (i = 0; i < 500; i++)
+			(void)read(devfd, buf, sizeof(buf));
+		close(devfd);
+		ASSERT_GT(skel->bss->calc_costs, 0, "model_priced");
+		/*
+		 * the usage stat is flushed by the iocost timer, give
+		 * the current period a chance to end before reading
+		 */
+		sleep(3);
+		ASSERT_OK(read_cost_usage("/sys/fs/cgroup/iocost_pairing/io.stat",
+					  dev, &u2), "cost_usage_post");
+		ASSERT_GT(u2, u1, "cost_charged");
+		ASSERT_GT(skel->bss->ioc_ticks, 0, "tick_seen");
+
+		/*
+		 * a cgroup created while the model is attached gets its
+		 * iocg_init() when its first IO onlines the blkg
+		 */
+		inits1 = skel->bss->iocg_inits;
+		ASSERT_OK(cgroup_mkdir("/sys/fs/cgroup/iocost_late"),
+			  "mkdir_late");
+		ASSERT_OK(configfs_write(
+				"/sys/fs/cgroup/iocost_late/cgroup.procs",
+				pidbuf), "procs_late");
+		devfd = open(path, O_RDONLY);
+		if (ASSERT_OK(devfd < 0 ? -errno : 0, "open_dev_late")) {
+			for (i = 0; i < 32; i++)
+				(void)read(devfd, buf, sizeof(buf));
+			close(devfd);
+		}
+		ASSERT_GT(skel->bss->iocg_inits, inits1, "late_init");
+		ASSERT_OK(configfs_write("/sys/fs/cgroup/cgroup.procs",
+					 pidbuf), "procs_back");
+		rmdir("/sys/fs/cgroup/iocost_late");
+
+		/*
+		 * a second device is independent: enabling it shows
+		 * the builtin model, not the one attached to the
+		 * first
+		 */
+		if (ASSERT_OK(nullb_create(IOCOST_NULLB2, dev2,
+					   sizeof(dev2)), "nullb_create2")) {
+			snprintf(qbuf, sizeof(qbuf),
+				 "%s enable=1 ctrl=auto", dev2);
+			ASSERT_OK(configfs_write(
+					"/sys/fs/cgroup/io.cost.qos", qbuf),
+				  "qos_nullb2");
+			err = readback_model(dev2, model, sizeof(model));
+			if (ASSERT_OK(err, "readback_nullb2"))
+				ASSERT_STREQ(model, "linear",
+					     "model_linear_nullb2");
+			snprintf(qbuf, sizeof(qbuf), "%s enable=0", dev2);
+			ASSERT_OK(configfs_write(
+					"/sys/fs/cgroup/io.cost.qos", qbuf),
+				  "qos_nullb2_off");
+		}
+		nullb_remove(IOCOST_NULLB2);
+
+		/*
+		 * attachment and model selection are independent:
+		 * ctrl=bpf is never accepted, model=linear switches
+		 * back to the builtin without detaching, model=bpf
+		 * switches to the attached model again
+		 */
+		err = write_cost_model(dev, "ctrl=bpf");
+		ASSERT_ERR(err, "ctrl_bpf_rejected");
+
+		err = write_cost_model(dev, "model=linear");
+		ASSERT_OK(err, "model_linear_write");
+		err = readback_model(dev, model, sizeof(model));
+		if (ASSERT_OK(err, "readback_linear"))
+			ASSERT_STREQ(model, "linear", "model_linear");
+
+		/* still attached: the switch must not free the slot */
+		second = iocost_model__open();
+		if (ASSERT_OK_PTR(second, "second_open2")) {
+			second->struct_ops.iocost_2x->dev =
+				(__u32)makedev(maj, min);
+			err = iocost_model__load(second);
+			if (ASSERT_OK(err, "second_load2")) {
+				struct bpf_link *l2;
+
+				l2 = bpf_map__attach_struct_ops(
+						second->maps.iocost_2x);
+				if (!ASSERT_ERR_PTR(l2, "second_ebusy2"))
+					bpf_link__destroy(l2);
+				else
+					ASSERT_EQ(libbpf_get_error(l2), -EBUSY,
+						  "second_ebusy2_errno");
+			}
+			iocost_model__destroy(second);
+		}
+
+		/* still attached: switching back to the model works */
+		err = write_cost_model(dev, "model=bpf");
+		ASSERT_OK(err, "model_bpf_write");
+		err = readback_model(dev, model, sizeof(model));
+		if (ASSERT_OK(err, "readback_bpf"))
+			ASSERT_STREQ(model, "bpf", "model_bpf");
+
+		/*
+		 * detach implicitly by closing the link fd instead of
+		 * calling bpf_link__destroy() explicitly: the release
+		 * path must detach the model just like the explicit
+		 * call does
+		 */
+		lfd = bpf_link__fd(skel->links.iocost_2x);
+		bpf_link__disconnect(skel->links.iocost_2x);
+		close(lfd);
+		skel->links.iocost_2x = NULL;
+
+		/* after detach, model=bpf fails: nothing is attached */
+		err = write_cost_model(dev, "model=bpf");
+		ASSERT_ERR(err, "model_bpf_after_detach");
+
+		err = readback_model(dev, model, sizeof(model));
+		if (ASSERT_OK(err, "readback_after_detach"))
+			ASSERT_STREQ(model, "linear", "model_linear");
+
+		/* the detach walk delivered the matching iocg_free()s */
+		ASSERT_EQ(skel->bss->iocg_frees, skel->bss->iocg_inits,
+			  "pair_free");
+	}
+
+	iocost_model__destroy(skel);
+	(void)configfs_write("/sys/fs/cgroup/cgroup.procs", pidbuf);
+	snprintf(qbuf, sizeof(qbuf), "%s enable=0", dev);
+	(void)configfs_write("/sys/fs/cgroup/io.cost.qos", qbuf);
+	if (rmdir("/sys/fs/cgroup/iocost_pairing") && errno != ENOENT)
+		printf("%s: rmdir cgroup failed: %s\n", __func__,
+		       strerror(errno));
+	nullb_remove(IOCOST_NULLB);
+}
+/*
+ * Same check for the multi-stream example model. Only one model can be
+ * attached to a device at a time; each test creates its own device and
+ * attaches and detaches, so they are serial.
+ */
+void serial_test_iocost_model_streams(void)
+{
+	struct iocost_ms *skel;
+	unsigned int maj, min;
+	int err;
+	char model[32], dev[32];
+
+	if (!iocost_test_env())
+		return;
+	if (!ASSERT_OK(nullb_create(IOCOST_NULLB, dev, sizeof(dev)),
+		       "nullb_create"))
+		return;
+	if (sscanf(dev, "%u:%u", &maj, &min) != 2) {
+		ASSERT_TRUE(false, "dev_parse");
+		nullb_remove(IOCOST_NULLB);
+		return;
+	}
+
+	skel = iocost_ms__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open")) {
+		nullb_remove(IOCOST_NULLB);
+		return;
+	}
+
+	skel->struct_ops.iocost_ms->dev = (__u32)makedev(maj, min);
+
+	err = iocost_ms__load(skel);
+	if (!ASSERT_OK(err, "skel_load")) {
+		iocost_ms__destroy(skel);
+		nullb_remove(IOCOST_NULLB);
+		return;
+	}
+
+	err = iocost_ms__attach(skel);
+	if (err == -EOPNOTSUPP || err == -ENODEV) {
+		printf("%s:SKIP:device is not blk-mq, has no iocost or does not exist\n",
+		       __func__);
+		iocost_ms__destroy(skel);
+		nullb_remove(IOCOST_NULLB);
+		test__skip();
+		return;
+	}
+	if (ASSERT_OK(err, "attach")) {
+		err = readback_model(dev, model, sizeof(model));
+		if (ASSERT_OK(err, "readback"))
+			ASSERT_STREQ(model, "bpf", "model_bpf");
+
+		bpf_link__destroy(skel->links.iocost_ms);
+		skel->links.iocost_ms = NULL;
+
+		err = readback_model(dev, model, sizeof(model));
+		if (ASSERT_OK(err, "readback_after_detach"))
+			ASSERT_STREQ(model, "linear", "model_linear");
+	}
+
+	iocost_ms__destroy(skel);
+	nullb_remove(IOCOST_NULLB);
+}
+
+/*
+ * Removal of the device while a model is attached: the test builds
+ * its own null_blk instance through configfs like the other tests,
+ * attaches to it, then removes the device out from under the model.
+ * The ejection must drop the device from the cgroup interface,
+ * down, and a recreated device must be attachable again.
+ */
+void serial_test_iocost_model_removal(void)
+{
+	struct iocost_model *skel, *skel2;
+	unsigned int maj, min;
+	char dev[32], model[32];
+	int err;
+
+	if (!iocost_test_env())
+		return;
+
+	skel = iocost_model__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		return;
+
+	err = nullb_create("iocost_removal", dev, sizeof(dev));
+	if (!ASSERT_OK(err, "nullb_create"))
+		goto out;
+	if (sscanf(dev, "%u:%u", &maj, &min) != 2) {
+		ASSERT_TRUE(false, "dev_parse");
+		goto out;
+	}
+
+	skel->struct_ops.iocost_2x->dev = (__u32)makedev(maj, min);
+	err = iocost_model__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto out;
+
+	err = iocost_model__attach(skel);
+	if (!ASSERT_OK(err, "attach"))
+		goto out;
+
+	err = readback_model(dev, model, sizeof(model));
+	if (ASSERT_OK(err, "readback"))
+		ASSERT_STREQ(model, "bpf", "model_bpf");
+
+	/* remove the device out from under the attached model */
+	if (!ASSERT_OK(nullb_remove("iocost_removal"), "nullb_remove"))
+		goto detach;
+
+	/* the ejection drops the device from the cgroup files */
+	err = readback_model(dev, model, sizeof(model));
+	ASSERT_ERR(err, "readback_after_removal");
+
+	err = write_cost_model(dev, "model=bpf");
+	ASSERT_ERR(err, "model_bpf_after_removal");
+
+	/*
+	 * the ejection path must deliver iocg_free() for every blkg
+	 * which was online on the removed device: pd_free() runs from
+	 * blkcg_deactivate_policy() before ioc_bpf_eject() drops the
+	 * map reference
+	 */
+	ASSERT_GT(skel->bss->iocg_inits, 0, "eject_had_inits");
+	ASSERT_EQ(skel->bss->iocg_frees, skel->bss->iocg_inits,
+		  "eject_balanced");
+
+detach:
+	/*
+	 * detaching after the device is gone must be a clean no-op;
+	 * this exercises the eject-then-detach ordering in the kernel
+	 */
+	bpf_link__destroy(skel->links.iocost_2x);
+	skel->links.iocost_2x = NULL;
+
+	/* the same attach and detach against a recreated device */
+	skel2 = iocost_model__open();
+	if (!ASSERT_OK_PTR(skel2, "skel2_open"))
+		goto out;
+
+	err = nullb_create("iocost_removal", dev, sizeof(dev));
+	if (!ASSERT_OK(err, "nullb_create_again"))
+		goto out2;
+	if (sscanf(dev, "%u:%u", &maj, &min) != 2) {
+		ASSERT_TRUE(false, "dev_parse_again");
+		goto out2;
+	}
+
+	skel2->struct_ops.iocost_2x->dev = (__u32)makedev(maj, min);
+	err = iocost_model__load(skel2);
+	if (!ASSERT_OK(err, "skel2_load"))
+		goto out2;
+
+	err = iocost_model__attach(skel2);
+	if (!ASSERT_OK(err, "attach_again"))
+		goto out2;
+
+	err = readback_model(dev, model, sizeof(model));
+	if (ASSERT_OK(err, "readback_again"))
+		ASSERT_STREQ(model, "bpf", "model_bpf_again");
+	ASSERT_GT(skel2->bss->iocg_inits, 0, "pair_init_again");
+
+	bpf_link__destroy(skel2->links.iocost_2x);
+	skel2->links.iocost_2x = NULL;
+
+	err = readback_model(dev, model, sizeof(model));
+	if (ASSERT_OK(err, "readback_after_detach_again"))
+		ASSERT_STREQ(model, "linear", "model_linear_again");
+
+	/* the re-attach walk delivered init for the recreated device */
+	ASSERT_EQ(skel2->bss->iocg_frees, skel2->bss->iocg_inits,
+		  "pair_balanced_again");
+
+out2:
+	iocost_model__destroy(skel2);
+out:
+	nullb_remove("iocost_removal");
+	iocost_model__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/iocost_model.c b/tools/testing/selftests/bpf/progs/iocost_model.c
new file mode 100644
index 000000000000..8875fb0f4417
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/iocost_model.c
@@ -0,0 +1,201 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (C) 2026 KylinSoft Corporation. */
+/*
+ * Example iocost cost model: the builtin linear HDD formula with all costs
+ * doubled, for one device given by the dev member of the struct_ops.
+ *
+ * The constants mirror what calc_lcoefs() derives from the AUTOP_HDD defaults
+ * (rbps=174019176 rseqiops=41708 rrandiops=370, w-side analog) in vtime units
+ * where 1s == 2^37. On a rotational device still on ctrl=auto, a device with
+ * this model attached charges twice the builtin model under the same
+ * read/write workload (the builtin prices flushes as zero and adds a fixed
+ * flush term to data-bearing preflushes, which this model omits; this model
+ * prices flushes as one-page writes), so the doubled cost is a direct check
+ * that accounting goes through the BPF path. On a non-rotational device, or
+ * one with user-pinned coefficients, the ratio to the builtin model is
+ * arbitrary.
+ *
+ * The cursor handling follows the builtin: a zero cursor means "no previous
+ * IO", and the cursor is advanced for the bios the builtin prices
+ * (READ/WRITE/ZONE_APPEND with a non-zero size), so flushes and discards leave
+ * it alone and long merged streams do not drift past the 16MB seek threshold.
+ * One known deviation, mispricing only: the builtin does not move its cursor
+ * for an IO whose cgroup fails to activate; this model does not see the
+ * activation result.
+ *
+ * The model implements the full linear formula itself, including flushes:
+ * there is no fallback to the builtin model, which prices a dataless
+ * WRITE|REQ_PREFLUSH as zero unless flushiops is configured; this model prices
+ * it as a one-page sequential write. What a flush actually costs depends on
+ * the IOs before it, which a static formula cannot express - pricing it as a
+ * small constant is a crude penalty for unnecessarily frequent flushes, and a
+ * stateful model would be the way to do better. Discards and other unpriced
+ * operations pass through at zero cost, like the builtin.
+ */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+/*
+ * VTIME_PER_SEC, IOC_PAGE_SIZE/SHIFT, IOC_SECT_TO_PAGE_SHIFT,
+ * LCOEF_RANDIO_PAGES (the 16MB seek threshold) and IOCOST_COST_F_MERGE
+ * come from vmlinux.h (BTF constants; the enum64 ones need pahole
+ * >= 1.23 to be emitted)
+ */
+#define IOCOST_REQ_OP_MASK	0xff		/* REQ_OP_MASK, not in BTF */
+
+/*
+ * DIV64_U64_ROUND_UP / DIV_ROUND_UP_ULL equivalents, folded at
+ * compile time
+ */
+#define RU(x, y)		((x) / (y) + (((x) % (y)) ? 1 : 0))
+
+#define RBPS	174019176ULL
+#define RSEQIOPS	41708ULL
+#define RRANDIOPS	370ULL
+#define WBPS	178075866ULL
+#define WSEQIOPS	42705ULL
+#define WRANDIOPS	378ULL
+
+#define RPAGE	(RU(VTIME_PER_SEC, RU(RBPS, IOC_PAGE_SIZE)))
+#define RSEQIO	(RU(VTIME_PER_SEC, RSEQIOPS) - RPAGE)
+#define RRANDIO	(RU(VTIME_PER_SEC, RRANDIOPS) - RPAGE)
+#define WPAGE	(RU(VTIME_PER_SEC, RU(WBPS, IOC_PAGE_SIZE)))
+#define WSEQIO	(RU(VTIME_PER_SEC, WSEQIOPS) - WPAGE)
+#define WRANDIO	(RU(VTIME_PER_SEC, WRANDIOPS) - WPAGE)
+
+/*
+ * per-cgroup cursor storage: keyed by the cgroup and freed with it,
+ * so a cgroup that comes back starts with no cursor
+ */
+struct {
+	__uint(type, BPF_MAP_TYPE_CGRP_STORAGE);
+	__uint(map_flags, BPF_F_NO_PREALLOC);
+	__type(key, int);
+	__type(value, __u64);
+} cursor_store SEC(".maps");
+
+/*
+ * cgroup callback counters: the test reads them back to verify the
+ * init/free pairing across attach, detach and device removal
+ */
+__u64 iocg_inits;
+__u64 iocg_frees;
+
+/*
+ * incremented by calc_cost(): the test asserts pricing went
+ * through the model while it is attached and the controller is
+ * enabled
+ */
+__u64 calc_costs;
+
+/* counts iocost_ioc_tick emissions observed from BPF side */
+__u64 ioc_ticks;
+
+SEC("struct_ops")
+void BPF_PROG(iocost_2x_iocg_init, struct blkcg *blkcg,
+	      struct request_queue *q)
+{
+	iocg_inits++;
+}
+
+SEC("struct_ops")
+void BPF_PROG(iocost_2x_iocg_free, struct blkcg *blkcg,
+	      struct request_queue *q)
+{
+	iocg_frees++;
+}
+
+SEC("struct_ops")
+u64 BPF_PROG(iocost_2x_calc_cost, struct bio *bio, u64 model_flags)
+{
+	u64 opf = bio->bi_opf, nbytes = bio->bi_iter.bi_size;
+	u64 sector = bio->bi_iter.bi_sector;
+	struct blkcg *blkcg = bio->bi_blkg->blkcg;
+	u64 pages, seek_pages = 0, base, coef_page, randio, cost;
+	__u64 *cursor, cur;
+	int priced;
+
+	calc_costs++;
+
+	/* builtin truncates: max(sectors >> IOC_SECT_TO_PAGE_SHIFT, 1) */
+	pages = nbytes >> IOC_PAGE_SHIFT;
+	if (!pages)
+		pages = 1;
+
+	if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ) {
+		base = RSEQIO; coef_page = RPAGE; randio = RRANDIO;
+	} else if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE ||
+		   (opf & IOCOST_REQ_OP_MASK) == REQ_OP_ZONE_APPEND) {
+		base = WSEQIO; coef_page = WPAGE; randio = WRANDIO;
+	} else {
+		/*
+		 * the builtin prices other operations at zero and
+		 * leaves them unthrottled; pass them through the same
+		 * way instead of inventing costs
+		 */
+		return 0;
+	}
+
+	/* mirror the builtin cursor semantics described above */
+	priced = (opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ ||
+		 (opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE ||
+		 (opf & IOCOST_REQ_OP_MASK) == REQ_OP_ZONE_APPEND;
+	cursor = bpf_cgrp_storage_get(&cursor_store,
+				      blkcg->css.cgroup, NULL,
+				      BPF_LOCAL_STORAGE_GET_F_CREATE);
+	if (!cursor) {
+		if (model_flags & IOCOST_COST_F_MERGE)
+			base = 0;
+		return 2 * (base + pages * coef_page);
+	}
+	cur = *cursor;
+	/*
+	 * a size-zero IO never moves the cursor, so it must not be
+	 * judged against it either; the builtin skips it entirely,
+	 * this model prices it as a one-page sequential write
+	 */
+	if (cur && priced && nbytes) {
+		seek_pages = sector > cur ? sector - cur
+					   : cur - sector;
+		seek_pages >>= IOC_SECT_TO_PAGE_SHIFT;
+		if (seek_pages > LCOEF_RANDIO_PAGES)
+			base = randio;
+	}
+	/*
+	 * the builtin moves the cursor unconditionally on submit, but
+	 * on a merge only for a back merge into the request ending at
+	 * it; a merged bio starts right at the cursor exactly then,
+	 * and a zero cursor means no history so it never matches
+	 */
+	if (priced && nbytes &&
+	    (!(model_flags & IOCOST_COST_F_MERGE) || (cur && cur == sector)))
+		*cursor = sector + (nbytes >> 9);
+
+	if (model_flags & IOCOST_COST_F_MERGE)
+		base = 0;
+
+	cost = 2 * (base + pages * coef_page);
+	return cost;
+}
+
+/* consumes the new iocost_ioc_tick tracepoint from BPF side */
+SEC("tp_btf/iocost_ioc_tick")
+__u64 BPF_PROG(iocost_2x_tick_count, struct ioc *ioc,
+	       const struct ioc_now *now, int nr_active, u64 usage_us_sum)
+{
+	ioc_ticks++;
+	return 0;
+}
+
+SEC(".struct_ops.link") /* detach on close */
+struct iocost_model_ops iocost_2x = {
+	.read_vtime_per_page = 2 * RPAGE,
+	.write_vtime_per_page = 2 * WPAGE,
+	.calc_cost = (void *)iocost_2x_calc_cost,
+	.iocg_init = (void *)iocost_2x_iocg_init,
+	.iocg_free = (void *)iocost_2x_iocg_free,
+};
+
+char LICENSE[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/iocost_ms.c b/tools/testing/selftests/bpf/progs/iocost_ms.c
new file mode 100644
index 000000000000..87d4abcbf693
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/iocost_ms.c
@@ -0,0 +1,175 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (C) 2026 KylinSoft Corporation. */
+/*
+ * Example multi-stream sequentiality detection cost model.
+ *
+ * The builtin model keeps a single cursor per cgroup, so two interleaved
+ * sequential readers in one cgroup are all priced random (measured 89x
+ * overcharge, 12.9x throughput collapse), while random IO inside a hot window
+ * smaller than the 16MB seek threshold is priced sequential (measured 107x
+ * undercharge). This model replaces the single cursor with a per-cgroup table
+ * of stream slots: an IO is sequential iff its sector matches the expected
+ * next sector of any tracked stream. Interleaved streams keep their own slots,
+ * and windowed random IO rarely matches a moving expectation.
+ *
+ * Stream state lives in a CGRP_STORAGE map, so it is created and freed with
+ * the cgroup. The model implements the builtin linear formula itself; unlike
+ * the builtin it has no flush term, so preflushed IOs are priced as plain
+ * writes (mispricing only, the test never issues flushes).
+ */
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+/*
+ * VTIME_PER_SEC, IOC_PAGE_SIZE/SHIFT, IOC_SECT_TO_PAGE_SHIFT and
+ * IOCOST_COST_F_MERGE come from vmlinux.h (BTF enum constants)
+ */
+#define IOCOST_REQ_OP_MASK	0xff		/* REQ_OP_MASK, not in BTF */
+
+/*
+ * DIV64_U64_ROUND_UP / DIV_ROUND_UP_ULL equivalents, folded at
+ * compile time
+ */
+#define RU(x, y)		((x) / (y) + (((x) % (y)) ? 1 : 0))
+
+#define RBPS	174019176ULL
+#define RSEQIOPS	41708ULL
+#define RRANDIOPS	370ULL
+#define WBPS	178075866ULL
+#define WSEQIOPS	42705ULL
+#define WRANDIOPS	378ULL
+
+#define RPAGE	(RU(VTIME_PER_SEC, RU(RBPS, IOC_PAGE_SIZE)))
+#define RSEQIO	(RU(VTIME_PER_SEC, RSEQIOPS) - RPAGE)
+#define RRANDIO	(RU(VTIME_PER_SEC, RRANDIOPS) - RPAGE)
+#define WPAGE	(RU(VTIME_PER_SEC, RU(WBPS, IOC_PAGE_SIZE)))
+#define WSEQIO	(RU(VTIME_PER_SEC, WSEQIOPS) - WPAGE)
+#define WRANDIO	(RU(VTIME_PER_SEC, WRANDIOPS) - WPAGE)
+
+#define NSLOTS	4
+
+struct streams {
+	__u64 expected[NSLOTS];	/* next expected sector, per stream */
+	__u64 stamp[NSLOTS];	/* LRU stamp, 0 = empty */
+};
+
+/*
+ * per-cgroup stream table: keyed by the cgroup, freed with it
+ */
+struct {
+	__uint(type, BPF_MAP_TYPE_CGRP_STORAGE);
+	__uint(map_flags, BPF_F_NO_PREALLOC);
+	__type(key, int);
+	__type(value, struct streams);
+} stream_tab SEC(".maps");
+
+SEC("struct_ops")
+u64 BPF_PROG(iocost_ms_calc_cost, struct bio *bio, u64 model_flags)
+{
+	u64 opf = bio->bi_opf, nbytes = bio->bi_iter.bi_size;
+	u64 sector = bio->bi_iter.bi_sector;
+	struct blkcg *blkcg = bio->bi_blkg->blkcg;
+	struct streams *s;
+	u64 pages, base, coef_page, randio, advance, now;
+	u32 i, victim = 0, found = 0xFFFFFFFF;
+
+	if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ) {
+		base = RSEQIO; coef_page = RPAGE; randio = RRANDIO;
+	} else if ((opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE ||
+		   (opf & IOCOST_REQ_OP_MASK) == REQ_OP_ZONE_APPEND) {
+		base = WSEQIO; coef_page = WPAGE; randio = WRANDIO;
+	} else {
+		/*
+		 * the builtin prices other operations at zero and
+		 * leaves them unthrottled; pass them through the same
+		 * way instead of inventing costs
+		 */
+		return 0;
+	}
+	advance = nbytes >> 9;	/* sectors, truncated like the builtin */
+
+	/*
+	 * only bios the builtin prices participate in stream tracking;
+	 * the early return below skips the merge discount, which only
+	 * applies to tracked data bios
+	 */
+	if (!(((opf & IOCOST_REQ_OP_MASK) == REQ_OP_READ ||
+	       (opf & IOCOST_REQ_OP_MASK) == REQ_OP_WRITE ||
+	       (opf & IOCOST_REQ_OP_MASK) == REQ_OP_ZONE_APPEND) && nbytes)) {
+		pages = nbytes >> IOC_PAGE_SHIFT;
+		if (!pages)
+			pages = 1;
+		return base + pages * coef_page;
+	}
+
+	s = bpf_cgrp_storage_get(&stream_tab, blkcg->css.cgroup, NULL,
+				 BPF_LOCAL_STORAGE_GET_F_CREATE);
+	if (!s) {
+		/* no storage: price per page, truncating like the builtin */
+		pages = nbytes >> IOC_PAGE_SHIFT;
+		if (!pages)
+			pages = 1;
+		return base + pages * coef_page;
+	}
+
+	/*
+	 * Slot access is lockless, mirroring the builtin single-cursor
+	 * update in ioc_rqos_throttle(): concurrent CPUs submitting for
+	 * the same cgroup can race on slot updates; mispricing is
+	 * bounded and acceptable for an example model.
+	 */
+	now = bpf_ktime_get_ns();
+	for (i = 0; i < NSLOTS; i++) {
+		if (s->expected[i] == sector && s->stamp[i]) {
+			found = i;
+			break;
+		}
+	}
+	if (found != 0xFFFFFFFF) {
+		/* sequential: keep the seq base from the op branch */
+		s->expected[found] = sector + advance;
+		s->stamp[found] = now;
+	} else {
+		/*
+		 * a merge which matches no slot is not a new stream:
+		 * replacing a slot could evict a live stream, so merged
+		 * bios are priced without touching the table (and skip
+		 * the base cost below)
+		 */
+		if (!(model_flags & IOCOST_COST_F_MERGE)) {
+			base = randio;
+			for (i = 1; i < NSLOTS; i++) {
+				if (s->stamp[i] < s->stamp[victim])
+					victim = i;
+			}
+			s->expected[victim] = sector + advance;
+			s->stamp[victim] = now;
+		}
+	}
+
+	/* builtin truncates: max(sectors >> IOC_SECT_TO_PAGE_SHIFT, 1) */
+	pages = nbytes >> IOC_PAGE_SHIFT;
+	if (!pages)
+		pages = 1;
+	if (model_flags & IOCOST_COST_F_MERGE) {
+		/*
+		 * merged bios skip the base cost but still advance
+		 * the stream position above, so a merge at the
+		 * expected sector does not make the following new IO
+		 * look random
+		 */
+		base = 0;
+	}
+
+	return base + pages * coef_page;
+}
+
+SEC(".struct_ops.link") /* detach on close */
+struct iocost_model_ops iocost_ms = {
+	.read_vtime_per_page = RPAGE,
+	.write_vtime_per_page = WPAGE,
+	.calc_cost = (void *)iocost_ms_calc_cost,
+};
+
+char LICENSE[] SEC("license") = "GPL";
-- 
2.43.0

[PATCH 4/5] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

Add iocost_ioc_tick, emitted once per period from ioc_timer_fn()
with the overall controller state: period_us, vrate, busy_level,
active iocg count and usage percentage.  It reads the values the
period ran in: the event is emitted before the vrate adjustment and
the period transition, so each tick reports the completed period.

It fires every period the controller is running, including steady
states, plus one final tick before the controller goes idle, which
reads active=0, making the transition to dormancy (e.g. a device
saturated entirely by uncharged IO) directly visible.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 MAINTAINERS                   |  1 +
 block/blk-iocost.c            | 13 +++++++++++
 include/trace/events/iocost.h | 43 +++++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 598a5f65423d..a6a39279d266 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6832,6 +6832,7 @@ F:	block/blk-iocost.h
 F:	block/blk-iolatency.c
 F:	block/blk-throttle.c
 F:	include/linux/blk-cgroup.h
+F:	include/trace/events/iocost.h
 
 CONTROL GROUP - CPUSET
 M:	Waiman Long <longman@redhat.com>
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 1e84232e6740..9f7f5e5f0ebb 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -1839,6 +1839,7 @@ static void ioc_timer_fn(struct timer_list *timer)
 	struct ioc_now now;
 	LIST_HEAD(surpluses);
 	int nr_debtors, nr_shortages = 0, nr_lagging = 0;
+	int nr_active = 0;
 	u64 usage_us_sum = 0;
 	u32 ppm_rthr;
 	u32 ppm_wthr;
@@ -1875,6 +1876,8 @@ static void ioc_timer_fn(struct timer_list *timer)
 		u64 vdone, vtime, usage_us;
 		u32 hw_active, hw_inuse;
 
+		nr_active++;
+
 		/*
 		 * Collect unused and wind vtime closer to vnow to prevent
 		 * iocgs from accumulating a large amount of budget.
@@ -2036,6 +2039,16 @@ static void ioc_timer_fn(struct timer_list *timer)
 
 	ioc->busy_level = clamp(ioc->busy_level, -1000, 1000);
 
+	/*
+	 * Everything the tick reports is final here: busy_level was just
+	 * computed, cur_period hasn't changed, nr_active and usage_us_sum are
+	 * complete, and vrate and period_us still hold the values this period
+	 * ran in. usage is wall time over wall time, summed over iocgs, and is
+	 * not capped at 100. Emit before the refresh below so the event reads
+	 * the completed period directly.
+	 */
+	trace_iocost_ioc_tick(ioc, &now, nr_active, usage_us_sum);
+
 	ioc_adjust_base_vrate(ioc, rq_wait_pct, nr_lagging, nr_shortages,
 			      prev_busy_level, missed_ppm);
 
diff --git a/include/trace/events/iocost.h b/include/trace/events/iocost.h
index e772b1bc60d6..39229605f499 100644
--- a/include/trace/events/iocost.h
+++ b/include/trace/events/iocost.h
@@ -178,6 +178,49 @@ TRACE_EVENT(iocost_ioc_vrate_adj,
 	)
 );
 
+/*
+ * Periodic per-device summary, emitted once per period before the vrate
+ * adjustment and the period transition in ioc_timer_fn(). Unlike the
+ * state-change events above, this fires every period the controller is
+ * running, including steady states, and carries the overall controller state
+ * so basic monitoring doesn't require drgn.
+ */
+TRACE_EVENT(iocost_ioc_tick,
+
+	TP_PROTO(struct ioc *ioc, const struct ioc_now *now,
+		 int nr_active, u64 usage_us_sum),
+
+	TP_ARGS(ioc, now, nr_active, usage_us_sum),
+
+	TP_STRUCT__entry (
+		__string(devname, ioc_name(ioc))
+		__field(u64, cur_period)
+		__field(u32, period_us)
+		__field(u64, vrate)
+		__field(int, busy_level)
+		__field(int, nr_active)
+		__field(u32, usage_pct)
+	),
+
+	TP_fast_assign(
+		__assign_str(devname);
+		__entry->cur_period = atomic64_read(&ioc->cur_period);
+		__entry->period_us = ioc->period_us;
+		__entry->vrate = ioc->vtime_base_rate;
+		__entry->busy_level = ioc->busy_level;
+		__entry->nr_active = nr_active;
+		__entry->usage_pct = now->now > ioc->period_at ?
+			div64_u64(usage_us_sum * 100,
+				  now->now - ioc->period_at) : 0;
+	),
+
+	TP_printk("[%s] period=%llu:%uus vrate=%llu busy=%d active=%d usage=%u%%",
+		__get_str(devname), __entry->cur_period, __entry->period_us,
+		__entry->vrate, __entry->busy_level, __entry->nr_active,
+		__entry->usage_pct
+	)
+);
+
 TRACE_EVENT(iocost_iocg_forgive_debt,
 
 	TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
-- 
2.43.0

[PATCH 5/5] docs: cgroup-v2: document the iocost BPF cost model attachment

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

Document the BPF cost model attachment in the io.cost.model section
of the cgroup v2 documentation: attaching an iocost_model_ops
struct_ops to a device by its major:minor, the model=bpf readback
while attached (ctrl keeps describing the coefficients), the builtin
model coming back on detach, and ctrl writes never selecting a
model.  The device's line is populated by attaching too, and the
attachment paragraph is split into attaching, model selection and
detaching.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 Documentation/admin-guide/cgroup-v2.rst | 46 ++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 8 deletions(-)

diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 9e3c67b02ae2..fd274037a9df 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -2056,8 +2056,9 @@ IO Interface Files
 	currently implements "io.weight" proportional control.  Lines
 	are keyed by $MAJ:$MIN device numbers and not ordered.  The
 	line for a given device is populated on the first write for
-	the device on "io.cost.qos" or "io.cost.model".  The following
-	nested keys are defined.
+	the device on "io.cost.qos" or "io.cost.model", or when a BPF
+	cost model is attached to it.  The following nested keys are
+	defined.
 
 	  ======	=====================================
 	  enable	Weight-based control enable
@@ -2111,13 +2112,41 @@ IO Interface Files
 	implements "io.weight" proportional control.  Lines are keyed
 	by $MAJ:$MIN device numbers and not ordered.  The line for a
 	given device is populated on the first write for the device on
-	"io.cost.qos" or "io.cost.model".  The following nested keys
-	are defined.
+	"io.cost.qos" or "io.cost.model", or when a BPF cost model is
+	attached to it.  The following nested keys are defined.
 
-	  =====		================================
+	  =====		==========================================
 	  ctrl		"auto" or "user"
-	  model		The cost model in use - "linear"
-	  =====		================================
+	  model		The cost model in use - "linear" or "bpf"
+	  =====		==========================================
+
+	When CONFIG_BLK_CGROUP_IOCOST_BPF is enabled, a BPF cost
+	model can be bound to a device by attaching an
+	"iocost_model_ops" struct_ops carrying the whole disk's
+	major:minor in its "dev" member (a partition's major:minor is
+	rejected).  Attaching creates the controller if needed, like
+	an io.cost.model write does, switches the device's pricing to
+	the model and populates the device's line in this file and in
+	"io.cost.qos" if it did not exist yet.  Attaching fails with
+	-EBUSY if the device already has a model attached.  The
+	attached model prices IOs only while the controller is
+	enabled through "io.cost.qos"; attaching does not enable it.
+
+	While a model is attached, "model" selects between it and the
+	builtin model: "model=bpf" switches to the attached model and
+	"model=linear" switches back to the builtin model, and neither
+	detaches the struct_ops.
+
+	Only detaching removes the model, after which "model=bpf"
+	fails and the device prices with the builtin model again.
+	Device removal ejects the attached model as well, while the
+	struct_ops link remains until userspace destroys it.  "ctrl"
+	accepts only "auto" and "user" and never selects a model.  It
+	keeps describing the builtin coefficients, which take effect
+	again when switched back.  The automatic profile stepping does
+	not switch profiles while the BPF model is in use; as with
+	user-set parameters, the profile selected from the device
+	class still applies.
 
 	When "ctrl" is "auto", the kernel may change all parameters
 	dynamically.  When "ctrl" is set to "user" or any other
@@ -2125,7 +2154,8 @@ IO Interface Files
 	automatic changes are disabled.
 
 	When "model" is "linear", the following model parameters are
-	defined.
+	defined.  They remain readable and writable while "model" is
+	"bpf" but take effect only after switching back to "linear".
 
 	  =============	========================================
 	  [r|w]bps	The maximum sequential IO throughput
-- 
2.43.0

Re: [PATCH 3/5] selftests/bpf: add iocost cost model test

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2188 bytes --]

> selftests/bpf: add iocost cost model test

TL;DR: Nit only: in the iocost_model selftest, the comment before the final
"pair_balanced_again" assertion describes the re-attach init check, not the
detach-time init/free pairing it actually tests.

> diff --git a/tools/testing/selftests/bpf/prog_tests/iocost_model.c b/tools/testing/selftests/bpf/prog_tests/iocost_model.c
> new file mode 100644
> index 000000000000..5373d52d93a6
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/iocost_model.c
> @@ -0,0 +1,806 @@

[ ... ]

> +void serial_test_iocost_model_removal(void)
> +{

[ ... ]

> +	err = iocost_model__attach(skel2);
> +	if (!ASSERT_OK(err, "attach_again"))
> +		goto out2;
> +
> +	err = readback_model(dev, model, sizeof(model));
> +	if (ASSERT_OK(err, "readback_again"))
> +		ASSERT_STREQ(model, "bpf", "model_bpf_again");
> +	ASSERT_GT(skel2->bss->iocg_inits, 0, "pair_init_again");
> +
> +	bpf_link__destroy(skel2->links.iocost_2x);
> +	skel2->links.iocost_2x = NULL;
> +
> +	err = readback_model(dev, model, sizeof(model));
> +	if (ASSERT_OK(err, "readback_after_detach_again"))
> +		ASSERT_STREQ(model, "linear", "model_linear_again");
> +
> +	/* the re-attach walk delivered init for the recreated device */
> +	ASSERT_EQ(skel2->bss->iocg_frees, skel2->bss->iocg_inits,
> +		  "pair_balanced_again");

This isn't a bug, but the comment above the final assertion doesn't seem to
describe the check below it.

The comment says "the re-attach walk delivered init for the recreated
device", while the assertion that follows compares iocg_frees against
iocg_inits after bpf_link__destroy(), which is the detach walk delivering a
matching iocg_free() for every iocg_init().  The init side is the
ASSERT_GT(skel2->bss->iocg_inits, 0, "pair_init_again") check a few lines
earlier, right after the attach.

Should this comment sit next to the "pair_init_again" check, or say that the
detach walk paired every init with a free?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40710751825

Re: [PATCH 2/5] blk-iocost: add BPF struct_ops cost model support

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 4585 bytes --]

> blk-iocost: add BPF struct_ops cost model support

TL;DR: Nit only, not a bug: the long iocost_model_ops interface comment in
blk-iocost.h repeats attach/teardown details documented elsewhere and names
call sites; suggests trimming it to the model-author contract.

> diff --git a/block/blk-iocost.h b/block/blk-iocost.h
> index e4e517ef6a0f..6d62b4fd0c54 100644
> --- a/block/blk-iocost.h
> +++ b/block/blk-iocost.h
> @@ -22,6 +22,79 @@
>  #include <asm/local64.h>
>  #include "blk-rq-qos.h"
>
> +struct bpf_link;
> +
> +/*
> + * Pluggable cost model interface for blk-iocost.
> + *
> + * A BPF struct_ops implementation can take over the cost model of one device.
> + * The device is identified by the dev member, which is set from userspace
> + * before the program is loaded; the struct_ops core owns the program lifetime.
> + * Attaching creates the ioc if needed, like an io.cost.model write, and
> + * switches the device to the BPF model with the queue frozen before
> + * rq_qos_mutex, like an io.cost.qos write, and delivers iocg_init() to the
> + * cgroups which already exist on the device. Enabling and disabling the
> + * controller stays with io.cost.qos.
> + *
> + * While a model is attached, io.cost.model selects between it and the builtin
> + * model: "model=bpf" switches to the attached model and "model=linear" back to
> + * the builtin. Neither detaches the struct_ops; only detaching does. The
> + * builtin linear coefficients are kept while the BPF model is in use and take
> + * effect again when switched back.
> + *
> + * While in use, the BPF model prices every charged IO on the device, including
> + * flushes, from the bio charging path. calc_cost() is called from the IO
> + * submission path with RCU read lock held and must not sleep. It receives the
> + * bio itself so that the model can use what it needs (operation flags, size,
> + * sector, the issuing cgroup through bio->bi_blkg, which is guaranteed
> + * non-NULL) and returns the cost of the IO in vtime units, where 1 second
> + * of device time equals VTIME_PER_SEC (2^37,
> + * available to BPF programs through vmlinux.h). The return value is clamped to
> + * 1 second of device time per IO.
> + *
> + * The struct_ops also carries the transfer cost coefficients, vtime per page
> + * for reads and writes. While the BPF model is in use, they replace the
> + * builtin coefficients in the completion-time request sizing of READ and
> + * WRITE requests, with ZONE_APPEND counted as a write. Taking over the QoS
> + * side (latency tracking, vrate control) is left for later extensions.
> + *
> + * The cgroup callbacks are bound to the iocg policy lifetime, one (cgroup,
> + * device) pair per invocation. iocg_init() is delivered from ioc_pd_online()
> + * and on attach to the cgroups already online on the device; iocg_free() on
> + * detach to the cgroups which still exist then and at pd_free() time for the
> + * rest, so that init and free pair up exactly. iocg_free() is gated on a
> + * per-iocg delivered flag, as bios can still be priced between the blkg going
> + * offline and its free. Both callbacks are called with ioc->lock held and must
> + * not sleep; the delivery points nest inside queue_lock except for iocg_free()
> + * from ioc_pd_free() in blkg_free_workfn().
> + */

[ ... ]

this isn't a bug, but could this interface comment be cut down to the
contract a model author needs (calling context, units, clamping, locking)?

The attach description repeats what is already at the top of
block/blk-iocost-bpf.c:

 * attaches the model to that device, creating the ioc if needed and switching
 * to the model with the queue frozen before rq_qos_mutex, like io.cost.qos
 * writes, and .unreg detaches it; enabling and disabling the controller stays
 * with io.cost.qos.

The init/free pairing rules and the "bios can still be priced between
the blkg going offline and its free" reasoning are repeated in the comments
above ioc_pd_online(), ioc_pd_free() and ioc_bpf_deliver_iocg_callbacks().

The comment also names implementation call sites:

 * not sleep; the delivery points nest inside queue_lock except for iocg_free()
 * from ioc_pd_free() in blkg_free_workfn().

Would it be better to leave the attach and teardown mechanics to the
comments next to that code, so this one does not go stale when those paths
change?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40710751825

Re: [RFC PATCH v11 0/5] blk-iocost: BPF struct_ops cost model

[Alexei Starovoitov]

On Wed, Oct 28, 2026 at 07:35 AM Tao Cui <cui.tao@linux.dev> wrote:
> The cgroup callbacks are bound to the iocg policy lifetime and pair
> up exactly: iocg_init() is delivered from ioc_pd_online(), which
> blkg_create() only calls after the blkg has been inserted into the
> radix tree and the queue's blkg list, and on attach to every cgroup
> which is already online on the device; iocg_free() is delivered at
> detach to the cgroups which still exist then and at pd_free() time
> for the rest, gated per iocg on a delivered flag.  The attach and
> detach walks go over q->blkg_list under q->blkcg_mutex and
> q->queue_lock, the same locking blkg creation and destruction use.

Who needs iocg_init() and iocg_free()?
Both example models keep their state in BPF_MAP_TYPE_CGRP_STORAGE.
The only user of the callbacks is a pair of counters in the selftest.

One struct_ops map is bound to one device. So the cgroup storage of
a model is per (cgroup, device) already.
The blkg holds the blkcg css until blkg_free_workfn(). The storage is
freed with the cgroup or with the map. So it outlives the iocg.

The v11 changelog says that calc_cost() can run before iocg_init().
So a model has to create its state in calc_cost() anyway.
iocg_init() runs under queue_lock and ioc->lock. An allocation there
is not better than in calc_cost().

v8, v9, v10 and v11 all reworked the pairing of these two callbacks.

Lets drop them. A struct_ops can get new members later, when a model
needs them.
I'd like to hear from Tejun whether such a model is planned.

Without the callbacks there is no walk of q->blkg_list:

  what                v11                          without callbacks
  ------------------  ---------------------------  ------------------
  attach takes        freeze, rq_qos_mutex,        freeze,
                      quiesce, blkcg_mutex,        rq_qos_mutex,
                      queue_lock, ioc->lock        ioc->lock
  detach takes        rq_qos_mutex, queue_lock,    rq_qos_mutex,
                      ioc->lock to check, then     ioc->lock
                      freeze and the same again,
                      quiesce, blkcg_mutex,
                      queue_lock, ioc->lock
  state per iocg      bpf_iocg_inited              none
  hooks in            ioc_pd_online(),             none
                      ioc_pd_free()
  teardown            ioc_bpf_detach(),            ioc_detach_model()
                      ioc_bpf_eject()

calc_cost() runs under rcu_read_lock(). bpf_map_put() frees a struct_ops
map after an RCU gp. So detach doesn't need to freeze the queue.
The rest fits into blk-iocost.c and patch 1 is not needed.

The diff keeps the map reference, the owning link and the
model=linear/bpf switch. It has no docs, no selftests and no tracepoint.

Something like this (compile tested only):

diff --git a/block/Kconfig b/block/Kconfig
index 70e4a66..26badf5 100644
--- a/block/Kconfig
+++ b/block/Kconfig
@@ -161,6 +161,10 @@ config BLK_CGROUP_IOCOST
 	distributes IO capacity between different groups based on
 	their share of the overall weight distribution.

+config BLK_CGROUP_IOCOST_BPF
+	def_bool y
+	depends on BLK_CGROUP_IOCOST && BPF_SYSCALL && BPF_JIT && DEBUG_INFO_BTF
+
 config BLK_CGROUP_IOPRIO
 	bool "Cgroup I/O controller for assigning an I/O priority class"
 	depends on BLK_CGROUP
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index a81488c..5e65bf4 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -178,8 +178,12 @@
 #include <linux/time64.h>
 #include <linux/parser.h>
 #include <linux/sched/signal.h>
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf.h>
 #include <asm/local.h>
 #include <asm/local64.h>
+#include "blk.h"
 #include "blk-rq-qos.h"
 #include "blk-stat.h"
 #include "blk-wbt.h"
@@ -404,6 +408,22 @@ struct ioc_pcpu_stat {
 	u64				last_rq_wait_ns;
 };

+enum {
+	IOCOST_COST_F_MERGE	= 1 << 0,	/* called from the merge path */
+};
+
+/* bpf cost model of one device */
+struct iocost_model_ops {
+	/* whole disk, in the encoding of new_decode_dev() */
+	dev_t dev;
+
+	/* used by the completion path instead of LCOEF_[RW]PAGE */
+	u64 read_vtime_per_page;
+	u64 write_vtime_per_page;
+
+	u64 (*calc_cost)(struct bio *bio, u64 model_flags);
+};
+
 /* per device */
 struct ioc {
 	struct rq_qos			rqos;
@@ -447,6 +467,12 @@ struct ioc {
 	int				autop_idx;
 	bool				user_qos_params:1;
 	bool				user_cost_model:1;
+
+	/* holds a reference on the struct_ops map, NULL if none is attached */
+	const struct iocost_model_ops	*attached;
+	struct bpf_link			*attached_link;
+	/* either NULL (linear) or ->attached */
+	const struct iocost_model_ops __rcu *model;
 };

 struct iocg_pcpu_stat {
@@ -669,6 +695,13 @@ static struct ioc *q_to_ioc(struct request_queue *q)
 	return rqos_to_ioc(rq_qos_id(q, RQ_QOS_COST));
 }

+static const struct iocost_model_ops *ioc_model(struct ioc *ioc)
+{
+	if (!IS_ENABLED(CONFIG_BLK_CGROUP_IOCOST_BPF))
+		return NULL;
+	return rcu_dereference_check(ioc->model, lockdep_is_held(&ioc->lock));
+}
+
 static const char __maybe_unused *ioc_name(struct ioc *ioc)
 {
 	struct gendisk *disk = ioc->rqos.disk;
@@ -806,7 +839,7 @@ static int ioc_autop_idx(struct ioc *ioc, struct gendisk *disk)
 		return AUTOP_SSD_DFL;

 	/* if user is overriding anything, maintain what was there */
-	if (ioc->user_qos_params || ioc->user_cost_model)
+	if (ioc->user_qos_params || ioc->user_cost_model || ioc_model(ioc))
 		return idx;

 	/* step up/down based on the vrate */
@@ -2586,8 +2619,16 @@ out:

 static u64 calc_vtime_cost(struct bio *bio, struct ioc_gq *iocg, bool is_merge)
 {
+	const struct iocost_model_ops *model;
 	u64 cost;

+	guard(rcu)();
+	model = ioc_model(iocg->ioc);
+	if (model) {
+		cost = model->calc_cost(bio, is_merge ? IOCOST_COST_F_MERGE : 0);
+		return min(cost, VTIME_PER_SEC);
+	}
+
 	calc_vtime_cost_builtin(bio, iocg, is_merge, &cost);
 	return cost;
 }
@@ -2612,8 +2653,16 @@ static void calc_size_vtime_cost_builtin(struct request *rq, struct ioc *ioc,

 static u64 calc_size_vtime_cost(struct request *rq, struct ioc *ioc)
 {
+	const struct iocost_model_ops *model;
 	u64 cost;

+	guard(rcu)();
+	model = ioc_model(ioc);
+	if (model)
+		return (blk_rq_stats_sectors(rq) >> IOC_SECT_TO_PAGE_SHIFT) *
+			(req_op(rq) == REQ_OP_READ ? model->read_vtime_per_page :
+						     model->write_vtime_per_page);
+
 	calc_size_vtime_cost_builtin(rq, ioc, &cost);
 	return cost;
 }
@@ -2908,10 +2957,33 @@ static void ioc_rqos_queue_depth_changed(struct rq_qos *rqos)
 	spin_unlock_irq(&ioc->lock);
 }

+/* Called by .unreg and by device removal, whichever is first. */
+static void ioc_detach_model(struct ioc *ioc)
+{
+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+	const struct iocost_model_ops *ops = ioc->attached;
+
+	lockdep_assert_held(&ioc->rqos.disk->queue->rq_qos_mutex);
+
+	if (!ops)
+		return;
+
+	spin_lock_irq(&ioc->lock);
+	rcu_assign_pointer(ioc->model, NULL);
+	ioc->attached = NULL;
+	ioc->attached_link = NULL;
+	spin_unlock_irq(&ioc->lock);
+
+	/* calc_cost() can still run. The map is freed after an RCU gp. */
+	bpf_struct_ops_put(ops);
+#endif
+}
+
 static void ioc_rqos_exit(struct rq_qos *rqos)
 {
 	struct ioc *ioc = rqos_to_ioc(rqos);

+	ioc_detach_model(ioc);
 	blkcg_deactivate_policy(rqos->disk, &blkcg_policy_iocost);

 	spin_lock_irq(&ioc->lock);
@@ -3458,11 +3530,12 @@ static u64 ioc_cost_model_prfill(struct seq_file *sf,
 		return 0;

 	spin_lock_irq(&ioc->lock);
-	seq_printf(sf, "%s ctrl=%s model=linear "
+	seq_printf(sf, "%s ctrl=%s model=%s "
 		   "rbps=%llu rseqiops=%llu rrandiops=%llu "
 		   "wbps=%llu wseqiops=%llu wrandiops=%llu "
 		   "flushiops=%llu\n",
 		   dname, ioc->user_cost_model ? "user" : "auto",
+		   ioc_model(ioc) ? "bpf" : "linear",
 		   u[I_LCOEF_RBPS], u[I_LCOEF_RSEQIOPS], u[I_LCOEF_RRANDIOPS],
 		   u[I_LCOEF_WBPS], u[I_LCOEF_WSEQIOPS], u[I_LCOEF_WRANDIOPS],
 		   u[I_LCOEF_FLUSHIOPS]);
@@ -3499,6 +3572,7 @@ static const match_table_t i_lcoef_tokens = {
 static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
 				    size_t nbytes, loff_t off)
 {
+	const struct iocost_model_ops *model;
 	struct blkg_conf_ctx ctx;
 	struct request_queue *q;
 	unsigned int memflags;
@@ -3535,6 +3609,7 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
 	spin_lock_irq(&ioc->lock);
 	memcpy(u, ioc->params.i_lcoefs, sizeof(u));
 	user = ioc->user_cost_model;
+	model = ioc_model(ioc);

 	ret = -EINVAL;

@@ -3559,7 +3634,11 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
 			continue;
 		case COST_MODEL:
 			match_strlcpy(buf, &args[0], sizeof(buf));
-			if (strcmp(buf, "linear"))
+			if (!strcmp(buf, "linear"))
+				model = NULL;
+			else if (!strcmp(buf, "bpf") && ioc->attached)
+				model = ioc->attached;
+			else
 				goto unlock;
 			continue;
 		}
@@ -3579,6 +3658,7 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
 	} else {
 		ioc->user_cost_model = false;
 	}
+	rcu_assign_pointer(ioc->model, model);
 	ioc_refresh_params(ioc, true);

 	ret = 0;
@@ -3626,6 +3706,158 @@ static struct blkcg_policy blkcg_policy_iocost = {
 	.pd_stat_fn	= ioc_pd_stat,
 };

+#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF
+static int ioc_bpf_reg(void *kdata, struct bpf_link *link)
+{
+	struct iocost_model_ops *ops = kdata;
+	struct block_device *bdev;
+	struct request_queue *q;
+	unsigned int memflags;
+	struct ioc *ioc;
+	int ret;
+
+	bdev = blkdev_get_no_open(new_decode_dev(ops->dev), false);
+	if (!bdev)
+		return -ENODEV;
+
+	q = bdev_get_queue(bdev);
+	if (bdev_is_partition(bdev) || !queue_is_mq(q)) {
+		blkdev_put_no_open(bdev);
+		return -EOPNOTSUPP;
+	}
+
+	/* the order of ioc_qos_write() */
+	memflags = blk_mq_freeze_queue(q);
+	mutex_lock(&q->rq_qos_mutex);
+
+	ret = -ENODEV;
+	if (!disk_live(bdev->bd_disk))
+		goto out;
+
+	ioc = q_to_ioc(q);
+	if (!ioc) {
+		ret = blk_iocost_init(bdev->bd_disk);
+		if (ret)
+			goto out;
+		ioc = q_to_ioc(q);
+	}
+
+	ret = -EBUSY;
+	if (ioc->attached || !bpf_struct_ops_get(ops))
+		goto out;
+
+	spin_lock_irq(&ioc->lock);
+	ioc->attached = ops;
+	ioc->attached_link = link;
+	rcu_assign_pointer(ioc->model, ops);
+	spin_unlock_irq(&ioc->lock);
+	ret = 0;
+out:
+	mutex_unlock(&q->rq_qos_mutex);
+	blk_mq_unfreeze_queue(q, memflags);
+	blkdev_put_no_open(bdev);
+	return ret;
+}
+
+static void ioc_bpf_unreg(void *kdata, struct bpf_link *link)
+{
+	struct iocost_model_ops *ops = kdata;
+	struct block_device *bdev;
+	struct request_queue *q;
+	struct ioc *ioc;
+
+	/* del_gendisk() is in progress or done. ioc_rqos_exit() detaches. */
+	bdev = blkdev_get_no_open(new_decode_dev(ops->dev), false);
+	if (!bdev)
+		return;
+
+	q = bdev_get_queue(bdev);
+	mutex_lock(&q->rq_qos_mutex);
+	ioc = q_to_ioc(q);
+	/* the dev_t can belong to another disk and another model by now */
+	if (ioc && ioc->attached == ops && ioc->attached_link == link)
+		ioc_detach_model(ioc);
+	mutex_unlock(&q->rq_qos_mutex);
+	blkdev_put_no_open(bdev);
+}
+
+static int ioc_bpf_init(struct btf *btf)
+{
+	return 0;
+}
+
+static int ioc_bpf_check_member(const struct btf_type *t,
+				const struct btf_member *member,
+				const struct bpf_prog *prog)
+{
+	return prog->sleepable ? -EINVAL : 0;
+}
+
+static int ioc_bpf_init_member(const struct btf_type *t,
+			       const struct btf_member *member,
+			       void *kdata, const void *udata)
+{
+	const struct iocost_model_ops *uops = udata;
+	struct iocost_model_ops *ops = kdata;
+
+	switch (__btf_member_bit_offset(t, member) / 8) {
+	case offsetof(struct iocost_model_ops, dev):
+		ops->dev = uops->dev;
+		return 1;
+	case offsetof(struct iocost_model_ops, read_vtime_per_page):
+		ops->read_vtime_per_page = uops->read_vtime_per_page;
+		return 1;
+	case offsetof(struct iocost_model_ops, write_vtime_per_page):
+		ops->write_vtime_per_page = uops->write_vtime_per_page;
+		return 1;
+	}
+	return 0;
+}
+
+static int ioc_bpf_validate(void *kdata)
+{
+	struct iocost_model_ops *ops = kdata;
+
+	if (!ops->dev || !ops->calc_cost ||
+	    !ops->read_vtime_per_page || !ops->write_vtime_per_page)
+		return -EINVAL;
+	return 0;
+}
+
+static u64 ioc_bpf_calc_cost_stub(struct bio *bio, u64 model_flags)
+{
+	return 0;
+}
+
+static struct iocost_model_ops ioc_bpf_stubs = {
+	.calc_cost	= ioc_bpf_calc_cost_stub,
+};
+
+static const struct bpf_verifier_ops ioc_bpf_verifier_ops = {
+	.get_func_proto		= bpf_base_func_proto,
+	.is_valid_access	= bpf_tracing_btf_ctx_access,
+};
+
+static struct bpf_struct_ops ioc_bpf_struct_ops = {
+	.verifier_ops	= &ioc_bpf_verifier_ops,
+	.init		= ioc_bpf_init,
+	.check_member	= ioc_bpf_check_member,
+	.init_member	= ioc_bpf_init_member,
+	.validate	= ioc_bpf_validate,
+	.reg		= ioc_bpf_reg,
+	.unreg		= ioc_bpf_unreg,
+	.cfi_stubs	= &ioc_bpf_stubs,
+	.name		= "iocost_model_ops",
+	.owner		= THIS_MODULE,
+};
+
+static int __init ioc_bpf_struct_ops_init(void)
+{
+	return register_bpf_struct_ops(&ioc_bpf_struct_ops, iocost_model_ops);
+}
+late_initcall(ioc_bpf_struct_ops_init);
+#endif
+
 static int __init ioc_init(void)
 {
 	return blkcg_policy_register(&blkcg_policy_iocost);

>  3/5: selftests with two example models (the full builtin linear
>       HDD formula at double cost, and a multi-stream sequentiality
>       model) plus a runner and the selftest kernel config entries
>  4/5: add an iocost_ioc_tick tracepoint emitting the per-period
>       controller state, so model quality can be evaluated without
>       drgn (existing events are state-change driven and silent in
>       steady state)

Also progs/iocost_model.c in patch 3 has SEC("tp_btf/iocost_ioc_tick").
Patch 4 adds this tracepoint. At patch 3 iocost_model__load() fails.
Swap the two patches.

pw-bot: cr

Re: [RFC PATCH v11 0/5] blk-iocost: BPF struct_ops cost model

[Tejun Heo]

Hello, Alexei.

On Wed, Oct 28, 2026 at 03:58:57PM +0000, Alexei Starovoitov wrote:
> Lets drop them. A struct_ops can get new members later, when a model
> needs them.
> I'd like to hear from Tejun whether such a model is planned.

The callbacks would only be worth having if they could do something
calc_cost() can't, e.g. sleep, but I forgot that pd allocations are
opportunistic to begin with. A blkg and its pds are created on the bio
path under queue_lock with GFP_NOWAIT, so iocg_init() can't sleep
either. Agreed, let's drop them.

> ...
> calc_cost() runs under rcu_read_lock(). bpf_map_put() frees a struct_ops
> map after an RCU gp. So detach doesn't need to freeze the queue.
> The rest fits into blk-iocost.c and patch 1 is not needed.

Agreed on both.

Thanks.

--
tejun

Re: [RFC PATCH v11 0/5] blk-iocost: BPF struct_ops cost model

[Tejun Heo]

Hello, Tao.

Besides the restructuring agreed in my reply to Alexei, dropping
iocg_init()/iocg_free() and moving everything into blk-iocost.c, the
series still has problems, which I replied to on the patches. The cover
letter is still long and narrates mechanics, and it still carries the
v9 and v10 changelogs.

Thanks.

--
tejun

Re: [PATCH 1/5] blk-iocost: move shared definitions to blk-iocost.h

[Tejun Heo]

Hello, Tao.

On Wed, Oct 28, 2026 at 10:35:30PM +0800, Tao Cui wrote:
> -static struct blkcg_policy blkcg_policy_iocost;
> -
> -/* accessors and helpers */
> -static struct ioc *rqos_to_ioc(struct rq_qos *rqos)

This isn't a pure move. The comment is dropped here.

>  }
>
> -
>  static int ioc_weight_show(struct seq_file *sf, void *v)

The blank line removal is unrelated to the move.

> +	bool				user_qos_params:1;
> +	bool				user_cost_model:1;
> +
> +};

The blank line is new, and 2/5 then replaces it.

Thanks.

--
tejun

Re: [PATCH 2/5] blk-iocost: add BPF struct_ops cost model support

[Tejun Heo]

Hello, Tao.

On Wed, Oct 28, 2026 at 10:35:31PM +0800, Tao Cui wrote:
> +/*
> +/*
> + * Deliver iocg_init()/iocg_free() to the cgroups already online on the

The comment opens with "/*" twice. clang warns "'/*' within block comment
[-Wcomment]" and CONFIG_WERROR builds break.

> +			/*
> +			 * A blkg which never went online never reached the
> +			 * radix tree, so it must not receive an init either;
> ...
> +			if (!iocg || !iocg->pd.online)
> +				continue;

blkg_create() adds a blkg to q->blkg_list only after radix_tree_insert()
succeeds, so the blkgs skipped here are the ones blkg_destroy() already
took offline, not ones that never reached the radix tree.

> +	if (!disk_live(disk)) {
> +		mutex_unlock(&q->rq_qos_mutex);
> +		blk_mq_unfreeze_queue(q, memflags);
> +		blkdev_put_no_open(bdev);
> +		return -ENODEV;
> +	}

The guard() suggestion from v10 was only partly applied. This unlock,
unfreeze and put sequence is repeated on three error paths.

> +	/*
> +	 * Scope the mutex so the tail bdev put runs with blkcg_mutex
> +	 * already released: dropping the last device reference can
> +	 * reach disk_release() -> blkcg_exit_disk() ->
> +	 * blkg_destroy_all(), which takes q->blkcg_mutex.
> +	 */
> +	{
> +		guard(mutex)(&q->blkcg_mutex);

The bare { } block isn't necessary, and ioc_bpf_detach() has the same
block with the same comment. Can you drop them?

> +	/*
> +	 * The attachment is checked under rq_qos_mutex first, so an unmatched
> +	 * detach exits without freezing.
> +	 */
> +	mutex_lock(&q->rq_qos_mutex);
> +	spin_lock_irq(&q->queue_lock);
> +	ioc = q_to_ioc(q);
> +	if (ioc) {
> +		spin_lock(&ioc->lock);
> +		matched = ioc_attached_model_locked(ioc) == ops &&
> +			  ioc->bpf_link == link;
> ...

The same 11 lines check the attachment again after the freeze. The check
under rq_qos_mutex after the freeze is the one that matters.

> +/*
> + * Pluggable cost model interface for blk-iocost.
> + *
> + * A BPF struct_ops implementation can take over the cost model of one device.
> ...
> + * offline and its free. Both callbacks are called with ioc->lock held and must
> + * not sleep; the delivery points nest inside queue_lock except for iocg_free()
> + * from ioc_pd_free() in blkg_free_workfn().
> + */

This restates the attach mechanics and lists the call sites. The comments
and descriptions in general are still long and narrate mechanics.

Thanks.

--
tejun

Re: [PATCH 3/5] selftests/bpf: add iocost cost model test

[Tejun Heo]

Hello, Tao.

On Wed, Oct 28, 2026 at 10:35:32PM +0800, Tao Cui wrote:
> +static bool iocost_bpf_supported(void)
> +{
> +	FILE *f;
> +	char line[128];
> +	bool any = false, on = false;
> ...
> +		if (strstr(line, "CONFIG_"))
> +			any = true;
> ...
> +	return on;

any is set but never read, so every test skips without /proc/config.gz.

> +	/* isolate the configfs and cgroup mounts from the host */
> +	if (unshare(CLONE_NEWNS)) {

unshare(CLONE_NEWNS) alone doesn't make the mounts private, so the
configfs mount propagates to the host, one more per test.

> +	ASSERT_OK(configfs_write("/sys/fs/cgroup/cgroup.subtree_control",
> +				 "+io"), "subtree_io");
> +	if (!ASSERT_OK(cgroup_mkdir("/sys/fs/cgroup/iocost_pairing"),

The "+io" write to the root cgroup's cgroup.subtree_control is never
reverted, the cgroups are created directly under /sys/fs/cgroup and
test_progs stays in the root cgroup. cgroup_helpers handles all of this.

> +	err = iocost_model__attach(skel);
> +	if (err == -EOPNOTSUPP || err == -ENODEV) {
> ...
> +		test__skip();

-ENODEV and -EOPNOTSUPP from attaching to the test's own null_blk device
are failures, not skips. The same goes for
serial_test_iocost_model_streams().

> +		ASSERT_GT(skel->bss->iocg_inits, 0, "pair_init");

This passes on the root blkg alone, so it doesn't show that the test
cgroup got iocg_init(). The same goes for eject_had_inits and
pair_init_again.

> +		/*
> +		 * detach implicitly by closing the link fd instead of
> +		 * calling bpf_link__destroy() explicitly: the release
> +		 * path must detach the model just like the explicit
> +		 * call does
> +		 */
> +		lfd = bpf_link__fd(skel->links.iocost_2x);
> +		bpf_link__disconnect(skel->links.iocost_2x);
> +		close(lfd);
> +		skel->links.iocost_2x = NULL;

This leaks libbpf's link struct. The reason in the comment doesn't hold
either. bpf_link__destroy() on a struct_ops link does the same close().

> +void serial_test_iocost_model_streams(void)
> ...
> +	if (ASSERT_OK(err, "attach")) {
> +		err = readback_model(dev, model, sizeof(model));
> +		if (ASSERT_OK(err, "readback"))
> +			ASSERT_STREQ(model, "bpf", "model_bpf");
> +
> +		bpf_link__destroy(skel->links.iocost_ms);

The test never enables the controller, so iocost_ms's calc_cost() never
runs. The multi-stream results in the cover letter aren't exercised by
the test.

Thanks.

--
tejun

Re: [PATCH 4/5] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary

[Tejun Heo]

Hello, Tao.

On Wed, Oct 28, 2026 at 10:35:33PM +0800, Tao Cui wrote:
>  		u64 vdone, vtime, usage_us;
>  		u32 hw_active, hw_inuse;
>
> +		nr_active++;

nr_active is counted even when the tracepoint is disabled.

> +	/*
> +	 * Everything the tick reports is final here: busy_level was just
> +	 * computed, cur_period hasn't changed, nr_active and usage_us_sum are
> +	 * complete, and vrate and period_us still hold the values this period
> +	 * ran in. usage is wall time over wall time, summed over iocgs, and is

An io.cost.qos write in the middle of a period changes both period_us
and vrate.

Thanks.

--
tejun

Re: [PATCH 5/5] docs: cgroup-v2: document the iocost BPF cost model attachment

[Tejun Heo]

Hello, Tao.

On Wed, Oct 28, 2026 at 10:35:34PM +0800, Tao Cui wrote:
> +	When CONFIG_BLK_CGROUP_IOCOST_BPF is enabled, a BPF cost
> +	model can be bound to a device by attaching an

Users can't select CONFIG_BLK_CGROUP_IOCOST_BPF.

> +	Only detaching removes the model, after which "model=bpf"
> +	fails and the device prices with the builtin model again.
> +	Device removal ejects the attached model as well, while the
> +	struct_ops link remains until userspace destroys it.  "ctrl"

"Only detaching removes the model" is followed by "Device removal ejects
the attached model as well".

> +	accepts only "auto" and "user" and never selects a model.  It
> +	keeps describing the builtin coefficients, which take effect
> +	again when switched back.  The automatic profile stepping does
> +	not switch profiles while the BPF model is in use; as with
> +	user-set parameters, the profile selected from the device
> +	class still applies.

cgroup-v2.rst doesn't describe automatic profile stepping anywhere else.

> +	defined.  They remain readable and writable while "model" is
> +	"bpf" but take effect only after switching back to "linear".

The builtin coefficients taking effect again after switching back is
already said above.

Thanks.

--
tejun