[PATCH v3] block/blk-iolatency: fix always-zero cur_stat mean on non-SSD

[PATCH v3] block/blk-iolatency: fix always-zero cur_stat mean on non-SSD

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

iolatency_check_latencies() accumulates the per-window latency stat
into iolat->cur_stat via latency_stat_sum(), which for non-SSD devices
calls blk_rq_stat_sum().  But blk_rq_stat_sum() is contracted on a raw
per-cpu src: it folds src->batch into the new mean.  By the time the
per-window stat reaches cur_stat it is already aggregated, with batch
left at 0.  Each window therefore contributes zero latency to
cur_stat->mean, which on non-SSD devices stays at 0; the
latency_sum_ok(&cur_stat) check that gates scaling up is always true,
so the scale-up hysteresis never engages.  SSD devices use the
percentile path and are unaffected.

Add latency_stat_merge(), which merges two already-aggregated stats by
their reconstructed totals (mean * nr_samples) instead of src->batch,
and use it only for the cur_stat accumulation; the earlier per-cpu sum
keeps latency_stat_sum(), whose src is still raw.

Fixes: d70675121546 ("block: introduce blk-iolatency io controller")
Signed-off-by: Tao Cui <cuitao@kylinos.cn>

---
Changes in v3:
- Rebase onto current linux-next head (no code change).
- Verified with a two-kernel QEMU run (virtio-blk forced rotational,
  cgroup v2 io.latency target set below the achievable latency so the
  scale path runs every window): on the unpatched kernel cur_stat->mean
  stayed 0 across all windows while the per-window mean was 120-250us;
  with this patch cur_stat->mean tracked the per-window mean.

Changes in v2:
- Move the fix from blk-stat.c (blk_rq_stat_sum) into blk-iolatency.c,
  per Tang Yizhou's review: blk_rq_stat_sum() is contracted on a raw
  per-cpu src, so the aggregated-stat merge belongs in the iolatency
  caller, not the shared helper.

v1: https://lore.kernel.org/all/20260720133813.45150-1-cui.tao@linux.dev/
v2: https://lore.kernel.org/all/20260721030312.85655-1-cui.tao@linux.dev/
---
 block/blk-iolatency.c | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index 2caa79a008ad..853608a4f187 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -216,6 +216,31 @@ static inline void latency_stat_sum(struct iolatency_grp *iolat,
 		blk_rq_stat_sum(&sum->rqs, &stat->rqs);
 }
 
+/*
+ * Merge an already-aggregated @stat into @sum by reconstructed totals
+ * (mean * nr_samples); blk_rq_stat_sum() reads src->batch, which is 0
+ * here. The overflow check also rejects the 0 samples case.
+ */
+static inline void latency_stat_merge(struct iolatency_grp *iolat,
+				      struct latency_stat *sum,
+				      struct latency_stat *stat)
+{
+	if (iolat->ssd) {
+		sum->ps.total += stat->ps.total;
+		sum->ps.missed += stat->ps.missed;
+	} else {
+		if (sum->rqs.nr_samples + stat->rqs.nr_samples <=
+		    sum->rqs.nr_samples)
+			return;
+		sum->rqs.mean = div_u64(sum->rqs.mean * sum->rqs.nr_samples +
+					stat->rqs.mean * stat->rqs.nr_samples,
+					sum->rqs.nr_samples + stat->rqs.nr_samples);
+		sum->rqs.min = min(sum->rqs.min, stat->rqs.min);
+		sum->rqs.max = max(sum->rqs.max, stat->rqs.max);
+		sum->rqs.nr_samples += stat->rqs.nr_samples;
+	}
+}
+
 static inline void latency_stat_record_time(struct iolatency_grp *iolat,
 					    u64 req_time)
 {
@@ -547,7 +572,7 @@ static void iolatency_check_latencies(struct iolatency_grp *iolat, u64 now)
 	/* Somebody beat us to the punch, just bail. */
 	spin_lock_irqsave(&lat_info->lock, flags);
 
-	latency_stat_sum(iolat, &iolat->cur_stat, &stat);
+	latency_stat_merge(iolat, &iolat->cur_stat, &stat);
 	lat_info->nr_samples -= iolat->nr_samples;
 	lat_info->nr_samples += latency_stat_samples(iolat, &iolat->cur_stat);
 	iolat->nr_samples = latency_stat_samples(iolat, &iolat->cur_stat);
-- 
2.43.0