[PATCH 0/4] media: verisilicon: Keep the CAPTURE format in line with the decoder

[PATCH 0/4] media: verisilicon: Keep the CAPTURE format in line with the decoder

[Arun Iyengar via B4 Relay]

Pre-existing bugs where the decoder writes more than the CAPTURE format
or an auxiliary buffer provides: the VP8 segment map when the format
grows while streaming (1), the AV1 motion vector area after film grain
is turned off (2), and an 8-bit CAPTURE format for a 10-bit stream
(3, 4). Found by Sashiko on my hantro-dec-size-check v1 [1].

Tested on an Orange Pi 5 (RK3588S) with a v7.3-based kernel, on the AV1
decoder: after film grain is turned off, the CAPTURE format is the same
as a fresh one (stock: 3133440 instead of 3525632 bytes); an 8-bit
CAPTURE format set for a 10-bit sequence is replaced by the 10-bit one,
and the 10-bit format comes back after the sequence is set again (stock:
8-bit in both cases). VP8 and AV1 fluster: no new failures; no splats.
HEVC and VP9 (G2) are not tested: the RK3588 has no G2 core, so AV1 is
the only 10-bit path here.
Patch 1 is not reproduced: the test kernel limits RK3588 VP8 to
1920x1088, and frames up to that size fit the single page the map gets.

Benjamin Gaignard's "[PATCH v1 5/7] media: verisilicon: vp8: Use
alloc/free helpers for auxiliary buffers" (2026-09-16) has the lines
patch 1 changes as context: a trivial conflict for whichever lands
second.

Developed with Claude (Anthropic), which wrote the code and the commit
messages. I reviewed every change.

[1] https://lore.kernel.org/r/20261112-hantro-dec-size-check-v1-1-e5f1180b625f@gmail.com

Signed-off-by: Arun Iyengar <aiyengar1990@gmail.com>
---
Arun Iyengar (4):
      media: verisilicon: Size the VP8 segment map for the largest frame
      media: verisilicon: Set need_postproc before resetting the CAPTURE format
      media: verisilicon: Keep the CAPTURE format at the stream's bit depth
      media: verisilicon: Set the bit depth again when its control is rewritten

 drivers/media/platform/verisilicon/hantro_drv.c  | 14 ++++++++++++--
 drivers/media/platform/verisilicon/hantro_v4l2.c | 20 +++++++++++++++++---
 drivers/media/platform/verisilicon/hantro_vp8.c  |  6 +++---
 3 files changed, 32 insertions(+), 8 deletions(-)
---
base-commit: ad6549a46748d00818483b535e5cd2a704685c98
change-id: 20261113-hantro-fixes-aace04d5268b

Best regards,
--  
Arun Iyengar <aiyengar1990@gmail.com>

[PATCH 1/4] media: verisilicon: Size the VP8 segment map for the largest frame

[Arun Iyengar via B4 Relay]

From: Arun Iyengar <aiyengar1990@gmail.com>

The segment map is allocated at STREAMON for the CAPTURE format of that
time, but the format can grow while streaming, and the decoder writes
2 bits per macroblock of the new size. The map of a small format is one
page, which a frame of more than 16384 macroblocks (3840x2160, for
example) overruns.

Size it for the largest frame the variant decodes, as VP9 does.

Found by Sashiko in its review of hantro-dec-size-check v1.

Fixes: bbd267daf4fc ("media: verisilicon: vp9: Allow to change resolution while streaming")
Cc: stable@vger.kernel.org
Signed-off-by: Arun Iyengar <aiyengar1990@gmail.com>
---
 drivers/media/platform/verisilicon/hantro_vp8.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/media/platform/verisilicon/hantro_vp8.c b/drivers/media/platform/verisilicon/hantro_vp8.c
index 381bc1d3bfda..12907f000a92 100644
--- a/drivers/media/platform/verisilicon/hantro_vp8.c
+++ b/drivers/media/platform/verisilicon/hantro_vp8.c
@@ -150,9 +150,9 @@ int hantro_vp8_dec_init(struct hantro_ctx *ctx)
 	size_t segment_map_size;
 	int ret;
 
-	/* segment map table size calculation */
-	mb_width = DIV_ROUND_UP(ctx->dst_fmt.width, 16);
-	mb_height = DIV_ROUND_UP(ctx->dst_fmt.height, 16);
+	/* The format can grow while streaming: size for the largest frame. */
+	mb_width = DIV_ROUND_UP(ctx->vpu_src_fmt->frmsize.max_width, 16);
+	mb_height = DIV_ROUND_UP(ctx->vpu_src_fmt->frmsize.max_height, 16);
 	segment_map_size = round_up(DIV_ROUND_UP(mb_width * mb_height, 4), 64);
 
 	/*

-- 
2.55.0

[PATCH 2/4] media: verisilicon: Set need_postproc before resetting the CAPTURE format

[Arun Iyengar via B4 Relay]

From: Arun Iyengar <aiyengar1990@gmail.com>

hantro_reset_raw_fmt() stores need_postproc only after
hantro_set_fmt_cap(), so hantro_try_fmt() sizes the new format with the
old value. When an AV1 stream drops film grain, the post-processor goes
away but the CAPTURE format still lacks the motion vector area the
decoder then writes after the picture.

Set the bit depth and need_postproc first and restore them on error.
The next patch needs the bit depth set there too.

Found by Sashiko in its review of hantro-dec-size-check v1.

Fixes: 80c7373a456e ("media: verisilicon: Conditionally ignore native formats")
Cc: stable@vger.kernel.org
Signed-off-by: Arun Iyengar <aiyengar1990@gmail.com>
---
 drivers/media/platform/verisilicon/hantro_v4l2.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/media/platform/verisilicon/hantro_v4l2.c b/drivers/media/platform/verisilicon/hantro_v4l2.c
index 83af9fa1ce94..2baab4cf3bbf 100644
--- a/drivers/media/platform/verisilicon/hantro_v4l2.c
+++ b/drivers/media/platform/verisilicon/hantro_v4l2.c
@@ -449,6 +449,8 @@ hantro_reset_raw_fmt(struct hantro_ctx *ctx, int bit_depth, bool need_postproc)
 {
 	const struct hantro_fmt *raw_vpu_fmt;
 	struct v4l2_pix_format_mplane raw_fmt, *encoded_fmt;
+	bool old_need_postproc = ctx->need_postproc;
+	int old_bit_depth = ctx->bit_depth;
 	int ret;
 
 	raw_vpu_fmt = hantro_get_default_fmt(ctx, false, bit_depth, need_postproc);
@@ -465,14 +467,18 @@ hantro_reset_raw_fmt(struct hantro_ctx *ctx, int bit_depth, bool need_postproc)
 	hantro_reset_fmt(&raw_fmt, raw_vpu_fmt);
 	raw_fmt.width = encoded_fmt->width;
 	raw_fmt.height = encoded_fmt->height;
+
+	/* hantro_try_fmt() uses these. */
+	ctx->bit_depth = bit_depth;
+	ctx->need_postproc = need_postproc;
 	if (ctx->is_encoder)
 		ret = hantro_set_fmt_out(ctx, &raw_fmt, need_postproc);
 	else
 		ret = hantro_set_fmt_cap(ctx, &raw_fmt);
 
-	if (!ret) {
-		ctx->bit_depth = bit_depth;
-		ctx->need_postproc = need_postproc;
+	if (ret) {
+		ctx->bit_depth = old_bit_depth;
+		ctx->need_postproc = old_need_postproc;
 	}
 
 	return ret;

-- 
2.55.0

[PATCH 3/4] media: verisilicon: Keep the CAPTURE format at the stream's bit depth

[Arun Iyengar via B4 Relay]

From: Arun Iyengar <aiyengar1990@gmail.com>

ENUM_FMT on the CAPTURE queue of a decoder lists only the formats that
match the bit depth of the stream, but S_FMT and TRY_FMT accept any of
them, so userspace can select an 8-bit format for a 10-bit stream and
the decoder writes 10-bit pictures into buffers sized for 8 bits.

Replace a format of the wrong depth with the default one for the depth,
as for an unknown format.

Found by Sashiko in its review of hantro-dec-size-check v1.

Fixes: 8d03c2ab881b ("media: hantro: Support format filtering by depth")
Cc: stable@vger.kernel.org
Signed-off-by: Arun Iyengar <aiyengar1990@gmail.com>
---
 drivers/media/platform/verisilicon/hantro_v4l2.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/media/platform/verisilicon/hantro_v4l2.c b/drivers/media/platform/verisilicon/hantro_v4l2.c
index 2baab4cf3bbf..362fbc747eea 100644
--- a/drivers/media/platform/verisilicon/hantro_v4l2.c
+++ b/drivers/media/platform/verisilicon/hantro_v4l2.c
@@ -342,6 +342,14 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
 		pix_mp->pixelformat = fmt->fourcc;
 	}
 
+	if (!coded && !ctx->is_encoder &&
+	    !hantro_check_depth_match(fmt, ctx->bit_depth)) {
+		fmt = hantro_get_default_fmt(ctx, false, ctx->bit_depth, ctx->need_postproc);
+		if (!fmt)
+			return -EINVAL;
+		pix_mp->pixelformat = fmt->fourcc;
+	}
+
 	if (coded) {
 		pix_mp->num_planes = 1;
 		vpu_fmt = fmt;

-- 
2.55.0

[PATCH 4/4] media: verisilicon: Set the bit depth again when its control is rewritten

[Arun Iyengar via B4 Relay]

From: Arun Iyengar <aiyengar1990@gmail.com>

S_FMT on the OUTPUT queue resets the CAPTURE format to 8 bits, but the
HEVC SPS, VP9 frame and AV1 sequence controls keep their 10-bit value.
Writing the same value again doesn't call s_ctrl, so the CAPTURE format
stays at 8 bits while the decoder writes 10-bit pictures. The same holds
for the AV1 film grain flag.

Mark these controls EXECUTE_ON_WRITE. Every write of them then sends a
V4L2_EVENT_CTRL value change, and VIDIOC_QUERY_EXT_CTRL shows the flag.

Found by Sashiko in its review of hantro-dec-size-check v1.

Fixes: 3c32d94c9c52 ("media: verisilicon: Do not change context bit depth before validating the format")
Cc: stable@vger.kernel.org
Signed-off-by: Arun Iyengar <aiyengar1990@gmail.com>
---
 drivers/media/platform/verisilicon/hantro_drv.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/media/platform/verisilicon/hantro_drv.c b/drivers/media/platform/verisilicon/hantro_drv.c
index 26e5d5547466..dffba6c43a0f 100644
--- a/drivers/media/platform/verisilicon/hantro_drv.c
+++ b/drivers/media/platform/verisilicon/hantro_drv.c
@@ -534,6 +534,7 @@ static const struct hantro_ctrl controls[] = {
 		.cfg = {
 			.id = V4L2_CID_STATELESS_HEVC_SPS,
 			.ops = &hantro_hevc_ctrl_ops,
+			.flags = V4L2_CTRL_FLAG_EXECUTE_ON_WRITE,
 		},
 	}, {
 		.codec = HANTRO_HEVC_DECODER,
@@ -555,6 +556,7 @@ static const struct hantro_ctrl controls[] = {
 		.cfg = {
 			.id = V4L2_CID_STATELESS_VP9_FRAME,
 			.ops = &hantro_vp9_ctrl_ops,
+			.flags = V4L2_CTRL_FLAG_EXECUTE_ON_WRITE,
 		},
 	}, {
 		.codec = HANTRO_VP9_DECODER,
@@ -577,6 +579,7 @@ static const struct hantro_ctrl controls[] = {
 		.cfg = {
 			.id = V4L2_CID_STATELESS_AV1_SEQUENCE,
 			.ops = &hantro_av1_ctrl_ops,
+			.flags = V4L2_CTRL_FLAG_EXECUTE_ON_WRITE,
 		},
 	}, {
 		.codec = HANTRO_AV1_DECODER,
@@ -591,6 +594,7 @@ static int hantro_ctrls_setup(struct hantro_dev *vpu,
 			      int allowed_codecs)
 {
 	int i, num_ctrls = ARRAY_SIZE(controls);
+	struct v4l2_ctrl *ctrl;
 
 	v4l2_ctrl_handler_init(&ctx->ctrl_handler, num_ctrls);
 
@@ -598,8 +602,8 @@ static int hantro_ctrls_setup(struct hantro_dev *vpu,
 		if (!(allowed_codecs & controls[i].codec))
 			continue;
 
-		v4l2_ctrl_new_custom(&ctx->ctrl_handler,
-				     &controls[i].cfg, NULL);
+		ctrl = v4l2_ctrl_new_custom(&ctx->ctrl_handler,
+					    &controls[i].cfg, NULL);
 		if (ctx->ctrl_handler.error) {
 			vpu_err("Adding control (%d) failed %d\n",
 				controls[i].cfg.id,
@@ -607,6 +611,12 @@ static int hantro_ctrls_setup(struct hantro_dev *vpu,
 			v4l2_ctrl_handler_free(&ctx->ctrl_handler);
 			return ctx->ctrl_handler.error;
 		}
+
+		/*
+		 * Standard controls get their flags from the core. S_FMT on
+		 * OUTPUT resets the bit depth: set it again on every write.
+		 */
+		ctrl->flags |= controls[i].cfg.flags & V4L2_CTRL_FLAG_EXECUTE_ON_WRITE;
 	}
 	return v4l2_ctrl_handler_setup(&ctx->ctrl_handler);
 }

-- 
2.55.0