[PATCH 3/4] media: verisilicon: Keep the CAPTURE format at the stream's bit depth

[Arun Iyengar via B4 Relay <devnull+aiyengar1990.gmail.com@kernel.org>]

From: Arun Iyengar <aiyengar1990@gmail.com>

ENUM_FMT on the CAPTURE queue of a decoder lists only the formats that
match the bit depth of the stream, but S_FMT and TRY_FMT accept any of
them, so userspace can select an 8-bit format for a 10-bit stream and
the decoder writes 10-bit pictures into buffers sized for 8 bits.

Replace a format of the wrong depth with the default one for the depth,
as for an unknown format.

Found by Sashiko in its review of hantro-dec-size-check v1.

Fixes: 8d03c2ab881b ("media: hantro: Support format filtering by depth")
Cc: stable@vger.kernel.org
Signed-off-by: Arun Iyengar <aiyengar1990@gmail.com>
---
 drivers/media/platform/verisilicon/hantro_v4l2.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/media/platform/verisilicon/hantro_v4l2.c b/drivers/media/platform/verisilicon/hantro_v4l2.c
index 2baab4cf3bbf..362fbc747eea 100644
--- a/drivers/media/platform/verisilicon/hantro_v4l2.c
+++ b/drivers/media/platform/verisilicon/hantro_v4l2.c
@@ -342,6 +342,14 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
 		pix_mp->pixelformat = fmt->fourcc;
 	}
 
+	if (!coded && !ctx->is_encoder &&
+	    !hantro_check_depth_match(fmt, ctx->bit_depth)) {
+		fmt = hantro_get_default_fmt(ctx, false, ctx->bit_depth, ctx->need_postproc);
+		if (!fmt)
+			return -EINVAL;
+		pix_mp->pixelformat = fmt->fourcc;
+	}
+
 	if (coded) {
 		pix_mp->num_planes = 1;
 		vpu_fmt = fmt;

-- 
2.55.0