[PATCH] media: meson: vdec: fix NULL pointer deref in vdec_try_fmt_common

[PATCH] media: meson: vdec: fix NULL pointer deref in vdec_try_fmt_common

[Christian Hewitt]

When VIDIOC_TRY_FMT is called with an unsupported pixel format on the
OUTPUT queue, vdec_try_fmt_common() falls back to V4L2_PIX_FMT_MPEG2.
However, if a distro has locally patched MPEG2 support out (as it has
been broken for some time) the platform format table does not contain
MPEG2 so find_format() returns NULL and the subsequent dereference of
fmt_out->max_width triggers a NULL pointer dereference.

Fix this by falling back to the first format in the platform's format
array instead of hardcoding V4L2_PIX_FMT_MPEG2. This is always valid
since every platform defines at least one format.

Fixes: 3e7f51bd9607 ("media: meson: add v4l2 m2m video decoder driver")
Signed-off-by: Christian Hewitt <christianshewitt@gmail.com>

---
This fixes use of v4l2-ctl and v4l2-compliance tools when MPEG2 support
has been disabled via vdec_platform.c - NB: although this is a fix it's
also a rather niche scenario so there's no need to backport it to older
stable kernels.

 drivers/staging/media/meson/vdec/vdec.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/media/meson/vdec/vdec.c b/drivers/staging/media/meson/vdec/vdec.c
index 4b77ec1af5a7..f3e7072f221a 100644
--- a/drivers/staging/media/meson/vdec/vdec.c
+++ b/drivers/staging/media/meson/vdec/vdec.c
@@ -504,8 +504,8 @@ vdec_try_fmt_common(struct amvdec_session *sess, u32 size,
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
 		fmt_out = find_format(fmts, size, pixmp->pixelformat);
 		if (!fmt_out) {
-			pixmp->pixelformat = V4L2_PIX_FMT_MPEG2;
-			fmt_out = find_format(fmts, size, pixmp->pixelformat);
+			pixmp->pixelformat = fmts[0].pixfmt;
+			fmt_out = &fmts[0];
 		}
 		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-- 
2.43.0


_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: [PATCH] media: meson: vdec: fix NULL pointer deref in vdec_try_fmt_common

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Missing V4L2 queue busy check in vdec_s_fmt allows format changes during active streaming, leading to type confusion and memory corruption in the ISR.
--

commit 8aa33adb9b745ad7b1a26721ccafdbeb5f0768bb
Author: Christian Hewitt <christianshewitt@gmail.com>

media: meson: vdec: fix NULL pointer deref in vdec_try_fmt_common

This commit fixes a NULL pointer dereference in vdec_try_fmt_common() by
falling back to the first format in the platform's format array instead of
hardcoding MPEG2. This prevents crashes on systems where MPEG2 support has
been locally patched out.

> diff --git a/drivers/staging/media/meson/vdec/vdec.c b/drivers/staging/media/meson/vdec/vdec.c
> index a039d925c0fe5..c18bf352b1996 100644
> --- a/drivers/staging/media/meson/vdec/vdec.c
> +++ b/drivers/staging/media/meson/vdec/vdec.c

[ ... ]

[Severity: High]
This is a pre-existing issue, but while looking at the format handling in
this driver, I noticed a potential race condition.

Does vdec_s_fmt() allow format changes during active streaming?

drivers/staging/media/meson/vdec/vdec.c:vdec_s_fmt() {
        ...
	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
		sess->fmt_out = fmt_out;
	else if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
		sess->pixfmt_cap = format.fmt.pix_mp.pixelformat;
        ...
}

It appears there is no check to ensure the V4L2 queues are idle (e.g.,
via vb2_is_busy()) before overwriting the session format parameters.

If a userspace application is actively streaming and concurrently calls
VIDIOC_S_FMT with a different format, could this lead to type confusion
and memory corruption in the hardirq handler?

drivers/staging/media/meson/vdec/vdec.c:vdec_isr() {
        ...
	return sess->fmt_out->codec_ops->isr(sess);
}

Because the hardirq handler dereferences sess->fmt_out without locks,
a mid-stream format change might cause the ISR of the newly selected codec
to execute on a session state (sess->priv) initialized for the previous
codec.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260624065809.2460720-1-christianshewitt@gmail.com?part=1

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: [PATCH] media: meson: vdec: fix NULL pointer deref in vdec_try_fmt_common

[Nicolas Dufresne]

[-- Attachment #1.1: Type: text/plain, Size: 1988 bytes --]

Le mercredi 24 juin 2026 à 06:58 +0000, Christian Hewitt a écrit :
> When VIDIOC_TRY_FMT is called with an unsupported pixel format on the
> OUTPUT queue, vdec_try_fmt_common() falls back to V4L2_PIX_FMT_MPEG2.
> However, if a distro has locally patched MPEG2 support out (as it has
> been broken for some time) the platform format table does not contain
> MPEG2 so find_format() returns NULL and the subsequent dereference of
> fmt_out->max_width triggers a NULL pointer dereference.
> 
> Fix this by falling back to the first format in the platform's format
> array instead of hardcoding V4L2_PIX_FMT_MPEG2. This is always valid
> since every platform defines at least one format.
> 
> Fixes: 3e7f51bd9607 ("media: meson: add v4l2 m2m video decoder driver")
> Signed-off-by: Christian Hewitt <christianshewitt@gmail.com>

Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>

> 
> ---
> This fixes use of v4l2-ctl and v4l2-compliance tools when MPEG2 support
> has been disabled via vdec_platform.c - NB: although this is a fix it's
> also a rather niche scenario so there's no need to backport it to older
> stable kernels.
> 
>  drivers/staging/media/meson/vdec/vdec.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/staging/media/meson/vdec/vdec.c
> b/drivers/staging/media/meson/vdec/vdec.c
> index 4b77ec1af5a7..f3e7072f221a 100644
> --- a/drivers/staging/media/meson/vdec/vdec.c
> +++ b/drivers/staging/media/meson/vdec/vdec.c
> @@ -504,8 +504,8 @@ vdec_try_fmt_common(struct amvdec_session *sess, u32 size,
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>  		fmt_out = find_format(fmts, size, pixmp->pixelformat);
>  		if (!fmt_out) {
> -			pixmp->pixelformat = V4L2_PIX_FMT_MPEG2;
> -			fmt_out = find_format(fmts, size, pixmp-
> >pixelformat);
> +			pixmp->pixelformat = fmts[0].pixfmt;
> +			fmt_out = &fmts[0];
>  		}
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

[-- Attachment #2: Type: text/plain, Size: 167 bytes --]

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic