[PATCH] media: vicodec: fix out-of-bounds write in FWHT encoder

[PATCH] media: vicodec: fix out-of-bounds write in FWHT encoder

[Junrui Luo]

vidioc_s_fmt_vid_out() sizes the encoder CAPTURE buffer from the
compressed descriptor pixfmt_fwht, whose sizeimage_mult is 3:
coded_w * coded_h * 3 + sizeof(struct fwht_cframe_hdr). fwht_encode_frame()
encodes one plane per component, and an incompressible plane takes the
FWHT_FRAME_UNENCODED path in encode_plane(), copying the plane verbatim.

For a 4-component pixel format all four planes are full resolution
(width_div == height_div == 1), so a frame that forces every plane
through the unencoded fallback writes
sizeof(struct fwht_cframe_hdr) + 4 * coded_w * coded_h bytes, overrunning
the plane by coded_w * coded_h, which can result in corruption
of adjacent kernel heap memory.

Bump pixfmt_fwht.sizeimage_mult from 3 to 4, matching the largest
components_num among the supported raw formats, so the capture buffer is
always large enough for the unencoded fallback.

Fixes: 29a7a5e99080 ("media: vicodec: add support for more pixel formats")
Reported-by: Yuhao Jiang <danisjiang@gmail.com>
Cc: stable@vger.kernel.org
Signed-off-by: Junrui Luo <moonafterrain@outlook.com>
---
 drivers/media/test-drivers/vicodec/vicodec-core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/media/test-drivers/vicodec/vicodec-core.c b/drivers/media/test-drivers/vicodec/vicodec-core.c
index 318e8330f16a..3ff2d06742c2 100644
--- a/drivers/media/test-drivers/vicodec/vicodec-core.c
+++ b/drivers/media/test-drivers/vicodec/vicodec-core.c
@@ -63,7 +63,7 @@ struct pixfmt_info {
 };
 
 static const struct v4l2_fwht_pixfmt_info pixfmt_fwht = {
-	V4L2_PIX_FMT_FWHT, 0, 3, 1, 1, 1, 1, 1, 0, 1
+	V4L2_PIX_FMT_FWHT, 0, 4, 1, 1, 1, 1, 1, 0, 1
 };
 
 static const struct v4l2_fwht_pixfmt_info pixfmt_stateless_fwht = {

---
base-commit: e43ffb69e0438cddd72aaa30898b4dc446f664f8
change-id: 20260601-fixes-7416dda13cfd

Best regards,
-- 
Junrui Luo <moonafterrain@outlook.com>

Re: [PATCH] media: vicodec: fix out-of-bounds write in FWHT encoder

[Hans Verkuil]

Hi Junrui,

On 01/06/2026 16:57, Junrui Luo wrote:
> vidioc_s_fmt_vid_out() sizes the encoder CAPTURE buffer from the
> compressed descriptor pixfmt_fwht, whose sizeimage_mult is 3:
> coded_w * coded_h * 3 + sizeof(struct fwht_cframe_hdr). fwht_encode_frame()
> encodes one plane per component, and an incompressible plane takes the
> FWHT_FRAME_UNENCODED path in encode_plane(), copying the plane verbatim.
> 
> For a 4-component pixel format all four planes are full resolution
> (width_div == height_div == 1), so a frame that forces every plane
> through the unencoded fallback writes
> sizeof(struct fwht_cframe_hdr) + 4 * coded_w * coded_h bytes, overrunning
> the plane by coded_w * coded_h, which can result in corruption
> of adjacent kernel heap memory.
> 
> Bump pixfmt_fwht.sizeimage_mult from 3 to 4, matching the largest
> components_num among the supported raw formats, so the capture buffer is
> always large enough for the unencoded fallback.
> 
> Fixes: 29a7a5e99080 ("media: vicodec: add support for more pixel formats")

Actually, it's commit b40dc2bf3581 ("media: vicodec: add support for 4 new
RGB32 pixelformats") that introduced this.

Please update the Fixes tag.

> Reported-by: Yuhao Jiang <danisjiang@gmail.com>

Where was that reported? Is there a URL to the that report?

> Cc: stable@vger.kernel.org
> Signed-off-by: Junrui Luo <moonafterrain@outlook.com>
> ---
>  drivers/media/test-drivers/vicodec/vicodec-core.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/media/test-drivers/vicodec/vicodec-core.c b/drivers/media/test-drivers/vicodec/vicodec-core.c
> index 318e8330f16a..3ff2d06742c2 100644
> --- a/drivers/media/test-drivers/vicodec/vicodec-core.c
> +++ b/drivers/media/test-drivers/vicodec/vicodec-core.c
> @@ -63,7 +63,7 @@ struct pixfmt_info {
>  };
>  
>  static const struct v4l2_fwht_pixfmt_info pixfmt_fwht = {
> -	V4L2_PIX_FMT_FWHT, 0, 3, 1, 1, 1, 1, 1, 0, 1
> +	V4L2_PIX_FMT_FWHT, 0, 4, 1, 1, 1, 1, 1, 0, 1
>  };
>  
>  static const struct v4l2_fwht_pixfmt_info pixfmt_stateless_fwht = {

It should be changed here as well, otherwise the test-media regression script
in v4l-utils will fail.

Regards,

	Hans

> 
> ---
> base-commit: e43ffb69e0438cddd72aaa30898b4dc446f664f8
> change-id: 20260601-fixes-7416dda13cfd
> 
> Best regards,

Re: [PATCH] media: vicodec: fix out-of-bounds write in FWHT encoder

[Junrui Luo]

Hi Hans,

Thanks for the review.

On Fri, Jul 10, 2026 at 10:32:09AM +0200, Hans Verkuil wrote:
> Actually, it's commit b40dc2bf3581 ("media: vicodec: add support for 4 new
> RGB32 pixelformats") that introduced this.
> 
> Please update the Fixes tag.

I traced this back a bit further. Commit 16ecf6dff97c ("media:
vicodec: Add support for 4 planes formats") already added ARGB32 and
ABGR32 as four-component formats. Would this be more appropriate?

> 
> > Reported-by: Yuhao Jiang <danisjiang@gmail.com>
> 
> Where was that reported? Is there a URL to the that report?

It was reported to me privately by Yuhao, so there is no public URL.

> 
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Junrui Luo <moonafterrain@outlook.com>
> > ---
> >  drivers/media/test-drivers/vicodec/vicodec-core.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/drivers/media/test-drivers/vicodec/vicodec-core.c b/drivers/media/test-drivers/vicodec/vicodec-core.c
> > index 318e8330f16a..3ff2d06742c2 100644
> > --- a/drivers/media/test-drivers/vicodec/vicodec-core.c
> > +++ b/drivers/media/test-drivers/vicodec/vicodec-core.c
> > @@ -63,7 +63,7 @@ struct pixfmt_info {
> >  };
> >  
> >  static const struct v4l2_fwht_pixfmt_info pixfmt_fwht = {
> > -	V4L2_PIX_FMT_FWHT, 0, 3, 1, 1, 1, 1, 1, 0, 1
> > +	V4L2_PIX_FMT_FWHT, 0, 4, 1, 1, 1, 1, 1, 0, 1
> >  };
> >  
> >  static const struct v4l2_fwht_pixfmt_info pixfmt_stateless_fwht = {
> 
> It should be changed here as well, otherwise the test-media regression script
> in v4l-utils will fail.

I will update in v2.

Thanks,
Junrui Luo

Re: [PATCH] media: vicodec: fix out-of-bounds write in FWHT encoder

[Hans Verkuil]

On 11/07/2026 08:29, Junrui Luo wrote:
> Hi Hans,
> 
> Thanks for the review.
> 
> On Fri, Jul 10, 2026 at 10:32:09AM +0200, Hans Verkuil wrote:
>> Actually, it's commit b40dc2bf3581 ("media: vicodec: add support for 4 new
>> RGB32 pixelformats") that introduced this.
>>
>> Please update the Fixes tag.
> 
> I traced this back a bit further. Commit 16ecf6dff97c ("media:
> vicodec: Add support for 4 planes formats") already added ARGB32 and
> ABGR32 as four-component formats. Would this be more appropriate?

Yes, you are right. That's when support for this was added.
Good catch.

> 
>>
>>> Reported-by: Yuhao Jiang <danisjiang@gmail.com>
>>
>> Where was that reported? Is there a URL to the that report?
> 
> It was reported to me privately by Yuhao, so there is no public URL.
> 
>>
>>> Cc: stable@vger.kernel.org
>>> Signed-off-by: Junrui Luo <moonafterrain@outlook.com>
>>> ---
>>>  drivers/media/test-drivers/vicodec/vicodec-core.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/media/test-drivers/vicodec/vicodec-core.c b/drivers/media/test-drivers/vicodec/vicodec-core.c
>>> index 318e8330f16a..3ff2d06742c2 100644
>>> --- a/drivers/media/test-drivers/vicodec/vicodec-core.c
>>> +++ b/drivers/media/test-drivers/vicodec/vicodec-core.c
>>> @@ -63,7 +63,7 @@ struct pixfmt_info {
>>>  };
>>>  
>>>  static const struct v4l2_fwht_pixfmt_info pixfmt_fwht = {
>>> -	V4L2_PIX_FMT_FWHT, 0, 3, 1, 1, 1, 1, 1, 0, 1
>>> +	V4L2_PIX_FMT_FWHT, 0, 4, 1, 1, 1, 1, 1, 0, 1
>>>  };
>>>  
>>>  static const struct v4l2_fwht_pixfmt_info pixfmt_stateless_fwht = {
>>
>> It should be changed here as well, otherwise the test-media regression script
>> in v4l-utils will fail.
> 
> I will update in v2.

Regards,

	Hans

[PATCH v2] media: vicodec: fix out-of-bounds write in FWHT encoder

[Junrui Luo]

vidioc_s_fmt_vid_out() sizes the encoder CAPTURE buffer from the
compressed descriptor pixfmt_fwht, whose sizeimage_mult is 3:
coded_w * coded_h * 3 + sizeof(struct fwht_cframe_hdr). fwht_encode_frame()
encodes one plane per component, and an incompressible plane takes the
FWHT_FRAME_UNENCODED path in encode_plane(), copying the plane verbatim.

For a 4-component pixel format all four planes are full resolution
(width_div == height_div == 1), so a frame that forces every plane
through the unencoded fallback writes
sizeof(struct fwht_cframe_hdr) + 4 * coded_w * coded_h bytes, overrunning
the plane by coded_w * coded_h, which can result in corruption
of adjacent kernel heap memory.

Bump pixfmt_fwht.sizeimage_mult from 3 to 4, matching the largest
components_num among the supported raw formats, so the capture buffer is
always large enough for the unencoded fallback.

Fixes: 16ecf6dff97c ("media: vicodec: Add support for 4 planes formats")
Reported-by: Yuhao Jiang <danisjiang@gmail.com>
Cc: stable@vger.kernel.org
Signed-off-by: Junrui Luo <moonafterrain@outlook.com>
---
Changes in v2:
- Correct the Fixes tag.
- Set pixfmt_stateless_fwht.sizeimage_mult to 4 as well.
---
 drivers/media/test-drivers/vicodec/vicodec-core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/media/test-drivers/vicodec/vicodec-core.c b/drivers/media/test-drivers/vicodec/vicodec-core.c
index 318e8330f16a..ff9d50fb05fd 100644
--- a/drivers/media/test-drivers/vicodec/vicodec-core.c
+++ b/drivers/media/test-drivers/vicodec/vicodec-core.c
@@ -63,11 +63,11 @@ struct pixfmt_info {
 };
 
 static const struct v4l2_fwht_pixfmt_info pixfmt_fwht = {
-	V4L2_PIX_FMT_FWHT, 0, 3, 1, 1, 1, 1, 1, 0, 1
+	V4L2_PIX_FMT_FWHT, 0, 4, 1, 1, 1, 1, 1, 0, 1
 };
 
 static const struct v4l2_fwht_pixfmt_info pixfmt_stateless_fwht = {
-	V4L2_PIX_FMT_FWHT_STATELESS, 0, 3, 1, 1, 1, 1, 1, 0, 1
+	V4L2_PIX_FMT_FWHT_STATELESS, 0, 4, 1, 1, 1, 1, 1, 0, 1
 };
 
 static void vicodec_dev_release(struct device *dev)

---
base-commit: 9716c086c8e8b141d35aa61f2e96a2e83de212a7
change-id: 20260714-fixes-aa8cf63bfd95

Best regards,
-- 
Junrui Luo <moonafterrain@outlook.com>