[PATCH v2 0/3] media: verisilicon: fix VEPU1 JPEG encoder writes past the CAPTURE buffer

[Sergei Khomenkov via B4 Relay <devnull+shomenkow.gmail.com@kernel.org>]

The H1 JPEG encoder is given its stream buffer limit in bytes, but the
register counts 64-bit words, so a stream that does not fit the CAPTURE
buffer is written past its end, up to eight times the buffer size. A
quality 100 frame of noise does not fit the default sizeimage, and
userspace can also set a smaller one.

Patch 1 stops the VEPU1 encoder after a buffer-full interrupt, which
leaves it paused. Patch 2 programs the limit in 64-bit words, rounded
down to whole 64-byte bursts. Patch 3 makes the smallest CAPTURE
sizeimage the 624-byte header plus one burst.

Testing, on an RK3026 (VEPU1) only: without the series the encoder
wrote 230 KB past a default-sized 640x480 buffer and 1.6 MB past a
1920x1088 one. With it, 17 overflowing frames, four of them queued back
to back, stopped at the programmed limit without writing past the plane
and came back with the ERROR flag, one interrupt each and no timeout;
the jobs after them completed, and frames that fit are byte for byte the
same as before. A sizeimage of 1 or 600 now reads back 688.
v4l2-compliance 1.30.1 on the encoder node shows no new failures.

The VEPU2 JPEG encoder writes its limit in bytes the same way, but I
cannot test it, so its limit and interrupt handler are unchanged; the
common patch 3 applies to it as well. The STM32MP25 VENC runs the H1
code of patch 2 and gets patch 3, but not the stop of patch 1. Neither
is tested.

The bugs were found and the patches drafted with an LLM coding
assistant while porting mainline Linux to an RK3026 board.

---
Changes in v2:
- Trim the commit messages and code comments (Nicolas).
- 1/3: clear only the enable bit of H1_REG_ENC_CTRL instead of writing
  0 to the register, and add a rate-limited warning per fault bit, as
  hantro_g2_irq() does; add the H1_REG_INTERRUPT_* fault bits (Nicolas).
- 3/3: keep max(): it is a floor of header + one burst, the 64-byte
  rounding is on the stream limit in 2/3 (Nicolas).
- Move the test numbers from the commit messages to this cover letter.
- Link to v1: https://lore.kernel.org/r/20261109-hantro-buflimit-v1-0-f0b86ce61e66@gmail.com

---
Sergei Khomenkov (3):
      media: verisilicon: stop the VEPU1 JPEG encoder after an unfinished frame
      media: verisilicon: program the H1 JPEG stream limit in 64-bit words
      media: verisilicon: leave room for the JPEG header and one stream burst

 .../media/platform/verisilicon/hantro_h1_jpeg_enc.c    |  7 ++++++-
 drivers/media/platform/verisilicon/hantro_h1_regs.h    |  4 ++++
 drivers/media/platform/verisilicon/hantro_v4l2.c       |  9 +++++++++
 drivers/media/platform/verisilicon/rockchip_vpu_hw.c   | 18 ++++++++++++++++++
 4 files changed, 37 insertions(+), 1 deletion(-)
---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
change-id: 20261106-hantro-buflimit-055e8029be23

Best regards,
-- 
Sergei Khomenkov <shomenkow@gmail.com>