[PATCH 0/2] media: rkvdec: vdpu381: compress the co-located motion vectors

[PATCH 0/2] media: rkvdec: vdpu381: compress the co-located motion vectors

[Arun Iyengar via B4 Relay]

The VDPU381 can store the co-located motion vectors of a frame
compressed (colmv_compress_en in register 12). The Rockchip MPP enables
it for H.264 streams without field pictures or MBAFF and always for
HEVC; mainline never does. B-frames are what limits H.264 decoding at
4K on one core, and the motion vector traffic is part of that.

Patch 1 enables it for H.264 and reserves the worst case of the
compressed layout, which can be larger than the uncompressed one (the
hardware wrote up to 556128 bytes compressed against 522240
uncompressed for the motion vectors of a 4K HEVC stream): a 4K CAPTURE
buffer gets 8.3 MB instead of 4.1 MB of motion vector space on VDPU381.
Patch 2 enables it for HEVC.

Tested on an Orange Pi 5 (RK3588S), GStreamer into fakevideosink,
240-frame clips: H.264 3840x2160 65.4 -> 68.4 fps, 1920x1080 214 -> 234
fps, HEVC 3840x2160 98.6-102.8 -> 104.6 fps. Every frame's checksum is
the same with compression off and on, the decoded frames are bit-exact
with FFmpeg's software decoders, and fluster is unchanged
(JCT-VC-HEVC_V1 146/147, JVT-AVC_V1 129/135, JVT-FR-EXT 65/69). The
bytes the hardware writes into the motion vector area were measured
with a poisoned buffer to size the compressed layout.

This series was developed with Claude (Anthropic) as a coding assistant:
it wrote the code and the commit messages, and I reviewed and tested
every change as described above.

Signed-off-by: Arun Iyengar <aiyengar1990@gmail.com>
---
Arun Iyengar (2):
      media: rkvdec: vdpu381: Compress the co-located motion vectors of H.264
      media: rkvdec: vdpu381: Compress the co-located motion vectors of HEVC

 .../platform/rockchip/rkvdec/rkvdec-vdpu381-h264.c     | 10 ++++++++++
 .../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c     |  3 +++
 drivers/media/platform/rockchip/rkvdec/rkvdec.c        | 18 +++++++++++++++++-
 3 files changed, 30 insertions(+), 1 deletion(-)
---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
change-id: 20261110-rkvdec-colmv-455057cb3974

Best regards,
--  
Arun Iyengar <aiyengar1990@gmail.com>

[PATCH 1/2] media: rkvdec: vdpu381: Compress the co-located motion vectors of H.264

[Arun Iyengar via B4 Relay]

From: Arun Iyengar <aiyengar1990@gmail.com>

The VDPU381 can store the co-located motion vectors of a frame
compressed (reg012 colmv_compress_en). The vendor code (MPP,
hal_h264d_vdpu34x.c) enables it for H.264 streams without field
pictures or MBAFF, and for HEVC. Mainline never does.

B-frames are what limits H.264 decoding at 4K: the core is 99% busy,
and a B-frame of a 48 Mbit/s 4K stream takes 16 ms of hardware time
against 10.8 ms for a P-frame and 7.0 ms for an I-frame. Storing the
motion vectors compressed cuts that memory traffic. Decoding the same
240 frames to fakevideosink on RK3588 goes from 65.4 to 68.4 fps at
3840x2160 (+4.7%) and from 214 to 234 fps at 1920x1080 (+9%). The
decoded frames are identical (same checksum for all 240 frames, and
bit-exact with FFmpeg).

The compressed layout is made of 64x16-pixel segments, each with a
header and up to 1 KiB of payload. How much of that is used depends on
the content and can exceed the uncompressed layout: the hardware wrote
up to 556128 bytes compressed against 522240 uncompressed for the
motion vectors of a 4K HEVC stream. So reserve what the vendor code
reserves for the compressed layout on VDPU381, which is twice the
current 128 bytes per macroblock: 8.3 MB instead of 4.1 MB per 4K
CAPTURE buffer.

Signed-off-by: Arun Iyengar <aiyengar1990@gmail.com>
---
 .../platform/rockchip/rkvdec/rkvdec-vdpu381-h264.c     | 10 ++++++++++
 drivers/media/platform/rockchip/rkvdec/rkvdec.c        | 18 +++++++++++++++++-
 2 files changed, 27 insertions(+), 1 deletion(-)

diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-h264.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-h264.c
index b961fddc8583..06d83a6ed617 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-h264.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-h264.c
@@ -237,6 +237,16 @@ static void config_registers(struct rkvdec_ctx *ctx,
 	regs->common.reg011_important_en.dec_timeout_e = 1;
 	regs->common.reg011_important_en.pix_range_det_e = 1;
 
+	/*
+	 * Store the co-located motion vectors compressed, as the vendor code
+	 * does for frame-only streams. This cuts the memory traffic of
+	 * B-frames and the hardware time of 4K streams by about 5%. The
+	 * compressed data fits the space reserved for the uncompressed
+	 * layout. Field and MBAFF streams use the uncompressed layout.
+	 */
+	regs->common.reg012_secondary_en.colmv_compress_en =
+		!!(run->sps->flags & V4L2_H264_SPS_FLAG_FRAME_MBS_ONLY);
+
 	/*
 	 * Even though the scan list address can be set in RPS,
 	 * with some frames, it will try to use the address set in the register.
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec.c b/drivers/media/platform/rockchip/rkvdec/rkvdec.c
index fbcd0f175ad3..5f7e876da6eb 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec.c
@@ -20,6 +20,7 @@
 #include <linux/platform_device.h>
 #include <linux/pm.h>
 #include <linux/pm_runtime.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/videodev2.h>
 #include <linux/workqueue.h>
@@ -93,6 +94,21 @@ static u32 rkvdec_colmv_size(u16 width, u16 height)
 	return 128 * DIV_ROUND_UP(width, 16) * DIV_ROUND_UP(height, 16);
 }
 
+/*
+ * The VDPU381 stores co-located motion vectors compressed, in segments of
+ * 64x16 pixels, with a segment header (rows padded to 16 headers)
+ * and up to 1 KiB of payload. That's the bound the vendor code allocates;
+ * how much is used depends on the content, and can be more than the
+ * uncompressed layout of rkvdec_colmv_size() takes.
+ */
+static u32 vdpu381_colmv_size(u16 width, u16 height)
+{
+	u32 seg_w = DIV_ROUND_UP(width, 64);
+	u32 seg_h = DIV_ROUND_UP(height, 16);
+
+	return ALIGN(ALIGN(seg_w, 16) * seg_h + seg_w * seg_h * SZ_1K, 128);
+}
+
 static u32 vdpu383_colmv_size(u16 width, u16 height)
 {
 	return ALIGN(width, 64) * ALIGN(height, 16);
@@ -1700,7 +1716,7 @@ static const struct rcb_size_info vdpu381_rcb_sizes[] = {
 
 static const struct rkvdec_variant_ops vdpu381_variant_ops = {
 	.irq_handler = vdpu381_irq_handler,
-	.colmv_size = rkvdec_colmv_size,
+	.colmv_size = vdpu381_colmv_size,
 	.flatten_matrices = transpose_and_flatten_matrices,
 };
 

-- 
2.55.0

[PATCH 2/2] media: rkvdec: vdpu381: Compress the co-located motion vectors of HEVC

[Arun Iyengar via B4 Relay]

From: Arun Iyengar <aiyengar1990@gmail.com>

Like for H.264, store the co-located motion vectors of HEVC frames
compressed, which the vendor code (MPP, hal_h265d_vdpu34x.c) always
does. The previous commit reserved space for the compressed layout.

On RK3588, a 24 Mbit/s 3840x2160 HEVC stream (240 frames, x265
ultrafast) decodes at 104.6 fps instead of 98.6-102.8 fps, and the
decoded frames are identical.

Signed-off-by: Arun Iyengar <aiyengar1990@gmail.com>
---
 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
index e1936e87f45b..2473c176b319 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
@@ -404,6 +404,9 @@ static void config_registers(struct rkvdec_ctx *ctx,
 	/* Set HEVC mode */
 	regs->common.reg009_dec_mode.dec_mode = VDPU381_MODE_HEVC;
 
+	/* Store the co-located motion vectors compressed, as for H.264 */
+	regs->common.reg012_secondary_en.colmv_compress_en = 1;
+
 	/* Set config */
 	regs->common.reg011_important_en.buf_empty_en = 1;
 	regs->common.reg011_important_en.dec_clkgate_e = 1;

-- 
2.55.0