[RESEND PATCH v7 0/4] Add features to an existing driver

[RESEND PATCH v7 0/4] Add features to an existing driver

[Jackson.lee]

The wave5 codec driver is a stateful encoder/decoder.
The following patches is for supporting yuv422 inpuy format,
supporting runtime suspend/resume feature and extra things.

v4l2-compliance results:
========================

v4l2-compliance 1.24.1, 64 bits, 64-bit time_t

Buffer ioctls:
       warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported
       warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
    test VIDIOC_EXPBUF: OK
    test Requests: OK (Not Supported)

Total for wave5-dec device /dev/video0: 45, Succeeded: 45, Failed: 0, Warnings: 2
Total for wave5-enc device /dev/video1: 45, Succeeded: 45, Failed: 0, Warnings: 0

Fluster test results:
=====================

Running test suite JCT-VC-HEVC_V1 with decoder GStreamer-H.265-V4L2-Gst1.0 Using 1 parallel job(s)
Ran 132/147 tests successfully               in 89.870 secs

(1 test fails because of not supporting to parse multi frames, 1 test fails because of a missing frame and slight corruption,
 2 tests fail because of sizes which are incompatible with the IP, 11 tests fail because of unsupported 10 bit format)

Running test suite JVT-AVC_V1 with decoder GStreamer-H.264-V4L2-Gst1.0 Using 1 parallel job(s)
Ran 77/135 tests successfully               in 30.364 secs

(58 fail because the hardware is unable to decode  MBAFF / FMO / Field / Extended profile streams.)

Change since v6:
================
* For [PATCH v4 2/4] media: chips-media: wave5: Support runtime suspend/resume
 - Change autosuspend delay to 100ms
 - Add to enable/disable hrtimer in the runtime suspend/resume function for hw not using irq 

Change since v5:
================
* For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to calculate bytesperline and sizeimage.
 - Fix v4l2-compliance error for the vidioc_enum_framesizes

* For [PATCH v4 1/4] media: chips-media: wave5: Support SPS/PPS generation for each IDR
 - Remove warning messages for the checkpatch.pl script

Change since v4:
================
* For [PATCH v4 2/4] media: chips-media: wave5: Support runtime suspend/resume
 - Fix warning message

* For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to calculate bytesperline and sizeimage.
 - Fix warning message
 - add Reviewed-By tag

* For [PATCH v4 4/4] media: chips-media: wave5: Support YUV422 raw pixel-formats on the encoder
 - add Reviewed-By tag

Change since v3:
=================

* For [PATCH v4 1/4] media: chips-media: wave5: Support SPS/PPS generation for each IDR
 - add Reviewed-By tag

* For [PATCH v4 2/4] media: chips-media: wave5: Support runtime suspend/resume
 - add Reviewed-By tag

* For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to calculate bytesperline and sizeimage.
 - modify the commit message
 - define three framesize structures for decoder

* For [PATCH v4 4/4] media: chips-media: wave5: Support YUV422 raw pixel-formats on the encoder
 - modify the commit message
 - use the v4l2_format_info to calculate luma, chroma size

Change since v2:
=================

* For [PATCH v3 0/4] media: chips-media: wave5: Support SPS/PPS generation for each IDR
 - add the suggested _SHIFT suffix

* For [PATCH v3 1/4] media: chips-media: wave5: Support runtime suspend/resume
 - change a commit message

* For [PATCH v3 2/4] media: chips-media: wave5: Use helpers to calculate bytesperline and sizeimage
 - add pix_fmt_type parameter into wave5_update_pix_fmt function
 - add min/max width/height values into dec_fmt_list 

Change since v1:
=================

* For [PATCH v2 0/4] media: chips-media: wave5: Support SPS/PPS generation for each IDR
 - define a macro for register addresses

* For [PATCH v2 1/4] media: chips-media: wave5: Support runtime suspend/resume
 - add auto suspend/resume

* For [PATCH v2 2/4] media: chips-media: wave5: Use helpers to calculate bytesperline and sizeimage
 - use helper functions to calculate bytesperline and sizeimage

* For [PATCH v2 3/4] media: chips-media: wave5: Support YUV422 raw pixel-formats on the encoder
 - remove unnecessary codes

Change since v0:
=================
The DEFAULT_SRC_SIZE macro was defined using multiple lines, To make a simple define, tab and multiple lines has been removed, The macro is defined using one line.

Jackson.lee (4):
  media: chips-media: wave5: Support SPS/PPS generation for each IDR
  media: chips-media: wave5: Support runtime suspend/resume
  media: chips-media: wave5: Use helpers to calculate bytesperline and
    sizeimage.
  media: chips-media: wave5: Support YUV422 raw pixel-formats on the
    encoder.

 .../platform/chips-media/wave5/wave5-helper.c |  37 +-
 .../platform/chips-media/wave5/wave5-helper.h |   5 +
 .../platform/chips-media/wave5/wave5-hw.c     |  30 +-
 .../chips-media/wave5/wave5-vpu-dec.c         | 321 +++++++-----------
 .../chips-media/wave5/wave5-vpu-enc.c         | 313 +++++++++--------
 .../platform/chips-media/wave5/wave5-vpu.c    |  50 +++
 .../platform/chips-media/wave5/wave5-vpu.h    |   5 +-
 .../platform/chips-media/wave5/wave5-vpuapi.c |  33 +-
 .../platform/chips-media/wave5/wave5-vpuapi.h |   1 +
 .../chips-media/wave5/wave5-vpuconfig.h       |  27 +-
 .../media/platform/chips-media/wave5/wave5.h  |   3 +
 11 files changed, 460 insertions(+), 365 deletions(-)

-- 
2.43.0

[RESEND PATCH v7 1/4] media: chips-media: wave5: Support SPS/PPS generation for each IDR

[Jackson.lee]

Provide a control to toggle (0 = off / 1 = on), whether the SPS and
PPS are generated for every IDR.

Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
---
 .../platform/chips-media/wave5/wave5-hw.c     | 26 +++++++++++++++----
 .../chips-media/wave5/wave5-vpu-enc.c         |  7 +++++
 .../platform/chips-media/wave5/wave5-vpuapi.h |  1 +
 3 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/drivers/media/platform/chips-media/wave5/wave5-hw.c b/drivers/media/platform/chips-media/wave5/wave5-hw.c
index c89aafabc742..2a98bab446d0 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-hw.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-hw.c
@@ -23,6 +23,15 @@
 #define W521_FEATURE_AVC_ENCODER	BIT(1)
 #define W521_FEATURE_HEVC_ENCODER	BIT(0)
 
+#define ENC_AVC_INTRA_IDR_PARAM_MASK	0x7ff
+#define ENC_AVC_INTRA_PERIOD_SHIFT		6
+#define ENC_AVC_IDR_PERIOD_SHIFT		17
+#define ENC_AVC_FORCED_IDR_HEADER_SHIFT		28
+
+#define ENC_HEVC_INTRA_QP_SHIFT			3
+#define ENC_HEVC_FORCED_IDR_HEADER_SHIFT	9
+#define ENC_HEVC_INTRA_PERIOD_SHIFT		16
+
 /* Decoder support fields */
 #define W521_FEATURE_AVC_DECODER	BIT(3)
 #define W521_FEATURE_HEVC_DECODER	BIT(2)
@@ -35,7 +44,7 @@
 
 #define REMAP_CTRL_MAX_SIZE_BITS	((W5_REMAP_MAX_SIZE >> 12) & 0x1ff)
 #define REMAP_CTRL_REGISTER_VALUE(index)	(			\
-	(BIT(31) | (index << 12) | BIT(11) | REMAP_CTRL_MAX_SIZE_BITS)	\
+	(BIT(31) | ((index) << 12) | BIT(11) | REMAP_CTRL_MAX_SIZE_BITS)\
 )
 
 #define FASTIO_ADDRESS_MASK		GENMASK(15, 0)
@@ -1772,12 +1781,19 @@ int wave5_vpu_enc_init_seq(struct vpu_instance *inst)
 
 	if (inst->std == W_AVC_ENC)
 		vpu_write_reg(inst->dev, W5_CMD_ENC_SEQ_INTRA_PARAM, p_param->intra_qp |
-				((p_param->intra_period & 0x7ff) << 6) |
-				((p_param->avc_idr_period & 0x7ff) << 17));
+			      ((p_param->intra_period & ENC_AVC_INTRA_IDR_PARAM_MASK)
+				<< ENC_AVC_INTRA_PERIOD_SHIFT) |
+			      ((p_param->avc_idr_period & ENC_AVC_INTRA_IDR_PARAM_MASK)
+				<< ENC_AVC_IDR_PERIOD_SHIFT) |
+			      (p_param->forced_idr_header_enable
+			       << ENC_AVC_FORCED_IDR_HEADER_SHIFT));
 	else if (inst->std == W_HEVC_ENC)
 		vpu_write_reg(inst->dev, W5_CMD_ENC_SEQ_INTRA_PARAM,
-			      p_param->decoding_refresh_type | (p_param->intra_qp << 3) |
-				(p_param->intra_period << 16));
+			      p_param->decoding_refresh_type |
+			      (p_param->intra_qp << ENC_HEVC_INTRA_QP_SHIFT) |
+			      (p_param->forced_idr_header_enable
+			       << ENC_HEVC_FORCED_IDR_HEADER_SHIFT) |
+			      (p_param->intra_period << ENC_HEVC_INTRA_PERIOD_SHIFT));
 
 	reg_val = (p_param->rdo_skip << 2) |
 		(p_param->lambda_scaling_enable << 3) |
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
index 3e35a05c2d8d..b731decbfda6 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
@@ -1061,6 +1061,9 @@ static int wave5_vpu_enc_s_ctrl(struct v4l2_ctrl *ctrl)
 	case V4L2_CID_MPEG_VIDEO_H264_ENTROPY_MODE:
 		inst->enc_param.entropy_coding_mode = ctrl->val;
 		break;
+	case V4L2_CID_MPEG_VIDEO_PREPEND_SPSPPS_TO_IDR:
+		inst->enc_param.forced_idr_header_enable = ctrl->val;
+		break;
 	case V4L2_CID_MIN_BUFFERS_FOR_OUTPUT:
 		break;
 	default:
@@ -1219,6 +1222,7 @@ static void wave5_set_enc_openparam(struct enc_open_param *open_param,
 		else
 			open_param->wave_param.intra_refresh_arg = num_ctu_row;
 	}
+	open_param->wave_param.forced_idr_header_enable = input.forced_idr_header_enable;
 }
 
 static int initialize_sequence(struct vpu_instance *inst)
@@ -1701,6 +1705,9 @@ static int wave5_vpu_open_enc(struct file *filp)
 			  0, 1, 1, 0);
 	v4l2_ctrl_new_std(v4l2_ctrl_hdl, &wave5_vpu_enc_ctrl_ops,
 			  V4L2_CID_MIN_BUFFERS_FOR_OUTPUT, 1, 32, 1, 1);
+	v4l2_ctrl_new_std(v4l2_ctrl_hdl, &wave5_vpu_enc_ctrl_ops,
+			  V4L2_CID_MPEG_VIDEO_PREPEND_SPSPPS_TO_IDR,
+			  0, 1, 1, 0);
 
 	if (v4l2_ctrl_hdl->error) {
 		ret = -ENODEV;
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.h b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.h
index d2370511faf8..45615c15beca 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.h
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.h
@@ -568,6 +568,7 @@ struct enc_wave_param {
 	u32 lambda_scaling_enable: 1; /* enable lambda scaling using custom GOP */
 	u32 transform8x8_enable: 1; /* enable 8x8 intra prediction and 8x8 transform */
 	u32 mb_level_rc_enable: 1; /* enable MB-level rate control */
+	u32 forced_idr_header_enable: 1; /* enable header encoding before IDR frame */
 };
 
 struct enc_open_param {
-- 
2.43.0

[RESEND PATCH v7 2/4] media: chips-media: wave5: Support runtime suspend/resume

[Jackson.lee]

Add support for runtime suspend/resume in the encoder and decoder. This is
achieved by saving the VPU state and powering it off while the VPU idle.

Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
---
 .../platform/chips-media/wave5/wave5-helper.c | 13 -----
 .../platform/chips-media/wave5/wave5-hw.c     |  4 +-
 .../chips-media/wave5/wave5-vpu-dec.c         | 21 ++++++--
 .../chips-media/wave5/wave5-vpu-enc.c         | 20 ++++++--
 .../platform/chips-media/wave5/wave5-vpu.c    | 50 +++++++++++++++++++
 .../platform/chips-media/wave5/wave5-vpuapi.c | 33 ++++++++++--
 .../media/platform/chips-media/wave5/wave5.h  |  3 ++
 7 files changed, 118 insertions(+), 26 deletions(-)

diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.c b/drivers/media/platform/chips-media/wave5/wave5-helper.c
index d60841c54a80..a20d84dbe3aa 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
@@ -58,7 +58,6 @@ int wave5_vpu_release_device(struct file *filp,
 			     char *name)
 {
 	struct vpu_instance *inst = wave5_to_vpu_inst(filp->private_data);
-	struct vpu_device *dev = inst->dev;
 	int ret = 0;
 
 	v4l2_m2m_ctx_release(inst->v4l2_fh.m2m_ctx);
@@ -78,18 +77,6 @@ int wave5_vpu_release_device(struct file *filp,
 	}
 
 	wave5_cleanup_instance(inst);
-	if (dev->irq < 0) {
-		ret = mutex_lock_interruptible(&dev->dev_lock);
-		if (ret)
-			return ret;
-
-		if (list_empty(&dev->instances)) {
-			dev_dbg(dev->dev, "Disabling the hrtimer\n");
-			hrtimer_cancel(&dev->hrtimer);
-		}
-
-		mutex_unlock(&dev->dev_lock);
-	}
 
 	return ret;
 }
diff --git a/drivers/media/platform/chips-media/wave5/wave5-hw.c b/drivers/media/platform/chips-media/wave5/wave5-hw.c
index 2a98bab446d0..c8a905994109 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-hw.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-hw.c
@@ -1228,8 +1228,8 @@ int wave5_vpu_re_init(struct device *dev, u8 *fw, size_t size)
 	return setup_wave5_properties(dev);
 }
 
-static int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const uint16_t *code,
-				size_t size)
+int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const uint16_t *code,
+			 size_t size)
 {
 	u32 reg_val;
 	struct vpu_buf *common_vb;
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
index 0c5c9a8de91f..698c83e3082e 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
@@ -5,6 +5,7 @@
  * Copyright (C) 2021-2023 CHIPS&MEDIA INC
  */
 
+#include <linux/pm_runtime.h>
 #include "wave5-helper.h"
 
 #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
@@ -518,6 +519,8 @@ static void wave5_vpu_dec_finish_decode(struct vpu_instance *inst)
 	if (q_status.report_queue_count == 0 &&
 	    (q_status.instance_queue_count == 0 || dec_info.sequence_changed)) {
 		dev_dbg(inst->dev->dev, "%s: finishing job.\n", __func__);
+		pm_runtime_mark_last_busy(inst->dev->dev);
+		pm_runtime_put_autosuspend(inst->dev->dev);
 		v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
 	}
 }
@@ -1398,6 +1401,7 @@ static int wave5_vpu_dec_start_streaming(struct vb2_queue *q, unsigned int count
 	int ret = 0;
 
 	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
+	pm_runtime_resume_and_get(inst->dev->dev);
 
 	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
 
@@ -1429,13 +1433,15 @@ static int wave5_vpu_dec_start_streaming(struct vb2_queue *q, unsigned int count
 		if (ret)
 			goto return_buffers;
 	}
-
+	pm_runtime_mark_last_busy(inst->dev->dev);
+	pm_runtime_put_autosuspend(inst->dev->dev);
 	return ret;
 
 free_bitstream_vbuf:
 	wave5_vdi_free_dma_memory(inst->dev, &inst->bitstream_vbuf);
 return_buffers:
 	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
+	pm_runtime_put_autosuspend(inst->dev->dev);
 	return ret;
 }
 
@@ -1521,6 +1527,7 @@ static void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
 	bool check_cmd = TRUE;
 
 	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
+	pm_runtime_resume_and_get(inst->dev->dev);
 
 	while (check_cmd) {
 		struct queue_status_info q_status;
@@ -1544,6 +1551,9 @@ static void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
 		streamoff_output(q);
 	else
 		streamoff_capture(q);
+
+	pm_runtime_mark_last_busy(inst->dev->dev);
+	pm_runtime_put_autosuspend(inst->dev->dev);
 }
 
 static const struct vb2_ops wave5_vpu_dec_vb2_ops = {
@@ -1630,7 +1640,7 @@ static void wave5_vpu_dec_device_run(void *priv)
 	int ret = 0;
 
 	dev_dbg(inst->dev->dev, "%s: Fill the ring buffer with new bitstream data", __func__);
-
+	pm_runtime_resume_and_get(inst->dev->dev);
 	ret = fill_ringbuffer(inst);
 	if (ret) {
 		dev_warn(inst->dev->dev, "Filling ring buffer failed\n");
@@ -1713,6 +1723,8 @@ static void wave5_vpu_dec_device_run(void *priv)
 
 finish_job_and_return:
 	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
+	pm_runtime_mark_last_busy(inst->dev->dev);
+	pm_runtime_put_autosuspend(inst->dev->dev);
 	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
 }
 
@@ -1879,9 +1891,8 @@ static int wave5_vpu_open_dec(struct file *filp)
 	if (ret)
 		goto cleanup_inst;
 
-	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) && list_empty(&dev->instances))
-		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev->vpu_poll_interval * NSEC_PER_MSEC),
-			      HRTIMER_MODE_REL_PINNED);
+	if (list_empty(&dev->instances))
+		pm_runtime_use_autosuspend(inst->dev->dev);
 
 	list_add_tail(&inst->list, &dev->instances);
 
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
index b731decbfda6..985de0c293e2 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
@@ -5,6 +5,7 @@
  * Copyright (C) 2021-2023 CHIPS&MEDIA INC
  */
 
+#include <linux/pm_runtime.h>
 #include "wave5-helper.h"
 
 #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
@@ -1310,6 +1311,7 @@ static int wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
 	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
 	int ret = 0;
 
+	pm_runtime_resume_and_get(inst->dev->dev);
 	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
 
 	if (inst->state == VPU_INST_STATE_NONE && q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
@@ -1364,9 +1366,13 @@ static int wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
 	if (ret)
 		goto return_buffers;
 
+	pm_runtime_mark_last_busy(inst->dev->dev);
+	pm_runtime_put_autosuspend(inst->dev->dev);
 	return 0;
 return_buffers:
 	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
+	pm_runtime_mark_last_busy(inst->dev->dev);
+	pm_runtime_put_autosuspend(inst->dev->dev);
 	return ret;
 }
 
@@ -1408,6 +1414,7 @@ static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
 	 */
 
 	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
+	pm_runtime_resume_and_get(inst->dev->dev);
 
 	if (wave5_vpu_both_queues_are_streaming(inst))
 		switch_state(inst, VPU_INST_STATE_STOP);
@@ -1432,6 +1439,9 @@ static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
 		streamoff_output(inst, q);
 	else
 		streamoff_capture(inst, q);
+
+	pm_runtime_mark_last_busy(inst->dev->dev);
+	pm_runtime_put_autosuspend(inst->dev->dev);
 }
 
 static const struct vb2_ops wave5_vpu_enc_vb2_ops = {
@@ -1478,6 +1488,7 @@ static void wave5_vpu_enc_device_run(void *priv)
 	u32 fail_res = 0;
 	int ret = 0;
 
+	pm_runtime_resume_and_get(inst->dev->dev);
 	switch (inst->state) {
 	case VPU_INST_STATE_PIC_RUN:
 		ret = start_encode(inst, &fail_res);
@@ -1491,6 +1502,8 @@ static void wave5_vpu_enc_device_run(void *priv)
 			break;
 		}
 		dev_dbg(inst->dev->dev, "%s: leave with active job", __func__);
+		pm_runtime_mark_last_busy(inst->dev->dev);
+		pm_runtime_put_autosuspend(inst->dev->dev);
 		return;
 	default:
 		WARN(1, "Execution of a job in state %s is invalid.\n",
@@ -1498,6 +1511,8 @@ static void wave5_vpu_enc_device_run(void *priv)
 		break;
 	}
 	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
+	pm_runtime_mark_last_busy(inst->dev->dev);
+	pm_runtime_put_autosuspend(inst->dev->dev);
 	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
 }
 
@@ -1739,9 +1754,8 @@ static int wave5_vpu_open_enc(struct file *filp)
 	if (ret)
 		goto cleanup_inst;
 
-	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) && list_empty(&dev->instances))
-		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev->vpu_poll_interval * NSEC_PER_MSEC),
-			      HRTIMER_MODE_REL_PINNED);
+	if (list_empty(&dev->instances))
+		pm_runtime_use_autosuspend(inst->dev->dev);
 
 	list_add_tail(&inst->list, &dev->instances);
 
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu.c b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
index 7273254ecb03..41c4bf64f27d 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
@@ -10,6 +10,7 @@
 #include <linux/clk.h>
 #include <linux/firmware.h>
 #include <linux/interrupt.h>
+#include <linux/pm_runtime.h>
 #include <linux/reset.h>
 #include "wave5-vpu.h"
 #include "wave5-regdefine.h"
@@ -153,6 +154,45 @@ static int wave5_vpu_load_firmware(struct device *dev, const char *fw_name,
 	return 0;
 }
 
+static __maybe_unused int wave5_pm_suspend(struct device *dev)
+{
+	struct vpu_device *vpu = dev_get_drvdata(dev);
+
+	if (pm_runtime_suspended(dev))
+		return 0;
+
+	if (vpu->irq < 0)
+		hrtimer_cancel(&vpu->hrtimer);
+
+	wave5_vpu_sleep_wake(dev, true, NULL, 0);
+	clk_bulk_disable_unprepare(vpu->num_clks, vpu->clks);
+
+	return 0;
+}
+
+static __maybe_unused int wave5_pm_resume(struct device *dev)
+{
+	struct vpu_device *vpu = dev_get_drvdata(dev);
+	int ret = 0;
+
+	wave5_vpu_sleep_wake(dev, false, NULL, 0);
+	ret = clk_bulk_prepare_enable(vpu->num_clks, vpu->clks);
+	if (ret) {
+		dev_err(dev, "Enabling clocks, fail: %d\n", ret);
+		return ret;
+	}
+
+	if (vpu->irq < 0 && !hrtimer_active(&vpu->hrtimer))
+		hrtimer_start(&vpu->hrtimer, ns_to_ktime(vpu->vpu_poll_interval * NSEC_PER_MSEC),
+				HRTIMER_MODE_REL_PINNED);
+
+	return ret;
+}
+
+static const struct dev_pm_ops wave5_pm_ops = {
+	SET_RUNTIME_PM_OPS(wave5_pm_suspend, wave5_pm_resume, NULL)
+};
+
 static int wave5_vpu_probe(struct platform_device *pdev)
 {
 	int ret;
@@ -281,6 +321,12 @@ static int wave5_vpu_probe(struct platform_device *pdev)
 		 (match_data->flags & WAVE5_IS_DEC) ? "'DECODE'" : "");
 	dev_info(&pdev->dev, "Product Code:      0x%x\n", dev->product_code);
 	dev_info(&pdev->dev, "Firmware Revision: %u\n", fw_revision);
+
+	pm_runtime_set_autosuspend_delay(&pdev->dev, 100);
+	pm_runtime_use_autosuspend(&pdev->dev);
+	pm_runtime_enable(&pdev->dev);
+	wave5_vpu_sleep_wake(&pdev->dev, true, NULL, 0);
+
 	return 0;
 
 err_enc_unreg:
@@ -310,6 +356,9 @@ static void wave5_vpu_remove(struct platform_device *pdev)
 		hrtimer_cancel(&dev->hrtimer);
 	}
 
+	pm_runtime_put_sync(&pdev->dev);
+	pm_runtime_disable(&pdev->dev);
+
 	mutex_destroy(&dev->dev_lock);
 	mutex_destroy(&dev->hw_lock);
 	reset_control_assert(dev->resets);
@@ -337,6 +386,7 @@ static struct platform_driver wave5_vpu_driver = {
 	.driver = {
 		.name = VPU_PLATFORM_DEVICE_NAME,
 		.of_match_table = of_match_ptr(wave5_dt_ids),
+		.pm = &wave5_pm_ops,
 		},
 	.probe = wave5_vpu_probe,
 	.remove_new = wave5_vpu_remove,
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
index 1a3efb638dde..e16b990041c2 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
@@ -6,6 +6,8 @@
  */
 
 #include <linux/bug.h>
+#include <linux/pm_runtime.h>
+#include <linux/delay.h>
 #include "wave5-vpuapi.h"
 #include "wave5-regdefine.h"
 #include "wave5.h"
@@ -195,14 +197,20 @@ int wave5_vpu_dec_close(struct vpu_instance *inst, u32 *fail_res)
 	int retry = 0;
 	struct vpu_device *vpu_dev = inst->dev;
 	int i;
+	int inst_count = 0;
+	struct vpu_instance *inst_elm;
 
 	*fail_res = 0;
 	if (!inst->codec_info)
 		return -EINVAL;
 
+	pm_runtime_resume_and_get(inst->dev->dev);
+
 	ret = mutex_lock_interruptible(&vpu_dev->hw_lock);
-	if (ret)
+	if (ret) {
+		pm_runtime_put_sync(inst->dev->dev);
 		return ret;
+	}
 
 	do {
 		ret = wave5_vpu_dec_finish_seq(inst, fail_res);
@@ -232,9 +240,14 @@ int wave5_vpu_dec_close(struct vpu_instance *inst, u32 *fail_res)
 
 	wave5_vdi_free_dma_memory(vpu_dev, &p_dec_info->vb_task);
 
+	list_for_each_entry(inst_elm, &vpu_dev->instances, list)
+		inst_count++;
+	if (inst_count == 1)
+		pm_runtime_dont_use_autosuspend(vpu_dev->dev);
+
 unlock_and_return:
 	mutex_unlock(&vpu_dev->hw_lock);
-
+	pm_runtime_put_sync(inst->dev->dev);
 	return ret;
 }
 
@@ -697,25 +710,33 @@ int wave5_vpu_enc_close(struct vpu_instance *inst, u32 *fail_res)
 	int ret;
 	int retry = 0;
 	struct vpu_device *vpu_dev = inst->dev;
+	int inst_count = 0;
+	struct vpu_instance *inst_elm;
 
 	*fail_res = 0;
 	if (!inst->codec_info)
 		return -EINVAL;
 
+	pm_runtime_resume_and_get(inst->dev->dev);
+
 	ret = mutex_lock_interruptible(&vpu_dev->hw_lock);
-	if (ret)
+	if (ret) {
+		pm_runtime_resume_and_get(inst->dev->dev);
 		return ret;
+	}
 
 	do {
 		ret = wave5_vpu_enc_finish_seq(inst, fail_res);
 		if (ret < 0 && *fail_res != WAVE5_SYSERR_VPU_STILL_RUNNING) {
 			dev_warn(inst->dev->dev, "enc_finish_seq timed out\n");
+			pm_runtime_resume_and_get(inst->dev->dev);
 			mutex_unlock(&vpu_dev->hw_lock);
 			return ret;
 		}
 
 		if (*fail_res == WAVE5_SYSERR_VPU_STILL_RUNNING &&
 		    retry++ >= MAX_FIRMWARE_CALL_RETRY) {
+			pm_runtime_resume_and_get(inst->dev->dev);
 			mutex_unlock(&vpu_dev->hw_lock);
 			return -ETIMEDOUT;
 		}
@@ -734,7 +755,13 @@ int wave5_vpu_enc_close(struct vpu_instance *inst, u32 *fail_res)
 
 	wave5_vdi_free_dma_memory(vpu_dev, &p_enc_info->vb_task);
 
+	list_for_each_entry(inst_elm, &vpu_dev->instances, list)
+		inst_count++;
+	if (inst_count == 1)
+		pm_runtime_dont_use_autosuspend(vpu_dev->dev);
+
 	mutex_unlock(&vpu_dev->hw_lock);
+	pm_runtime_put_sync(inst->dev->dev);
 
 	return 0;
 }
diff --git a/drivers/media/platform/chips-media/wave5/wave5.h b/drivers/media/platform/chips-media/wave5/wave5.h
index 2a29b9164f97..2caab356f3e1 100644
--- a/drivers/media/platform/chips-media/wave5/wave5.h
+++ b/drivers/media/platform/chips-media/wave5/wave5.h
@@ -62,6 +62,9 @@ int wave5_vpu_get_version(struct vpu_device *vpu_dev, u32 *revision);
 
 int wave5_vpu_init(struct device *dev, u8 *fw, size_t size);
 
+int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const uint16_t *code,
+			 size_t size);
+
 int wave5_vpu_reset(struct device *dev, enum sw_reset_mode reset_mode);
 
 int wave5_vpu_build_up_dec_param(struct vpu_instance *inst, struct dec_open_param *param);
-- 
2.43.0

Re: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support runtime suspend/resume

[Nicolas Dufresne]

Hi,

Le lundi 12 août 2024 à 16:08 +0900, Jackson.lee a écrit :
> Add support for runtime suspend/resume in the encoder and decoder. This is
> achieved by saving the VPU state and powering it off while the VPU idle.

If you don't, I'll edit to "while the VPU *is* idle".

regards,
Nicolas

> 
> Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
> Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
> Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
> ---
>  .../platform/chips-media/wave5/wave5-helper.c | 13 -----
>  .../platform/chips-media/wave5/wave5-hw.c     |  4 +-
>  .../chips-media/wave5/wave5-vpu-dec.c         | 21 ++++++--
>  .../chips-media/wave5/wave5-vpu-enc.c         | 20 ++++++--
>  .../platform/chips-media/wave5/wave5-vpu.c    | 50 +++++++++++++++++++
>  .../platform/chips-media/wave5/wave5-vpuapi.c | 33 ++++++++++--
>  .../media/platform/chips-media/wave5/wave5.h  |  3 ++
>  7 files changed, 118 insertions(+), 26 deletions(-)
> 
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.c b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> index d60841c54a80..a20d84dbe3aa 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> @@ -58,7 +58,6 @@ int wave5_vpu_release_device(struct file *filp,
>  			     char *name)
>  {
>  	struct vpu_instance *inst = wave5_to_vpu_inst(filp->private_data);
> -	struct vpu_device *dev = inst->dev;
>  	int ret = 0;
>  
>  	v4l2_m2m_ctx_release(inst->v4l2_fh.m2m_ctx);
> @@ -78,18 +77,6 @@ int wave5_vpu_release_device(struct file *filp,
>  	}
>  
>  	wave5_cleanup_instance(inst);
> -	if (dev->irq < 0) {
> -		ret = mutex_lock_interruptible(&dev->dev_lock);
> -		if (ret)
> -			return ret;
> -
> -		if (list_empty(&dev->instances)) {
> -			dev_dbg(dev->dev, "Disabling the hrtimer\n");
> -			hrtimer_cancel(&dev->hrtimer);
> -		}
> -
> -		mutex_unlock(&dev->dev_lock);
> -	}
>  
>  	return ret;
>  }
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-hw.c b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> index 2a98bab446d0..c8a905994109 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-hw.c
> +++ b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> @@ -1228,8 +1228,8 @@ int wave5_vpu_re_init(struct device *dev, u8 *fw, size_t size)
>  	return setup_wave5_properties(dev);
>  }
>  
> -static int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const uint16_t *code,
> -				size_t size)
> +int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const uint16_t *code,
> +			 size_t size)
>  {
>  	u32 reg_val;
>  	struct vpu_buf *common_vb;
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> index 0c5c9a8de91f..698c83e3082e 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> @@ -5,6 +5,7 @@
>   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
>   */
>  
> +#include <linux/pm_runtime.h>
>  #include "wave5-helper.h"
>  
>  #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
> @@ -518,6 +519,8 @@ static void wave5_vpu_dec_finish_decode(struct vpu_instance *inst)
>  	if (q_status.report_queue_count == 0 &&
>  	    (q_status.instance_queue_count == 0 || dec_info.sequence_changed)) {
>  		dev_dbg(inst->dev->dev, "%s: finishing job.\n", __func__);
> +		pm_runtime_mark_last_busy(inst->dev->dev);
> +		pm_runtime_put_autosuspend(inst->dev->dev);
>  		v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
>  	}
>  }
> @@ -1398,6 +1401,7 @@ static int wave5_vpu_dec_start_streaming(struct vb2_queue *q, unsigned int count
>  	int ret = 0;
>  
>  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> +	pm_runtime_resume_and_get(inst->dev->dev);
>  
>  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
>  
> @@ -1429,13 +1433,15 @@ static int wave5_vpu_dec_start_streaming(struct vb2_queue *q, unsigned int count
>  		if (ret)
>  			goto return_buffers;
>  	}
> -
> +	pm_runtime_mark_last_busy(inst->dev->dev);
> +	pm_runtime_put_autosuspend(inst->dev->dev);
>  	return ret;
>  
>  free_bitstream_vbuf:
>  	wave5_vdi_free_dma_memory(inst->dev, &inst->bitstream_vbuf);
>  return_buffers:
>  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> +	pm_runtime_put_autosuspend(inst->dev->dev);
>  	return ret;
>  }
>  
> @@ -1521,6 +1527,7 @@ static void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
>  	bool check_cmd = TRUE;
>  
>  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> +	pm_runtime_resume_and_get(inst->dev->dev);
>  
>  	while (check_cmd) {
>  		struct queue_status_info q_status;
> @@ -1544,6 +1551,9 @@ static void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
>  		streamoff_output(q);
>  	else
>  		streamoff_capture(q);
> +
> +	pm_runtime_mark_last_busy(inst->dev->dev);
> +	pm_runtime_put_autosuspend(inst->dev->dev);
>  }
>  
>  static const struct vb2_ops wave5_vpu_dec_vb2_ops = {
> @@ -1630,7 +1640,7 @@ static void wave5_vpu_dec_device_run(void *priv)
>  	int ret = 0;
>  
>  	dev_dbg(inst->dev->dev, "%s: Fill the ring buffer with new bitstream data", __func__);
> -
> +	pm_runtime_resume_and_get(inst->dev->dev);
>  	ret = fill_ringbuffer(inst);
>  	if (ret) {
>  		dev_warn(inst->dev->dev, "Filling ring buffer failed\n");
> @@ -1713,6 +1723,8 @@ static void wave5_vpu_dec_device_run(void *priv)
>  
>  finish_job_and_return:
>  	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
> +	pm_runtime_mark_last_busy(inst->dev->dev);
> +	pm_runtime_put_autosuspend(inst->dev->dev);
>  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
>  }
>  
> @@ -1879,9 +1891,8 @@ static int wave5_vpu_open_dec(struct file *filp)
>  	if (ret)
>  		goto cleanup_inst;
>  
> -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) && list_empty(&dev->instances))
> -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev->vpu_poll_interval * NSEC_PER_MSEC),
> -			      HRTIMER_MODE_REL_PINNED);
> +	if (list_empty(&dev->instances))
> +		pm_runtime_use_autosuspend(inst->dev->dev);
>  
>  	list_add_tail(&inst->list, &dev->instances);
>  
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> index b731decbfda6..985de0c293e2 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> @@ -5,6 +5,7 @@
>   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
>   */
>  
> +#include <linux/pm_runtime.h>
>  #include "wave5-helper.h"
>  
>  #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
> @@ -1310,6 +1311,7 @@ static int wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
>  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
>  	int ret = 0;
>  
> +	pm_runtime_resume_and_get(inst->dev->dev);
>  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
>  
>  	if (inst->state == VPU_INST_STATE_NONE && q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
> @@ -1364,9 +1366,13 @@ static int wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
>  	if (ret)
>  		goto return_buffers;
>  
> +	pm_runtime_mark_last_busy(inst->dev->dev);
> +	pm_runtime_put_autosuspend(inst->dev->dev);
>  	return 0;
>  return_buffers:
>  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> +	pm_runtime_mark_last_busy(inst->dev->dev);
> +	pm_runtime_put_autosuspend(inst->dev->dev);
>  	return ret;
>  }
>  
> @@ -1408,6 +1414,7 @@ static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
>  	 */
>  
>  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> +	pm_runtime_resume_and_get(inst->dev->dev);
>  
>  	if (wave5_vpu_both_queues_are_streaming(inst))
>  		switch_state(inst, VPU_INST_STATE_STOP);
> @@ -1432,6 +1439,9 @@ static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
>  		streamoff_output(inst, q);
>  	else
>  		streamoff_capture(inst, q);
> +
> +	pm_runtime_mark_last_busy(inst->dev->dev);
> +	pm_runtime_put_autosuspend(inst->dev->dev);
>  }
>  
>  static const struct vb2_ops wave5_vpu_enc_vb2_ops = {
> @@ -1478,6 +1488,7 @@ static void wave5_vpu_enc_device_run(void *priv)
>  	u32 fail_res = 0;
>  	int ret = 0;
>  
> +	pm_runtime_resume_and_get(inst->dev->dev);
>  	switch (inst->state) {
>  	case VPU_INST_STATE_PIC_RUN:
>  		ret = start_encode(inst, &fail_res);
> @@ -1491,6 +1502,8 @@ static void wave5_vpu_enc_device_run(void *priv)
>  			break;
>  		}
>  		dev_dbg(inst->dev->dev, "%s: leave with active job", __func__);
> +		pm_runtime_mark_last_busy(inst->dev->dev);
> +		pm_runtime_put_autosuspend(inst->dev->dev);
>  		return;
>  	default:
>  		WARN(1, "Execution of a job in state %s is invalid.\n",
> @@ -1498,6 +1511,8 @@ static void wave5_vpu_enc_device_run(void *priv)
>  		break;
>  	}
>  	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
> +	pm_runtime_mark_last_busy(inst->dev->dev);
> +	pm_runtime_put_autosuspend(inst->dev->dev);
>  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
>  }
>  
> @@ -1739,9 +1754,8 @@ static int wave5_vpu_open_enc(struct file *filp)
>  	if (ret)
>  		goto cleanup_inst;
>  
> -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) && list_empty(&dev->instances))
> -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev->vpu_poll_interval * NSEC_PER_MSEC),
> -			      HRTIMER_MODE_REL_PINNED);
> +	if (list_empty(&dev->instances))
> +		pm_runtime_use_autosuspend(inst->dev->dev);
>  
>  	list_add_tail(&inst->list, &dev->instances);
>  
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu.c b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> index 7273254ecb03..41c4bf64f27d 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> @@ -10,6 +10,7 @@
>  #include <linux/clk.h>
>  #include <linux/firmware.h>
>  #include <linux/interrupt.h>
> +#include <linux/pm_runtime.h>
>  #include <linux/reset.h>
>  #include "wave5-vpu.h"
>  #include "wave5-regdefine.h"
> @@ -153,6 +154,45 @@ static int wave5_vpu_load_firmware(struct device *dev, const char *fw_name,
>  	return 0;
>  }
>  
> +static __maybe_unused int wave5_pm_suspend(struct device *dev)
> +{
> +	struct vpu_device *vpu = dev_get_drvdata(dev);
> +
> +	if (pm_runtime_suspended(dev))
> +		return 0;
> +
> +	if (vpu->irq < 0)
> +		hrtimer_cancel(&vpu->hrtimer);
> +
> +	wave5_vpu_sleep_wake(dev, true, NULL, 0);
> +	clk_bulk_disable_unprepare(vpu->num_clks, vpu->clks);
> +
> +	return 0;
> +}
> +
> +static __maybe_unused int wave5_pm_resume(struct device *dev)
> +{
> +	struct vpu_device *vpu = dev_get_drvdata(dev);
> +	int ret = 0;
> +
> +	wave5_vpu_sleep_wake(dev, false, NULL, 0);
> +	ret = clk_bulk_prepare_enable(vpu->num_clks, vpu->clks);
> +	if (ret) {
> +		dev_err(dev, "Enabling clocks, fail: %d\n", ret);
> +		return ret;
> +	}
> +
> +	if (vpu->irq < 0 && !hrtimer_active(&vpu->hrtimer))
> +		hrtimer_start(&vpu->hrtimer, ns_to_ktime(vpu->vpu_poll_interval * NSEC_PER_MSEC),
> +				HRTIMER_MODE_REL_PINNED);
> +
> +	return ret;
> +}
> +
> +static const struct dev_pm_ops wave5_pm_ops = {
> +	SET_RUNTIME_PM_OPS(wave5_pm_suspend, wave5_pm_resume, NULL)
> +};
> +
>  static int wave5_vpu_probe(struct platform_device *pdev)
>  {
>  	int ret;
> @@ -281,6 +321,12 @@ static int wave5_vpu_probe(struct platform_device *pdev)
>  		 (match_data->flags & WAVE5_IS_DEC) ? "'DECODE'" : "");
>  	dev_info(&pdev->dev, "Product Code:      0x%x\n", dev->product_code);
>  	dev_info(&pdev->dev, "Firmware Revision: %u\n", fw_revision);
> +
> +	pm_runtime_set_autosuspend_delay(&pdev->dev, 100);
> +	pm_runtime_use_autosuspend(&pdev->dev);
> +	pm_runtime_enable(&pdev->dev);
> +	wave5_vpu_sleep_wake(&pdev->dev, true, NULL, 0);
> +
>  	return 0;
>  
>  err_enc_unreg:
> @@ -310,6 +356,9 @@ static void wave5_vpu_remove(struct platform_device *pdev)
>  		hrtimer_cancel(&dev->hrtimer);
>  	}
>  
> +	pm_runtime_put_sync(&pdev->dev);
> +	pm_runtime_disable(&pdev->dev);
> +
>  	mutex_destroy(&dev->dev_lock);
>  	mutex_destroy(&dev->hw_lock);
>  	reset_control_assert(dev->resets);
> @@ -337,6 +386,7 @@ static struct platform_driver wave5_vpu_driver = {
>  	.driver = {
>  		.name = VPU_PLATFORM_DEVICE_NAME,
>  		.of_match_table = of_match_ptr(wave5_dt_ids),
> +		.pm = &wave5_pm_ops,
>  		},
>  	.probe = wave5_vpu_probe,
>  	.remove_new = wave5_vpu_remove,
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> index 1a3efb638dde..e16b990041c2 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> +++ b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> @@ -6,6 +6,8 @@
>   */
>  
>  #include <linux/bug.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/delay.h>
>  #include "wave5-vpuapi.h"
>  #include "wave5-regdefine.h"
>  #include "wave5.h"
> @@ -195,14 +197,20 @@ int wave5_vpu_dec_close(struct vpu_instance *inst, u32 *fail_res)
>  	int retry = 0;
>  	struct vpu_device *vpu_dev = inst->dev;
>  	int i;
> +	int inst_count = 0;
> +	struct vpu_instance *inst_elm;
>  
>  	*fail_res = 0;
>  	if (!inst->codec_info)
>  		return -EINVAL;
>  
> +	pm_runtime_resume_and_get(inst->dev->dev);
> +
>  	ret = mutex_lock_interruptible(&vpu_dev->hw_lock);
> -	if (ret)
> +	if (ret) {
> +		pm_runtime_put_sync(inst->dev->dev);
>  		return ret;
> +	}
>  
>  	do {
>  		ret = wave5_vpu_dec_finish_seq(inst, fail_res);
> @@ -232,9 +240,14 @@ int wave5_vpu_dec_close(struct vpu_instance *inst, u32 *fail_res)
>  
>  	wave5_vdi_free_dma_memory(vpu_dev, &p_dec_info->vb_task);
>  
> +	list_for_each_entry(inst_elm, &vpu_dev->instances, list)
> +		inst_count++;
> +	if (inst_count == 1)
> +		pm_runtime_dont_use_autosuspend(vpu_dev->dev);
> +
>  unlock_and_return:
>  	mutex_unlock(&vpu_dev->hw_lock);
> -
> +	pm_runtime_put_sync(inst->dev->dev);
>  	return ret;
>  }
>  
> @@ -697,25 +710,33 @@ int wave5_vpu_enc_close(struct vpu_instance *inst, u32 *fail_res)
>  	int ret;
>  	int retry = 0;
>  	struct vpu_device *vpu_dev = inst->dev;
> +	int inst_count = 0;
> +	struct vpu_instance *inst_elm;
>  
>  	*fail_res = 0;
>  	if (!inst->codec_info)
>  		return -EINVAL;
>  
> +	pm_runtime_resume_and_get(inst->dev->dev);
> +
>  	ret = mutex_lock_interruptible(&vpu_dev->hw_lock);
> -	if (ret)
> +	if (ret) {
> +		pm_runtime_resume_and_get(inst->dev->dev);
>  		return ret;
> +	}
>  
>  	do {
>  		ret = wave5_vpu_enc_finish_seq(inst, fail_res);
>  		if (ret < 0 && *fail_res != WAVE5_SYSERR_VPU_STILL_RUNNING) {
>  			dev_warn(inst->dev->dev, "enc_finish_seq timed out\n");
> +			pm_runtime_resume_and_get(inst->dev->dev);
>  			mutex_unlock(&vpu_dev->hw_lock);
>  			return ret;
>  		}
>  
>  		if (*fail_res == WAVE5_SYSERR_VPU_STILL_RUNNING &&
>  		    retry++ >= MAX_FIRMWARE_CALL_RETRY) {
> +			pm_runtime_resume_and_get(inst->dev->dev);
>  			mutex_unlock(&vpu_dev->hw_lock);
>  			return -ETIMEDOUT;
>  		}
> @@ -734,7 +755,13 @@ int wave5_vpu_enc_close(struct vpu_instance *inst, u32 *fail_res)
>  
>  	wave5_vdi_free_dma_memory(vpu_dev, &p_enc_info->vb_task);
>  
> +	list_for_each_entry(inst_elm, &vpu_dev->instances, list)
> +		inst_count++;
> +	if (inst_count == 1)
> +		pm_runtime_dont_use_autosuspend(vpu_dev->dev);
> +
>  	mutex_unlock(&vpu_dev->hw_lock);
> +	pm_runtime_put_sync(inst->dev->dev);
>  
>  	return 0;
>  }
> diff --git a/drivers/media/platform/chips-media/wave5/wave5.h b/drivers/media/platform/chips-media/wave5/wave5.h
> index 2a29b9164f97..2caab356f3e1 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5.h
> +++ b/drivers/media/platform/chips-media/wave5/wave5.h
> @@ -62,6 +62,9 @@ int wave5_vpu_get_version(struct vpu_device *vpu_dev, u32 *revision);
>  
>  int wave5_vpu_init(struct device *dev, u8 *fw, size_t size);
>  
> +int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const uint16_t *code,
> +			 size_t size);
> +
>  int wave5_vpu_reset(struct device *dev, enum sw_reset_mode reset_mode);
>  
>  int wave5_vpu_build_up_dec_param(struct vpu_instance *inst, struct dec_open_param *param);

Re: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support runtime suspend/resume

[Nicolas Dufresne]

Hi Again,

Le vendredi 06 septembre 2024 à 13:08 -0400, Nicolas Dufresne a écrit :
> Hi,
> 
> Le lundi 12 août 2024 à 16:08 +0900, Jackson.lee a écrit :
> > Add support for runtime suspend/resume in the encoder and decoder. This is
> > achieved by saving the VPU state and powering it off while the VPU idle.
> 
> If you don't, I'll edit to "while the VPU *is* idle".
> 
> regards,
> Nicolas
> 
> > 
> > Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
> > Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
> > Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
> > ---
> >  .../platform/chips-media/wave5/wave5-helper.c | 13 -----
> >  .../platform/chips-media/wave5/wave5-hw.c     |  4 +-
> >  .../chips-media/wave5/wave5-vpu-dec.c         | 21 ++++++--
> >  .../chips-media/wave5/wave5-vpu-enc.c         | 20 ++++++--
> >  .../platform/chips-media/wave5/wave5-vpu.c    | 50 +++++++++++++++++++
> >  .../platform/chips-media/wave5/wave5-vpuapi.c | 33 ++++++++++--
> >  .../media/platform/chips-media/wave5/wave5.h  |  3 ++
> >  7 files changed, 118 insertions(+), 26 deletions(-)
> > 
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.c b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > index d60841c54a80..a20d84dbe3aa 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > @@ -58,7 +58,6 @@ int wave5_vpu_release_device(struct file *filp,
> >  			     char *name)
> >  {
> >  	struct vpu_instance *inst = wave5_to_vpu_inst(filp->private_data);
> > -	struct vpu_device *dev = inst->dev;
> >  	int ret = 0;
> >  
> >  	v4l2_m2m_ctx_release(inst->v4l2_fh.m2m_ctx);
> > @@ -78,18 +77,6 @@ int wave5_vpu_release_device(struct file *filp,
> >  	}
> >  
> >  	wave5_cleanup_instance(inst);
> > -	if (dev->irq < 0) {
> > -		ret = mutex_lock_interruptible(&dev->dev_lock);
> > -		if (ret)
> > -			return ret;
> > -
> > -		if (list_empty(&dev->instances)) {
> > -			dev_dbg(dev->dev, "Disabling the hrtimer\n");
> > -			hrtimer_cancel(&dev->hrtimer);
> > -		}
> > -
> > -		mutex_unlock(&dev->dev_lock);
> > -	}
> >  
> >  	return ret;
> >  }
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-hw.c b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > index 2a98bab446d0..c8a905994109 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > @@ -1228,8 +1228,8 @@ int wave5_vpu_re_init(struct device *dev, u8 *fw, size_t size)
> >  	return setup_wave5_properties(dev);
> >  }
> >  
> > -static int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const uint16_t *code,
> > -				size_t size)
> > +int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const uint16_t *code,
> > +			 size_t size)
> >  {
> >  	u32 reg_val;
> >  	struct vpu_buf *common_vb;
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > index 0c5c9a8de91f..698c83e3082e 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > @@ -5,6 +5,7 @@
> >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
> >   */
> >  
> > +#include <linux/pm_runtime.h>
> >  #include "wave5-helper.h"
> >  
> >  #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
> > @@ -518,6 +519,8 @@ static void wave5_vpu_dec_finish_decode(struct vpu_instance *inst)
> >  	if (q_status.report_queue_count == 0 &&
> >  	    (q_status.instance_queue_count == 0 || dec_info.sequence_changed)) {
> >  		dev_dbg(inst->dev->dev, "%s: finishing job.\n", __func__);
> > +		pm_runtime_mark_last_busy(inst->dev->dev);
> > +		pm_runtime_put_autosuspend(inst->dev->dev);
> >  		v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
> >  	}
> >  }
> > @@ -1398,6 +1401,7 @@ static int wave5_vpu_dec_start_streaming(struct vb2_queue *q, unsigned int count
> >  	int ret = 0;
> >  
> >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >  
> >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
> >  
> > @@ -1429,13 +1433,15 @@ static int wave5_vpu_dec_start_streaming(struct vb2_queue *q, unsigned int count
> >  		if (ret)
> >  			goto return_buffers;
> >  	}
> > -
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	return ret;
> >  
> >  free_bitstream_vbuf:
> >  	wave5_vdi_free_dma_memory(inst->dev, &inst->bitstream_vbuf);
> >  return_buffers:
> >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	return ret;
> >  }
> >  
> > @@ -1521,6 +1527,7 @@ static void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
> >  	bool check_cmd = TRUE;
> >  
> >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >  
> >  	while (check_cmd) {
> >  		struct queue_status_info q_status;
> > @@ -1544,6 +1551,9 @@ static void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
> >  		streamoff_output(q);
> >  	else
> >  		streamoff_capture(q);
> > +
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  }
> >  
> >  static const struct vb2_ops wave5_vpu_dec_vb2_ops = {
> > @@ -1630,7 +1640,7 @@ static void wave5_vpu_dec_device_run(void *priv)
> >  	int ret = 0;
> >  
> >  	dev_dbg(inst->dev->dev, "%s: Fill the ring buffer with new bitstream data", __func__);
> > -
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >  	ret = fill_ringbuffer(inst);
> >  	if (ret) {
> >  		dev_warn(inst->dev->dev, "Filling ring buffer failed\n");
> > @@ -1713,6 +1723,8 @@ static void wave5_vpu_dec_device_run(void *priv)
> >  
> >  finish_job_and_return:
> >  	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
> >  }
> >  
> > @@ -1879,9 +1891,8 @@ static int wave5_vpu_open_dec(struct file *filp)
> >  	if (ret)
> >  		goto cleanup_inst;
> >  
> > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) && list_empty(&dev->instances))
> > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev->vpu_poll_interval * NSEC_PER_MSEC),
> > -			      HRTIMER_MODE_REL_PINNED);
> > +	if (list_empty(&dev->instances))
> > +		pm_runtime_use_autosuspend(inst->dev->dev);
> >  
> >  	list_add_tail(&inst->list, &dev->instances);
> >  
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > index b731decbfda6..985de0c293e2 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > @@ -5,6 +5,7 @@
> >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
> >   */
> >  
> > +#include <linux/pm_runtime.h>
> >  #include "wave5-helper.h"
> >  
> >  #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
> > @@ -1310,6 +1311,7 @@ static int wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
> >  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
> >  	int ret = 0;
> >  
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
> >  
> >  	if (inst->state == VPU_INST_STATE_NONE && q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
> > @@ -1364,9 +1366,13 @@ static int wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
> >  	if (ret)
> >  		goto return_buffers;
> >  
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	return 0;
> >  return_buffers:
> >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	return ret;
> >  }
> >  
> > @@ -1408,6 +1414,7 @@ static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
> >  	 */
> >  
> >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >  
> >  	if (wave5_vpu_both_queues_are_streaming(inst))
> >  		switch_state(inst, VPU_INST_STATE_STOP);
> > @@ -1432,6 +1439,9 @@ static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
> >  		streamoff_output(inst, q);
> >  	else
> >  		streamoff_capture(inst, q);
> > +
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  }
> >  
> >  static const struct vb2_ops wave5_vpu_enc_vb2_ops = {
> > @@ -1478,6 +1488,7 @@ static void wave5_vpu_enc_device_run(void *priv)
> >  	u32 fail_res = 0;
> >  	int ret = 0;
> >  
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >  	switch (inst->state) {
> >  	case VPU_INST_STATE_PIC_RUN:
> >  		ret = start_encode(inst, &fail_res);
> > @@ -1491,6 +1502,8 @@ static void wave5_vpu_enc_device_run(void *priv)
> >  			break;
> >  		}
> >  		dev_dbg(inst->dev->dev, "%s: leave with active job", __func__);
> > +		pm_runtime_mark_last_busy(inst->dev->dev);
> > +		pm_runtime_put_autosuspend(inst->dev->dev);
> >  		return;
> >  	default:
> >  		WARN(1, "Execution of a job in state %s is invalid.\n",
> > @@ -1498,6 +1511,8 @@ static void wave5_vpu_enc_device_run(void *priv)
> >  		break;
> >  	}
> >  	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
> >  }
> >  
> > @@ -1739,9 +1754,8 @@ static int wave5_vpu_open_enc(struct file *filp)
> >  	if (ret)
> >  		goto cleanup_inst;
> >  
> > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) && list_empty(&dev->instances))
> > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev->vpu_poll_interval * NSEC_PER_MSEC),
> > -			      HRTIMER_MODE_REL_PINNED);
> > +	if (list_empty(&dev->instances))
> > +		pm_runtime_use_autosuspend(inst->dev->dev);
> >  
> >  	list_add_tail(&inst->list, &dev->instances);
> >  
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu.c b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > index 7273254ecb03..41c4bf64f27d 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > @@ -10,6 +10,7 @@
> >  #include <linux/clk.h>
> >  #include <linux/firmware.h>
> >  #include <linux/interrupt.h>
> > +#include <linux/pm_runtime.h>
> >  #include <linux/reset.h>
> >  #include "wave5-vpu.h"
> >  #include "wave5-regdefine.h"
> > @@ -153,6 +154,45 @@ static int wave5_vpu_load_firmware(struct device *dev, const char *fw_name,
> >  	return 0;
> >  }
> >  
> > +static __maybe_unused int wave5_pm_suspend(struct device *dev)
> > +{
> > +	struct vpu_device *vpu = dev_get_drvdata(dev);
> > +
> > +	if (pm_runtime_suspended(dev))
> > +		return 0;
> > +
> > +	if (vpu->irq < 0)
> > +		hrtimer_cancel(&vpu->hrtimer);
> > +
> > +	wave5_vpu_sleep_wake(dev, true, NULL, 0);
> > +	clk_bulk_disable_unprepare(vpu->num_clks, vpu->clks);
> > +
> > +	return 0;
> > +}
> > +
> > +static __maybe_unused int wave5_pm_resume(struct device *dev)
> > +{
> > +	struct vpu_device *vpu = dev_get_drvdata(dev);
> > +	int ret = 0;
> > +
> > +	wave5_vpu_sleep_wake(dev, false, NULL, 0);
> > +	ret = clk_bulk_prepare_enable(vpu->num_clks, vpu->clks);
> > +	if (ret) {
> > +		dev_err(dev, "Enabling clocks, fail: %d\n", ret);
> > +		return ret;
> > +	}
> > +
> > +	if (vpu->irq < 0 && !hrtimer_active(&vpu->hrtimer))
> > +		hrtimer_start(&vpu->hrtimer, ns_to_ktime(vpu->vpu_poll_interval * NSEC_PER_MSEC),
> > +				HRTIMER_MODE_REL_PINNED);

I have fixed locally this style error. Alignment should match open parenthesis.
Checkpatch caught that, it saves times if you run checkpatch before your
submissions.

https://gitlab.freedesktop.org/ndufresne/media-staging/-/jobs/63283654

regards,
Nicolas

> > +
> > +	return ret;
> > +}
> > +
> > +static const struct dev_pm_ops wave5_pm_ops = {
> > +	SET_RUNTIME_PM_OPS(wave5_pm_suspend, wave5_pm_resume, NULL)
> > +};
> > +
> >  static int wave5_vpu_probe(struct platform_device *pdev)
> >  {
> >  	int ret;
> > @@ -281,6 +321,12 @@ static int wave5_vpu_probe(struct platform_device *pdev)
> >  		 (match_data->flags & WAVE5_IS_DEC) ? "'DECODE'" : "");
> >  	dev_info(&pdev->dev, "Product Code:      0x%x\n", dev->product_code);
> >  	dev_info(&pdev->dev, "Firmware Revision: %u\n", fw_revision);
> > +
> > +	pm_runtime_set_autosuspend_delay(&pdev->dev, 100);
> > +	pm_runtime_use_autosuspend(&pdev->dev);
> > +	pm_runtime_enable(&pdev->dev);
> > +	wave5_vpu_sleep_wake(&pdev->dev, true, NULL, 0);
> > +
> >  	return 0;
> >  
> >  err_enc_unreg:
> > @@ -310,6 +356,9 @@ static void wave5_vpu_remove(struct platform_device *pdev)
> >  		hrtimer_cancel(&dev->hrtimer);
> >  	}
> >  
> > +	pm_runtime_put_sync(&pdev->dev);
> > +	pm_runtime_disable(&pdev->dev);
> > +
> >  	mutex_destroy(&dev->dev_lock);
> >  	mutex_destroy(&dev->hw_lock);
> >  	reset_control_assert(dev->resets);
> > @@ -337,6 +386,7 @@ static struct platform_driver wave5_vpu_driver = {
> >  	.driver = {
> >  		.name = VPU_PLATFORM_DEVICE_NAME,
> >  		.of_match_table = of_match_ptr(wave5_dt_ids),
> > +		.pm = &wave5_pm_ops,
> >  		},
> >  	.probe = wave5_vpu_probe,
> >  	.remove_new = wave5_vpu_remove,
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > index 1a3efb638dde..e16b990041c2 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > @@ -6,6 +6,8 @@
> >   */
> >  
> >  #include <linux/bug.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/delay.h>
> >  #include "wave5-vpuapi.h"
> >  #include "wave5-regdefine.h"
> >  #include "wave5.h"
> > @@ -195,14 +197,20 @@ int wave5_vpu_dec_close(struct vpu_instance *inst, u32 *fail_res)
> >  	int retry = 0;
> >  	struct vpu_device *vpu_dev = inst->dev;
> >  	int i;
> > +	int inst_count = 0;
> > +	struct vpu_instance *inst_elm;
> >  
> >  	*fail_res = 0;
> >  	if (!inst->codec_info)
> >  		return -EINVAL;
> >  
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> > +
> >  	ret = mutex_lock_interruptible(&vpu_dev->hw_lock);
> > -	if (ret)
> > +	if (ret) {
> > +		pm_runtime_put_sync(inst->dev->dev);
> >  		return ret;
> > +	}
> >  
> >  	do {
> >  		ret = wave5_vpu_dec_finish_seq(inst, fail_res);
> > @@ -232,9 +240,14 @@ int wave5_vpu_dec_close(struct vpu_instance *inst, u32 *fail_res)
> >  
> >  	wave5_vdi_free_dma_memory(vpu_dev, &p_dec_info->vb_task);
> >  
> > +	list_for_each_entry(inst_elm, &vpu_dev->instances, list)
> > +		inst_count++;
> > +	if (inst_count == 1)
> > +		pm_runtime_dont_use_autosuspend(vpu_dev->dev);
> > +
> >  unlock_and_return:
> >  	mutex_unlock(&vpu_dev->hw_lock);
> > -
> > +	pm_runtime_put_sync(inst->dev->dev);
> >  	return ret;
> >  }
> >  
> > @@ -697,25 +710,33 @@ int wave5_vpu_enc_close(struct vpu_instance *inst, u32 *fail_res)
> >  	int ret;
> >  	int retry = 0;
> >  	struct vpu_device *vpu_dev = inst->dev;
> > +	int inst_count = 0;
> > +	struct vpu_instance *inst_elm;
> >  
> >  	*fail_res = 0;
> >  	if (!inst->codec_info)
> >  		return -EINVAL;
> >  
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> > +
> >  	ret = mutex_lock_interruptible(&vpu_dev->hw_lock);
> > -	if (ret)
> > +	if (ret) {
> > +		pm_runtime_resume_and_get(inst->dev->dev);
> >  		return ret;
> > +	}
> >  
> >  	do {
> >  		ret = wave5_vpu_enc_finish_seq(inst, fail_res);
> >  		if (ret < 0 && *fail_res != WAVE5_SYSERR_VPU_STILL_RUNNING) {
> >  			dev_warn(inst->dev->dev, "enc_finish_seq timed out\n");
> > +			pm_runtime_resume_and_get(inst->dev->dev);
> >  			mutex_unlock(&vpu_dev->hw_lock);
> >  			return ret;
> >  		}
> >  
> >  		if (*fail_res == WAVE5_SYSERR_VPU_STILL_RUNNING &&
> >  		    retry++ >= MAX_FIRMWARE_CALL_RETRY) {
> > +			pm_runtime_resume_and_get(inst->dev->dev);
> >  			mutex_unlock(&vpu_dev->hw_lock);
> >  			return -ETIMEDOUT;
> >  		}
> > @@ -734,7 +755,13 @@ int wave5_vpu_enc_close(struct vpu_instance *inst, u32 *fail_res)
> >  
> >  	wave5_vdi_free_dma_memory(vpu_dev, &p_enc_info->vb_task);
> >  
> > +	list_for_each_entry(inst_elm, &vpu_dev->instances, list)
> > +		inst_count++;
> > +	if (inst_count == 1)
> > +		pm_runtime_dont_use_autosuspend(vpu_dev->dev);
> > +
> >  	mutex_unlock(&vpu_dev->hw_lock);
> > +	pm_runtime_put_sync(inst->dev->dev);
> >  
> >  	return 0;
> >  }
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5.h b/drivers/media/platform/chips-media/wave5/wave5.h
> > index 2a29b9164f97..2caab356f3e1 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5.h
> > +++ b/drivers/media/platform/chips-media/wave5/wave5.h
> > @@ -62,6 +62,9 @@ int wave5_vpu_get_version(struct vpu_device *vpu_dev, u32 *revision);
> >  
> >  int wave5_vpu_init(struct device *dev, u8 *fw, size_t size);
> >  
> > +int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const uint16_t *code,
> > +			 size_t size);
> > +
> >  int wave5_vpu_reset(struct device *dev, enum sw_reset_mode reset_mode);
> >  
> >  int wave5_vpu_build_up_dec_param(struct vpu_instance *inst, struct dec_open_param *param);
> 

RE: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support runtime suspend/resume

[jackson.lee]

Hi Nicolas

> -----Original Message-----
> From: Nicolas Dufresne <nicolas@ndufresne.ca>
> Sent: Saturday, September 7, 2024 4:27 AM
> To: jackson.lee <jackson.lee@chipsnmedia.com>; mchehab@kernel.org;
> sebastian.fricke@collabora.com
> Cc: linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
> hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com
> Subject: Re: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support
> runtime suspend/resume
> 
> Hi Again,
> 
> Le vendredi 06 septembre 2024 à 13:08 -0400, Nicolas Dufresne a écrit :
> > Hi,
> >
> > Le lundi 12 août 2024 à 16:08 +0900, Jackson.lee a écrit :
> > > Add support for runtime suspend/resume in the encoder and decoder.
> > > This is achieved by saving the VPU state and powering it off while the
> VPU idle.
> >
> > If you don't, I'll edit to "while the VPU *is* idle".
> >
> > regards,
> > Nicolas
> >
> > >
> > > Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
> > > Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
> > > Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
> > > ---
> > >  .../platform/chips-media/wave5/wave5-helper.c | 13 -----
> > >  .../platform/chips-media/wave5/wave5-hw.c     |  4 +-
> > >  .../chips-media/wave5/wave5-vpu-dec.c         | 21 ++++++--
> > >  .../chips-media/wave5/wave5-vpu-enc.c         | 20 ++++++--
> > >  .../platform/chips-media/wave5/wave5-vpu.c    | 50 +++++++++++++++++++
> > >  .../platform/chips-media/wave5/wave5-vpuapi.c | 33 ++++++++++--
> > > .../media/platform/chips-media/wave5/wave5.h  |  3 ++
> > >  7 files changed, 118 insertions(+), 26 deletions(-)
> > >
> > > diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > > b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > > index d60841c54a80..a20d84dbe3aa 100644
> > > --- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > > @@ -58,7 +58,6 @@ int wave5_vpu_release_device(struct file *filp,
> > >  			     char *name)
> > >  {
> > >  	struct vpu_instance *inst = wave5_to_vpu_inst(filp->private_data);
> > > -	struct vpu_device *dev = inst->dev;
> > >  	int ret = 0;
> > >
> > >  	v4l2_m2m_ctx_release(inst->v4l2_fh.m2m_ctx);
> > > @@ -78,18 +77,6 @@ int wave5_vpu_release_device(struct file *filp,
> > >  	}
> > >
> > >  	wave5_cleanup_instance(inst);
> > > -	if (dev->irq < 0) {
> > > -		ret = mutex_lock_interruptible(&dev->dev_lock);
> > > -		if (ret)
> > > -			return ret;
> > > -
> > > -		if (list_empty(&dev->instances)) {
> > > -			dev_dbg(dev->dev, "Disabling the hrtimer\n");
> > > -			hrtimer_cancel(&dev->hrtimer);
> > > -		}
> > > -
> > > -		mutex_unlock(&dev->dev_lock);
> > > -	}
> > >
> > >  	return ret;
> > >  }
> > > diff --git a/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > > b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > > index 2a98bab446d0..c8a905994109 100644
> > > --- a/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > > @@ -1228,8 +1228,8 @@ int wave5_vpu_re_init(struct device *dev, u8 *fw,
> size_t size)
> > >  	return setup_wave5_properties(dev);  }
> > >
> > > -static int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake,
> const uint16_t *code,
> > > -				size_t size)
> > > +int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const
> uint16_t *code,
> > > +			 size_t size)
> > >  {
> > >  	u32 reg_val;
> > >  	struct vpu_buf *common_vb;
> > > diff --git
> > > a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > > index 0c5c9a8de91f..698c83e3082e 100644
> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > > @@ -5,6 +5,7 @@
> > >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
> > >   */
> > >
> > > +#include <linux/pm_runtime.h>
> > >  #include "wave5-helper.h"
> > >
> > >  #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
> > > @@ -518,6 +519,8 @@ static void wave5_vpu_dec_finish_decode(struct
> vpu_instance *inst)
> > >  	if (q_status.report_queue_count == 0 &&
> > >  	    (q_status.instance_queue_count == 0 ||
> dec_info.sequence_changed)) {
> > >  		dev_dbg(inst->dev->dev, "%s: finishing job.\n", __func__);
> > > +		pm_runtime_mark_last_busy(inst->dev->dev);
> > > +		pm_runtime_put_autosuspend(inst->dev->dev);
> > >  		v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
> > >  	}
> > >  }
> > > @@ -1398,6 +1401,7 @@ static int wave5_vpu_dec_start_streaming(struct
> vb2_queue *q, unsigned int count
> > >  	int ret = 0;
> > >
> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >
> > >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
> > >
> > > @@ -1429,13 +1433,15 @@ static int
> wave5_vpu_dec_start_streaming(struct vb2_queue *q, unsigned int count
> > >  		if (ret)
> > >  			goto return_buffers;
> > >  	}
> > > -
> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >  	return ret;
> > >
> > >  free_bitstream_vbuf:
> > >  	wave5_vdi_free_dma_memory(inst->dev, &inst->bitstream_vbuf);
> > >  return_buffers:
> > >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >  	return ret;
> > >  }
> > >
> > > @@ -1521,6 +1527,7 @@ static void wave5_vpu_dec_stop_streaming(struct
> vb2_queue *q)
> > >  	bool check_cmd = TRUE;
> > >
> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >
> > >  	while (check_cmd) {
> > >  		struct queue_status_info q_status; @@ -1544,6 +1551,9 @@
> static
> > > void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
> > >  		streamoff_output(q);
> > >  	else
> > >  		streamoff_capture(q);
> > > +
> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >  }
> > >
> > >  static const struct vb2_ops wave5_vpu_dec_vb2_ops = { @@ -1630,7
> > > +1640,7 @@ static void wave5_vpu_dec_device_run(void *priv)
> > >  	int ret = 0;
> > >
> > >  	dev_dbg(inst->dev->dev, "%s: Fill the ring buffer with new
> > > bitstream data", __func__);
> > > -
> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >  	ret = fill_ringbuffer(inst);
> > >  	if (ret) {
> > >  		dev_warn(inst->dev->dev, "Filling ring buffer failed\n"); @@
> > > -1713,6 +1723,8 @@ static void wave5_vpu_dec_device_run(void *priv)
> > >
> > >  finish_job_and_return:
> > >  	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);  }
> > >
> > > @@ -1879,9 +1891,8 @@ static int wave5_vpu_open_dec(struct file *filp)
> > >  	if (ret)
> > >  		goto cleanup_inst;
> > >
> > > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) &&
> list_empty(&dev->instances))
> > > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev-
> >vpu_poll_interval * NSEC_PER_MSEC),
> > > -			      HRTIMER_MODE_REL_PINNED);
> > > +	if (list_empty(&dev->instances))
> > > +		pm_runtime_use_autosuspend(inst->dev->dev);
> > >
> > >  	list_add_tail(&inst->list, &dev->instances);
> > >
> > > diff --git
> > > a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > > index b731decbfda6..985de0c293e2 100644
> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > > @@ -5,6 +5,7 @@
> > >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
> > >   */
> > >
> > > +#include <linux/pm_runtime.h>
> > >  #include "wave5-helper.h"
> > >
> > >  #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
> > > @@ -1310,6 +1311,7 @@ static int wave5_vpu_enc_start_streaming(struct
> vb2_queue *q, unsigned int count
> > >  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
> > >  	int ret = 0;
> > >
> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
> > >
> > >  	if (inst->state == VPU_INST_STATE_NONE && q->type ==
> > > V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { @@ -1364,9 +1366,13 @@ static int
> wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
> > >  	if (ret)
> > >  		goto return_buffers;
> > >
> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >  	return 0;
> > >  return_buffers:
> > >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >  	return ret;
> > >  }
> > >
> > > @@ -1408,6 +1414,7 @@ static void wave5_vpu_enc_stop_streaming(struct
> vb2_queue *q)
> > >  	 */
> > >
> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >
> > >  	if (wave5_vpu_both_queues_are_streaming(inst))
> > >  		switch_state(inst, VPU_INST_STATE_STOP); @@ -1432,6 +1439,9
> @@
> > > static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
> > >  		streamoff_output(inst, q);
> > >  	else
> > >  		streamoff_capture(inst, q);
> > > +
> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >  }
> > >
> > >  static const struct vb2_ops wave5_vpu_enc_vb2_ops = { @@ -1478,6
> > > +1488,7 @@ static void wave5_vpu_enc_device_run(void *priv)
> > >  	u32 fail_res = 0;
> > >  	int ret = 0;
> > >
> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >  	switch (inst->state) {
> > >  	case VPU_INST_STATE_PIC_RUN:
> > >  		ret = start_encode(inst, &fail_res); @@ -1491,6 +1502,8 @@
> static
> > > void wave5_vpu_enc_device_run(void *priv)
> > >  			break;
> > >  		}
> > >  		dev_dbg(inst->dev->dev, "%s: leave with active job",
> __func__);
> > > +		pm_runtime_mark_last_busy(inst->dev->dev);
> > > +		pm_runtime_put_autosuspend(inst->dev->dev);
> > >  		return;
> > >  	default:
> > >  		WARN(1, "Execution of a job in state %s is invalid.\n", @@
> > > -1498,6 +1511,8 @@ static void wave5_vpu_enc_device_run(void *priv)
> > >  		break;
> > >  	}
> > >  	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);  }
> > >
> > > @@ -1739,9 +1754,8 @@ static int wave5_vpu_open_enc(struct file *filp)
> > >  	if (ret)
> > >  		goto cleanup_inst;
> > >
> > > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) &&
> list_empty(&dev->instances))
> > > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev-
> >vpu_poll_interval * NSEC_PER_MSEC),
> > > -			      HRTIMER_MODE_REL_PINNED);
> > > +	if (list_empty(&dev->instances))
> > > +		pm_runtime_use_autosuspend(inst->dev->dev);
> > >
> > >  	list_add_tail(&inst->list, &dev->instances);
> > >
> > > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > > index 7273254ecb03..41c4bf64f27d 100644
> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > > @@ -10,6 +10,7 @@
> > >  #include <linux/clk.h>
> > >  #include <linux/firmware.h>
> > >  #include <linux/interrupt.h>
> > > +#include <linux/pm_runtime.h>
> > >  #include <linux/reset.h>
> > >  #include "wave5-vpu.h"
> > >  #include "wave5-regdefine.h"
> > > @@ -153,6 +154,45 @@ static int wave5_vpu_load_firmware(struct device
> *dev, const char *fw_name,
> > >  	return 0;
> > >  }
> > >
> > > +static __maybe_unused int wave5_pm_suspend(struct device *dev) {
> > > +	struct vpu_device *vpu = dev_get_drvdata(dev);
> > > +
> > > +	if (pm_runtime_suspended(dev))
> > > +		return 0;
> > > +
> > > +	if (vpu->irq < 0)
> > > +		hrtimer_cancel(&vpu->hrtimer);
> > > +
> > > +	wave5_vpu_sleep_wake(dev, true, NULL, 0);
> > > +	clk_bulk_disable_unprepare(vpu->num_clks, vpu->clks);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static __maybe_unused int wave5_pm_resume(struct device *dev) {
> > > +	struct vpu_device *vpu = dev_get_drvdata(dev);
> > > +	int ret = 0;
> > > +
> > > +	wave5_vpu_sleep_wake(dev, false, NULL, 0);
> > > +	ret = clk_bulk_prepare_enable(vpu->num_clks, vpu->clks);
> > > +	if (ret) {
> > > +		dev_err(dev, "Enabling clocks, fail: %d\n", ret);
> > > +		return ret;
> > > +	}
> > > +
> > > +	if (vpu->irq < 0 && !hrtimer_active(&vpu->hrtimer))
> > > +		hrtimer_start(&vpu->hrtimer, ns_to_ktime(vpu-
> >vpu_poll_interval * NSEC_PER_MSEC),
> > > +				HRTIMER_MODE_REL_PINNED);
> 
> I have fixed locally this style error. Alignment should match open
> parenthesis.
> Checkpatch caught that, it saves times if you run checkpatch before your
> submissions.
> 
> https://gitlab.freedesktop.org/ndufresne/media-staging/-/jobs/63283654
> 


Sorry for that, thanks for your feedback.
Should I make v8 for that ?





> regards,
> Nicolas
> 
> > > +
> > > +	return ret;
> > > +}
> > > +
> > > +static const struct dev_pm_ops wave5_pm_ops = {
> > > +	SET_RUNTIME_PM_OPS(wave5_pm_suspend, wave5_pm_resume, NULL) };
> > > +
> > >  static int wave5_vpu_probe(struct platform_device *pdev)  {
> > >  	int ret;
> > > @@ -281,6 +321,12 @@ static int wave5_vpu_probe(struct platform_device
> *pdev)
> > >  		 (match_data->flags & WAVE5_IS_DEC) ? "'DECODE'" : "");
> > >  	dev_info(&pdev->dev, "Product Code:      0x%x\n", dev->product_code);
> > >  	dev_info(&pdev->dev, "Firmware Revision: %u\n", fw_revision);
> > > +
> > > +	pm_runtime_set_autosuspend_delay(&pdev->dev, 100);
> > > +	pm_runtime_use_autosuspend(&pdev->dev);
> > > +	pm_runtime_enable(&pdev->dev);
> > > +	wave5_vpu_sleep_wake(&pdev->dev, true, NULL, 0);
> > > +
> > >  	return 0;
> > >
> > >  err_enc_unreg:
> > > @@ -310,6 +356,9 @@ static void wave5_vpu_remove(struct
> platform_device *pdev)
> > >  		hrtimer_cancel(&dev->hrtimer);
> > >  	}
> > >
> > > +	pm_runtime_put_sync(&pdev->dev);
> > > +	pm_runtime_disable(&pdev->dev);
> > > +
> > >  	mutex_destroy(&dev->dev_lock);
> > >  	mutex_destroy(&dev->hw_lock);
> > >  	reset_control_assert(dev->resets);
> > > @@ -337,6 +386,7 @@ static struct platform_driver wave5_vpu_driver = {
> > >  	.driver = {
> > >  		.name = VPU_PLATFORM_DEVICE_NAME,
> > >  		.of_match_table = of_match_ptr(wave5_dt_ids),
> > > +		.pm = &wave5_pm_ops,
> > >  		},
> > >  	.probe = wave5_vpu_probe,
> > >  	.remove_new = wave5_vpu_remove,
> > > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > > b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > > index 1a3efb638dde..e16b990041c2 100644
> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > > @@ -6,6 +6,8 @@
> > >   */
> > >
> > >  #include <linux/bug.h>
> > > +#include <linux/pm_runtime.h>
> > > +#include <linux/delay.h>
> > >  #include "wave5-vpuapi.h"
> > >  #include "wave5-regdefine.h"
> > >  #include "wave5.h"
> > > @@ -195,14 +197,20 @@ int wave5_vpu_dec_close(struct vpu_instance
> *inst, u32 *fail_res)
> > >  	int retry = 0;
> > >  	struct vpu_device *vpu_dev = inst->dev;
> > >  	int i;
> > > +	int inst_count = 0;
> > > +	struct vpu_instance *inst_elm;
> > >
> > >  	*fail_res = 0;
> > >  	if (!inst->codec_info)
> > >  		return -EINVAL;
> > >
> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > > +
> > >  	ret = mutex_lock_interruptible(&vpu_dev->hw_lock);
> > > -	if (ret)
> > > +	if (ret) {
> > > +		pm_runtime_put_sync(inst->dev->dev);
> > >  		return ret;
> > > +	}
> > >
> > >  	do {
> > >  		ret = wave5_vpu_dec_finish_seq(inst, fail_res); @@ -232,9
> +240,14
> > > @@ int wave5_vpu_dec_close(struct vpu_instance *inst, u32 *fail_res)
> > >
> > >  	wave5_vdi_free_dma_memory(vpu_dev, &p_dec_info->vb_task);
> > >
> > > +	list_for_each_entry(inst_elm, &vpu_dev->instances, list)
> > > +		inst_count++;
> > > +	if (inst_count == 1)
> > > +		pm_runtime_dont_use_autosuspend(vpu_dev->dev);
> > > +
> > >  unlock_and_return:
> > >  	mutex_unlock(&vpu_dev->hw_lock);
> > > -
> > > +	pm_runtime_put_sync(inst->dev->dev);
> > >  	return ret;
> > >  }
> > >
> > > @@ -697,25 +710,33 @@ int wave5_vpu_enc_close(struct vpu_instance
> *inst, u32 *fail_res)
> > >  	int ret;
> > >  	int retry = 0;
> > >  	struct vpu_device *vpu_dev = inst->dev;
> > > +	int inst_count = 0;
> > > +	struct vpu_instance *inst_elm;
> > >
> > >  	*fail_res = 0;
> > >  	if (!inst->codec_info)
> > >  		return -EINVAL;
> > >
> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > > +
> > >  	ret = mutex_lock_interruptible(&vpu_dev->hw_lock);
> > > -	if (ret)
> > > +	if (ret) {
> > > +		pm_runtime_resume_and_get(inst->dev->dev);
> > >  		return ret;
> > > +	}
> > >
> > >  	do {
> > >  		ret = wave5_vpu_enc_finish_seq(inst, fail_res);
> > >  		if (ret < 0 && *fail_res != WAVE5_SYSERR_VPU_STILL_RUNNING)
> {
> > >  			dev_warn(inst->dev->dev, "enc_finish_seq timed out\n");
> > > +			pm_runtime_resume_and_get(inst->dev->dev);
> > >  			mutex_unlock(&vpu_dev->hw_lock);
> > >  			return ret;
> > >  		}
> > >
> > >  		if (*fail_res == WAVE5_SYSERR_VPU_STILL_RUNNING &&
> > >  		    retry++ >= MAX_FIRMWARE_CALL_RETRY) {
> > > +			pm_runtime_resume_and_get(inst->dev->dev);
> > >  			mutex_unlock(&vpu_dev->hw_lock);
> > >  			return -ETIMEDOUT;
> > >  		}
> > > @@ -734,7 +755,13 @@ int wave5_vpu_enc_close(struct vpu_instance
> > > *inst, u32 *fail_res)
> > >
> > >  	wave5_vdi_free_dma_memory(vpu_dev, &p_enc_info->vb_task);
> > >
> > > +	list_for_each_entry(inst_elm, &vpu_dev->instances, list)
> > > +		inst_count++;
> > > +	if (inst_count == 1)
> > > +		pm_runtime_dont_use_autosuspend(vpu_dev->dev);
> > > +
> > >  	mutex_unlock(&vpu_dev->hw_lock);
> > > +	pm_runtime_put_sync(inst->dev->dev);
> > >
> > >  	return 0;
> > >  }
> > > diff --git a/drivers/media/platform/chips-media/wave5/wave5.h
> > > b/drivers/media/platform/chips-media/wave5/wave5.h
> > > index 2a29b9164f97..2caab356f3e1 100644
> > > --- a/drivers/media/platform/chips-media/wave5/wave5.h
> > > +++ b/drivers/media/platform/chips-media/wave5/wave5.h
> > > @@ -62,6 +62,9 @@ int wave5_vpu_get_version(struct vpu_device
> > > *vpu_dev, u32 *revision);
> > >
> > >  int wave5_vpu_init(struct device *dev, u8 *fw, size_t size);
> > >
> > > +int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const
> uint16_t *code,
> > > +			 size_t size);
> > > +
> > >  int wave5_vpu_reset(struct device *dev, enum sw_reset_mode
> > > reset_mode);
> > >
> > >  int wave5_vpu_build_up_dec_param(struct vpu_instance *inst, struct
> > > dec_open_param *param);
> >

Re: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support runtime suspend/resume

[sebastian.fricke]

Hey Jackson,

On 09.09.2024 01:21, jackson.lee wrote:
>Hi Nicolas
>
>> -----Original Message-----
>> From: Nicolas Dufresne <nicolas@ndufresne.ca>
>> Sent: Saturday, September 7, 2024 4:27 AM
>> To: jackson.lee <jackson.lee@chipsnmedia.com>; mchehab@kernel.org;
>> sebastian.fricke@collabora.com
>> Cc: linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
>> hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
>> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com
>> Subject: Re: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support
>> runtime suspend/resume
>>
>> Hi Again,
>>
>> Le vendredi 06 septembre 2024 à 13:08 -0400, Nicolas Dufresne a écrit :
>> > Hi,
>> >
>> > Le lundi 12 août 2024 à 16:08 +0900, Jackson.lee a écrit :
>> > > Add support for runtime suspend/resume in the encoder and decoder.
>> > > This is achieved by saving the VPU state and powering it off while the
>> VPU idle.
>> >
>> > If you don't, I'll edit to "while the VPU *is* idle".
>> >
>> > regards,
>> > Nicolas
>> >
>> > >
>> > > Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
>> > > Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
>> > > Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
>> > > ---
>> > >  .../platform/chips-media/wave5/wave5-helper.c | 13 -----
>> > >  .../platform/chips-media/wave5/wave5-hw.c     |  4 +-
>> > >  .../chips-media/wave5/wave5-vpu-dec.c         | 21 ++++++--
>> > >  .../chips-media/wave5/wave5-vpu-enc.c         | 20 ++++++--
>> > >  .../platform/chips-media/wave5/wave5-vpu.c    | 50 +++++++++++++++++++
>> > >  .../platform/chips-media/wave5/wave5-vpuapi.c | 33 ++++++++++--
>> > > .../media/platform/chips-media/wave5/wave5.h  |  3 ++
>> > >  7 files changed, 118 insertions(+), 26 deletions(-)
>> > >
>> > > diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.c
>> > > b/drivers/media/platform/chips-media/wave5/wave5-helper.c
>> > > index d60841c54a80..a20d84dbe3aa 100644
>> > > --- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
>> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
>> > > @@ -58,7 +58,6 @@ int wave5_vpu_release_device(struct file *filp,
>> > >  			     char *name)
>> > >  {
>> > >  	struct vpu_instance *inst = wave5_to_vpu_inst(filp->private_data);
>> > > -	struct vpu_device *dev = inst->dev;
>> > >  	int ret = 0;
>> > >
>> > >  	v4l2_m2m_ctx_release(inst->v4l2_fh.m2m_ctx);
>> > > @@ -78,18 +77,6 @@ int wave5_vpu_release_device(struct file *filp,
>> > >  	}
>> > >
>> > >  	wave5_cleanup_instance(inst);
>> > > -	if (dev->irq < 0) {
>> > > -		ret = mutex_lock_interruptible(&dev->dev_lock);
>> > > -		if (ret)
>> > > -			return ret;
>> > > -
>> > > -		if (list_empty(&dev->instances)) {
>> > > -			dev_dbg(dev->dev, "Disabling the hrtimer\n");
>> > > -			hrtimer_cancel(&dev->hrtimer);
>> > > -		}
>> > > -
>> > > -		mutex_unlock(&dev->dev_lock);
>> > > -	}
>> > >
>> > >  	return ret;
>> > >  }
>> > > diff --git a/drivers/media/platform/chips-media/wave5/wave5-hw.c
>> > > b/drivers/media/platform/chips-media/wave5/wave5-hw.c
>> > > index 2a98bab446d0..c8a905994109 100644
>> > > --- a/drivers/media/platform/chips-media/wave5/wave5-hw.c
>> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-hw.c
>> > > @@ -1228,8 +1228,8 @@ int wave5_vpu_re_init(struct device *dev, u8 *fw,
>> size_t size)
>> > >  	return setup_wave5_properties(dev);  }
>> > >
>> > > -static int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake,
>> const uint16_t *code,
>> > > -				size_t size)
>> > > +int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const
>> uint16_t *code,
>> > > +			 size_t size)
>> > >  {
>> > >  	u32 reg_val;
>> > >  	struct vpu_buf *common_vb;
>> > > diff --git
>> > > a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
>> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
>> > > index 0c5c9a8de91f..698c83e3082e 100644
>> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
>> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
>> > > @@ -5,6 +5,7 @@
>> > >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
>> > >   */
>> > >
>> > > +#include <linux/pm_runtime.h>
>> > >  #include "wave5-helper.h"
>> > >
>> > >  #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
>> > > @@ -518,6 +519,8 @@ static void wave5_vpu_dec_finish_decode(struct
>> vpu_instance *inst)
>> > >  	if (q_status.report_queue_count == 0 &&
>> > >  	    (q_status.instance_queue_count == 0 ||
>> dec_info.sequence_changed)) {
>> > >  		dev_dbg(inst->dev->dev, "%s: finishing job.\n", __func__);
>> > > +		pm_runtime_mark_last_busy(inst->dev->dev);
>> > > +		pm_runtime_put_autosuspend(inst->dev->dev);
>> > >  		v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
>> > >  	}
>> > >  }
>> > > @@ -1398,6 +1401,7 @@ static int wave5_vpu_dec_start_streaming(struct
>> vb2_queue *q, unsigned int count
>> > >  	int ret = 0;
>> > >
>> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
>> > > +	pm_runtime_resume_and_get(inst->dev->dev);
>> > >
>> > >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
>> > >
>> > > @@ -1429,13 +1433,15 @@ static int
>> wave5_vpu_dec_start_streaming(struct vb2_queue *q, unsigned int count
>> > >  		if (ret)
>> > >  			goto return_buffers;
>> > >  	}
>> > > -
>> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
>> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
>> > >  	return ret;
>> > >
>> > >  free_bitstream_vbuf:
>> > >  	wave5_vdi_free_dma_memory(inst->dev, &inst->bitstream_vbuf);
>> > >  return_buffers:
>> > >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
>> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
>> > >  	return ret;
>> > >  }
>> > >
>> > > @@ -1521,6 +1527,7 @@ static void wave5_vpu_dec_stop_streaming(struct
>> vb2_queue *q)
>> > >  	bool check_cmd = TRUE;
>> > >
>> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
>> > > +	pm_runtime_resume_and_get(inst->dev->dev);
>> > >
>> > >  	while (check_cmd) {
>> > >  		struct queue_status_info q_status; @@ -1544,6 +1551,9 @@
>> static
>> > > void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
>> > >  		streamoff_output(q);
>> > >  	else
>> > >  		streamoff_capture(q);
>> > > +
>> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
>> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
>> > >  }
>> > >
>> > >  static const struct vb2_ops wave5_vpu_dec_vb2_ops = { @@ -1630,7
>> > > +1640,7 @@ static void wave5_vpu_dec_device_run(void *priv)
>> > >  	int ret = 0;
>> > >
>> > >  	dev_dbg(inst->dev->dev, "%s: Fill the ring buffer with new
>> > > bitstream data", __func__);
>> > > -
>> > > +	pm_runtime_resume_and_get(inst->dev->dev);
>> > >  	ret = fill_ringbuffer(inst);
>> > >  	if (ret) {
>> > >  		dev_warn(inst->dev->dev, "Filling ring buffer failed\n"); @@
>> > > -1713,6 +1723,8 @@ static void wave5_vpu_dec_device_run(void *priv)
>> > >
>> > >  finish_job_and_return:
>> > >  	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
>> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
>> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
>> > >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);  }
>> > >
>> > > @@ -1879,9 +1891,8 @@ static int wave5_vpu_open_dec(struct file *filp)
>> > >  	if (ret)
>> > >  		goto cleanup_inst;
>> > >
>> > > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) &&
>> list_empty(&dev->instances))
>> > > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev-
>> >vpu_poll_interval * NSEC_PER_MSEC),
>> > > -			      HRTIMER_MODE_REL_PINNED);
>> > > +	if (list_empty(&dev->instances))
>> > > +		pm_runtime_use_autosuspend(inst->dev->dev);
>> > >
>> > >  	list_add_tail(&inst->list, &dev->instances);
>> > >
>> > > diff --git
>> > > a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
>> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
>> > > index b731decbfda6..985de0c293e2 100644
>> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
>> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
>> > > @@ -5,6 +5,7 @@
>> > >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
>> > >   */
>> > >
>> > > +#include <linux/pm_runtime.h>
>> > >  #include "wave5-helper.h"
>> > >
>> > >  #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
>> > > @@ -1310,6 +1311,7 @@ static int wave5_vpu_enc_start_streaming(struct
>> vb2_queue *q, unsigned int count
>> > >  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
>> > >  	int ret = 0;
>> > >
>> > > +	pm_runtime_resume_and_get(inst->dev->dev);
>> > >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
>> > >
>> > >  	if (inst->state == VPU_INST_STATE_NONE && q->type ==
>> > > V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { @@ -1364,9 +1366,13 @@ static int
>> wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
>> > >  	if (ret)
>> > >  		goto return_buffers;
>> > >
>> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
>> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
>> > >  	return 0;
>> > >  return_buffers:
>> > >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
>> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
>> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
>> > >  	return ret;
>> > >  }
>> > >
>> > > @@ -1408,6 +1414,7 @@ static void wave5_vpu_enc_stop_streaming(struct
>> vb2_queue *q)
>> > >  	 */
>> > >
>> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
>> > > +	pm_runtime_resume_and_get(inst->dev->dev);
>> > >
>> > >  	if (wave5_vpu_both_queues_are_streaming(inst))
>> > >  		switch_state(inst, VPU_INST_STATE_STOP); @@ -1432,6 +1439,9
>> @@
>> > > static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
>> > >  		streamoff_output(inst, q);
>> > >  	else
>> > >  		streamoff_capture(inst, q);
>> > > +
>> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
>> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
>> > >  }
>> > >
>> > >  static const struct vb2_ops wave5_vpu_enc_vb2_ops = { @@ -1478,6
>> > > +1488,7 @@ static void wave5_vpu_enc_device_run(void *priv)
>> > >  	u32 fail_res = 0;
>> > >  	int ret = 0;
>> > >
>> > > +	pm_runtime_resume_and_get(inst->dev->dev);
>> > >  	switch (inst->state) {
>> > >  	case VPU_INST_STATE_PIC_RUN:
>> > >  		ret = start_encode(inst, &fail_res); @@ -1491,6 +1502,8 @@
>> static
>> > > void wave5_vpu_enc_device_run(void *priv)
>> > >  			break;
>> > >  		}
>> > >  		dev_dbg(inst->dev->dev, "%s: leave with active job",
>> __func__);
>> > > +		pm_runtime_mark_last_busy(inst->dev->dev);
>> > > +		pm_runtime_put_autosuspend(inst->dev->dev);
>> > >  		return;
>> > >  	default:
>> > >  		WARN(1, "Execution of a job in state %s is invalid.\n", @@
>> > > -1498,6 +1511,8 @@ static void wave5_vpu_enc_device_run(void *priv)
>> > >  		break;
>> > >  	}
>> > >  	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
>> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
>> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
>> > >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);  }
>> > >
>> > > @@ -1739,9 +1754,8 @@ static int wave5_vpu_open_enc(struct file *filp)
>> > >  	if (ret)
>> > >  		goto cleanup_inst;
>> > >
>> > > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) &&
>> list_empty(&dev->instances))
>> > > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev-
>> >vpu_poll_interval * NSEC_PER_MSEC),
>> > > -			      HRTIMER_MODE_REL_PINNED);
>> > > +	if (list_empty(&dev->instances))
>> > > +		pm_runtime_use_autosuspend(inst->dev->dev);
>> > >
>> > >  	list_add_tail(&inst->list, &dev->instances);
>> > >
>> > > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
>> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
>> > > index 7273254ecb03..41c4bf64f27d 100644
>> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
>> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
>> > > @@ -10,6 +10,7 @@
>> > >  #include <linux/clk.h>
>> > >  #include <linux/firmware.h>
>> > >  #include <linux/interrupt.h>
>> > > +#include <linux/pm_runtime.h>
>> > >  #include <linux/reset.h>
>> > >  #include "wave5-vpu.h"
>> > >  #include "wave5-regdefine.h"
>> > > @@ -153,6 +154,45 @@ static int wave5_vpu_load_firmware(struct device
>> *dev, const char *fw_name,
>> > >  	return 0;
>> > >  }
>> > >
>> > > +static __maybe_unused int wave5_pm_suspend(struct device *dev) {
>> > > +	struct vpu_device *vpu = dev_get_drvdata(dev);
>> > > +
>> > > +	if (pm_runtime_suspended(dev))
>> > > +		return 0;
>> > > +
>> > > +	if (vpu->irq < 0)
>> > > +		hrtimer_cancel(&vpu->hrtimer);
>> > > +
>> > > +	wave5_vpu_sleep_wake(dev, true, NULL, 0);
>> > > +	clk_bulk_disable_unprepare(vpu->num_clks, vpu->clks);
>> > > +
>> > > +	return 0;
>> > > +}
>> > > +
>> > > +static __maybe_unused int wave5_pm_resume(struct device *dev) {
>> > > +	struct vpu_device *vpu = dev_get_drvdata(dev);
>> > > +	int ret = 0;
>> > > +
>> > > +	wave5_vpu_sleep_wake(dev, false, NULL, 0);
>> > > +	ret = clk_bulk_prepare_enable(vpu->num_clks, vpu->clks);
>> > > +	if (ret) {
>> > > +		dev_err(dev, "Enabling clocks, fail: %d\n", ret);
>> > > +		return ret;
>> > > +	}
>> > > +
>> > > +	if (vpu->irq < 0 && !hrtimer_active(&vpu->hrtimer))
>> > > +		hrtimer_start(&vpu->hrtimer, ns_to_ktime(vpu-
>> >vpu_poll_interval * NSEC_PER_MSEC),
>> > > +				HRTIMER_MODE_REL_PINNED);
>>
>> I have fixed locally this style error. Alignment should match open
>> parenthesis.
>> Checkpatch caught that, it saves times if you run checkpatch before your
>> submissions.
>>
>> https://gitlab.freedesktop.org/ndufresne/media-staging/-/jobs/63283654
>>
>
>
>Sorry for that, thanks for your feedback.
>Should I make v8 for that ?

That is not required in this case, when he says "I have fixed locally",
that means that the required change is applied by the maintainer before
merging the changes upstream.

Regards,
Sebastian

RE: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support runtime suspend/resume

[jackson.lee]

Hi Sebastian

Many thanks

Jackson

> -----Original Message-----
> From: sebastian.fricke@collabora.com <sebastian.fricke@collabora.com>
> Sent: Monday, September 9, 2024 2:47 PM
> To: jackson.lee <jackson.lee@chipsnmedia.com>
> Cc: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org; linux-
> media@vger.kernel.org; linux-kernel@vger.kernel.org; hverkuil@xs4all.nl;
> Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com
> Subject: Re: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support
> runtime suspend/resume
> 
> Hey Jackson,
> 
> On 09.09.2024 01:21, jackson.lee wrote:
> >Hi Nicolas
> >
> >> -----Original Message-----
> >> From: Nicolas Dufresne <nicolas@ndufresne.ca>
> >> Sent: Saturday, September 7, 2024 4:27 AM
> >> To: jackson.lee <jackson.lee@chipsnmedia.com>; mchehab@kernel.org;
> >> sebastian.fricke@collabora.com
> >> Cc: linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
> >> hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
> >> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com
> >> Subject: Re: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support
> >> runtime suspend/resume
> >>
> >> Hi Again,
> >>
> >> Le vendredi 06 septembre 2024 à 13:08 -0400, Nicolas Dufresne a écrit :
> >> > Hi,
> >> >
> >> > Le lundi 12 août 2024 à 16:08 +0900, Jackson.lee a écrit :
> >> > > Add support for runtime suspend/resume in the encoder and decoder.
> >> > > This is achieved by saving the VPU state and powering it off
> >> > > while the
> >> VPU idle.
> >> >
> >> > If you don't, I'll edit to "while the VPU *is* idle".
> >> >
> >> > regards,
> >> > Nicolas
> >> >
> >> > >
> >> > > Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
> >> > > Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
> >> > > Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
> >> > > ---
> >> > >  .../platform/chips-media/wave5/wave5-helper.c | 13 -----
> >> > >  .../platform/chips-media/wave5/wave5-hw.c     |  4 +-
> >> > >  .../chips-media/wave5/wave5-vpu-dec.c         | 21 ++++++--
> >> > >  .../chips-media/wave5/wave5-vpu-enc.c         | 20 ++++++--
> >> > >  .../platform/chips-media/wave5/wave5-vpu.c    | 50
> +++++++++++++++++++
> >> > >  .../platform/chips-media/wave5/wave5-vpuapi.c | 33 ++++++++++--
> >> > > .../media/platform/chips-media/wave5/wave5.h  |  3 ++
> >> > >  7 files changed, 118 insertions(+), 26 deletions(-)
> >> > >
> >> > > diff --git
> >> > > a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> >> > > b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> >> > > index d60841c54a80..a20d84dbe3aa 100644
> >> > > --- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> >> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> >> > > @@ -58,7 +58,6 @@ int wave5_vpu_release_device(struct file *filp,
> >> > >  			     char *name)
> >> > >  {
> >> > >  	struct vpu_instance *inst = wave5_to_vpu_inst(filp-
> >private_data);
> >> > > -	struct vpu_device *dev = inst->dev;
> >> > >  	int ret = 0;
> >> > >
> >> > >  	v4l2_m2m_ctx_release(inst->v4l2_fh.m2m_ctx);
> >> > > @@ -78,18 +77,6 @@ int wave5_vpu_release_device(struct file *filp,
> >> > >  	}
> >> > >
> >> > >  	wave5_cleanup_instance(inst);
> >> > > -	if (dev->irq < 0) {
> >> > > -		ret = mutex_lock_interruptible(&dev->dev_lock);
> >> > > -		if (ret)
> >> > > -			return ret;
> >> > > -
> >> > > -		if (list_empty(&dev->instances)) {
> >> > > -			dev_dbg(dev->dev, "Disabling the hrtimer\n");
> >> > > -			hrtimer_cancel(&dev->hrtimer);
> >> > > -		}
> >> > > -
> >> > > -		mutex_unlock(&dev->dev_lock);
> >> > > -	}
> >> > >
> >> > >  	return ret;
> >> > >  }
> >> > > diff --git a/drivers/media/platform/chips-media/wave5/wave5-hw.c
> >> > > b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> >> > > index 2a98bab446d0..c8a905994109 100644
> >> > > --- a/drivers/media/platform/chips-media/wave5/wave5-hw.c
> >> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> >> > > @@ -1228,8 +1228,8 @@ int wave5_vpu_re_init(struct device *dev,
> >> > > u8 *fw,
> >> size_t size)
> >> > >  	return setup_wave5_properties(dev);  }
> >> > >
> >> > > -static int wave5_vpu_sleep_wake(struct device *dev, bool
> >> > > i_sleep_wake,
> >> const uint16_t *code,
> >> > > -				size_t size)
> >> > > +int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake,
> >> > > +const
> >> uint16_t *code,
> >> > > +			 size_t size)
> >> > >  {
> >> > >  	u32 reg_val;
> >> > >  	struct vpu_buf *common_vb;
> >> > > diff --git
> >> > > a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> >> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> >> > > index 0c5c9a8de91f..698c83e3082e 100644
> >> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> >> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> >> > > @@ -5,6 +5,7 @@
> >> > >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
> >> > >   */
> >> > >
> >> > > +#include <linux/pm_runtime.h>
> >> > >  #include "wave5-helper.h"
> >> > >
> >> > >  #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
> >> > > @@ -518,6 +519,8 @@ static void
> >> > > wave5_vpu_dec_finish_decode(struct
> >> vpu_instance *inst)
> >> > >  	if (q_status.report_queue_count == 0 &&
> >> > >  	    (q_status.instance_queue_count == 0 ||
> >> dec_info.sequence_changed)) {
> >> > >  		dev_dbg(inst->dev->dev, "%s: finishing job.\n",
> __func__);
> >> > > +		pm_runtime_mark_last_busy(inst->dev->dev);
> >> > > +		pm_runtime_put_autosuspend(inst->dev->dev);
> >> > >  		v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
> >> > >  	}
> >> > >  }
> >> > > @@ -1398,6 +1401,7 @@ static int
> >> > > wave5_vpu_dec_start_streaming(struct
> >> vb2_queue *q, unsigned int count
> >> > >  	int ret = 0;
> >> > >
> >> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> >> > >
> >> > >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
> >> > >
> >> > > @@ -1429,13 +1433,15 @@ static int
> >> wave5_vpu_dec_start_streaming(struct vb2_queue *q, unsigned int count
> >> > >  		if (ret)
> >> > >  			goto return_buffers;
> >> > >  	}
> >> > > -
> >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >> > >  	return ret;
> >> > >
> >> > >  free_bitstream_vbuf:
> >> > >  	wave5_vdi_free_dma_memory(inst->dev, &inst->bitstream_vbuf);
> >> > >  return_buffers:
> >> > >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >> > >  	return ret;
> >> > >  }
> >> > >
> >> > > @@ -1521,6 +1527,7 @@ static void
> >> > > wave5_vpu_dec_stop_streaming(struct
> >> vb2_queue *q)
> >> > >  	bool check_cmd = TRUE;
> >> > >
> >> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> >> > >
> >> > >  	while (check_cmd) {
> >> > >  		struct queue_status_info q_status; @@ -1544,6 +1551,9
> @@
> >> static
> >> > > void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
> >> > >  		streamoff_output(q);
> >> > >  	else
> >> > >  		streamoff_capture(q);
> >> > > +
> >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >> > >  }
> >> > >
> >> > >  static const struct vb2_ops wave5_vpu_dec_vb2_ops = { @@ -1630,7
> >> > > +1640,7 @@ static void wave5_vpu_dec_device_run(void *priv)
> >> > >  	int ret = 0;
> >> > >
> >> > >  	dev_dbg(inst->dev->dev, "%s: Fill the ring buffer with new
> >> > > bitstream data", __func__);
> >> > > -
> >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> >> > >  	ret = fill_ringbuffer(inst);
> >> > >  	if (ret) {
> >> > >  		dev_warn(inst->dev->dev, "Filling ring buffer
> failed\n"); @@
> >> > > -1713,6 +1723,8 @@ static void wave5_vpu_dec_device_run(void
> >> > > *priv)
> >> > >
> >> > >  finish_job_and_return:
> >> > >  	dev_dbg(inst->dev->dev, "%s: leave and finish job",
> __func__);
> >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >> > >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);  }
> >> > >
> >> > > @@ -1879,9 +1891,8 @@ static int wave5_vpu_open_dec(struct file
> *filp)
> >> > >  	if (ret)
> >> > >  		goto cleanup_inst;
> >> > >
> >> > > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) &&
> >> list_empty(&dev->instances))
> >> > > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev-
> >> >vpu_poll_interval * NSEC_PER_MSEC),
> >> > > -			      HRTIMER_MODE_REL_PINNED);
> >> > > +	if (list_empty(&dev->instances))
> >> > > +		pm_runtime_use_autosuspend(inst->dev->dev);
> >> > >
> >> > >  	list_add_tail(&inst->list, &dev->instances);
> >> > >
> >> > > diff --git
> >> > > a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> >> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> >> > > index b731decbfda6..985de0c293e2 100644
> >> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> >> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> >> > > @@ -5,6 +5,7 @@
> >> > >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
> >> > >   */
> >> > >
> >> > > +#include <linux/pm_runtime.h>
> >> > >  #include "wave5-helper.h"
> >> > >
> >> > >  #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
> >> > > @@ -1310,6 +1311,7 @@ static int
> >> > > wave5_vpu_enc_start_streaming(struct
> >> vb2_queue *q, unsigned int count
> >> > >  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
> >> > >  	int ret = 0;
> >> > >
> >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> >> > >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
> >> > >
> >> > >  	if (inst->state == VPU_INST_STATE_NONE && q->type ==
> >> > > V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { @@ -1364,9 +1366,13 @@
> >> > > static int
> >> wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
> >> > >  	if (ret)
> >> > >  		goto return_buffers;
> >> > >
> >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >> > >  	return 0;
> >> > >  return_buffers:
> >> > >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >> > >  	return ret;
> >> > >  }
> >> > >
> >> > > @@ -1408,6 +1414,7 @@ static void
> >> > > wave5_vpu_enc_stop_streaming(struct
> >> vb2_queue *q)
> >> > >  	 */
> >> > >
> >> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> >> > >
> >> > >  	if (wave5_vpu_both_queues_are_streaming(inst))
> >> > >  		switch_state(inst, VPU_INST_STATE_STOP); @@ -1432,6
> +1439,9
> >> @@
> >> > > static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
> >> > >  		streamoff_output(inst, q);
> >> > >  	else
> >> > >  		streamoff_capture(inst, q);
> >> > > +
> >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >> > >  }
> >> > >
> >> > >  static const struct vb2_ops wave5_vpu_enc_vb2_ops = { @@ -1478,6
> >> > > +1488,7 @@ static void wave5_vpu_enc_device_run(void *priv)
> >> > >  	u32 fail_res = 0;
> >> > >  	int ret = 0;
> >> > >
> >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> >> > >  	switch (inst->state) {
> >> > >  	case VPU_INST_STATE_PIC_RUN:
> >> > >  		ret = start_encode(inst, &fail_res); @@ -1491,6
> +1502,8 @@
> >> static
> >> > > void wave5_vpu_enc_device_run(void *priv)
> >> > >  			break;
> >> > >  		}
> >> > >  		dev_dbg(inst->dev->dev, "%s: leave with active job",
> >> __func__);
> >> > > +		pm_runtime_mark_last_busy(inst->dev->dev);
> >> > > +		pm_runtime_put_autosuspend(inst->dev->dev);
> >> > >  		return;
> >> > >  	default:
> >> > >  		WARN(1, "Execution of a job in state %s is invalid.\n",
> @@
> >> > > -1498,6 +1511,8 @@ static void wave5_vpu_enc_device_run(void *priv)
> >> > >  		break;
> >> > >  	}
> >> > >  	dev_dbg(inst->dev->dev, "%s: leave and finish job",
> __func__);
> >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >> > >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);  }
> >> > >
> >> > > @@ -1739,9 +1754,8 @@ static int wave5_vpu_open_enc(struct file
> *filp)
> >> > >  	if (ret)
> >> > >  		goto cleanup_inst;
> >> > >
> >> > > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) &&
> >> list_empty(&dev->instances))
> >> > > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev-
> >> >vpu_poll_interval * NSEC_PER_MSEC),
> >> > > -			      HRTIMER_MODE_REL_PINNED);
> >> > > +	if (list_empty(&dev->instances))
> >> > > +		pm_runtime_use_autosuspend(inst->dev->dev);
> >> > >
> >> > >  	list_add_tail(&inst->list, &dev->instances);
> >> > >
> >> > > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> >> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> >> > > index 7273254ecb03..41c4bf64f27d 100644
> >> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> >> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> >> > > @@ -10,6 +10,7 @@
> >> > >  #include <linux/clk.h>
> >> > >  #include <linux/firmware.h>
> >> > >  #include <linux/interrupt.h>
> >> > > +#include <linux/pm_runtime.h>
> >> > >  #include <linux/reset.h>
> >> > >  #include "wave5-vpu.h"
> >> > >  #include "wave5-regdefine.h"
> >> > > @@ -153,6 +154,45 @@ static int wave5_vpu_load_firmware(struct
> >> > > device
> >> *dev, const char *fw_name,
> >> > >  	return 0;
> >> > >  }
> >> > >
> >> > > +static __maybe_unused int wave5_pm_suspend(struct device *dev) {
> >> > > +	struct vpu_device *vpu = dev_get_drvdata(dev);
> >> > > +
> >> > > +	if (pm_runtime_suspended(dev))
> >> > > +		return 0;
> >> > > +
> >> > > +	if (vpu->irq < 0)
> >> > > +		hrtimer_cancel(&vpu->hrtimer);
> >> > > +
> >> > > +	wave5_vpu_sleep_wake(dev, true, NULL, 0);
> >> > > +	clk_bulk_disable_unprepare(vpu->num_clks, vpu->clks);
> >> > > +
> >> > > +	return 0;
> >> > > +}
> >> > > +
> >> > > +static __maybe_unused int wave5_pm_resume(struct device *dev) {
> >> > > +	struct vpu_device *vpu = dev_get_drvdata(dev);
> >> > > +	int ret = 0;
> >> > > +
> >> > > +	wave5_vpu_sleep_wake(dev, false, NULL, 0);
> >> > > +	ret = clk_bulk_prepare_enable(vpu->num_clks, vpu->clks);
> >> > > +	if (ret) {
> >> > > +		dev_err(dev, "Enabling clocks, fail: %d\n", ret);
> >> > > +		return ret;
> >> > > +	}
> >> > > +
> >> > > +	if (vpu->irq < 0 && !hrtimer_active(&vpu->hrtimer))
> >> > > +		hrtimer_start(&vpu->hrtimer, ns_to_ktime(vpu-
> >> >vpu_poll_interval * NSEC_PER_MSEC),
> >> > > +				HRTIMER_MODE_REL_PINNED);
> >>
> >> I have fixed locally this style error. Alignment should match open
> >> parenthesis.
> >> Checkpatch caught that, it saves times if you run checkpatch before
> >> your submissions.
> >>
> >> https://gitlab.freedesktop.org/ndufresne/media-staging/-/jobs/6328365
> >> 4
> >>
> >
> >
> >Sorry for that, thanks for your feedback.
> >Should I make v8 for that ?
> 
> That is not required in this case, when he says "I have fixed locally",
> that means that the required change is applied by the maintainer before
> merging the changes upstream.
> 
> Regards,
> Sebastian

RE: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support runtime suspend/resume

[jackson.lee]

Hi Sebastian and Nicolas

Can you please pull our patch?

Thanks
Jackson

> -----Original Message-----
> From: jackson.lee
> Sent: Monday, September 9, 2024 4:56 PM
> To: sebastian.fricke@collabora.com
> Cc: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org; linux-
> media@vger.kernel.org; linux-kernel@vger.kernel.org; hverkuil@xs4all.nl;
> Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com
> Subject: RE: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support
> runtime suspend/resume
> 
> Hi Sebastian
> 
> Many thanks
> 
> Jackson
> 
> > -----Original Message-----
> > From: sebastian.fricke@collabora.com <sebastian.fricke@collabora.com>
> > Sent: Monday, September 9, 2024 2:47 PM
> > To: jackson.lee <jackson.lee@chipsnmedia.com>
> > Cc: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org;
> > linux- media@vger.kernel.org; linux-kernel@vger.kernel.org;
> > hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
> > <lafley.kim@chipsnmedia.com>; b-brnich@ti.com
> > Subject: Re: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support
> > runtime suspend/resume
> >
> > Hey Jackson,
> >
> > On 09.09.2024 01:21, jackson.lee wrote:
> > >Hi Nicolas
> > >
> > >> -----Original Message-----
> > >> From: Nicolas Dufresne <nicolas@ndufresne.ca>
> > >> Sent: Saturday, September 7, 2024 4:27 AM
> > >> To: jackson.lee <jackson.lee@chipsnmedia.com>; mchehab@kernel.org;
> > >> sebastian.fricke@collabora.com
> > >> Cc: linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
> > >> hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>;
> > >> lafley.kim <lafley.kim@chipsnmedia.com>; b-brnich@ti.com
> > >> Subject: Re: [RESEND PATCH v7 2/4] media: chips-media: wave5:
> > >> Support runtime suspend/resume
> > >>
> > >> Hi Again,
> > >>
> > >> Le vendredi 06 septembre 2024 à 13:08 -0400, Nicolas Dufresne a
> écrit :
> > >> > Hi,
> > >> >
> > >> > Le lundi 12 août 2024 à 16:08 +0900, Jackson.lee a écrit :
> > >> > > Add support for runtime suspend/resume in the encoder and decoder.
> > >> > > This is achieved by saving the VPU state and powering it off
> > >> > > while the
> > >> VPU idle.
> > >> >
> > >> > If you don't, I'll edit to "while the VPU *is* idle".
> > >> >
> > >> > regards,
> > >> > Nicolas
> > >> >
> > >> > >
> > >> > > Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
> > >> > > Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
> > >> > > Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
> > >> > > ---
> > >> > >  .../platform/chips-media/wave5/wave5-helper.c | 13 -----
> > >> > >  .../platform/chips-media/wave5/wave5-hw.c     |  4 +-
> > >> > >  .../chips-media/wave5/wave5-vpu-dec.c         | 21 ++++++--
> > >> > >  .../chips-media/wave5/wave5-vpu-enc.c         | 20 ++++++--
> > >> > >  .../platform/chips-media/wave5/wave5-vpu.c    | 50
> > +++++++++++++++++++
> > >> > >  .../platform/chips-media/wave5/wave5-vpuapi.c | 33
> > >> > > ++++++++++-- .../media/platform/chips-media/wave5/wave5.h  |  3
> > >> > > ++
> > >> > >  7 files changed, 118 insertions(+), 26 deletions(-)
> > >> > >
> > >> > > diff --git
> > >> > > a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > >> > > b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > >> > > index d60841c54a80..a20d84dbe3aa 100644
> > >> > > --- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > >> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > >> > > @@ -58,7 +58,6 @@ int wave5_vpu_release_device(struct file *filp,
> > >> > >  			     char *name)
> > >> > >  {
> > >> > >  	struct vpu_instance *inst = wave5_to_vpu_inst(filp-
> > >private_data);
> > >> > > -	struct vpu_device *dev = inst->dev;
> > >> > >  	int ret = 0;
> > >> > >
> > >> > >  	v4l2_m2m_ctx_release(inst->v4l2_fh.m2m_ctx);
> > >> > > @@ -78,18 +77,6 @@ int wave5_vpu_release_device(struct file *filp,
> > >> > >  	}
> > >> > >
> > >> > >  	wave5_cleanup_instance(inst);
> > >> > > -	if (dev->irq < 0) {
> > >> > > -		ret = mutex_lock_interruptible(&dev->dev_lock);
> > >> > > -		if (ret)
> > >> > > -			return ret;
> > >> > > -
> > >> > > -		if (list_empty(&dev->instances)) {
> > >> > > -			dev_dbg(dev->dev, "Disabling the hrtimer\n");
> > >> > > -			hrtimer_cancel(&dev->hrtimer);
> > >> > > -		}
> > >> > > -
> > >> > > -		mutex_unlock(&dev->dev_lock);
> > >> > > -	}
> > >> > >
> > >> > >  	return ret;
> > >> > >  }
> > >> > > diff --git
> > >> > > a/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > >> > > b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > >> > > index 2a98bab446d0..c8a905994109 100644
> > >> > > --- a/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > >> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > >> > > @@ -1228,8 +1228,8 @@ int wave5_vpu_re_init(struct device *dev,
> > >> > > u8 *fw,
> > >> size_t size)
> > >> > >  	return setup_wave5_properties(dev);  }
> > >> > >
> > >> > > -static int wave5_vpu_sleep_wake(struct device *dev, bool
> > >> > > i_sleep_wake,
> > >> const uint16_t *code,
> > >> > > -				size_t size)
> > >> > > +int wave5_vpu_sleep_wake(struct device *dev, bool
> > >> > > +i_sleep_wake, const
> > >> uint16_t *code,
> > >> > > +			 size_t size)
> > >> > >  {
> > >> > >  	u32 reg_val;
> > >> > >  	struct vpu_buf *common_vb;
> > >> > > diff --git
> > >> > > a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > >> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > >> > > index 0c5c9a8de91f..698c83e3082e 100644
> > >> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > >> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > >> > > @@ -5,6 +5,7 @@
> > >> > >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
> > >> > >   */
> > >> > >
> > >> > > +#include <linux/pm_runtime.h>
> > >> > >  #include "wave5-helper.h"
> > >> > >
> > >> > >  #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
> > >> > > @@ -518,6 +519,8 @@ static void
> > >> > > wave5_vpu_dec_finish_decode(struct
> > >> vpu_instance *inst)
> > >> > >  	if (q_status.report_queue_count == 0 &&
> > >> > >  	    (q_status.instance_queue_count == 0 ||
> > >> dec_info.sequence_changed)) {
> > >> > >  		dev_dbg(inst->dev->dev, "%s: finishing job.\n",
> > __func__);
> > >> > > +		pm_runtime_mark_last_busy(inst->dev->dev);
> > >> > > +		pm_runtime_put_autosuspend(inst->dev->dev);
> > >> > >  		v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
> > >> > >  	}
> > >> > >  }
> > >> > > @@ -1398,6 +1401,7 @@ static int
> > >> > > wave5_vpu_dec_start_streaming(struct
> > >> vb2_queue *q, unsigned int count
> > >> > >  	int ret = 0;
> > >> > >
> > >> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >> > >
> > >> > >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
> > >> > >
> > >> > > @@ -1429,13 +1433,15 @@ static int
> > >> wave5_vpu_dec_start_streaming(struct vb2_queue *q, unsigned int
> > >> count
> > >> > >  		if (ret)
> > >> > >  			goto return_buffers;
> > >> > >  	}
> > >> > > -
> > >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >> > >  	return ret;
> > >> > >
> > >> > >  free_bitstream_vbuf:
> > >> > >  	wave5_vdi_free_dma_memory(inst->dev, &inst->bitstream_vbuf);
> > >> > >  return_buffers:
> > >> > >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> > >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >> > >  	return ret;
> > >> > >  }
> > >> > >
> > >> > > @@ -1521,6 +1527,7 @@ static void
> > >> > > wave5_vpu_dec_stop_streaming(struct
> > >> vb2_queue *q)
> > >> > >  	bool check_cmd = TRUE;
> > >> > >
> > >> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >> > >
> > >> > >  	while (check_cmd) {
> > >> > >  		struct queue_status_info q_status; @@ -1544,6 +1551,9
> > @@
> > >> static
> > >> > > void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
> > >> > >  		streamoff_output(q);
> > >> > >  	else
> > >> > >  		streamoff_capture(q);
> > >> > > +
> > >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >> > >  }
> > >> > >
> > >> > >  static const struct vb2_ops wave5_vpu_dec_vb2_ops = { @@
> > >> > > -1630,7
> > >> > > +1640,7 @@ static void wave5_vpu_dec_device_run(void *priv)
> > >> > >  	int ret = 0;
> > >> > >
> > >> > >  	dev_dbg(inst->dev->dev, "%s: Fill the ring buffer with new
> > >> > > bitstream data", __func__);
> > >> > > -
> > >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >> > >  	ret = fill_ringbuffer(inst);
> > >> > >  	if (ret) {
> > >> > >  		dev_warn(inst->dev->dev, "Filling ring buffer
> > failed\n"); @@
> > >> > > -1713,6 +1723,8 @@ static void wave5_vpu_dec_device_run(void
> > >> > > *priv)
> > >> > >
> > >> > >  finish_job_and_return:
> > >> > >  	dev_dbg(inst->dev->dev, "%s: leave and finish job",
> > __func__);
> > >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >> > >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);  }
> > >> > >
> > >> > > @@ -1879,9 +1891,8 @@ static int wave5_vpu_open_dec(struct file
> > *filp)
> > >> > >  	if (ret)
> > >> > >  		goto cleanup_inst;
> > >> > >
> > >> > > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) &&
> > >> list_empty(&dev->instances))
> > >> > > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev-
> > >> >vpu_poll_interval * NSEC_PER_MSEC),
> > >> > > -			      HRTIMER_MODE_REL_PINNED);
> > >> > > +	if (list_empty(&dev->instances))
> > >> > > +		pm_runtime_use_autosuspend(inst->dev->dev);
> > >> > >
> > >> > >  	list_add_tail(&inst->list, &dev->instances);
> > >> > >
> > >> > > diff --git
> > >> > > a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > >> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > >> > > index b731decbfda6..985de0c293e2 100644
> > >> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > >> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > >> > > @@ -5,6 +5,7 @@
> > >> > >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
> > >> > >   */
> > >> > >
> > >> > > +#include <linux/pm_runtime.h>
> > >> > >  #include "wave5-helper.h"
> > >> > >
> > >> > >  #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
> > >> > > @@ -1310,6 +1311,7 @@ static int
> > >> > > wave5_vpu_enc_start_streaming(struct
> > >> vb2_queue *q, unsigned int count
> > >> > >  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
> > >> > >  	int ret = 0;
> > >> > >
> > >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >> > >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
> > >> > >
> > >> > >  	if (inst->state == VPU_INST_STATE_NONE && q->type ==
> > >> > > V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { @@ -1364,9 +1366,13 @@
> > >> > > static int
> > >> wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int
> > >> count
> > >> > >  	if (ret)
> > >> > >  		goto return_buffers;
> > >> > >
> > >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >> > >  	return 0;
> > >> > >  return_buffers:
> > >> > >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> > >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >> > >  	return ret;
> > >> > >  }
> > >> > >
> > >> > > @@ -1408,6 +1414,7 @@ static void
> > >> > > wave5_vpu_enc_stop_streaming(struct
> > >> vb2_queue *q)
> > >> > >  	 */
> > >> > >
> > >> > >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >> > >
> > >> > >  	if (wave5_vpu_both_queues_are_streaming(inst))
> > >> > >  		switch_state(inst, VPU_INST_STATE_STOP); @@ -1432,6
> > +1439,9
> > >> @@
> > >> > > static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
> > >> > >  		streamoff_output(inst, q);
> > >> > >  	else
> > >> > >  		streamoff_capture(inst, q);
> > >> > > +
> > >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >> > >  }
> > >> > >
> > >> > >  static const struct vb2_ops wave5_vpu_enc_vb2_ops = { @@
> > >> > > -1478,6
> > >> > > +1488,7 @@ static void wave5_vpu_enc_device_run(void *priv)
> > >> > >  	u32 fail_res = 0;
> > >> > >  	int ret = 0;
> > >> > >
> > >> > > +	pm_runtime_resume_and_get(inst->dev->dev);
> > >> > >  	switch (inst->state) {
> > >> > >  	case VPU_INST_STATE_PIC_RUN:
> > >> > >  		ret = start_encode(inst, &fail_res); @@ -1491,6
> > +1502,8 @@
> > >> static
> > >> > > void wave5_vpu_enc_device_run(void *priv)
> > >> > >  			break;
> > >> > >  		}
> > >> > >  		dev_dbg(inst->dev->dev, "%s: leave with active job",
> > >> __func__);
> > >> > > +		pm_runtime_mark_last_busy(inst->dev->dev);
> > >> > > +		pm_runtime_put_autosuspend(inst->dev->dev);
> > >> > >  		return;
> > >> > >  	default:
> > >> > >  		WARN(1, "Execution of a job in state %s is invalid.\n",
> > @@
> > >> > > -1498,6 +1511,8 @@ static void wave5_vpu_enc_device_run(void
> *priv)
> > >> > >  		break;
> > >> > >  	}
> > >> > >  	dev_dbg(inst->dev->dev, "%s: leave and finish job",
> > __func__);
> > >> > > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > >> > > +	pm_runtime_put_autosuspend(inst->dev->dev);
> > >> > >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);  }
> > >> > >
> > >> > > @@ -1739,9 +1754,8 @@ static int wave5_vpu_open_enc(struct file
> > *filp)
> > >> > >  	if (ret)
> > >> > >  		goto cleanup_inst;
> > >> > >
> > >> > > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) &&
> > >> list_empty(&dev->instances))
> > >> > > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev-
> > >> >vpu_poll_interval * NSEC_PER_MSEC),
> > >> > > -			      HRTIMER_MODE_REL_PINNED);
> > >> > > +	if (list_empty(&dev->instances))
> > >> > > +		pm_runtime_use_autosuspend(inst->dev->dev);
> > >> > >
> > >> > >  	list_add_tail(&inst->list, &dev->instances);
> > >> > >
> > >> > > diff --git
> > >> > > a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > >> > > b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > >> > > index 7273254ecb03..41c4bf64f27d 100644
> > >> > > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > >> > > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > >> > > @@ -10,6 +10,7 @@
> > >> > >  #include <linux/clk.h>
> > >> > >  #include <linux/firmware.h>
> > >> > >  #include <linux/interrupt.h>
> > >> > > +#include <linux/pm_runtime.h>
> > >> > >  #include <linux/reset.h>
> > >> > >  #include "wave5-vpu.h"
> > >> > >  #include "wave5-regdefine.h"
> > >> > > @@ -153,6 +154,45 @@ static int wave5_vpu_load_firmware(struct
> > >> > > device
> > >> *dev, const char *fw_name,
> > >> > >  	return 0;
> > >> > >  }
> > >> > >
> > >> > > +static __maybe_unused int wave5_pm_suspend(struct device *dev) {
> > >> > > +	struct vpu_device *vpu = dev_get_drvdata(dev);
> > >> > > +
> > >> > > +	if (pm_runtime_suspended(dev))
> > >> > > +		return 0;
> > >> > > +
> > >> > > +	if (vpu->irq < 0)
> > >> > > +		hrtimer_cancel(&vpu->hrtimer);
> > >> > > +
> > >> > > +	wave5_vpu_sleep_wake(dev, true, NULL, 0);
> > >> > > +	clk_bulk_disable_unprepare(vpu->num_clks, vpu->clks);
> > >> > > +
> > >> > > +	return 0;
> > >> > > +}
> > >> > > +
> > >> > > +static __maybe_unused int wave5_pm_resume(struct device *dev) {
> > >> > > +	struct vpu_device *vpu = dev_get_drvdata(dev);
> > >> > > +	int ret = 0;
> > >> > > +
> > >> > > +	wave5_vpu_sleep_wake(dev, false, NULL, 0);
> > >> > > +	ret = clk_bulk_prepare_enable(vpu->num_clks, vpu->clks);
> > >> > > +	if (ret) {
> > >> > > +		dev_err(dev, "Enabling clocks, fail: %d\n", ret);
> > >> > > +		return ret;
> > >> > > +	}
> > >> > > +
> > >> > > +	if (vpu->irq < 0 && !hrtimer_active(&vpu->hrtimer))
> > >> > > +		hrtimer_start(&vpu->hrtimer, ns_to_ktime(vpu-
> > >> >vpu_poll_interval * NSEC_PER_MSEC),
> > >> > > +				HRTIMER_MODE_REL_PINNED);
> > >>
> > >> I have fixed locally this style error. Alignment should match open
> > >> parenthesis.
> > >> Checkpatch caught that, it saves times if you run checkpatch before
> > >> your submissions.
> > >>
> > >> https://gitlab.freedesktop.org/ndufresne/media-staging/-/jobs/63283
> > >> 65
> > >> 4
> > >>
> > >
> > >
> > >Sorry for that, thanks for your feedback.
> > >Should I make v8 for that ?
> >
> > That is not required in this case, when he says "I have fixed
> > locally", that means that the required change is applied by the
> > maintainer before merging the changes upstream.
> >
> > Regards,
> > Sebastian

RE: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support runtime suspend/resume

[jackson.lee]

Hi Nicolas

I agree it, thanks for your feedback.

> -----Original Message-----
> From: Nicolas Dufresne <nicolas@ndufresne.ca>
> Sent: Saturday, September 7, 2024 2:08 AM
> To: jackson.lee <jackson.lee@chipsnmedia.com>; mchehab@kernel.org;
> sebastian.fricke@collabora.com
> Cc: linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
> hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com
> Subject: Re: [RESEND PATCH v7 2/4] media: chips-media: wave5: Support
> runtime suspend/resume
> 
> Hi,
> 
> Le lundi 12 août 2024 à 16:08 +0900, Jackson.lee a écrit :
> > Add support for runtime suspend/resume in the encoder and decoder.
> > This is achieved by saving the VPU state and powering it off while the
> VPU idle.
> 
> If you don't, I'll edit to "while the VPU *is* idle".
> 
> regards,
> Nicolas
> 
> >
> > Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
> > Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
> > Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
> > ---
> >  .../platform/chips-media/wave5/wave5-helper.c | 13 -----
> >  .../platform/chips-media/wave5/wave5-hw.c     |  4 +-
> >  .../chips-media/wave5/wave5-vpu-dec.c         | 21 ++++++--
> >  .../chips-media/wave5/wave5-vpu-enc.c         | 20 ++++++--
> >  .../platform/chips-media/wave5/wave5-vpu.c    | 50 +++++++++++++++++++
> >  .../platform/chips-media/wave5/wave5-vpuapi.c | 33 ++++++++++--
> > .../media/platform/chips-media/wave5/wave5.h  |  3 ++
> >  7 files changed, 118 insertions(+), 26 deletions(-)
> >
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > index d60841c54a80..a20d84dbe3aa 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > @@ -58,7 +58,6 @@ int wave5_vpu_release_device(struct file *filp,
> >  			     char *name)
> >  {
> >  	struct vpu_instance *inst = wave5_to_vpu_inst(filp->private_data);
> > -	struct vpu_device *dev = inst->dev;
> >  	int ret = 0;
> >
> >  	v4l2_m2m_ctx_release(inst->v4l2_fh.m2m_ctx);
> > @@ -78,18 +77,6 @@ int wave5_vpu_release_device(struct file *filp,
> >  	}
> >
> >  	wave5_cleanup_instance(inst);
> > -	if (dev->irq < 0) {
> > -		ret = mutex_lock_interruptible(&dev->dev_lock);
> > -		if (ret)
> > -			return ret;
> > -
> > -		if (list_empty(&dev->instances)) {
> > -			dev_dbg(dev->dev, "Disabling the hrtimer\n");
> > -			hrtimer_cancel(&dev->hrtimer);
> > -		}
> > -
> > -		mutex_unlock(&dev->dev_lock);
> > -	}
> >
> >  	return ret;
> >  }
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > index 2a98bab446d0..c8a905994109 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-hw.c
> > @@ -1228,8 +1228,8 @@ int wave5_vpu_re_init(struct device *dev, u8 *fw,
> size_t size)
> >  	return setup_wave5_properties(dev);
> >  }
> >
> > -static int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake,
> const uint16_t *code,
> > -				size_t size)
> > +int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const
> uint16_t *code,
> > +			 size_t size)
> >  {
> >  	u32 reg_val;
> >  	struct vpu_buf *common_vb;
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > index 0c5c9a8de91f..698c83e3082e 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > @@ -5,6 +5,7 @@
> >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
> >   */
> >
> > +#include <linux/pm_runtime.h>
> >  #include "wave5-helper.h"
> >
> >  #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
> > @@ -518,6 +519,8 @@ static void wave5_vpu_dec_finish_decode(struct
> vpu_instance *inst)
> >  	if (q_status.report_queue_count == 0 &&
> >  	    (q_status.instance_queue_count == 0 ||
> dec_info.sequence_changed)) {
> >  		dev_dbg(inst->dev->dev, "%s: finishing job.\n", __func__);
> > +		pm_runtime_mark_last_busy(inst->dev->dev);
> > +		pm_runtime_put_autosuspend(inst->dev->dev);
> >  		v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);
> >  	}
> >  }
> > @@ -1398,6 +1401,7 @@ static int wave5_vpu_dec_start_streaming(struct
> vb2_queue *q, unsigned int count
> >  	int ret = 0;
> >
> >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >
> >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
> >
> > @@ -1429,13 +1433,15 @@ static int wave5_vpu_dec_start_streaming(struct
> vb2_queue *q, unsigned int count
> >  		if (ret)
> >  			goto return_buffers;
> >  	}
> > -
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	return ret;
> >
> >  free_bitstream_vbuf:
> >  	wave5_vdi_free_dma_memory(inst->dev, &inst->bitstream_vbuf);
> >  return_buffers:
> >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	return ret;
> >  }
> >
> > @@ -1521,6 +1527,7 @@ static void wave5_vpu_dec_stop_streaming(struct
> vb2_queue *q)
> >  	bool check_cmd = TRUE;
> >
> >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >
> >  	while (check_cmd) {
> >  		struct queue_status_info q_status;
> > @@ -1544,6 +1551,9 @@ static void wave5_vpu_dec_stop_streaming(struct
> vb2_queue *q)
> >  		streamoff_output(q);
> >  	else
> >  		streamoff_capture(q);
> > +
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  }
> >
> >  static const struct vb2_ops wave5_vpu_dec_vb2_ops = { @@ -1630,7
> > +1640,7 @@ static void wave5_vpu_dec_device_run(void *priv)
> >  	int ret = 0;
> >
> >  	dev_dbg(inst->dev->dev, "%s: Fill the ring buffer with new
> bitstream
> > data", __func__);
> > -
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >  	ret = fill_ringbuffer(inst);
> >  	if (ret) {
> >  		dev_warn(inst->dev->dev, "Filling ring buffer failed\n"); @@
> > -1713,6 +1723,8 @@ static void wave5_vpu_dec_device_run(void *priv)
> >
> >  finish_job_and_return:
> >  	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);  }
> >
> > @@ -1879,9 +1891,8 @@ static int wave5_vpu_open_dec(struct file *filp)
> >  	if (ret)
> >  		goto cleanup_inst;
> >
> > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) &&
> list_empty(&dev->instances))
> > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev-
> >vpu_poll_interval * NSEC_PER_MSEC),
> > -			      HRTIMER_MODE_REL_PINNED);
> > +	if (list_empty(&dev->instances))
> > +		pm_runtime_use_autosuspend(inst->dev->dev);
> >
> >  	list_add_tail(&inst->list, &dev->instances);
> >
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > index b731decbfda6..985de0c293e2 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > @@ -5,6 +5,7 @@
> >   * Copyright (C) 2021-2023 CHIPS&MEDIA INC
> >   */
> >
> > +#include <linux/pm_runtime.h>
> >  #include "wave5-helper.h"
> >
> >  #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
> > @@ -1310,6 +1311,7 @@ static int wave5_vpu_enc_start_streaming(struct
> vb2_queue *q, unsigned int count
> >  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
> >  	int ret = 0;
> >
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >  	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
> >
> >  	if (inst->state == VPU_INST_STATE_NONE && q->type ==
> > V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { @@ -1364,9 +1366,13 @@ static int
> wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
> >  	if (ret)
> >  		goto return_buffers;
> >
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	return 0;
> >  return_buffers:
> >  	wave5_return_bufs(q, VB2_BUF_STATE_QUEUED);
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	return ret;
> >  }
> >
> > @@ -1408,6 +1414,7 @@ static void wave5_vpu_enc_stop_streaming(struct
> vb2_queue *q)
> >  	 */
> >
> >  	dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >
> >  	if (wave5_vpu_both_queues_are_streaming(inst))
> >  		switch_state(inst, VPU_INST_STATE_STOP); @@ -1432,6 +1439,9
> @@
> > static void wave5_vpu_enc_stop_streaming(struct vb2_queue *q)
> >  		streamoff_output(inst, q);
> >  	else
> >  		streamoff_capture(inst, q);
> > +
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  }
> >
> >  static const struct vb2_ops wave5_vpu_enc_vb2_ops = { @@ -1478,6
> > +1488,7 @@ static void wave5_vpu_enc_device_run(void *priv)
> >  	u32 fail_res = 0;
> >  	int ret = 0;
> >
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> >  	switch (inst->state) {
> >  	case VPU_INST_STATE_PIC_RUN:
> >  		ret = start_encode(inst, &fail_res); @@ -1491,6 +1502,8 @@
> static
> > void wave5_vpu_enc_device_run(void *priv)
> >  			break;
> >  		}
> >  		dev_dbg(inst->dev->dev, "%s: leave with active job",
> __func__);
> > +		pm_runtime_mark_last_busy(inst->dev->dev);
> > +		pm_runtime_put_autosuspend(inst->dev->dev);
> >  		return;
> >  	default:
> >  		WARN(1, "Execution of a job in state %s is invalid.\n", @@ -
> 1498,6
> > +1511,8 @@ static void wave5_vpu_enc_device_run(void *priv)
> >  		break;
> >  	}
> >  	dev_dbg(inst->dev->dev, "%s: leave and finish job", __func__);
> > +	pm_runtime_mark_last_busy(inst->dev->dev);
> > +	pm_runtime_put_autosuspend(inst->dev->dev);
> >  	v4l2_m2m_job_finish(inst->v4l2_m2m_dev, m2m_ctx);  }
> >
> > @@ -1739,9 +1754,8 @@ static int wave5_vpu_open_enc(struct file *filp)
> >  	if (ret)
> >  		goto cleanup_inst;
> >
> > -	if (dev->irq < 0 && !hrtimer_active(&dev->hrtimer) &&
> list_empty(&dev->instances))
> > -		hrtimer_start(&dev->hrtimer, ns_to_ktime(dev-
> >vpu_poll_interval * NSEC_PER_MSEC),
> > -			      HRTIMER_MODE_REL_PINNED);
> > +	if (list_empty(&dev->instances))
> > +		pm_runtime_use_autosuspend(inst->dev->dev);
> >
> >  	list_add_tail(&inst->list, &dev->instances);
> >
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > index 7273254ecb03..41c4bf64f27d 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
> > @@ -10,6 +10,7 @@
> >  #include <linux/clk.h>
> >  #include <linux/firmware.h>
> >  #include <linux/interrupt.h>
> > +#include <linux/pm_runtime.h>
> >  #include <linux/reset.h>
> >  #include "wave5-vpu.h"
> >  #include "wave5-regdefine.h"
> > @@ -153,6 +154,45 @@ static int wave5_vpu_load_firmware(struct device
> *dev, const char *fw_name,
> >  	return 0;
> >  }
> >
> > +static __maybe_unused int wave5_pm_suspend(struct device *dev) {
> > +	struct vpu_device *vpu = dev_get_drvdata(dev);
> > +
> > +	if (pm_runtime_suspended(dev))
> > +		return 0;
> > +
> > +	if (vpu->irq < 0)
> > +		hrtimer_cancel(&vpu->hrtimer);
> > +
> > +	wave5_vpu_sleep_wake(dev, true, NULL, 0);
> > +	clk_bulk_disable_unprepare(vpu->num_clks, vpu->clks);
> > +
> > +	return 0;
> > +}
> > +
> > +static __maybe_unused int wave5_pm_resume(struct device *dev) {
> > +	struct vpu_device *vpu = dev_get_drvdata(dev);
> > +	int ret = 0;
> > +
> > +	wave5_vpu_sleep_wake(dev, false, NULL, 0);
> > +	ret = clk_bulk_prepare_enable(vpu->num_clks, vpu->clks);
> > +	if (ret) {
> > +		dev_err(dev, "Enabling clocks, fail: %d\n", ret);
> > +		return ret;
> > +	}
> > +
> > +	if (vpu->irq < 0 && !hrtimer_active(&vpu->hrtimer))
> > +		hrtimer_start(&vpu->hrtimer, ns_to_ktime(vpu-
> >vpu_poll_interval * NSEC_PER_MSEC),
> > +				HRTIMER_MODE_REL_PINNED);
> > +
> > +	return ret;
> > +}
> > +
> > +static const struct dev_pm_ops wave5_pm_ops = {
> > +	SET_RUNTIME_PM_OPS(wave5_pm_suspend, wave5_pm_resume, NULL) };
> > +
> >  static int wave5_vpu_probe(struct platform_device *pdev)  {
> >  	int ret;
> > @@ -281,6 +321,12 @@ static int wave5_vpu_probe(struct platform_device
> *pdev)
> >  		 (match_data->flags & WAVE5_IS_DEC) ? "'DECODE'" : "");
> >  	dev_info(&pdev->dev, "Product Code:      0x%x\n", dev->product_code);
> >  	dev_info(&pdev->dev, "Firmware Revision: %u\n", fw_revision);
> > +
> > +	pm_runtime_set_autosuspend_delay(&pdev->dev, 100);
> > +	pm_runtime_use_autosuspend(&pdev->dev);
> > +	pm_runtime_enable(&pdev->dev);
> > +	wave5_vpu_sleep_wake(&pdev->dev, true, NULL, 0);
> > +
> >  	return 0;
> >
> >  err_enc_unreg:
> > @@ -310,6 +356,9 @@ static void wave5_vpu_remove(struct platform_device
> *pdev)
> >  		hrtimer_cancel(&dev->hrtimer);
> >  	}
> >
> > +	pm_runtime_put_sync(&pdev->dev);
> > +	pm_runtime_disable(&pdev->dev);
> > +
> >  	mutex_destroy(&dev->dev_lock);
> >  	mutex_destroy(&dev->hw_lock);
> >  	reset_control_assert(dev->resets);
> > @@ -337,6 +386,7 @@ static struct platform_driver wave5_vpu_driver = {
> >  	.driver = {
> >  		.name = VPU_PLATFORM_DEVICE_NAME,
> >  		.of_match_table = of_match_ptr(wave5_dt_ids),
> > +		.pm = &wave5_pm_ops,
> >  		},
> >  	.probe = wave5_vpu_probe,
> >  	.remove_new = wave5_vpu_remove,
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > index 1a3efb638dde..e16b990041c2 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.c
> > @@ -6,6 +6,8 @@
> >   */
> >
> >  #include <linux/bug.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/delay.h>
> >  #include "wave5-vpuapi.h"
> >  #include "wave5-regdefine.h"
> >  #include "wave5.h"
> > @@ -195,14 +197,20 @@ int wave5_vpu_dec_close(struct vpu_instance *inst,
> u32 *fail_res)
> >  	int retry = 0;
> >  	struct vpu_device *vpu_dev = inst->dev;
> >  	int i;
> > +	int inst_count = 0;
> > +	struct vpu_instance *inst_elm;
> >
> >  	*fail_res = 0;
> >  	if (!inst->codec_info)
> >  		return -EINVAL;
> >
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> > +
> >  	ret = mutex_lock_interruptible(&vpu_dev->hw_lock);
> > -	if (ret)
> > +	if (ret) {
> > +		pm_runtime_put_sync(inst->dev->dev);
> >  		return ret;
> > +	}
> >
> >  	do {
> >  		ret = wave5_vpu_dec_finish_seq(inst, fail_res); @@ -232,9
> +240,14
> > @@ int wave5_vpu_dec_close(struct vpu_instance *inst, u32 *fail_res)
> >
> >  	wave5_vdi_free_dma_memory(vpu_dev, &p_dec_info->vb_task);
> >
> > +	list_for_each_entry(inst_elm, &vpu_dev->instances, list)
> > +		inst_count++;
> > +	if (inst_count == 1)
> > +		pm_runtime_dont_use_autosuspend(vpu_dev->dev);
> > +
> >  unlock_and_return:
> >  	mutex_unlock(&vpu_dev->hw_lock);
> > -
> > +	pm_runtime_put_sync(inst->dev->dev);
> >  	return ret;
> >  }
> >
> > @@ -697,25 +710,33 @@ int wave5_vpu_enc_close(struct vpu_instance *inst,
> u32 *fail_res)
> >  	int ret;
> >  	int retry = 0;
> >  	struct vpu_device *vpu_dev = inst->dev;
> > +	int inst_count = 0;
> > +	struct vpu_instance *inst_elm;
> >
> >  	*fail_res = 0;
> >  	if (!inst->codec_info)
> >  		return -EINVAL;
> >
> > +	pm_runtime_resume_and_get(inst->dev->dev);
> > +
> >  	ret = mutex_lock_interruptible(&vpu_dev->hw_lock);
> > -	if (ret)
> > +	if (ret) {
> > +		pm_runtime_resume_and_get(inst->dev->dev);
> >  		return ret;
> > +	}
> >
> >  	do {
> >  		ret = wave5_vpu_enc_finish_seq(inst, fail_res);
> >  		if (ret < 0 && *fail_res != WAVE5_SYSERR_VPU_STILL_RUNNING)
> {
> >  			dev_warn(inst->dev->dev, "enc_finish_seq timed out\n");
> > +			pm_runtime_resume_and_get(inst->dev->dev);
> >  			mutex_unlock(&vpu_dev->hw_lock);
> >  			return ret;
> >  		}
> >
> >  		if (*fail_res == WAVE5_SYSERR_VPU_STILL_RUNNING &&
> >  		    retry++ >= MAX_FIRMWARE_CALL_RETRY) {
> > +			pm_runtime_resume_and_get(inst->dev->dev);
> >  			mutex_unlock(&vpu_dev->hw_lock);
> >  			return -ETIMEDOUT;
> >  		}
> > @@ -734,7 +755,13 @@ int wave5_vpu_enc_close(struct vpu_instance
> > *inst, u32 *fail_res)
> >
> >  	wave5_vdi_free_dma_memory(vpu_dev, &p_enc_info->vb_task);
> >
> > +	list_for_each_entry(inst_elm, &vpu_dev->instances, list)
> > +		inst_count++;
> > +	if (inst_count == 1)
> > +		pm_runtime_dont_use_autosuspend(vpu_dev->dev);
> > +
> >  	mutex_unlock(&vpu_dev->hw_lock);
> > +	pm_runtime_put_sync(inst->dev->dev);
> >
> >  	return 0;
> >  }
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5.h
> > b/drivers/media/platform/chips-media/wave5/wave5.h
> > index 2a29b9164f97..2caab356f3e1 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5.h
> > +++ b/drivers/media/platform/chips-media/wave5/wave5.h
> > @@ -62,6 +62,9 @@ int wave5_vpu_get_version(struct vpu_device
> > *vpu_dev, u32 *revision);
> >
> >  int wave5_vpu_init(struct device *dev, u8 *fw, size_t size);
> >
> > +int wave5_vpu_sleep_wake(struct device *dev, bool i_sleep_wake, const
> uint16_t *code,
> > +			 size_t size);
> > +
> >  int wave5_vpu_reset(struct device *dev, enum sw_reset_mode
> > reset_mode);
> >
> >  int wave5_vpu_build_up_dec_param(struct vpu_instance *inst, struct
> > dec_open_param *param);

[RESEND PATCH v7 3/4] media: chips-media: wave5: Use helpers to calculate bytesperline and sizeimage.

[Jackson.lee]

Use v4l2-common helper functions to calculate bytesperline and sizeimage,
instead of calculating in a wave5 driver directly.

In case of raw(YUV) v4l2_pix_format, the wave5 driver updates
v4l2_pix_format_mplane struct through v4l2_fill_pixfmt_mp() function.

Encoder and Decoder need same bytesperline and sizeimage values
for same v4l2_pix_format.
So, a wave5_update_pix_fmt is refactored to support both together.

Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
---
 .../platform/chips-media/wave5/wave5-helper.c |  24 ++
 .../platform/chips-media/wave5/wave5-helper.h |   5 +
 .../chips-media/wave5/wave5-vpu-dec.c         | 300 +++++++-----------
 .../chips-media/wave5/wave5-vpu-enc.c         | 197 +++++-------
 .../platform/chips-media/wave5/wave5-vpu.h    |   5 +-
 .../chips-media/wave5/wave5-vpuconfig.h       |  27 +-
 6 files changed, 239 insertions(+), 319 deletions(-)

diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.c b/drivers/media/platform/chips-media/wave5/wave5-helper.c
index a20d84dbe3aa..2c9d8cbca6e4 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
@@ -7,6 +7,8 @@
 
 #include "wave5-helper.h"
 
+#define DEFAULT_BS_SIZE(width, height) ((width) * (height) / 8 * 3)
+
 const char *state_to_str(enum vpu_instance_state state)
 {
 	switch (state) {
@@ -217,3 +219,25 @@ void wave5_return_bufs(struct vb2_queue *q, u32 state)
 		v4l2_m2m_buf_done(vbuf, state);
 	}
 }
+
+void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp,
+			  int pix_fmt_type,
+			  unsigned int width,
+			  unsigned int height,
+			  const struct v4l2_frmsize_stepwise *frmsize)
+{
+	v4l2_apply_frmsize_constraints(&width, &height, frmsize);
+
+	if (pix_fmt_type == VPU_FMT_TYPE_CODEC) {
+		pix_mp->width = width;
+		pix_mp->height = height;
+		pix_mp->num_planes = 1;
+		pix_mp->plane_fmt[0].bytesperline = 0;
+		pix_mp->plane_fmt[0].sizeimage = max(DEFAULT_BS_SIZE(width, height),
+						     pix_mp->plane_fmt[0].sizeimage);
+	} else {
+		v4l2_fill_pixfmt_mp(pix_mp, pix_mp->pixelformat, width, height);
+	}
+	pix_mp->flags = 0;
+	pix_mp->field = V4L2_FIELD_NONE;
+}
diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.h b/drivers/media/platform/chips-media/wave5/wave5-helper.h
index 6cee1c14d3ce..9937fce553fc 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-helper.h
+++ b/drivers/media/platform/chips-media/wave5/wave5-helper.h
@@ -28,4 +28,9 @@ const struct vpu_format *wave5_find_vpu_fmt_by_idx(unsigned int idx,
 						   const struct vpu_format fmt_list[MAX_FMTS]);
 enum wave_std wave5_to_vpu_std(unsigned int v4l2_pix_fmt, enum vpu_instance_type type);
 void wave5_return_bufs(struct vb2_queue *q, u32 state);
+void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp,
+			  int pix_fmt_type,
+			  unsigned int width,
+			  unsigned int height,
+			  const struct v4l2_frmsize_stepwise *frmsize);
 #endif
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
index 698c83e3082e..99e929aab870 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
@@ -11,111 +11,92 @@
 #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
 #define VPU_DEC_DRV_NAME "wave5-dec"
 
-#define DEFAULT_SRC_SIZE(width, height) ({			\
-	(width) * (height) / 8 * 3;					\
-})
+static const struct v4l2_frmsize_stepwise dec_hevc_frmsize = {
+	.min_width = W5_MIN_DEC_PIC_8_WIDTH,
+	.max_width = W5_MAX_DEC_PIC_WIDTH,
+	.step_width = W5_DEC_CODEC_STEP_WIDTH,
+	.min_height = W5_MIN_DEC_PIC_8_HEIGHT,
+	.max_height = W5_MAX_DEC_PIC_HEIGHT,
+	.step_height = W5_DEC_CODEC_STEP_HEIGHT,
+};
+
+static const struct v4l2_frmsize_stepwise dec_h264_frmsize = {
+	.min_width = W5_MIN_DEC_PIC_32_WIDTH,
+	.max_width = W5_MAX_DEC_PIC_WIDTH,
+	.step_width = W5_DEC_CODEC_STEP_WIDTH,
+	.min_height = W5_MIN_DEC_PIC_32_HEIGHT,
+	.max_height = W5_MAX_DEC_PIC_HEIGHT,
+	.step_height = W5_DEC_CODEC_STEP_HEIGHT,
+};
+
+static const struct v4l2_frmsize_stepwise dec_raw_frmsize = {
+	.min_width = W5_MIN_DEC_PIC_8_WIDTH,
+	.max_width = W5_MAX_DEC_PIC_WIDTH,
+	.step_width = W5_DEC_RAW_STEP_WIDTH,
+	.min_height = W5_MIN_DEC_PIC_8_HEIGHT,
+	.max_height = W5_MAX_DEC_PIC_HEIGHT,
+	.step_height = W5_DEC_RAW_STEP_HEIGHT,
+};
 
 static const struct vpu_format dec_fmt_list[FMT_TYPES][MAX_FMTS] = {
 	[VPU_FMT_TYPE_CODEC] = {
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_HEVC,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_hevc_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_H264,
-			.max_width = 8192,
-			.min_width = 32,
-			.max_height = 4320,
-			.min_height = 32,
+			.v4l2_frmsize = &dec_h264_frmsize,
 		},
 	},
 	[VPU_FMT_TYPE_RAW] = {
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV422P,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV16,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV61,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420M,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12M,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21M,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV422M,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV16M,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV61M,
-			.max_width = 8192,
-			.min_width = 8,
-			.max_height = 4320,
-			.min_height = 8,
+			.v4l2_frmsize = &dec_raw_frmsize,
 		},
 	}
 };
@@ -234,74 +215,6 @@ static void wave5_handle_src_buffer(struct vpu_instance *inst, dma_addr_t rd_ptr
 	inst->remaining_consumed_bytes = consumed_bytes;
 }
 
-static void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp, unsigned int width,
-				 unsigned int height)
-{
-	switch (pix_mp->pixelformat) {
-	case V4L2_PIX_FMT_YUV420:
-	case V4L2_PIX_FMT_NV12:
-	case V4L2_PIX_FMT_NV21:
-		pix_mp->width = round_up(width, 32);
-		pix_mp->height = round_up(height, 16);
-		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[0].sizeimage = width * height * 3 / 2;
-		break;
-	case V4L2_PIX_FMT_YUV422P:
-	case V4L2_PIX_FMT_NV16:
-	case V4L2_PIX_FMT_NV61:
-		pix_mp->width = round_up(width, 32);
-		pix_mp->height = round_up(height, 16);
-		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[0].sizeimage = width * height * 2;
-		break;
-	case V4L2_PIX_FMT_YUV420M:
-		pix_mp->width = round_up(width, 32);
-		pix_mp->height = round_up(height, 16);
-		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[0].sizeimage = width * height;
-		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32) / 2;
-		pix_mp->plane_fmt[1].sizeimage = width * height / 4;
-		pix_mp->plane_fmt[2].bytesperline = round_up(width, 32) / 2;
-		pix_mp->plane_fmt[2].sizeimage = width * height / 4;
-		break;
-	case V4L2_PIX_FMT_NV12M:
-	case V4L2_PIX_FMT_NV21M:
-		pix_mp->width = round_up(width, 32);
-		pix_mp->height = round_up(height, 16);
-		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[0].sizeimage = width * height;
-		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[1].sizeimage = width * height / 2;
-		break;
-	case V4L2_PIX_FMT_YUV422M:
-		pix_mp->width = round_up(width, 32);
-		pix_mp->height = round_up(height, 16);
-		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[0].sizeimage = width * height;
-		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32) / 2;
-		pix_mp->plane_fmt[1].sizeimage = width * height / 2;
-		pix_mp->plane_fmt[2].bytesperline = round_up(width, 32) / 2;
-		pix_mp->plane_fmt[2].sizeimage = width * height / 2;
-		break;
-	case V4L2_PIX_FMT_NV16M:
-	case V4L2_PIX_FMT_NV61M:
-		pix_mp->width = round_up(width, 32);
-		pix_mp->height = round_up(height, 16);
-		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[0].sizeimage = width * height;
-		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[1].sizeimage = width * height;
-		break;
-	default:
-		pix_mp->width = width;
-		pix_mp->height = height;
-		pix_mp->plane_fmt[0].bytesperline = 0;
-		pix_mp->plane_fmt[0].sizeimage = max(DEFAULT_SRC_SIZE(width, height),
-						     pix_mp->plane_fmt[0].sizeimage);
-		break;
-	}
-}
-
 static int start_decode(struct vpu_instance *inst, u32 *fail_res)
 {
 	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
@@ -389,6 +302,8 @@ static int handle_dynamic_resolution_change(struct vpu_instance *inst)
 	}
 
 	if (p_dec_info->initial_info_obtained) {
+		const struct vpu_format *vpu_fmt;
+
 		inst->conf_win.left = initial_info->pic_crop_rect.left;
 		inst->conf_win.top = initial_info->pic_crop_rect.top;
 		inst->conf_win.width = initial_info->pic_width -
@@ -396,10 +311,27 @@ static int handle_dynamic_resolution_change(struct vpu_instance *inst)
 		inst->conf_win.height = initial_info->pic_height -
 			initial_info->pic_crop_rect.top - initial_info->pic_crop_rect.bottom;
 
-		wave5_update_pix_fmt(&inst->src_fmt, initial_info->pic_width,
-				     initial_info->pic_height);
-		wave5_update_pix_fmt(&inst->dst_fmt, initial_info->pic_width,
-				     initial_info->pic_height);
+		vpu_fmt = wave5_find_vpu_fmt(inst->src_fmt.pixelformat,
+					     dec_fmt_list[VPU_FMT_TYPE_CODEC]);
+		if (!vpu_fmt)
+			return -EINVAL;
+
+		wave5_update_pix_fmt(&inst->src_fmt,
+				     VPU_FMT_TYPE_CODEC,
+				     initial_info->pic_width,
+				     initial_info->pic_height,
+				     vpu_fmt->v4l2_frmsize);
+
+		vpu_fmt = wave5_find_vpu_fmt(inst->dst_fmt.pixelformat,
+					     dec_fmt_list[VPU_FMT_TYPE_RAW]);
+		if (!vpu_fmt)
+			return -EINVAL;
+
+		wave5_update_pix_fmt(&inst->dst_fmt,
+				     VPU_FMT_TYPE_RAW,
+				     initial_info->pic_width,
+				     initial_info->pic_height,
+				     vpu_fmt->v4l2_frmsize);
 	}
 
 	v4l2_event_queue_fh(fh, &vpu_event_src_ch);
@@ -548,12 +480,12 @@ static int wave5_vpu_dec_enum_framesizes(struct file *f, void *fh, struct v4l2_f
 	}
 
 	fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
-	fsize->stepwise.min_width = vpu_fmt->min_width;
-	fsize->stepwise.max_width = vpu_fmt->max_width;
-	fsize->stepwise.step_width = 1;
-	fsize->stepwise.min_height = vpu_fmt->min_height;
-	fsize->stepwise.max_height = vpu_fmt->max_height;
-	fsize->stepwise.step_height = 1;
+	fsize->stepwise.min_width = vpu_fmt->v4l2_frmsize->min_width;
+	fsize->stepwise.max_width = vpu_fmt->v4l2_frmsize->max_width;
+	fsize->stepwise.step_width = W5_DEC_CODEC_STEP_WIDTH;
+	fsize->stepwise.min_height = vpu_fmt->v4l2_frmsize->min_height;
+	fsize->stepwise.max_height = vpu_fmt->v4l2_frmsize->max_height;
+	fsize->stepwise.step_height = W5_DEC_CODEC_STEP_HEIGHT;
 
 	return 0;
 }
@@ -576,6 +508,7 @@ static int wave5_vpu_dec_try_fmt_cap(struct file *file, void *fh, struct v4l2_fo
 {
 	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
 	struct dec_info *p_dec_info = &inst->codec_info->dec_info;
+	const struct v4l2_frmsize_stepwise *frmsize;
 	const struct vpu_format *vpu_fmt;
 	int width, height;
 
@@ -589,14 +522,12 @@ static int wave5_vpu_dec_try_fmt_cap(struct file *file, void *fh, struct v4l2_fo
 		width = inst->dst_fmt.width;
 		height = inst->dst_fmt.height;
 		f->fmt.pix_mp.pixelformat = inst->dst_fmt.pixelformat;
-		f->fmt.pix_mp.num_planes = inst->dst_fmt.num_planes;
+		frmsize = &dec_raw_frmsize;
 	} else {
-		const struct v4l2_format_info *info = v4l2_format_info(vpu_fmt->v4l2_pix_fmt);
-
-		width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width, vpu_fmt->max_width);
-		height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height, vpu_fmt->max_height);
+		width = f->fmt.pix_mp.width;
+		height = f->fmt.pix_mp.height;
 		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
-		f->fmt.pix_mp.num_planes = info->mem_planes;
+		frmsize = vpu_fmt->v4l2_frmsize;
 	}
 
 	if (p_dec_info->initial_info_obtained) {
@@ -604,9 +535,8 @@ static int wave5_vpu_dec_try_fmt_cap(struct file *file, void *fh, struct v4l2_fo
 		height = inst->dst_fmt.height;
 	}
 
-	wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
-	f->fmt.pix_mp.flags = 0;
-	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
+	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_RAW,
+			     width, height, frmsize);
 	f->fmt.pix_mp.colorspace = inst->colorspace;
 	f->fmt.pix_mp.ycbcr_enc = inst->ycbcr_enc;
 	f->fmt.pix_mp.quantization = inst->quantization;
@@ -718,7 +648,9 @@ static int wave5_vpu_dec_enum_fmt_out(struct file *file, void *fh, struct v4l2_f
 static int wave5_vpu_dec_try_fmt_out(struct file *file, void *fh, struct v4l2_format *f)
 {
 	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
+	const struct v4l2_frmsize_stepwise *frmsize;
 	const struct vpu_format *vpu_fmt;
+	int width, height;
 
 	dev_dbg(inst->dev->dev,
 		"%s: fourcc: %u width: %u height: %u num_planes: %u colorspace: %u field: %u\n",
@@ -727,20 +659,19 @@ static int wave5_vpu_dec_try_fmt_out(struct file *file, void *fh, struct v4l2_fo
 
 	vpu_fmt = wave5_find_vpu_fmt(f->fmt.pix_mp.pixelformat, dec_fmt_list[VPU_FMT_TYPE_CODEC]);
 	if (!vpu_fmt) {
+		width = inst->src_fmt.width;
+		height = inst->src_fmt.height;
 		f->fmt.pix_mp.pixelformat = inst->src_fmt.pixelformat;
-		f->fmt.pix_mp.num_planes = inst->src_fmt.num_planes;
-		wave5_update_pix_fmt(&f->fmt.pix_mp, inst->src_fmt.width, inst->src_fmt.height);
+		frmsize = &dec_hevc_frmsize;
 	} else {
-		int width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width, vpu_fmt->max_width);
-		int height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height, vpu_fmt->max_height);
-
+		width = f->fmt.pix_mp.width;
+		height = f->fmt.pix_mp.height;
 		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
-		f->fmt.pix_mp.num_planes = 1;
-		wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
+		frmsize = vpu_fmt->v4l2_frmsize;
 	}
 
-	f->fmt.pix_mp.flags = 0;
-	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
+	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_CODEC,
+			     width, height, frmsize);
 
 	return 0;
 }
@@ -748,6 +679,7 @@ static int wave5_vpu_dec_try_fmt_out(struct file *file, void *fh, struct v4l2_fo
 static int wave5_vpu_dec_s_fmt_out(struct file *file, void *fh, struct v4l2_format *f)
 {
 	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
+	const struct vpu_format *vpu_fmt;
 	int i, ret;
 
 	dev_dbg(inst->dev->dev,
@@ -782,7 +714,13 @@ static int wave5_vpu_dec_s_fmt_out(struct file *file, void *fh, struct v4l2_form
 	inst->quantization = f->fmt.pix_mp.quantization;
 	inst->xfer_func = f->fmt.pix_mp.xfer_func;
 
-	wave5_update_pix_fmt(&inst->dst_fmt, f->fmt.pix_mp.width, f->fmt.pix_mp.height);
+	vpu_fmt = wave5_find_vpu_fmt(inst->dst_fmt.pixelformat, dec_fmt_list[VPU_FMT_TYPE_RAW]);
+	if (!vpu_fmt)
+		return -EINVAL;
+
+	wave5_update_pix_fmt(&inst->dst_fmt, VPU_FMT_TYPE_RAW,
+			     f->fmt.pix_mp.width, f->fmt.pix_mp.height,
+			     vpu_fmt->v4l2_frmsize);
 
 	return 0;
 }
@@ -1005,6 +943,7 @@ static int wave5_vpu_dec_queue_setup(struct vb2_queue *q, unsigned int *num_buff
 	struct vpu_instance *inst = vb2_get_drv_priv(q);
 	struct v4l2_pix_format_mplane inst_format =
 		(q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) ? inst->src_fmt : inst->dst_fmt;
+	unsigned int i;
 
 	dev_dbg(inst->dev->dev, "%s: num_buffers: %u | num_planes: %u | type: %u\n", __func__,
 		*num_buffers, *num_planes, q->type);
@@ -1018,31 +957,9 @@ static int wave5_vpu_dec_queue_setup(struct vb2_queue *q, unsigned int *num_buff
 		if (*num_buffers < inst->fbc_buf_count)
 			*num_buffers = inst->fbc_buf_count;
 
-		if (*num_planes == 1) {
-			if (inst->output_format == FORMAT_422)
-				sizes[0] = inst_format.width * inst_format.height * 2;
-			else
-				sizes[0] = inst_format.width * inst_format.height * 3 / 2;
-			dev_dbg(inst->dev->dev, "%s: size[0]: %u\n", __func__, sizes[0]);
-		} else if (*num_planes == 2) {
-			sizes[0] = inst_format.width * inst_format.height;
-			if (inst->output_format == FORMAT_422)
-				sizes[1] = inst_format.width * inst_format.height;
-			else
-				sizes[1] = inst_format.width * inst_format.height / 2;
-			dev_dbg(inst->dev->dev, "%s: size[0]: %u | size[1]: %u\n",
-				__func__, sizes[0], sizes[1]);
-		} else if (*num_planes == 3) {
-			sizes[0] = inst_format.width * inst_format.height;
-			if (inst->output_format == FORMAT_422) {
-				sizes[1] = inst_format.width * inst_format.height / 2;
-				sizes[2] = inst_format.width * inst_format.height / 2;
-			} else {
-				sizes[1] = inst_format.width * inst_format.height / 4;
-				sizes[2] = inst_format.width * inst_format.height / 4;
-			}
-			dev_dbg(inst->dev->dev, "%s: size[0]: %u | size[1]: %u | size[2]: %u\n",
-				__func__, sizes[0], sizes[1], sizes[2]);
+		for (i = 0; i < *num_planes; i++) {
+			sizes[i] = inst_format.plane_fmt[i].sizeimage;
+			dev_dbg(inst->dev->dev, "%s: size[%u]: %u\n", __func__, i, sizes[i]);
 		}
 	}
 
@@ -1568,20 +1485,15 @@ static const struct vb2_ops wave5_vpu_dec_vb2_ops = {
 static void wave5_set_default_format(struct v4l2_pix_format_mplane *src_fmt,
 				     struct v4l2_pix_format_mplane *dst_fmt)
 {
-	unsigned int dst_pix_fmt = dec_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
-	const struct v4l2_format_info *dst_fmt_info = v4l2_format_info(dst_pix_fmt);
-
 	src_fmt->pixelformat = dec_fmt_list[VPU_FMT_TYPE_CODEC][0].v4l2_pix_fmt;
-	src_fmt->field = V4L2_FIELD_NONE;
-	src_fmt->flags = 0;
-	src_fmt->num_planes = 1;
-	wave5_update_pix_fmt(src_fmt, 720, 480);
-
-	dst_fmt->pixelformat = dst_pix_fmt;
-	dst_fmt->field = V4L2_FIELD_NONE;
-	dst_fmt->flags = 0;
-	dst_fmt->num_planes = dst_fmt_info->mem_planes;
-	wave5_update_pix_fmt(dst_fmt, 736, 480);
+	wave5_update_pix_fmt(src_fmt, VPU_FMT_TYPE_CODEC,
+			     W5_DEF_DEC_PIC_WIDTH, W5_DEF_DEC_PIC_HEIGHT,
+			     &dec_hevc_frmsize);
+
+	dst_fmt->pixelformat = dec_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
+	wave5_update_pix_fmt(dst_fmt, VPU_FMT_TYPE_RAW,
+			     W5_DEF_DEC_PIC_WIDTH, W5_DEF_DEC_PIC_HEIGHT,
+			     &dec_raw_frmsize);
 }
 
 static int wave5_vpu_dec_queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq)
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
index 985de0c293e2..ef9aa1562346 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
@@ -11,65 +11,60 @@
 #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
 #define VPU_ENC_DRV_NAME "wave5-enc"
 
+static const struct v4l2_frmsize_stepwise enc_frmsize[FMT_TYPES] = {
+	[VPU_FMT_TYPE_CODEC] = {
+		.min_width = W5_MIN_ENC_PIC_WIDTH,
+		.max_width = W5_MAX_ENC_PIC_WIDTH,
+		.step_width = W5_ENC_CODEC_STEP_WIDTH,
+		.min_height = W5_MIN_ENC_PIC_HEIGHT,
+		.max_height = W5_MAX_ENC_PIC_HEIGHT,
+		.step_height = W5_ENC_CODEC_STEP_HEIGHT,
+	},
+	[VPU_FMT_TYPE_RAW] = {
+		.min_width = W5_MIN_ENC_PIC_WIDTH,
+		.max_width = W5_MAX_ENC_PIC_WIDTH,
+		.step_width = W5_ENC_RAW_STEP_WIDTH,
+		.min_height = W5_MIN_ENC_PIC_HEIGHT,
+		.max_height = W5_MAX_ENC_PIC_HEIGHT,
+		.step_height = W5_ENC_RAW_STEP_HEIGHT,
+	},
+};
+
 static const struct vpu_format enc_fmt_list[FMT_TYPES][MAX_FMTS] = {
 	[VPU_FMT_TYPE_CODEC] = {
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_HEVC,
-			.max_width = W5_MAX_ENC_PIC_WIDTH,
-			.min_width = W5_MIN_ENC_PIC_WIDTH,
-			.max_height = W5_MAX_ENC_PIC_HEIGHT,
-			.min_height = W5_MIN_ENC_PIC_HEIGHT,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_CODEC],
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_H264,
-			.max_width = W5_MAX_ENC_PIC_WIDTH,
-			.min_width = W5_MIN_ENC_PIC_WIDTH,
-			.max_height = W5_MAX_ENC_PIC_HEIGHT,
-			.min_height = W5_MIN_ENC_PIC_HEIGHT,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_CODEC],
 		},
 	},
 	[VPU_FMT_TYPE_RAW] = {
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420,
-			.max_width = W5_MAX_ENC_PIC_WIDTH,
-			.min_width = W5_MIN_ENC_PIC_WIDTH,
-			.max_height = W5_MAX_ENC_PIC_HEIGHT,
-			.min_height = W5_MIN_ENC_PIC_HEIGHT,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12,
-			.max_width = W5_MAX_ENC_PIC_WIDTH,
-			.min_width = W5_MIN_ENC_PIC_WIDTH,
-			.max_height = W5_MAX_ENC_PIC_HEIGHT,
-			.min_height = W5_MIN_ENC_PIC_HEIGHT,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21,
-			.max_width = W5_MAX_ENC_PIC_WIDTH,
-			.min_width = W5_MIN_ENC_PIC_WIDTH,
-			.max_height = W5_MAX_ENC_PIC_HEIGHT,
-			.min_height = W5_MIN_ENC_PIC_HEIGHT,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420M,
-			.max_width = W5_MAX_ENC_PIC_WIDTH,
-			.min_width = W5_MIN_ENC_PIC_WIDTH,
-			.max_height = W5_MAX_ENC_PIC_HEIGHT,
-			.min_height = W5_MIN_ENC_PIC_HEIGHT,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12M,
-			.max_width = W5_MAX_ENC_PIC_WIDTH,
-			.min_width = W5_MIN_ENC_PIC_WIDTH,
-			.max_height = W5_MAX_ENC_PIC_HEIGHT,
-			.min_height = W5_MIN_ENC_PIC_HEIGHT,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
 		},
 		{
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21M,
-			.max_width = W5_MAX_ENC_PIC_WIDTH,
-			.min_width = W5_MIN_ENC_PIC_WIDTH,
-			.max_height = W5_MAX_ENC_PIC_HEIGHT,
-			.min_height = W5_MIN_ENC_PIC_HEIGHT,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
 		},
 	}
 };
@@ -106,46 +101,6 @@ static int switch_state(struct vpu_instance *inst, enum vpu_instance_state state
 	return -EINVAL;
 }
 
-static void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp, unsigned int width,
-				 unsigned int height)
-{
-	switch (pix_mp->pixelformat) {
-	case V4L2_PIX_FMT_YUV420:
-	case V4L2_PIX_FMT_NV12:
-	case V4L2_PIX_FMT_NV21:
-		pix_mp->width = width;
-		pix_mp->height = height;
-		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[0].sizeimage = round_up(width, 32) * height * 3 / 2;
-		break;
-	case V4L2_PIX_FMT_YUV420M:
-		pix_mp->width = width;
-		pix_mp->height = height;
-		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[0].sizeimage = round_up(width, 32) * height;
-		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32) / 2;
-		pix_mp->plane_fmt[1].sizeimage = round_up(width, 32) * height / 4;
-		pix_mp->plane_fmt[2].bytesperline = round_up(width, 32) / 2;
-		pix_mp->plane_fmt[2].sizeimage = round_up(width, 32) * height / 4;
-		break;
-	case V4L2_PIX_FMT_NV12M:
-	case V4L2_PIX_FMT_NV21M:
-		pix_mp->width = width;
-		pix_mp->height = height;
-		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[0].sizeimage = round_up(width, 32) * height;
-		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32);
-		pix_mp->plane_fmt[1].sizeimage = round_up(width, 32) * height / 2;
-		break;
-	default:
-		pix_mp->width = width;
-		pix_mp->height = height;
-		pix_mp->plane_fmt[0].bytesperline = 0;
-		pix_mp->plane_fmt[0].sizeimage = width * height / 8 * 3;
-		break;
-	}
-}
-
 static int start_encode(struct vpu_instance *inst, u32 *fail_res)
 {
 	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
@@ -360,13 +315,8 @@ static int wave5_vpu_enc_enum_framesizes(struct file *f, void *fh, struct v4l2_f
 			return -EINVAL;
 	}
 
-	fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
-	fsize->stepwise.min_width = vpu_fmt->min_width;
-	fsize->stepwise.max_width = vpu_fmt->max_width;
-	fsize->stepwise.step_width = 1;
-	fsize->stepwise.min_height = vpu_fmt->min_height;
-	fsize->stepwise.max_height = vpu_fmt->max_height;
-	fsize->stepwise.step_height = 1;
+	fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
+	fsize->stepwise = enc_frmsize[VPU_FMT_TYPE_CODEC];
 
 	return 0;
 }
@@ -391,7 +341,9 @@ static int wave5_vpu_enc_enum_fmt_cap(struct file *file, void *fh, struct v4l2_f
 static int wave5_vpu_enc_try_fmt_cap(struct file *file, void *fh, struct v4l2_format *f)
 {
 	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
+	const struct v4l2_frmsize_stepwise *frmsize;
 	const struct vpu_format *vpu_fmt;
+	int width, height;
 
 	dev_dbg(inst->dev->dev, "%s: fourcc: %u width: %u height: %u num_planes: %u field: %u\n",
 		__func__, f->fmt.pix_mp.pixelformat, f->fmt.pix_mp.width, f->fmt.pix_mp.height,
@@ -399,20 +351,19 @@ static int wave5_vpu_enc_try_fmt_cap(struct file *file, void *fh, struct v4l2_fo
 
 	vpu_fmt = wave5_find_vpu_fmt(f->fmt.pix_mp.pixelformat, enc_fmt_list[VPU_FMT_TYPE_CODEC]);
 	if (!vpu_fmt) {
+		width = inst->dst_fmt.width;
+		height = inst->dst_fmt.height;
 		f->fmt.pix_mp.pixelformat = inst->dst_fmt.pixelformat;
-		f->fmt.pix_mp.num_planes = inst->dst_fmt.num_planes;
-		wave5_update_pix_fmt(&f->fmt.pix_mp, inst->dst_fmt.width, inst->dst_fmt.height);
+		frmsize = &enc_frmsize[VPU_FMT_TYPE_CODEC];
 	} else {
-		int width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width, vpu_fmt->max_width);
-		int height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height, vpu_fmt->max_height);
-
+		width = f->fmt.pix_mp.width;
+		height = f->fmt.pix_mp.height;
 		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
-		f->fmt.pix_mp.num_planes = 1;
-		wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
+		frmsize = vpu_fmt->v4l2_frmsize;
 	}
 
-	f->fmt.pix_mp.flags = 0;
-	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
+	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_CODEC,
+			     width, height, frmsize);
 	f->fmt.pix_mp.colorspace = inst->colorspace;
 	f->fmt.pix_mp.ycbcr_enc = inst->ycbcr_enc;
 	f->fmt.pix_mp.quantization = inst->quantization;
@@ -499,7 +450,9 @@ static int wave5_vpu_enc_enum_fmt_out(struct file *file, void *fh, struct v4l2_f
 static int wave5_vpu_enc_try_fmt_out(struct file *file, void *fh, struct v4l2_format *f)
 {
 	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
+	const struct v4l2_frmsize_stepwise *frmsize;
 	const struct vpu_format *vpu_fmt;
+	int width, height;
 
 	dev_dbg(inst->dev->dev, "%s: fourcc: %u width: %u height: %u num_planes: %u field: %u\n",
 		__func__, f->fmt.pix_mp.pixelformat, f->fmt.pix_mp.width, f->fmt.pix_mp.height,
@@ -507,28 +460,26 @@ static int wave5_vpu_enc_try_fmt_out(struct file *file, void *fh, struct v4l2_fo
 
 	vpu_fmt = wave5_find_vpu_fmt(f->fmt.pix_mp.pixelformat, enc_fmt_list[VPU_FMT_TYPE_RAW]);
 	if (!vpu_fmt) {
+		width = inst->src_fmt.width;
+		height = inst->src_fmt.height;
 		f->fmt.pix_mp.pixelformat = inst->src_fmt.pixelformat;
-		f->fmt.pix_mp.num_planes = inst->src_fmt.num_planes;
-		wave5_update_pix_fmt(&f->fmt.pix_mp, inst->src_fmt.width, inst->src_fmt.height);
+		frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW];
 	} else {
-		int width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width, vpu_fmt->max_width);
-		int height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height, vpu_fmt->max_height);
-		const struct v4l2_format_info *info = v4l2_format_info(vpu_fmt->v4l2_pix_fmt);
-
+		width = f->fmt.pix_mp.width;
+		height = f->fmt.pix_mp.height;
 		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
-		f->fmt.pix_mp.num_planes = info->mem_planes;
-		wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
+		frmsize = vpu_fmt->v4l2_frmsize;
 	}
 
-	f->fmt.pix_mp.flags = 0;
-	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
-
+	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_RAW,
+			     width, height, frmsize);
 	return 0;
 }
 
 static int wave5_vpu_enc_s_fmt_out(struct file *file, void *fh, struct v4l2_format *f)
 {
 	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
+	const struct vpu_format *vpu_fmt;
 	int i, ret;
 
 	dev_dbg(inst->dev->dev, "%s: fourcc: %u width: %u height: %u num_planes: %u field: %u\n",
@@ -568,7 +519,15 @@ static int wave5_vpu_enc_s_fmt_out(struct file *file, void *fh, struct v4l2_form
 	inst->quantization = f->fmt.pix_mp.quantization;
 	inst->xfer_func = f->fmt.pix_mp.xfer_func;
 
-	wave5_update_pix_fmt(&inst->dst_fmt, f->fmt.pix_mp.width, f->fmt.pix_mp.height);
+	vpu_fmt = wave5_find_vpu_fmt(inst->dst_fmt.pixelformat, enc_fmt_list[VPU_FMT_TYPE_CODEC]);
+	if (!vpu_fmt)
+		return -EINVAL;
+
+	wave5_update_pix_fmt(&inst->dst_fmt, VPU_FMT_TYPE_CODEC,
+			     f->fmt.pix_mp.width, f->fmt.pix_mp.height,
+			     vpu_fmt->v4l2_frmsize);
+	inst->conf_win.width = inst->dst_fmt.width;
+	inst->conf_win.height = inst->dst_fmt.height;
 
 	return 0;
 }
@@ -584,12 +543,17 @@ static int wave5_vpu_enc_g_selection(struct file *file, void *fh, struct v4l2_se
 	switch (s->target) {
 	case V4L2_SEL_TGT_CROP_DEFAULT:
 	case V4L2_SEL_TGT_CROP_BOUNDS:
-	case V4L2_SEL_TGT_CROP:
 		s->r.left = 0;
 		s->r.top = 0;
 		s->r.width = inst->dst_fmt.width;
 		s->r.height = inst->dst_fmt.height;
 		break;
+	case V4L2_SEL_TGT_CROP:
+		s->r.left = 0;
+		s->r.top = 0;
+		s->r.width = inst->conf_win.width;
+		s->r.height = inst->conf_win.height;
+		break;
 	default:
 		return -EINVAL;
 	}
@@ -612,8 +576,10 @@ static int wave5_vpu_enc_s_selection(struct file *file, void *fh, struct v4l2_se
 
 	s->r.left = 0;
 	s->r.top = 0;
-	s->r.width = inst->src_fmt.width;
-	s->r.height = inst->src_fmt.height;
+	s->r.width = min(s->r.width, inst->dst_fmt.width);
+	s->r.height = min(s->r.height, inst->dst_fmt.height);
+
+	inst->conf_win = s->r;
 
 	return 0;
 }
@@ -1151,8 +1117,8 @@ static void wave5_set_enc_openparam(struct enc_open_param *open_param,
 	open_param->wave_param.lambda_scaling_enable = 1;
 
 	open_param->line_buf_int_en = true;
-	open_param->pic_width = inst->dst_fmt.width;
-	open_param->pic_height = inst->dst_fmt.height;
+	open_param->pic_width = inst->conf_win.width;
+	open_param->pic_height = inst->conf_win.height;
 	open_param->frame_rate_info = inst->frame_rate;
 	open_param->rc_enable = inst->rc_enable;
 	if (inst->rc_enable) {
@@ -1456,20 +1422,15 @@ static const struct vb2_ops wave5_vpu_enc_vb2_ops = {
 static void wave5_set_default_format(struct v4l2_pix_format_mplane *src_fmt,
 				     struct v4l2_pix_format_mplane *dst_fmt)
 {
-	unsigned int src_pix_fmt = enc_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
-	const struct v4l2_format_info *src_fmt_info = v4l2_format_info(src_pix_fmt);
-
-	src_fmt->pixelformat = src_pix_fmt;
-	src_fmt->field = V4L2_FIELD_NONE;
-	src_fmt->flags = 0;
-	src_fmt->num_planes = src_fmt_info->mem_planes;
-	wave5_update_pix_fmt(src_fmt, 416, 240);
+	src_fmt->pixelformat = enc_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
+	wave5_update_pix_fmt(src_fmt, VPU_FMT_TYPE_RAW,
+			     W5_DEF_ENC_PIC_WIDTH, W5_DEF_ENC_PIC_HEIGHT,
+			     &enc_frmsize[VPU_FMT_TYPE_RAW]);
 
 	dst_fmt->pixelformat = enc_fmt_list[VPU_FMT_TYPE_CODEC][0].v4l2_pix_fmt;
-	dst_fmt->field = V4L2_FIELD_NONE;
-	dst_fmt->flags = 0;
-	dst_fmt->num_planes = 1;
-	wave5_update_pix_fmt(dst_fmt, 416, 240);
+	wave5_update_pix_fmt(dst_fmt, VPU_FMT_TYPE_CODEC,
+			     W5_DEF_ENC_PIC_WIDTH, W5_DEF_ENC_PIC_HEIGHT,
+			     &enc_frmsize[VPU_FMT_TYPE_CODEC]);
 }
 
 static int wave5_vpu_enc_queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq)
@@ -1733,6 +1694,8 @@ static int wave5_vpu_open_enc(struct file *filp)
 	v4l2_ctrl_handler_setup(v4l2_ctrl_hdl);
 
 	wave5_set_default_format(&inst->src_fmt, &inst->dst_fmt);
+	inst->conf_win.width = inst->dst_fmt.width;
+	inst->conf_win.height = inst->dst_fmt.height;
 	inst->colorspace = V4L2_COLORSPACE_REC709;
 	inst->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
 	inst->quantization = V4L2_QUANTIZATION_DEFAULT;
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu.h b/drivers/media/platform/chips-media/wave5/wave5-vpu.h
index 32b7fd3730b5..3847332551fc 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu.h
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.h
@@ -38,10 +38,7 @@ enum vpu_fmt_type {
 
 struct vpu_format {
 	unsigned int v4l2_pix_fmt;
-	unsigned int max_width;
-	unsigned int min_width;
-	unsigned int max_height;
-	unsigned int min_height;
+	const struct v4l2_frmsize_stepwise *v4l2_frmsize;
 };
 
 static inline struct vpu_instance *wave5_to_vpu_inst(struct v4l2_fh *vfh)
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h b/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
index e4bc2e467cb5..1ea9f5f31499 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
@@ -32,10 +32,29 @@
 
 #define MAX_NUM_INSTANCE                32
 
-#define W5_MIN_ENC_PIC_WIDTH            256
-#define W5_MIN_ENC_PIC_HEIGHT           128
-#define W5_MAX_ENC_PIC_WIDTH            8192
-#define W5_MAX_ENC_PIC_HEIGHT           8192
+#define W5_DEF_DEC_PIC_WIDTH            720U
+#define W5_DEF_DEC_PIC_HEIGHT           480U
+#define W5_MIN_DEC_PIC_8_WIDTH          8U
+#define W5_MIN_DEC_PIC_8_HEIGHT         8U
+#define W5_MIN_DEC_PIC_32_WIDTH         32U
+#define W5_MIN_DEC_PIC_32_HEIGHT        32U
+#define W5_MAX_DEC_PIC_WIDTH            8192U
+#define W5_MAX_DEC_PIC_HEIGHT           4320U
+#define W5_DEC_CODEC_STEP_WIDTH         1U
+#define W5_DEC_CODEC_STEP_HEIGHT        1U
+#define W5_DEC_RAW_STEP_WIDTH           32U
+#define W5_DEC_RAW_STEP_HEIGHT          16U
+
+#define W5_DEF_ENC_PIC_WIDTH            416U
+#define W5_DEF_ENC_PIC_HEIGHT           240U
+#define W5_MIN_ENC_PIC_WIDTH            256U
+#define W5_MIN_ENC_PIC_HEIGHT           128U
+#define W5_MAX_ENC_PIC_WIDTH            8192U
+#define W5_MAX_ENC_PIC_HEIGHT           8192U
+#define W5_ENC_CODEC_STEP_WIDTH         8U
+#define W5_ENC_CODEC_STEP_HEIGHT        8U
+#define W5_ENC_RAW_STEP_WIDTH           32U
+#define W5_ENC_RAW_STEP_HEIGHT          16U
 
 //  application specific configuration
 #define VPU_ENC_TIMEOUT                 60000
-- 
2.43.0

Re: [RESEND PATCH v7 3/4] media: chips-media: wave5: Use helpers to calculate bytesperline and sizeimage.

[nicolas.dufresne]

Hi,

Le lundi 12 août 2024 à 16:08 +0900, Jackson.lee a écrit :
> Use v4l2-common helper functions to calculate bytesperline and sizeimage,
> instead of calculating in a wave5 driver directly.
> 
> In case of raw(YUV) v4l2_pix_format, the wave5 driver updates
> v4l2_pix_format_mplane struct through v4l2_fill_pixfmt_mp() function.
> 
> Encoder and Decoder need same bytesperline and sizeimage values

                           *the* same

> for same v4l2_pix_format.

  *the* same

> So, a wave5_update_pix_fmt is refactored to support both together.

  So, wave5_update_pix_fmt function ... 

If you don't mind, I'll apply these edits, and reflow to 75 charaters.

Nicolas

> 
> Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
> Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
> Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
> ---
>  .../platform/chips-media/wave5/wave5-helper.c |  24 ++
>  .../platform/chips-media/wave5/wave5-helper.h |   5 +
>  .../chips-media/wave5/wave5-vpu-dec.c         | 300 +++++++-----------
>  .../chips-media/wave5/wave5-vpu-enc.c         | 197 +++++-------
>  .../platform/chips-media/wave5/wave5-vpu.h    |   5 +-
>  .../chips-media/wave5/wave5-vpuconfig.h       |  27 +-
>  6 files changed, 239 insertions(+), 319 deletions(-)
> 
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.c b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> index a20d84dbe3aa..2c9d8cbca6e4 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> @@ -7,6 +7,8 @@
>  
>  #include "wave5-helper.h"
>  
> +#define DEFAULT_BS_SIZE(width, height) ((width) * (height) / 8 * 3)
> +
>  const char *state_to_str(enum vpu_instance_state state)
>  {
>  	switch (state) {
> @@ -217,3 +219,25 @@ void wave5_return_bufs(struct vb2_queue *q, u32 state)
>  		v4l2_m2m_buf_done(vbuf, state);
>  	}
>  }
> +
> +void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp,
> +			  int pix_fmt_type,
> +			  unsigned int width,
> +			  unsigned int height,
> +			  const struct v4l2_frmsize_stepwise *frmsize)
> +{
> +	v4l2_apply_frmsize_constraints(&width, &height, frmsize);
> +
> +	if (pix_fmt_type == VPU_FMT_TYPE_CODEC) {
> +		pix_mp->width = width;
> +		pix_mp->height = height;
> +		pix_mp->num_planes = 1;
> +		pix_mp->plane_fmt[0].bytesperline = 0;
> +		pix_mp->plane_fmt[0].sizeimage = max(DEFAULT_BS_SIZE(width, height),
> +						     pix_mp->plane_fmt[0].sizeimage);
> +	} else {
> +		v4l2_fill_pixfmt_mp(pix_mp, pix_mp->pixelformat, width, height);
> +	}
> +	pix_mp->flags = 0;
> +	pix_mp->field = V4L2_FIELD_NONE;
> +}
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.h b/drivers/media/platform/chips-media/wave5/wave5-helper.h
> index 6cee1c14d3ce..9937fce553fc 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-helper.h
> +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.h
> @@ -28,4 +28,9 @@ const struct vpu_format *wave5_find_vpu_fmt_by_idx(unsigned int idx,
>  						   const struct vpu_format fmt_list[MAX_FMTS]);
>  enum wave_std wave5_to_vpu_std(unsigned int v4l2_pix_fmt, enum vpu_instance_type type);
>  void wave5_return_bufs(struct vb2_queue *q, u32 state);
> +void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp,
> +			  int pix_fmt_type,
> +			  unsigned int width,
> +			  unsigned int height,
> +			  const struct v4l2_frmsize_stepwise *frmsize);
>  #endif
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> index 698c83e3082e..99e929aab870 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> @@ -11,111 +11,92 @@
>  #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
>  #define VPU_DEC_DRV_NAME "wave5-dec"
>  
> -#define DEFAULT_SRC_SIZE(width, height) ({			\
> -	(width) * (height) / 8 * 3;					\
> -})
> +static const struct v4l2_frmsize_stepwise dec_hevc_frmsize = {
> +	.min_width = W5_MIN_DEC_PIC_8_WIDTH,
> +	.max_width = W5_MAX_DEC_PIC_WIDTH,
> +	.step_width = W5_DEC_CODEC_STEP_WIDTH,
> +	.min_height = W5_MIN_DEC_PIC_8_HEIGHT,
> +	.max_height = W5_MAX_DEC_PIC_HEIGHT,
> +	.step_height = W5_DEC_CODEC_STEP_HEIGHT,
> +};
> +
> +static const struct v4l2_frmsize_stepwise dec_h264_frmsize = {
> +	.min_width = W5_MIN_DEC_PIC_32_WIDTH,
> +	.max_width = W5_MAX_DEC_PIC_WIDTH,
> +	.step_width = W5_DEC_CODEC_STEP_WIDTH,
> +	.min_height = W5_MIN_DEC_PIC_32_HEIGHT,
> +	.max_height = W5_MAX_DEC_PIC_HEIGHT,
> +	.step_height = W5_DEC_CODEC_STEP_HEIGHT,
> +};
> +
> +static const struct v4l2_frmsize_stepwise dec_raw_frmsize = {
> +	.min_width = W5_MIN_DEC_PIC_8_WIDTH,
> +	.max_width = W5_MAX_DEC_PIC_WIDTH,
> +	.step_width = W5_DEC_RAW_STEP_WIDTH,
> +	.min_height = W5_MIN_DEC_PIC_8_HEIGHT,
> +	.max_height = W5_MAX_DEC_PIC_HEIGHT,
> +	.step_height = W5_DEC_RAW_STEP_HEIGHT,
> +};
>  
>  static const struct vpu_format dec_fmt_list[FMT_TYPES][MAX_FMTS] = {
>  	[VPU_FMT_TYPE_CODEC] = {
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_HEVC,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_hevc_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_H264,
> -			.max_width = 8192,
> -			.min_width = 32,
> -			.max_height = 4320,
> -			.min_height = 32,
> +			.v4l2_frmsize = &dec_h264_frmsize,
>  		},
>  	},
>  	[VPU_FMT_TYPE_RAW] = {
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV422P,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV16,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV61,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420M,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12M,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21M,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV422M,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV16M,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV61M,
> -			.max_width = 8192,
> -			.min_width = 8,
> -			.max_height = 4320,
> -			.min_height = 8,
> +			.v4l2_frmsize = &dec_raw_frmsize,
>  		},
>  	}
>  };
> @@ -234,74 +215,6 @@ static void wave5_handle_src_buffer(struct vpu_instance *inst, dma_addr_t rd_ptr
>  	inst->remaining_consumed_bytes = consumed_bytes;
>  }
>  
> -static void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp, unsigned int width,
> -				 unsigned int height)
> -{
> -	switch (pix_mp->pixelformat) {
> -	case V4L2_PIX_FMT_YUV420:
> -	case V4L2_PIX_FMT_NV12:
> -	case V4L2_PIX_FMT_NV21:
> -		pix_mp->width = round_up(width, 32);
> -		pix_mp->height = round_up(height, 16);
> -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[0].sizeimage = width * height * 3 / 2;
> -		break;
> -	case V4L2_PIX_FMT_YUV422P:
> -	case V4L2_PIX_FMT_NV16:
> -	case V4L2_PIX_FMT_NV61:
> -		pix_mp->width = round_up(width, 32);
> -		pix_mp->height = round_up(height, 16);
> -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[0].sizeimage = width * height * 2;
> -		break;
> -	case V4L2_PIX_FMT_YUV420M:
> -		pix_mp->width = round_up(width, 32);
> -		pix_mp->height = round_up(height, 16);
> -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[0].sizeimage = width * height;
> -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32) / 2;
> -		pix_mp->plane_fmt[1].sizeimage = width * height / 4;
> -		pix_mp->plane_fmt[2].bytesperline = round_up(width, 32) / 2;
> -		pix_mp->plane_fmt[2].sizeimage = width * height / 4;
> -		break;
> -	case V4L2_PIX_FMT_NV12M:
> -	case V4L2_PIX_FMT_NV21M:
> -		pix_mp->width = round_up(width, 32);
> -		pix_mp->height = round_up(height, 16);
> -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[0].sizeimage = width * height;
> -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[1].sizeimage = width * height / 2;
> -		break;
> -	case V4L2_PIX_FMT_YUV422M:
> -		pix_mp->width = round_up(width, 32);
> -		pix_mp->height = round_up(height, 16);
> -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[0].sizeimage = width * height;
> -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32) / 2;
> -		pix_mp->plane_fmt[1].sizeimage = width * height / 2;
> -		pix_mp->plane_fmt[2].bytesperline = round_up(width, 32) / 2;
> -		pix_mp->plane_fmt[2].sizeimage = width * height / 2;
> -		break;
> -	case V4L2_PIX_FMT_NV16M:
> -	case V4L2_PIX_FMT_NV61M:
> -		pix_mp->width = round_up(width, 32);
> -		pix_mp->height = round_up(height, 16);
> -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[0].sizeimage = width * height;
> -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[1].sizeimage = width * height;
> -		break;
> -	default:
> -		pix_mp->width = width;
> -		pix_mp->height = height;
> -		pix_mp->plane_fmt[0].bytesperline = 0;
> -		pix_mp->plane_fmt[0].sizeimage = max(DEFAULT_SRC_SIZE(width, height),
> -						     pix_mp->plane_fmt[0].sizeimage);
> -		break;
> -	}
> -}
> -
>  static int start_decode(struct vpu_instance *inst, u32 *fail_res)
>  {
>  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
> @@ -389,6 +302,8 @@ static int handle_dynamic_resolution_change(struct vpu_instance *inst)
>  	}
>  
>  	if (p_dec_info->initial_info_obtained) {
> +		const struct vpu_format *vpu_fmt;
> +
>  		inst->conf_win.left = initial_info->pic_crop_rect.left;
>  		inst->conf_win.top = initial_info->pic_crop_rect.top;
>  		inst->conf_win.width = initial_info->pic_width -
> @@ -396,10 +311,27 @@ static int handle_dynamic_resolution_change(struct vpu_instance *inst)
>  		inst->conf_win.height = initial_info->pic_height -
>  			initial_info->pic_crop_rect.top - initial_info->pic_crop_rect.bottom;
>  
> -		wave5_update_pix_fmt(&inst->src_fmt, initial_info->pic_width,
> -				     initial_info->pic_height);
> -		wave5_update_pix_fmt(&inst->dst_fmt, initial_info->pic_width,
> -				     initial_info->pic_height);
> +		vpu_fmt = wave5_find_vpu_fmt(inst->src_fmt.pixelformat,
> +					     dec_fmt_list[VPU_FMT_TYPE_CODEC]);
> +		if (!vpu_fmt)
> +			return -EINVAL;
> +
> +		wave5_update_pix_fmt(&inst->src_fmt,
> +				     VPU_FMT_TYPE_CODEC,
> +				     initial_info->pic_width,
> +				     initial_info->pic_height,
> +				     vpu_fmt->v4l2_frmsize);
> +
> +		vpu_fmt = wave5_find_vpu_fmt(inst->dst_fmt.pixelformat,
> +					     dec_fmt_list[VPU_FMT_TYPE_RAW]);
> +		if (!vpu_fmt)
> +			return -EINVAL;
> +
> +		wave5_update_pix_fmt(&inst->dst_fmt,
> +				     VPU_FMT_TYPE_RAW,
> +				     initial_info->pic_width,
> +				     initial_info->pic_height,
> +				     vpu_fmt->v4l2_frmsize);
>  	}
>  
>  	v4l2_event_queue_fh(fh, &vpu_event_src_ch);
> @@ -548,12 +480,12 @@ static int wave5_vpu_dec_enum_framesizes(struct file *f, void *fh, struct v4l2_f
>  	}
>  
>  	fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
> -	fsize->stepwise.min_width = vpu_fmt->min_width;
> -	fsize->stepwise.max_width = vpu_fmt->max_width;
> -	fsize->stepwise.step_width = 1;
> -	fsize->stepwise.min_height = vpu_fmt->min_height;
> -	fsize->stepwise.max_height = vpu_fmt->max_height;
> -	fsize->stepwise.step_height = 1;
> +	fsize->stepwise.min_width = vpu_fmt->v4l2_frmsize->min_width;
> +	fsize->stepwise.max_width = vpu_fmt->v4l2_frmsize->max_width;
> +	fsize->stepwise.step_width = W5_DEC_CODEC_STEP_WIDTH;
> +	fsize->stepwise.min_height = vpu_fmt->v4l2_frmsize->min_height;
> +	fsize->stepwise.max_height = vpu_fmt->v4l2_frmsize->max_height;
> +	fsize->stepwise.step_height = W5_DEC_CODEC_STEP_HEIGHT;
>  
>  	return 0;
>  }
> @@ -576,6 +508,7 @@ static int wave5_vpu_dec_try_fmt_cap(struct file *file, void *fh, struct v4l2_fo
>  {
>  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
>  	struct dec_info *p_dec_info = &inst->codec_info->dec_info;
> +	const struct v4l2_frmsize_stepwise *frmsize;
>  	const struct vpu_format *vpu_fmt;
>  	int width, height;
>  
> @@ -589,14 +522,12 @@ static int wave5_vpu_dec_try_fmt_cap(struct file *file, void *fh, struct v4l2_fo
>  		width = inst->dst_fmt.width;
>  		height = inst->dst_fmt.height;
>  		f->fmt.pix_mp.pixelformat = inst->dst_fmt.pixelformat;
> -		f->fmt.pix_mp.num_planes = inst->dst_fmt.num_planes;
> +		frmsize = &dec_raw_frmsize;
>  	} else {
> -		const struct v4l2_format_info *info = v4l2_format_info(vpu_fmt->v4l2_pix_fmt);
> -
> -		width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width, vpu_fmt->max_width);
> -		height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height, vpu_fmt->max_height);
> +		width = f->fmt.pix_mp.width;
> +		height = f->fmt.pix_mp.height;
>  		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
> -		f->fmt.pix_mp.num_planes = info->mem_planes;
> +		frmsize = vpu_fmt->v4l2_frmsize;
>  	}
>  
>  	if (p_dec_info->initial_info_obtained) {
> @@ -604,9 +535,8 @@ static int wave5_vpu_dec_try_fmt_cap(struct file *file, void *fh, struct v4l2_fo
>  		height = inst->dst_fmt.height;
>  	}
>  
> -	wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
> -	f->fmt.pix_mp.flags = 0;
> -	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
> +	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_RAW,
> +			     width, height, frmsize);
>  	f->fmt.pix_mp.colorspace = inst->colorspace;
>  	f->fmt.pix_mp.ycbcr_enc = inst->ycbcr_enc;
>  	f->fmt.pix_mp.quantization = inst->quantization;
> @@ -718,7 +648,9 @@ static int wave5_vpu_dec_enum_fmt_out(struct file *file, void *fh, struct v4l2_f
>  static int wave5_vpu_dec_try_fmt_out(struct file *file, void *fh, struct v4l2_format *f)
>  {
>  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> +	const struct v4l2_frmsize_stepwise *frmsize;
>  	const struct vpu_format *vpu_fmt;
> +	int width, height;
>  
>  	dev_dbg(inst->dev->dev,
>  		"%s: fourcc: %u width: %u height: %u num_planes: %u colorspace: %u field: %u\n",
> @@ -727,20 +659,19 @@ static int wave5_vpu_dec_try_fmt_out(struct file *file, void *fh, struct v4l2_fo
>  
>  	vpu_fmt = wave5_find_vpu_fmt(f->fmt.pix_mp.pixelformat, dec_fmt_list[VPU_FMT_TYPE_CODEC]);
>  	if (!vpu_fmt) {
> +		width = inst->src_fmt.width;
> +		height = inst->src_fmt.height;
>  		f->fmt.pix_mp.pixelformat = inst->src_fmt.pixelformat;
> -		f->fmt.pix_mp.num_planes = inst->src_fmt.num_planes;
> -		wave5_update_pix_fmt(&f->fmt.pix_mp, inst->src_fmt.width, inst->src_fmt.height);
> +		frmsize = &dec_hevc_frmsize;
>  	} else {
> -		int width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width, vpu_fmt->max_width);
> -		int height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height, vpu_fmt->max_height);
> -
> +		width = f->fmt.pix_mp.width;
> +		height = f->fmt.pix_mp.height;
>  		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
> -		f->fmt.pix_mp.num_planes = 1;
> -		wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
> +		frmsize = vpu_fmt->v4l2_frmsize;
>  	}
>  
> -	f->fmt.pix_mp.flags = 0;
> -	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
> +	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_CODEC,
> +			     width, height, frmsize);
>  
>  	return 0;
>  }
> @@ -748,6 +679,7 @@ static int wave5_vpu_dec_try_fmt_out(struct file *file, void *fh, struct v4l2_fo
>  static int wave5_vpu_dec_s_fmt_out(struct file *file, void *fh, struct v4l2_format *f)
>  {
>  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> +	const struct vpu_format *vpu_fmt;
>  	int i, ret;
>  
>  	dev_dbg(inst->dev->dev,
> @@ -782,7 +714,13 @@ static int wave5_vpu_dec_s_fmt_out(struct file *file, void *fh, struct v4l2_form
>  	inst->quantization = f->fmt.pix_mp.quantization;
>  	inst->xfer_func = f->fmt.pix_mp.xfer_func;
>  
> -	wave5_update_pix_fmt(&inst->dst_fmt, f->fmt.pix_mp.width, f->fmt.pix_mp.height);
> +	vpu_fmt = wave5_find_vpu_fmt(inst->dst_fmt.pixelformat, dec_fmt_list[VPU_FMT_TYPE_RAW]);
> +	if (!vpu_fmt)
> +		return -EINVAL;
> +
> +	wave5_update_pix_fmt(&inst->dst_fmt, VPU_FMT_TYPE_RAW,
> +			     f->fmt.pix_mp.width, f->fmt.pix_mp.height,
> +			     vpu_fmt->v4l2_frmsize);
>  
>  	return 0;
>  }
> @@ -1005,6 +943,7 @@ static int wave5_vpu_dec_queue_setup(struct vb2_queue *q, unsigned int *num_buff
>  	struct vpu_instance *inst = vb2_get_drv_priv(q);
>  	struct v4l2_pix_format_mplane inst_format =
>  		(q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) ? inst->src_fmt : inst->dst_fmt;
> +	unsigned int i;
>  
>  	dev_dbg(inst->dev->dev, "%s: num_buffers: %u | num_planes: %u | type: %u\n", __func__,
>  		*num_buffers, *num_planes, q->type);
> @@ -1018,31 +957,9 @@ static int wave5_vpu_dec_queue_setup(struct vb2_queue *q, unsigned int *num_buff
>  		if (*num_buffers < inst->fbc_buf_count)
>  			*num_buffers = inst->fbc_buf_count;
>  
> -		if (*num_planes == 1) {
> -			if (inst->output_format == FORMAT_422)
> -				sizes[0] = inst_format.width * inst_format.height * 2;
> -			else
> -				sizes[0] = inst_format.width * inst_format.height * 3 / 2;
> -			dev_dbg(inst->dev->dev, "%s: size[0]: %u\n", __func__, sizes[0]);
> -		} else if (*num_planes == 2) {
> -			sizes[0] = inst_format.width * inst_format.height;
> -			if (inst->output_format == FORMAT_422)
> -				sizes[1] = inst_format.width * inst_format.height;
> -			else
> -				sizes[1] = inst_format.width * inst_format.height / 2;
> -			dev_dbg(inst->dev->dev, "%s: size[0]: %u | size[1]: %u\n",
> -				__func__, sizes[0], sizes[1]);
> -		} else if (*num_planes == 3) {
> -			sizes[0] = inst_format.width * inst_format.height;
> -			if (inst->output_format == FORMAT_422) {
> -				sizes[1] = inst_format.width * inst_format.height / 2;
> -				sizes[2] = inst_format.width * inst_format.height / 2;
> -			} else {
> -				sizes[1] = inst_format.width * inst_format.height / 4;
> -				sizes[2] = inst_format.width * inst_format.height / 4;
> -			}
> -			dev_dbg(inst->dev->dev, "%s: size[0]: %u | size[1]: %u | size[2]: %u\n",
> -				__func__, sizes[0], sizes[1], sizes[2]);
> +		for (i = 0; i < *num_planes; i++) {
> +			sizes[i] = inst_format.plane_fmt[i].sizeimage;
> +			dev_dbg(inst->dev->dev, "%s: size[%u]: %u\n", __func__, i, sizes[i]);
>  		}
>  	}
>  
> @@ -1568,20 +1485,15 @@ static const struct vb2_ops wave5_vpu_dec_vb2_ops = {
>  static void wave5_set_default_format(struct v4l2_pix_format_mplane *src_fmt,
>  				     struct v4l2_pix_format_mplane *dst_fmt)
>  {
> -	unsigned int dst_pix_fmt = dec_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
> -	const struct v4l2_format_info *dst_fmt_info = v4l2_format_info(dst_pix_fmt);
> -
>  	src_fmt->pixelformat = dec_fmt_list[VPU_FMT_TYPE_CODEC][0].v4l2_pix_fmt;
> -	src_fmt->field = V4L2_FIELD_NONE;
> -	src_fmt->flags = 0;
> -	src_fmt->num_planes = 1;
> -	wave5_update_pix_fmt(src_fmt, 720, 480);
> -
> -	dst_fmt->pixelformat = dst_pix_fmt;
> -	dst_fmt->field = V4L2_FIELD_NONE;
> -	dst_fmt->flags = 0;
> -	dst_fmt->num_planes = dst_fmt_info->mem_planes;
> -	wave5_update_pix_fmt(dst_fmt, 736, 480);
> +	wave5_update_pix_fmt(src_fmt, VPU_FMT_TYPE_CODEC,
> +			     W5_DEF_DEC_PIC_WIDTH, W5_DEF_DEC_PIC_HEIGHT,
> +			     &dec_hevc_frmsize);
> +
> +	dst_fmt->pixelformat = dec_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
> +	wave5_update_pix_fmt(dst_fmt, VPU_FMT_TYPE_RAW,
> +			     W5_DEF_DEC_PIC_WIDTH, W5_DEF_DEC_PIC_HEIGHT,
> +			     &dec_raw_frmsize);
>  }
>  
>  static int wave5_vpu_dec_queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq)
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> index 985de0c293e2..ef9aa1562346 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> @@ -11,65 +11,60 @@
>  #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
>  #define VPU_ENC_DRV_NAME "wave5-enc"
>  
> +static const struct v4l2_frmsize_stepwise enc_frmsize[FMT_TYPES] = {
> +	[VPU_FMT_TYPE_CODEC] = {
> +		.min_width = W5_MIN_ENC_PIC_WIDTH,
> +		.max_width = W5_MAX_ENC_PIC_WIDTH,
> +		.step_width = W5_ENC_CODEC_STEP_WIDTH,
> +		.min_height = W5_MIN_ENC_PIC_HEIGHT,
> +		.max_height = W5_MAX_ENC_PIC_HEIGHT,
> +		.step_height = W5_ENC_CODEC_STEP_HEIGHT,
> +	},
> +	[VPU_FMT_TYPE_RAW] = {
> +		.min_width = W5_MIN_ENC_PIC_WIDTH,
> +		.max_width = W5_MAX_ENC_PIC_WIDTH,
> +		.step_width = W5_ENC_RAW_STEP_WIDTH,
> +		.min_height = W5_MIN_ENC_PIC_HEIGHT,
> +		.max_height = W5_MAX_ENC_PIC_HEIGHT,
> +		.step_height = W5_ENC_RAW_STEP_HEIGHT,
> +	},
> +};
> +
>  static const struct vpu_format enc_fmt_list[FMT_TYPES][MAX_FMTS] = {
>  	[VPU_FMT_TYPE_CODEC] = {
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_HEVC,
> -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_CODEC],
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_H264,
> -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_CODEC],
>  		},
>  	},
>  	[VPU_FMT_TYPE_RAW] = {
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420,
> -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12,
> -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21,
> -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420M,
> -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12M,
> -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
>  		},
>  		{
>  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21M,
> -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
>  		},
>  	}
>  };
> @@ -106,46 +101,6 @@ static int switch_state(struct vpu_instance *inst, enum vpu_instance_state state
>  	return -EINVAL;
>  }
>  
> -static void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp, unsigned int width,
> -				 unsigned int height)
> -{
> -	switch (pix_mp->pixelformat) {
> -	case V4L2_PIX_FMT_YUV420:
> -	case V4L2_PIX_FMT_NV12:
> -	case V4L2_PIX_FMT_NV21:
> -		pix_mp->width = width;
> -		pix_mp->height = height;
> -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[0].sizeimage = round_up(width, 32) * height * 3 / 2;
> -		break;
> -	case V4L2_PIX_FMT_YUV420M:
> -		pix_mp->width = width;
> -		pix_mp->height = height;
> -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[0].sizeimage = round_up(width, 32) * height;
> -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32) / 2;
> -		pix_mp->plane_fmt[1].sizeimage = round_up(width, 32) * height / 4;
> -		pix_mp->plane_fmt[2].bytesperline = round_up(width, 32) / 2;
> -		pix_mp->plane_fmt[2].sizeimage = round_up(width, 32) * height / 4;
> -		break;
> -	case V4L2_PIX_FMT_NV12M:
> -	case V4L2_PIX_FMT_NV21M:
> -		pix_mp->width = width;
> -		pix_mp->height = height;
> -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[0].sizeimage = round_up(width, 32) * height;
> -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32);
> -		pix_mp->plane_fmt[1].sizeimage = round_up(width, 32) * height / 2;
> -		break;
> -	default:
> -		pix_mp->width = width;
> -		pix_mp->height = height;
> -		pix_mp->plane_fmt[0].bytesperline = 0;
> -		pix_mp->plane_fmt[0].sizeimage = width * height / 8 * 3;
> -		break;
> -	}
> -}
> -
>  static int start_encode(struct vpu_instance *inst, u32 *fail_res)
>  {
>  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
> @@ -360,13 +315,8 @@ static int wave5_vpu_enc_enum_framesizes(struct file *f, void *fh, struct v4l2_f
>  			return -EINVAL;
>  	}
>  
> -	fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
> -	fsize->stepwise.min_width = vpu_fmt->min_width;
> -	fsize->stepwise.max_width = vpu_fmt->max_width;
> -	fsize->stepwise.step_width = 1;
> -	fsize->stepwise.min_height = vpu_fmt->min_height;
> -	fsize->stepwise.max_height = vpu_fmt->max_height;
> -	fsize->stepwise.step_height = 1;
> +	fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
> +	fsize->stepwise = enc_frmsize[VPU_FMT_TYPE_CODEC];
>  
>  	return 0;
>  }
> @@ -391,7 +341,9 @@ static int wave5_vpu_enc_enum_fmt_cap(struct file *file, void *fh, struct v4l2_f
>  static int wave5_vpu_enc_try_fmt_cap(struct file *file, void *fh, struct v4l2_format *f)
>  {
>  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> +	const struct v4l2_frmsize_stepwise *frmsize;
>  	const struct vpu_format *vpu_fmt;
> +	int width, height;
>  
>  	dev_dbg(inst->dev->dev, "%s: fourcc: %u width: %u height: %u num_planes: %u field: %u\n",
>  		__func__, f->fmt.pix_mp.pixelformat, f->fmt.pix_mp.width, f->fmt.pix_mp.height,
> @@ -399,20 +351,19 @@ static int wave5_vpu_enc_try_fmt_cap(struct file *file, void *fh, struct v4l2_fo
>  
>  	vpu_fmt = wave5_find_vpu_fmt(f->fmt.pix_mp.pixelformat, enc_fmt_list[VPU_FMT_TYPE_CODEC]);
>  	if (!vpu_fmt) {
> +		width = inst->dst_fmt.width;
> +		height = inst->dst_fmt.height;
>  		f->fmt.pix_mp.pixelformat = inst->dst_fmt.pixelformat;
> -		f->fmt.pix_mp.num_planes = inst->dst_fmt.num_planes;
> -		wave5_update_pix_fmt(&f->fmt.pix_mp, inst->dst_fmt.width, inst->dst_fmt.height);
> +		frmsize = &enc_frmsize[VPU_FMT_TYPE_CODEC];
>  	} else {
> -		int width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width, vpu_fmt->max_width);
> -		int height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height, vpu_fmt->max_height);
> -
> +		width = f->fmt.pix_mp.width;
> +		height = f->fmt.pix_mp.height;
>  		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
> -		f->fmt.pix_mp.num_planes = 1;
> -		wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
> +		frmsize = vpu_fmt->v4l2_frmsize;
>  	}
>  
> -	f->fmt.pix_mp.flags = 0;
> -	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
> +	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_CODEC,
> +			     width, height, frmsize);
>  	f->fmt.pix_mp.colorspace = inst->colorspace;
>  	f->fmt.pix_mp.ycbcr_enc = inst->ycbcr_enc;
>  	f->fmt.pix_mp.quantization = inst->quantization;
> @@ -499,7 +450,9 @@ static int wave5_vpu_enc_enum_fmt_out(struct file *file, void *fh, struct v4l2_f
>  static int wave5_vpu_enc_try_fmt_out(struct file *file, void *fh, struct v4l2_format *f)
>  {
>  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> +	const struct v4l2_frmsize_stepwise *frmsize;
>  	const struct vpu_format *vpu_fmt;
> +	int width, height;
>  
>  	dev_dbg(inst->dev->dev, "%s: fourcc: %u width: %u height: %u num_planes: %u field: %u\n",
>  		__func__, f->fmt.pix_mp.pixelformat, f->fmt.pix_mp.width, f->fmt.pix_mp.height,
> @@ -507,28 +460,26 @@ static int wave5_vpu_enc_try_fmt_out(struct file *file, void *fh, struct v4l2_fo
>  
>  	vpu_fmt = wave5_find_vpu_fmt(f->fmt.pix_mp.pixelformat, enc_fmt_list[VPU_FMT_TYPE_RAW]);
>  	if (!vpu_fmt) {
> +		width = inst->src_fmt.width;
> +		height = inst->src_fmt.height;
>  		f->fmt.pix_mp.pixelformat = inst->src_fmt.pixelformat;
> -		f->fmt.pix_mp.num_planes = inst->src_fmt.num_planes;
> -		wave5_update_pix_fmt(&f->fmt.pix_mp, inst->src_fmt.width, inst->src_fmt.height);
> +		frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW];
>  	} else {
> -		int width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width, vpu_fmt->max_width);
> -		int height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height, vpu_fmt->max_height);
> -		const struct v4l2_format_info *info = v4l2_format_info(vpu_fmt->v4l2_pix_fmt);
> -
> +		width = f->fmt.pix_mp.width;
> +		height = f->fmt.pix_mp.height;
>  		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
> -		f->fmt.pix_mp.num_planes = info->mem_planes;
> -		wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
> +		frmsize = vpu_fmt->v4l2_frmsize;
>  	}
>  
> -	f->fmt.pix_mp.flags = 0;
> -	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
> -
> +	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_RAW,
> +			     width, height, frmsize);
>  	return 0;
>  }
>  
>  static int wave5_vpu_enc_s_fmt_out(struct file *file, void *fh, struct v4l2_format *f)
>  {
>  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> +	const struct vpu_format *vpu_fmt;
>  	int i, ret;
>  
>  	dev_dbg(inst->dev->dev, "%s: fourcc: %u width: %u height: %u num_planes: %u field: %u\n",
> @@ -568,7 +519,15 @@ static int wave5_vpu_enc_s_fmt_out(struct file *file, void *fh, struct v4l2_form
>  	inst->quantization = f->fmt.pix_mp.quantization;
>  	inst->xfer_func = f->fmt.pix_mp.xfer_func;
>  
> -	wave5_update_pix_fmt(&inst->dst_fmt, f->fmt.pix_mp.width, f->fmt.pix_mp.height);
> +	vpu_fmt = wave5_find_vpu_fmt(inst->dst_fmt.pixelformat, enc_fmt_list[VPU_FMT_TYPE_CODEC]);
> +	if (!vpu_fmt)
> +		return -EINVAL;
> +
> +	wave5_update_pix_fmt(&inst->dst_fmt, VPU_FMT_TYPE_CODEC,
> +			     f->fmt.pix_mp.width, f->fmt.pix_mp.height,
> +			     vpu_fmt->v4l2_frmsize);
> +	inst->conf_win.width = inst->dst_fmt.width;
> +	inst->conf_win.height = inst->dst_fmt.height;
>  
>  	return 0;
>  }
> @@ -584,12 +543,17 @@ static int wave5_vpu_enc_g_selection(struct file *file, void *fh, struct v4l2_se
>  	switch (s->target) {
>  	case V4L2_SEL_TGT_CROP_DEFAULT:
>  	case V4L2_SEL_TGT_CROP_BOUNDS:
> -	case V4L2_SEL_TGT_CROP:
>  		s->r.left = 0;
>  		s->r.top = 0;
>  		s->r.width = inst->dst_fmt.width;
>  		s->r.height = inst->dst_fmt.height;
>  		break;
> +	case V4L2_SEL_TGT_CROP:
> +		s->r.left = 0;
> +		s->r.top = 0;
> +		s->r.width = inst->conf_win.width;
> +		s->r.height = inst->conf_win.height;
> +		break;
>  	default:
>  		return -EINVAL;
>  	}
> @@ -612,8 +576,10 @@ static int wave5_vpu_enc_s_selection(struct file *file, void *fh, struct v4l2_se
>  
>  	s->r.left = 0;
>  	s->r.top = 0;
> -	s->r.width = inst->src_fmt.width;
> -	s->r.height = inst->src_fmt.height;
> +	s->r.width = min(s->r.width, inst->dst_fmt.width);
> +	s->r.height = min(s->r.height, inst->dst_fmt.height);
> +
> +	inst->conf_win = s->r;
>  
>  	return 0;
>  }
> @@ -1151,8 +1117,8 @@ static void wave5_set_enc_openparam(struct enc_open_param *open_param,
>  	open_param->wave_param.lambda_scaling_enable = 1;
>  
>  	open_param->line_buf_int_en = true;
> -	open_param->pic_width = inst->dst_fmt.width;
> -	open_param->pic_height = inst->dst_fmt.height;
> +	open_param->pic_width = inst->conf_win.width;
> +	open_param->pic_height = inst->conf_win.height;
>  	open_param->frame_rate_info = inst->frame_rate;
>  	open_param->rc_enable = inst->rc_enable;
>  	if (inst->rc_enable) {
> @@ -1456,20 +1422,15 @@ static const struct vb2_ops wave5_vpu_enc_vb2_ops = {
>  static void wave5_set_default_format(struct v4l2_pix_format_mplane *src_fmt,
>  				     struct v4l2_pix_format_mplane *dst_fmt)
>  {
> -	unsigned int src_pix_fmt = enc_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
> -	const struct v4l2_format_info *src_fmt_info = v4l2_format_info(src_pix_fmt);
> -
> -	src_fmt->pixelformat = src_pix_fmt;
> -	src_fmt->field = V4L2_FIELD_NONE;
> -	src_fmt->flags = 0;
> -	src_fmt->num_planes = src_fmt_info->mem_planes;
> -	wave5_update_pix_fmt(src_fmt, 416, 240);
> +	src_fmt->pixelformat = enc_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
> +	wave5_update_pix_fmt(src_fmt, VPU_FMT_TYPE_RAW,
> +			     W5_DEF_ENC_PIC_WIDTH, W5_DEF_ENC_PIC_HEIGHT,
> +			     &enc_frmsize[VPU_FMT_TYPE_RAW]);
>  
>  	dst_fmt->pixelformat = enc_fmt_list[VPU_FMT_TYPE_CODEC][0].v4l2_pix_fmt;
> -	dst_fmt->field = V4L2_FIELD_NONE;
> -	dst_fmt->flags = 0;
> -	dst_fmt->num_planes = 1;
> -	wave5_update_pix_fmt(dst_fmt, 416, 240);
> +	wave5_update_pix_fmt(dst_fmt, VPU_FMT_TYPE_CODEC,
> +			     W5_DEF_ENC_PIC_WIDTH, W5_DEF_ENC_PIC_HEIGHT,
> +			     &enc_frmsize[VPU_FMT_TYPE_CODEC]);
>  }
>  
>  static int wave5_vpu_enc_queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq)
> @@ -1733,6 +1694,8 @@ static int wave5_vpu_open_enc(struct file *filp)
>  	v4l2_ctrl_handler_setup(v4l2_ctrl_hdl);
>  
>  	wave5_set_default_format(&inst->src_fmt, &inst->dst_fmt);
> +	inst->conf_win.width = inst->dst_fmt.width;
> +	inst->conf_win.height = inst->dst_fmt.height;
>  	inst->colorspace = V4L2_COLORSPACE_REC709;
>  	inst->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>  	inst->quantization = V4L2_QUANTIZATION_DEFAULT;
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu.h b/drivers/media/platform/chips-media/wave5/wave5-vpu.h
> index 32b7fd3730b5..3847332551fc 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-vpu.h
> +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.h
> @@ -38,10 +38,7 @@ enum vpu_fmt_type {
>  
>  struct vpu_format {
>  	unsigned int v4l2_pix_fmt;
> -	unsigned int max_width;
> -	unsigned int min_width;
> -	unsigned int max_height;
> -	unsigned int min_height;
> +	const struct v4l2_frmsize_stepwise *v4l2_frmsize;
>  };
>  
>  static inline struct vpu_instance *wave5_to_vpu_inst(struct v4l2_fh *vfh)
> diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h b/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
> index e4bc2e467cb5..1ea9f5f31499 100644
> --- a/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
> +++ b/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
> @@ -32,10 +32,29 @@
>  
>  #define MAX_NUM_INSTANCE                32
>  
> -#define W5_MIN_ENC_PIC_WIDTH            256
> -#define W5_MIN_ENC_PIC_HEIGHT           128
> -#define W5_MAX_ENC_PIC_WIDTH            8192
> -#define W5_MAX_ENC_PIC_HEIGHT           8192
> +#define W5_DEF_DEC_PIC_WIDTH            720U
> +#define W5_DEF_DEC_PIC_HEIGHT           480U
> +#define W5_MIN_DEC_PIC_8_WIDTH          8U
> +#define W5_MIN_DEC_PIC_8_HEIGHT         8U
> +#define W5_MIN_DEC_PIC_32_WIDTH         32U
> +#define W5_MIN_DEC_PIC_32_HEIGHT        32U
> +#define W5_MAX_DEC_PIC_WIDTH            8192U
> +#define W5_MAX_DEC_PIC_HEIGHT           4320U
> +#define W5_DEC_CODEC_STEP_WIDTH         1U
> +#define W5_DEC_CODEC_STEP_HEIGHT        1U
> +#define W5_DEC_RAW_STEP_WIDTH           32U
> +#define W5_DEC_RAW_STEP_HEIGHT          16U
> +
> +#define W5_DEF_ENC_PIC_WIDTH            416U
> +#define W5_DEF_ENC_PIC_HEIGHT           240U
> +#define W5_MIN_ENC_PIC_WIDTH            256U
> +#define W5_MIN_ENC_PIC_HEIGHT           128U
> +#define W5_MAX_ENC_PIC_WIDTH            8192U
> +#define W5_MAX_ENC_PIC_HEIGHT           8192U
> +#define W5_ENC_CODEC_STEP_WIDTH         8U
> +#define W5_ENC_CODEC_STEP_HEIGHT        8U
> +#define W5_ENC_RAW_STEP_WIDTH           32U
> +#define W5_ENC_RAW_STEP_HEIGHT          16U
>  
>  //  application specific configuration
>  #define VPU_ENC_TIMEOUT                 60000

RE: [RESEND PATCH v7 3/4] media: chips-media: wave5: Use helpers to calculate bytesperline and sizeimage.

[jackson.lee]

Hello Nicolas

I agree it, thanks for your feedback.

> -----Original Message-----
> From: nicolas.dufresne@collabora.corp-partner.google.com
> <nicolas.dufresne@collabora.corp-partner.google.com>
> Sent: Saturday, September 7, 2024 2:12 AM
> To: jackson.lee <jackson.lee@chipsnmedia.com>; mchehab@kernel.org;
> sebastian.fricke@collabora.com
> Cc: linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
> hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com
> Subject: Re: [RESEND PATCH v7 3/4] media: chips-media: wave5: Use helpers
> to calculate bytesperline and sizeimage.
> 
> Hi,
> 
> Le lundi 12 août 2024 à 16:08 +0900, Jackson.lee a écrit :
> > Use v4l2-common helper functions to calculate bytesperline and
> > sizeimage, instead of calculating in a wave5 driver directly.
> >
> > In case of raw(YUV) v4l2_pix_format, the wave5 driver updates
> > v4l2_pix_format_mplane struct through v4l2_fill_pixfmt_mp() function.
> >
> > Encoder and Decoder need same bytesperline and sizeimage values
> 
>                            *the* same
> 
> > for same v4l2_pix_format.
> 
>   *the* same
> 
> > So, a wave5_update_pix_fmt is refactored to support both together.
> 
>   So, wave5_update_pix_fmt function ...
> 
> If you don't mind, I'll apply these edits, and reflow to 75 charaters.
> 
> Nicolas
> 
> >
> > Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
> > Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
> > Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
> > ---
> >  .../platform/chips-media/wave5/wave5-helper.c |  24 ++
> >  .../platform/chips-media/wave5/wave5-helper.h |   5 +
> >  .../chips-media/wave5/wave5-vpu-dec.c         | 300 +++++++-----------
> >  .../chips-media/wave5/wave5-vpu-enc.c         | 197 +++++-------
> >  .../platform/chips-media/wave5/wave5-vpu.h    |   5 +-
> >  .../chips-media/wave5/wave5-vpuconfig.h       |  27 +-
> >  6 files changed, 239 insertions(+), 319 deletions(-)
> >
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > index a20d84dbe3aa..2c9d8cbca6e4 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
> > @@ -7,6 +7,8 @@
> >
> >  #include "wave5-helper.h"
> >
> > +#define DEFAULT_BS_SIZE(width, height) ((width) * (height) / 8 * 3)
> > +
> >  const char *state_to_str(enum vpu_instance_state state)  {
> >  	switch (state) {
> > @@ -217,3 +219,25 @@ void wave5_return_bufs(struct vb2_queue *q, u32
> state)
> >  		v4l2_m2m_buf_done(vbuf, state);
> >  	}
> >  }
> > +
> > +void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp,
> > +			  int pix_fmt_type,
> > +			  unsigned int width,
> > +			  unsigned int height,
> > +			  const struct v4l2_frmsize_stepwise *frmsize) {
> > +	v4l2_apply_frmsize_constraints(&width, &height, frmsize);
> > +
> > +	if (pix_fmt_type == VPU_FMT_TYPE_CODEC) {
> > +		pix_mp->width = width;
> > +		pix_mp->height = height;
> > +		pix_mp->num_planes = 1;
> > +		pix_mp->plane_fmt[0].bytesperline = 0;
> > +		pix_mp->plane_fmt[0].sizeimage = max(DEFAULT_BS_SIZE(width,
> height),
> > +						     pix_mp-
> >plane_fmt[0].sizeimage);
> > +	} else {
> > +		v4l2_fill_pixfmt_mp(pix_mp, pix_mp->pixelformat, width,
> height);
> > +	}
> > +	pix_mp->flags = 0;
> > +	pix_mp->field = V4L2_FIELD_NONE;
> > +}
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.h
> > b/drivers/media/platform/chips-media/wave5/wave5-helper.h
> > index 6cee1c14d3ce..9937fce553fc 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-helper.h
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-helper.h
> > @@ -28,4 +28,9 @@ const struct vpu_format
> *wave5_find_vpu_fmt_by_idx(unsigned int idx,
> >  						   const struct vpu_format
> fmt_list[MAX_FMTS]);  enum wave_std
> > wave5_to_vpu_std(unsigned int v4l2_pix_fmt, enum vpu_instance_type
> > type);  void wave5_return_bufs(struct vb2_queue *q, u32 state);
> > +void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp,
> > +			  int pix_fmt_type,
> > +			  unsigned int width,
> > +			  unsigned int height,
> > +			  const struct v4l2_frmsize_stepwise *frmsize);
> >  #endif
> > diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > index 698c83e3082e..99e929aab870 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
> > @@ -11,111 +11,92 @@
> >  #define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
> >  #define VPU_DEC_DRV_NAME "wave5-dec"
> >
> > -#define DEFAULT_SRC_SIZE(width, height) ({			\
> > -	(width) * (height) / 8 * 3;					\
> > -})
> > +static const struct v4l2_frmsize_stepwise dec_hevc_frmsize = {
> > +	.min_width = W5_MIN_DEC_PIC_8_WIDTH,
> > +	.max_width = W5_MAX_DEC_PIC_WIDTH,
> > +	.step_width = W5_DEC_CODEC_STEP_WIDTH,
> > +	.min_height = W5_MIN_DEC_PIC_8_HEIGHT,
> > +	.max_height = W5_MAX_DEC_PIC_HEIGHT,
> > +	.step_height = W5_DEC_CODEC_STEP_HEIGHT, };
> > +
> > +static const struct v4l2_frmsize_stepwise dec_h264_frmsize = {
> > +	.min_width = W5_MIN_DEC_PIC_32_WIDTH,
> > +	.max_width = W5_MAX_DEC_PIC_WIDTH,
> > +	.step_width = W5_DEC_CODEC_STEP_WIDTH,
> > +	.min_height = W5_MIN_DEC_PIC_32_HEIGHT,
> > +	.max_height = W5_MAX_DEC_PIC_HEIGHT,
> > +	.step_height = W5_DEC_CODEC_STEP_HEIGHT, };
> > +
> > +static const struct v4l2_frmsize_stepwise dec_raw_frmsize = {
> > +	.min_width = W5_MIN_DEC_PIC_8_WIDTH,
> > +	.max_width = W5_MAX_DEC_PIC_WIDTH,
> > +	.step_width = W5_DEC_RAW_STEP_WIDTH,
> > +	.min_height = W5_MIN_DEC_PIC_8_HEIGHT,
> > +	.max_height = W5_MAX_DEC_PIC_HEIGHT,
> > +	.step_height = W5_DEC_RAW_STEP_HEIGHT, };
> >
> >  static const struct vpu_format dec_fmt_list[FMT_TYPES][MAX_FMTS] = {
> >  	[VPU_FMT_TYPE_CODEC] = {
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_HEVC,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_hevc_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_H264,
> > -			.max_width = 8192,
> > -			.min_width = 32,
> > -			.max_height = 4320,
> > -			.min_height = 32,
> > +			.v4l2_frmsize = &dec_h264_frmsize,
> >  		},
> >  	},
> >  	[VPU_FMT_TYPE_RAW] = {
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV422P,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV16,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV61,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420M,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12M,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21M,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV422M,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV16M,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV61M,
> > -			.max_width = 8192,
> > -			.min_width = 8,
> > -			.max_height = 4320,
> > -			.min_height = 8,
> > +			.v4l2_frmsize = &dec_raw_frmsize,
> >  		},
> >  	}
> >  };
> > @@ -234,74 +215,6 @@ static void wave5_handle_src_buffer(struct
> vpu_instance *inst, dma_addr_t rd_ptr
> >  	inst->remaining_consumed_bytes = consumed_bytes;  }
> >
> > -static void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp,
> unsigned int width,
> > -				 unsigned int height)
> > -{
> > -	switch (pix_mp->pixelformat) {
> > -	case V4L2_PIX_FMT_YUV420:
> > -	case V4L2_PIX_FMT_NV12:
> > -	case V4L2_PIX_FMT_NV21:
> > -		pix_mp->width = round_up(width, 32);
> > -		pix_mp->height = round_up(height, 16);
> > -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[0].sizeimage = width * height * 3 / 2;
> > -		break;
> > -	case V4L2_PIX_FMT_YUV422P:
> > -	case V4L2_PIX_FMT_NV16:
> > -	case V4L2_PIX_FMT_NV61:
> > -		pix_mp->width = round_up(width, 32);
> > -		pix_mp->height = round_up(height, 16);
> > -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[0].sizeimage = width * height * 2;
> > -		break;
> > -	case V4L2_PIX_FMT_YUV420M:
> > -		pix_mp->width = round_up(width, 32);
> > -		pix_mp->height = round_up(height, 16);
> > -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[0].sizeimage = width * height;
> > -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32) / 2;
> > -		pix_mp->plane_fmt[1].sizeimage = width * height / 4;
> > -		pix_mp->plane_fmt[2].bytesperline = round_up(width, 32) / 2;
> > -		pix_mp->plane_fmt[2].sizeimage = width * height / 4;
> > -		break;
> > -	case V4L2_PIX_FMT_NV12M:
> > -	case V4L2_PIX_FMT_NV21M:
> > -		pix_mp->width = round_up(width, 32);
> > -		pix_mp->height = round_up(height, 16);
> > -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[0].sizeimage = width * height;
> > -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[1].sizeimage = width * height / 2;
> > -		break;
> > -	case V4L2_PIX_FMT_YUV422M:
> > -		pix_mp->width = round_up(width, 32);
> > -		pix_mp->height = round_up(height, 16);
> > -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[0].sizeimage = width * height;
> > -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32) / 2;
> > -		pix_mp->plane_fmt[1].sizeimage = width * height / 2;
> > -		pix_mp->plane_fmt[2].bytesperline = round_up(width, 32) / 2;
> > -		pix_mp->plane_fmt[2].sizeimage = width * height / 2;
> > -		break;
> > -	case V4L2_PIX_FMT_NV16M:
> > -	case V4L2_PIX_FMT_NV61M:
> > -		pix_mp->width = round_up(width, 32);
> > -		pix_mp->height = round_up(height, 16);
> > -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[0].sizeimage = width * height;
> > -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[1].sizeimage = width * height;
> > -		break;
> > -	default:
> > -		pix_mp->width = width;
> > -		pix_mp->height = height;
> > -		pix_mp->plane_fmt[0].bytesperline = 0;
> > -		pix_mp->plane_fmt[0].sizeimage = max(DEFAULT_SRC_SIZE(width,
> height),
> > -						     pix_mp-
> >plane_fmt[0].sizeimage);
> > -		break;
> > -	}
> > -}
> > -
> >  static int start_decode(struct vpu_instance *inst, u32 *fail_res)  {
> >  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx; @@ -389,6
> > +302,8 @@ static int handle_dynamic_resolution_change(struct
> vpu_instance *inst)
> >  	}
> >
> >  	if (p_dec_info->initial_info_obtained) {
> > +		const struct vpu_format *vpu_fmt;
> > +
> >  		inst->conf_win.left = initial_info->pic_crop_rect.left;
> >  		inst->conf_win.top = initial_info->pic_crop_rect.top;
> >  		inst->conf_win.width = initial_info->pic_width - @@ -396,10
> +311,27
> > @@ static int handle_dynamic_resolution_change(struct vpu_instance *inst)
> >  		inst->conf_win.height = initial_info->pic_height -
> >  			initial_info->pic_crop_rect.top -
> > initial_info->pic_crop_rect.bottom;
> >
> > -		wave5_update_pix_fmt(&inst->src_fmt, initial_info->pic_width,
> > -				     initial_info->pic_height);
> > -		wave5_update_pix_fmt(&inst->dst_fmt, initial_info->pic_width,
> > -				     initial_info->pic_height);
> > +		vpu_fmt = wave5_find_vpu_fmt(inst->src_fmt.pixelformat,
> > +					     dec_fmt_list[VPU_FMT_TYPE_CODEC]);
> > +		if (!vpu_fmt)
> > +			return -EINVAL;
> > +
> > +		wave5_update_pix_fmt(&inst->src_fmt,
> > +				     VPU_FMT_TYPE_CODEC,
> > +				     initial_info->pic_width,
> > +				     initial_info->pic_height,
> > +				     vpu_fmt->v4l2_frmsize);
> > +
> > +		vpu_fmt = wave5_find_vpu_fmt(inst->dst_fmt.pixelformat,
> > +					     dec_fmt_list[VPU_FMT_TYPE_RAW]);
> > +		if (!vpu_fmt)
> > +			return -EINVAL;
> > +
> > +		wave5_update_pix_fmt(&inst->dst_fmt,
> > +				     VPU_FMT_TYPE_RAW,
> > +				     initial_info->pic_width,
> > +				     initial_info->pic_height,
> > +				     vpu_fmt->v4l2_frmsize);
> >  	}
> >
> >  	v4l2_event_queue_fh(fh, &vpu_event_src_ch); @@ -548,12 +480,12 @@
> > static int wave5_vpu_dec_enum_framesizes(struct file *f, void *fh,
> struct v4l2_f
> >  	}
> >
> >  	fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
> > -	fsize->stepwise.min_width = vpu_fmt->min_width;
> > -	fsize->stepwise.max_width = vpu_fmt->max_width;
> > -	fsize->stepwise.step_width = 1;
> > -	fsize->stepwise.min_height = vpu_fmt->min_height;
> > -	fsize->stepwise.max_height = vpu_fmt->max_height;
> > -	fsize->stepwise.step_height = 1;
> > +	fsize->stepwise.min_width = vpu_fmt->v4l2_frmsize->min_width;
> > +	fsize->stepwise.max_width = vpu_fmt->v4l2_frmsize->max_width;
> > +	fsize->stepwise.step_width = W5_DEC_CODEC_STEP_WIDTH;
> > +	fsize->stepwise.min_height = vpu_fmt->v4l2_frmsize->min_height;
> > +	fsize->stepwise.max_height = vpu_fmt->v4l2_frmsize->max_height;
> > +	fsize->stepwise.step_height = W5_DEC_CODEC_STEP_HEIGHT;
> >
> >  	return 0;
> >  }
> > @@ -576,6 +508,7 @@ static int wave5_vpu_dec_try_fmt_cap(struct file
> > *file, void *fh, struct v4l2_fo  {
> >  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> >  	struct dec_info *p_dec_info = &inst->codec_info->dec_info;
> > +	const struct v4l2_frmsize_stepwise *frmsize;
> >  	const struct vpu_format *vpu_fmt;
> >  	int width, height;
> >
> > @@ -589,14 +522,12 @@ static int wave5_vpu_dec_try_fmt_cap(struct file
> *file, void *fh, struct v4l2_fo
> >  		width = inst->dst_fmt.width;
> >  		height = inst->dst_fmt.height;
> >  		f->fmt.pix_mp.pixelformat = inst->dst_fmt.pixelformat;
> > -		f->fmt.pix_mp.num_planes = inst->dst_fmt.num_planes;
> > +		frmsize = &dec_raw_frmsize;
> >  	} else {
> > -		const struct v4l2_format_info *info =
> v4l2_format_info(vpu_fmt->v4l2_pix_fmt);
> > -
> > -		width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width,
> vpu_fmt->max_width);
> > -		height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height,
> vpu_fmt->max_height);
> > +		width = f->fmt.pix_mp.width;
> > +		height = f->fmt.pix_mp.height;
> >  		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
> > -		f->fmt.pix_mp.num_planes = info->mem_planes;
> > +		frmsize = vpu_fmt->v4l2_frmsize;
> >  	}
> >
> >  	if (p_dec_info->initial_info_obtained) { @@ -604,9 +535,8 @@ static
> > int wave5_vpu_dec_try_fmt_cap(struct file *file, void *fh, struct
> v4l2_fo
> >  		height = inst->dst_fmt.height;
> >  	}
> >
> > -	wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
> > -	f->fmt.pix_mp.flags = 0;
> > -	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
> > +	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_RAW,
> > +			     width, height, frmsize);
> >  	f->fmt.pix_mp.colorspace = inst->colorspace;
> >  	f->fmt.pix_mp.ycbcr_enc = inst->ycbcr_enc;
> >  	f->fmt.pix_mp.quantization = inst->quantization; @@ -718,7 +648,9
> @@
> > static int wave5_vpu_dec_enum_fmt_out(struct file *file, void *fh,
> > struct v4l2_f  static int wave5_vpu_dec_try_fmt_out(struct file *file,
> > void *fh, struct v4l2_format *f)  {
> >  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> > +	const struct v4l2_frmsize_stepwise *frmsize;
> >  	const struct vpu_format *vpu_fmt;
> > +	int width, height;
> >
> >  	dev_dbg(inst->dev->dev,
> >  		"%s: fourcc: %u width: %u height: %u num_planes: %u
> colorspace: %u
> > field: %u\n", @@ -727,20 +659,19 @@ static int
> > wave5_vpu_dec_try_fmt_out(struct file *file, void *fh, struct v4l2_fo
> >
> >  	vpu_fmt = wave5_find_vpu_fmt(f->fmt.pix_mp.pixelformat,
> dec_fmt_list[VPU_FMT_TYPE_CODEC]);
> >  	if (!vpu_fmt) {
> > +		width = inst->src_fmt.width;
> > +		height = inst->src_fmt.height;
> >  		f->fmt.pix_mp.pixelformat = inst->src_fmt.pixelformat;
> > -		f->fmt.pix_mp.num_planes = inst->src_fmt.num_planes;
> > -		wave5_update_pix_fmt(&f->fmt.pix_mp, inst->src_fmt.width,
> inst->src_fmt.height);
> > +		frmsize = &dec_hevc_frmsize;
> >  	} else {
> > -		int width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width,
> vpu_fmt->max_width);
> > -		int height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height,
> vpu_fmt->max_height);
> > -
> > +		width = f->fmt.pix_mp.width;
> > +		height = f->fmt.pix_mp.height;
> >  		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
> > -		f->fmt.pix_mp.num_planes = 1;
> > -		wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
> > +		frmsize = vpu_fmt->v4l2_frmsize;
> >  	}
> >
> > -	f->fmt.pix_mp.flags = 0;
> > -	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
> > +	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_CODEC,
> > +			     width, height, frmsize);
> >
> >  	return 0;
> >  }
> > @@ -748,6 +679,7 @@ static int wave5_vpu_dec_try_fmt_out(struct file
> > *file, void *fh, struct v4l2_fo  static int
> > wave5_vpu_dec_s_fmt_out(struct file *file, void *fh, struct v4l2_format
> *f)  {
> >  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> > +	const struct vpu_format *vpu_fmt;
> >  	int i, ret;
> >
> >  	dev_dbg(inst->dev->dev,
> > @@ -782,7 +714,13 @@ static int wave5_vpu_dec_s_fmt_out(struct file
> *file, void *fh, struct v4l2_form
> >  	inst->quantization = f->fmt.pix_mp.quantization;
> >  	inst->xfer_func = f->fmt.pix_mp.xfer_func;
> >
> > -	wave5_update_pix_fmt(&inst->dst_fmt, f->fmt.pix_mp.width, f-
> >fmt.pix_mp.height);
> > +	vpu_fmt = wave5_find_vpu_fmt(inst->dst_fmt.pixelformat,
> dec_fmt_list[VPU_FMT_TYPE_RAW]);
> > +	if (!vpu_fmt)
> > +		return -EINVAL;
> > +
> > +	wave5_update_pix_fmt(&inst->dst_fmt, VPU_FMT_TYPE_RAW,
> > +			     f->fmt.pix_mp.width, f->fmt.pix_mp.height,
> > +			     vpu_fmt->v4l2_frmsize);
> >
> >  	return 0;
> >  }
> > @@ -1005,6 +943,7 @@ static int wave5_vpu_dec_queue_setup(struct
> vb2_queue *q, unsigned int *num_buff
> >  	struct vpu_instance *inst = vb2_get_drv_priv(q);
> >  	struct v4l2_pix_format_mplane inst_format =
> >  		(q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) ? inst-
> >src_fmt :
> > inst->dst_fmt;
> > +	unsigned int i;
> >
> >  	dev_dbg(inst->dev->dev, "%s: num_buffers: %u | num_planes: %u |
> type: %u\n", __func__,
> >  		*num_buffers, *num_planes, q->type); @@ -1018,31 +957,9 @@
> static
> > int wave5_vpu_dec_queue_setup(struct vb2_queue *q, unsigned int
> *num_buff
> >  		if (*num_buffers < inst->fbc_buf_count)
> >  			*num_buffers = inst->fbc_buf_count;
> >
> > -		if (*num_planes == 1) {
> > -			if (inst->output_format == FORMAT_422)
> > -				sizes[0] = inst_format.width *
> inst_format.height * 2;
> > -			else
> > -				sizes[0] = inst_format.width *
> inst_format.height * 3 / 2;
> > -			dev_dbg(inst->dev->dev, "%s: size[0]: %u\n", __func__,
> sizes[0]);
> > -		} else if (*num_planes == 2) {
> > -			sizes[0] = inst_format.width * inst_format.height;
> > -			if (inst->output_format == FORMAT_422)
> > -				sizes[1] = inst_format.width *
> inst_format.height;
> > -			else
> > -				sizes[1] = inst_format.width *
> inst_format.height / 2;
> > -			dev_dbg(inst->dev->dev, "%s: size[0]: %u |
> size[1]: %u\n",
> > -				__func__, sizes[0], sizes[1]);
> > -		} else if (*num_planes == 3) {
> > -			sizes[0] = inst_format.width * inst_format.height;
> > -			if (inst->output_format == FORMAT_422) {
> > -				sizes[1] = inst_format.width *
> inst_format.height / 2;
> > -				sizes[2] = inst_format.width *
> inst_format.height / 2;
> > -			} else {
> > -				sizes[1] = inst_format.width *
> inst_format.height / 4;
> > -				sizes[2] = inst_format.width *
> inst_format.height / 4;
> > -			}
> > -			dev_dbg(inst->dev->dev, "%s: size[0]: %u | size[1]: %u
> | size[2]: %u\n",
> > -				__func__, sizes[0], sizes[1], sizes[2]);
> > +		for (i = 0; i < *num_planes; i++) {
> > +			sizes[i] = inst_format.plane_fmt[i].sizeimage;
> > +			dev_dbg(inst->dev->dev, "%s: size[%u]: %u\n", __func__,
> i,
> > +sizes[i]);
> >  		}
> >  	}
> >
> > @@ -1568,20 +1485,15 @@ static const struct vb2_ops
> > wave5_vpu_dec_vb2_ops = {  static void wave5_set_default_format(struct
> v4l2_pix_format_mplane *src_fmt,
> >  				     struct v4l2_pix_format_mplane *dst_fmt)  {
> > -	unsigned int dst_pix_fmt =
> dec_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
> > -	const struct v4l2_format_info *dst_fmt_info =
> v4l2_format_info(dst_pix_fmt);
> > -
> >  	src_fmt->pixelformat =
> dec_fmt_list[VPU_FMT_TYPE_CODEC][0].v4l2_pix_fmt;
> > -	src_fmt->field = V4L2_FIELD_NONE;
> > -	src_fmt->flags = 0;
> > -	src_fmt->num_planes = 1;
> > -	wave5_update_pix_fmt(src_fmt, 720, 480);
> > -
> > -	dst_fmt->pixelformat = dst_pix_fmt;
> > -	dst_fmt->field = V4L2_FIELD_NONE;
> > -	dst_fmt->flags = 0;
> > -	dst_fmt->num_planes = dst_fmt_info->mem_planes;
> > -	wave5_update_pix_fmt(dst_fmt, 736, 480);
> > +	wave5_update_pix_fmt(src_fmt, VPU_FMT_TYPE_CODEC,
> > +			     W5_DEF_DEC_PIC_WIDTH, W5_DEF_DEC_PIC_HEIGHT,
> > +			     &dec_hevc_frmsize);
> > +
> > +	dst_fmt->pixelformat =
> dec_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
> > +	wave5_update_pix_fmt(dst_fmt, VPU_FMT_TYPE_RAW,
> > +			     W5_DEF_DEC_PIC_WIDTH, W5_DEF_DEC_PIC_HEIGHT,
> > +			     &dec_raw_frmsize);
> >  }
> >
> >  static int wave5_vpu_dec_queue_init(void *priv, struct vb2_queue
> > *src_vq, struct vb2_queue *dst_vq) diff --git
> > a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > index 985de0c293e2..ef9aa1562346 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
> > @@ -11,65 +11,60 @@
> >  #define VPU_ENC_DEV_NAME "C&M Wave5 VPU encoder"
> >  #define VPU_ENC_DRV_NAME "wave5-enc"
> >
> > +static const struct v4l2_frmsize_stepwise enc_frmsize[FMT_TYPES] = {
> > +	[VPU_FMT_TYPE_CODEC] = {
> > +		.min_width = W5_MIN_ENC_PIC_WIDTH,
> > +		.max_width = W5_MAX_ENC_PIC_WIDTH,
> > +		.step_width = W5_ENC_CODEC_STEP_WIDTH,
> > +		.min_height = W5_MIN_ENC_PIC_HEIGHT,
> > +		.max_height = W5_MAX_ENC_PIC_HEIGHT,
> > +		.step_height = W5_ENC_CODEC_STEP_HEIGHT,
> > +	},
> > +	[VPU_FMT_TYPE_RAW] = {
> > +		.min_width = W5_MIN_ENC_PIC_WIDTH,
> > +		.max_width = W5_MAX_ENC_PIC_WIDTH,
> > +		.step_width = W5_ENC_RAW_STEP_WIDTH,
> > +		.min_height = W5_MIN_ENC_PIC_HEIGHT,
> > +		.max_height = W5_MAX_ENC_PIC_HEIGHT,
> > +		.step_height = W5_ENC_RAW_STEP_HEIGHT,
> > +	},
> > +};
> > +
> >  static const struct vpu_format enc_fmt_list[FMT_TYPES][MAX_FMTS] = {
> >  	[VPU_FMT_TYPE_CODEC] = {
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_HEVC,
> > -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> > -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> > -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> > -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> > +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_CODEC],
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_H264,
> > -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> > -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> > -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> > -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> > +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_CODEC],
> >  		},
> >  	},
> >  	[VPU_FMT_TYPE_RAW] = {
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420,
> > -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> > -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> > -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> > -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> > +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12,
> > -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> > -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> > -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> > -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> > +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21,
> > -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> > -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> > -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> > -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> > +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV420M,
> > -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> > -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> > -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> > -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> > +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV12M,
> > -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> > -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> > -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> > -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> > +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
> >  		},
> >  		{
> >  			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21M,
> > -			.max_width = W5_MAX_ENC_PIC_WIDTH,
> > -			.min_width = W5_MIN_ENC_PIC_WIDTH,
> > -			.max_height = W5_MAX_ENC_PIC_HEIGHT,
> > -			.min_height = W5_MIN_ENC_PIC_HEIGHT,
> > +			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
> >  		},
> >  	}
> >  };
> > @@ -106,46 +101,6 @@ static int switch_state(struct vpu_instance *inst,
> enum vpu_instance_state state
> >  	return -EINVAL;
> >  }
> >
> > -static void wave5_update_pix_fmt(struct v4l2_pix_format_mplane *pix_mp,
> unsigned int width,
> > -				 unsigned int height)
> > -{
> > -	switch (pix_mp->pixelformat) {
> > -	case V4L2_PIX_FMT_YUV420:
> > -	case V4L2_PIX_FMT_NV12:
> > -	case V4L2_PIX_FMT_NV21:
> > -		pix_mp->width = width;
> > -		pix_mp->height = height;
> > -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[0].sizeimage = round_up(width, 32) *
> height * 3 / 2;
> > -		break;
> > -	case V4L2_PIX_FMT_YUV420M:
> > -		pix_mp->width = width;
> > -		pix_mp->height = height;
> > -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[0].sizeimage = round_up(width, 32) *
> height;
> > -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32) / 2;
> > -		pix_mp->plane_fmt[1].sizeimage = round_up(width, 32) *
> height / 4;
> > -		pix_mp->plane_fmt[2].bytesperline = round_up(width, 32) / 2;
> > -		pix_mp->plane_fmt[2].sizeimage = round_up(width, 32) *
> height / 4;
> > -		break;
> > -	case V4L2_PIX_FMT_NV12M:
> > -	case V4L2_PIX_FMT_NV21M:
> > -		pix_mp->width = width;
> > -		pix_mp->height = height;
> > -		pix_mp->plane_fmt[0].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[0].sizeimage = round_up(width, 32) *
> height;
> > -		pix_mp->plane_fmt[1].bytesperline = round_up(width, 32);
> > -		pix_mp->plane_fmt[1].sizeimage = round_up(width, 32) *
> height / 2;
> > -		break;
> > -	default:
> > -		pix_mp->width = width;
> > -		pix_mp->height = height;
> > -		pix_mp->plane_fmt[0].bytesperline = 0;
> > -		pix_mp->plane_fmt[0].sizeimage = width * height / 8 * 3;
> > -		break;
> > -	}
> > -}
> > -
> >  static int start_encode(struct vpu_instance *inst, u32 *fail_res)  {
> >  	struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx; @@ -360,13
> > +315,8 @@ static int wave5_vpu_enc_enum_framesizes(struct file *f, void
> *fh, struct v4l2_f
> >  			return -EINVAL;
> >  	}
> >
> > -	fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
> > -	fsize->stepwise.min_width = vpu_fmt->min_width;
> > -	fsize->stepwise.max_width = vpu_fmt->max_width;
> > -	fsize->stepwise.step_width = 1;
> > -	fsize->stepwise.min_height = vpu_fmt->min_height;
> > -	fsize->stepwise.max_height = vpu_fmt->max_height;
> > -	fsize->stepwise.step_height = 1;
> > +	fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
> > +	fsize->stepwise = enc_frmsize[VPU_FMT_TYPE_CODEC];
> >
> >  	return 0;
> >  }
> > @@ -391,7 +341,9 @@ static int wave5_vpu_enc_enum_fmt_cap(struct file
> > *file, void *fh, struct v4l2_f  static int
> > wave5_vpu_enc_try_fmt_cap(struct file *file, void *fh, struct
> v4l2_format *f)  {
> >  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> > +	const struct v4l2_frmsize_stepwise *frmsize;
> >  	const struct vpu_format *vpu_fmt;
> > +	int width, height;
> >
> >  	dev_dbg(inst->dev->dev, "%s: fourcc: %u width: %u height: %u
> num_planes: %u field: %u\n",
> >  		__func__, f->fmt.pix_mp.pixelformat, f->fmt.pix_mp.width,
> > f->fmt.pix_mp.height, @@ -399,20 +351,19 @@ static int
> > wave5_vpu_enc_try_fmt_cap(struct file *file, void *fh, struct v4l2_fo
> >
> >  	vpu_fmt = wave5_find_vpu_fmt(f->fmt.pix_mp.pixelformat,
> enc_fmt_list[VPU_FMT_TYPE_CODEC]);
> >  	if (!vpu_fmt) {
> > +		width = inst->dst_fmt.width;
> > +		height = inst->dst_fmt.height;
> >  		f->fmt.pix_mp.pixelformat = inst->dst_fmt.pixelformat;
> > -		f->fmt.pix_mp.num_planes = inst->dst_fmt.num_planes;
> > -		wave5_update_pix_fmt(&f->fmt.pix_mp, inst->dst_fmt.width,
> inst->dst_fmt.height);
> > +		frmsize = &enc_frmsize[VPU_FMT_TYPE_CODEC];
> >  	} else {
> > -		int width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width,
> vpu_fmt->max_width);
> > -		int height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height,
> vpu_fmt->max_height);
> > -
> > +		width = f->fmt.pix_mp.width;
> > +		height = f->fmt.pix_mp.height;
> >  		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
> > -		f->fmt.pix_mp.num_planes = 1;
> > -		wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
> > +		frmsize = vpu_fmt->v4l2_frmsize;
> >  	}
> >
> > -	f->fmt.pix_mp.flags = 0;
> > -	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
> > +	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_CODEC,
> > +			     width, height, frmsize);
> >  	f->fmt.pix_mp.colorspace = inst->colorspace;
> >  	f->fmt.pix_mp.ycbcr_enc = inst->ycbcr_enc;
> >  	f->fmt.pix_mp.quantization = inst->quantization; @@ -499,7 +450,9
> @@
> > static int wave5_vpu_enc_enum_fmt_out(struct file *file, void *fh,
> > struct v4l2_f  static int wave5_vpu_enc_try_fmt_out(struct file *file,
> > void *fh, struct v4l2_format *f)  {
> >  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> > +	const struct v4l2_frmsize_stepwise *frmsize;
> >  	const struct vpu_format *vpu_fmt;
> > +	int width, height;
> >
> >  	dev_dbg(inst->dev->dev, "%s: fourcc: %u width: %u height: %u
> num_planes: %u field: %u\n",
> >  		__func__, f->fmt.pix_mp.pixelformat, f->fmt.pix_mp.width,
> > f->fmt.pix_mp.height, @@ -507,28 +460,26 @@ static int
> > wave5_vpu_enc_try_fmt_out(struct file *file, void *fh, struct v4l2_fo
> >
> >  	vpu_fmt = wave5_find_vpu_fmt(f->fmt.pix_mp.pixelformat,
> enc_fmt_list[VPU_FMT_TYPE_RAW]);
> >  	if (!vpu_fmt) {
> > +		width = inst->src_fmt.width;
> > +		height = inst->src_fmt.height;
> >  		f->fmt.pix_mp.pixelformat = inst->src_fmt.pixelformat;
> > -		f->fmt.pix_mp.num_planes = inst->src_fmt.num_planes;
> > -		wave5_update_pix_fmt(&f->fmt.pix_mp, inst->src_fmt.width,
> inst->src_fmt.height);
> > +		frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW];
> >  	} else {
> > -		int width = clamp(f->fmt.pix_mp.width, vpu_fmt->min_width,
> vpu_fmt->max_width);
> > -		int height = clamp(f->fmt.pix_mp.height, vpu_fmt->min_height,
> vpu_fmt->max_height);
> > -		const struct v4l2_format_info *info =
> v4l2_format_info(vpu_fmt->v4l2_pix_fmt);
> > -
> > +		width = f->fmt.pix_mp.width;
> > +		height = f->fmt.pix_mp.height;
> >  		f->fmt.pix_mp.pixelformat = vpu_fmt->v4l2_pix_fmt;
> > -		f->fmt.pix_mp.num_planes = info->mem_planes;
> > -		wave5_update_pix_fmt(&f->fmt.pix_mp, width, height);
> > +		frmsize = vpu_fmt->v4l2_frmsize;
> >  	}
> >
> > -	f->fmt.pix_mp.flags = 0;
> > -	f->fmt.pix_mp.field = V4L2_FIELD_NONE;
> > -
> > +	wave5_update_pix_fmt(&f->fmt.pix_mp, VPU_FMT_TYPE_RAW,
> > +			     width, height, frmsize);
> >  	return 0;
> >  }
> >
> >  static int wave5_vpu_enc_s_fmt_out(struct file *file, void *fh,
> > struct v4l2_format *f)  {
> >  	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
> > +	const struct vpu_format *vpu_fmt;
> >  	int i, ret;
> >
> >  	dev_dbg(inst->dev->dev, "%s: fourcc: %u width: %u height: %u
> > num_planes: %u field: %u\n", @@ -568,7 +519,15 @@ static int
> wave5_vpu_enc_s_fmt_out(struct file *file, void *fh, struct v4l2_form
> >  	inst->quantization = f->fmt.pix_mp.quantization;
> >  	inst->xfer_func = f->fmt.pix_mp.xfer_func;
> >
> > -	wave5_update_pix_fmt(&inst->dst_fmt, f->fmt.pix_mp.width, f-
> >fmt.pix_mp.height);
> > +	vpu_fmt = wave5_find_vpu_fmt(inst->dst_fmt.pixelformat,
> enc_fmt_list[VPU_FMT_TYPE_CODEC]);
> > +	if (!vpu_fmt)
> > +		return -EINVAL;
> > +
> > +	wave5_update_pix_fmt(&inst->dst_fmt, VPU_FMT_TYPE_CODEC,
> > +			     f->fmt.pix_mp.width, f->fmt.pix_mp.height,
> > +			     vpu_fmt->v4l2_frmsize);
> > +	inst->conf_win.width = inst->dst_fmt.width;
> > +	inst->conf_win.height = inst->dst_fmt.height;
> >
> >  	return 0;
> >  }
> > @@ -584,12 +543,17 @@ static int wave5_vpu_enc_g_selection(struct file
> *file, void *fh, struct v4l2_se
> >  	switch (s->target) {
> >  	case V4L2_SEL_TGT_CROP_DEFAULT:
> >  	case V4L2_SEL_TGT_CROP_BOUNDS:
> > -	case V4L2_SEL_TGT_CROP:
> >  		s->r.left = 0;
> >  		s->r.top = 0;
> >  		s->r.width = inst->dst_fmt.width;
> >  		s->r.height = inst->dst_fmt.height;
> >  		break;
> > +	case V4L2_SEL_TGT_CROP:
> > +		s->r.left = 0;
> > +		s->r.top = 0;
> > +		s->r.width = inst->conf_win.width;
> > +		s->r.height = inst->conf_win.height;
> > +		break;
> >  	default:
> >  		return -EINVAL;
> >  	}
> > @@ -612,8 +576,10 @@ static int wave5_vpu_enc_s_selection(struct file
> > *file, void *fh, struct v4l2_se
> >
> >  	s->r.left = 0;
> >  	s->r.top = 0;
> > -	s->r.width = inst->src_fmt.width;
> > -	s->r.height = inst->src_fmt.height;
> > +	s->r.width = min(s->r.width, inst->dst_fmt.width);
> > +	s->r.height = min(s->r.height, inst->dst_fmt.height);
> > +
> > +	inst->conf_win = s->r;
> >
> >  	return 0;
> >  }
> > @@ -1151,8 +1117,8 @@ static void wave5_set_enc_openparam(struct
> enc_open_param *open_param,
> >  	open_param->wave_param.lambda_scaling_enable = 1;
> >
> >  	open_param->line_buf_int_en = true;
> > -	open_param->pic_width = inst->dst_fmt.width;
> > -	open_param->pic_height = inst->dst_fmt.height;
> > +	open_param->pic_width = inst->conf_win.width;
> > +	open_param->pic_height = inst->conf_win.height;
> >  	open_param->frame_rate_info = inst->frame_rate;
> >  	open_param->rc_enable = inst->rc_enable;
> >  	if (inst->rc_enable) {
> > @@ -1456,20 +1422,15 @@ static const struct vb2_ops
> > wave5_vpu_enc_vb2_ops = {  static void wave5_set_default_format(struct
> v4l2_pix_format_mplane *src_fmt,
> >  				     struct v4l2_pix_format_mplane *dst_fmt)  {
> > -	unsigned int src_pix_fmt =
> enc_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
> > -	const struct v4l2_format_info *src_fmt_info =
> v4l2_format_info(src_pix_fmt);
> > -
> > -	src_fmt->pixelformat = src_pix_fmt;
> > -	src_fmt->field = V4L2_FIELD_NONE;
> > -	src_fmt->flags = 0;
> > -	src_fmt->num_planes = src_fmt_info->mem_planes;
> > -	wave5_update_pix_fmt(src_fmt, 416, 240);
> > +	src_fmt->pixelformat =
> enc_fmt_list[VPU_FMT_TYPE_RAW][0].v4l2_pix_fmt;
> > +	wave5_update_pix_fmt(src_fmt, VPU_FMT_TYPE_RAW,
> > +			     W5_DEF_ENC_PIC_WIDTH, W5_DEF_ENC_PIC_HEIGHT,
> > +			     &enc_frmsize[VPU_FMT_TYPE_RAW]);
> >
> >  	dst_fmt->pixelformat =
> enc_fmt_list[VPU_FMT_TYPE_CODEC][0].v4l2_pix_fmt;
> > -	dst_fmt->field = V4L2_FIELD_NONE;
> > -	dst_fmt->flags = 0;
> > -	dst_fmt->num_planes = 1;
> > -	wave5_update_pix_fmt(dst_fmt, 416, 240);
> > +	wave5_update_pix_fmt(dst_fmt, VPU_FMT_TYPE_CODEC,
> > +			     W5_DEF_ENC_PIC_WIDTH, W5_DEF_ENC_PIC_HEIGHT,
> > +			     &enc_frmsize[VPU_FMT_TYPE_CODEC]);
> >  }
> >
> >  static int wave5_vpu_enc_queue_init(void *priv, struct vb2_queue
> > *src_vq, struct vb2_queue *dst_vq) @@ -1733,6 +1694,8 @@ static int
> wave5_vpu_open_enc(struct file *filp)
> >  	v4l2_ctrl_handler_setup(v4l2_ctrl_hdl);
> >
> >  	wave5_set_default_format(&inst->src_fmt, &inst->dst_fmt);
> > +	inst->conf_win.width = inst->dst_fmt.width;
> > +	inst->conf_win.height = inst->dst_fmt.height;
> >  	inst->colorspace = V4L2_COLORSPACE_REC709;
> >  	inst->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >  	inst->quantization = V4L2_QUANTIZATION_DEFAULT; diff --git
> > a/drivers/media/platform/chips-media/wave5/wave5-vpu.h
> > b/drivers/media/platform/chips-media/wave5/wave5-vpu.h
> > index 32b7fd3730b5..3847332551fc 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpu.h
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.h
> > @@ -38,10 +38,7 @@ enum vpu_fmt_type {
> >
> >  struct vpu_format {
> >  	unsigned int v4l2_pix_fmt;
> > -	unsigned int max_width;
> > -	unsigned int min_width;
> > -	unsigned int max_height;
> > -	unsigned int min_height;
> > +	const struct v4l2_frmsize_stepwise *v4l2_frmsize;
> >  };
> >
> >  static inline struct vpu_instance *wave5_to_vpu_inst(struct v4l2_fh
> > *vfh) diff --git
> > a/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
> > b/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
> > index e4bc2e467cb5..1ea9f5f31499 100644
> > --- a/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
> > +++ b/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
> > @@ -32,10 +32,29 @@
> >
> >  #define MAX_NUM_INSTANCE                32
> >
> > -#define W5_MIN_ENC_PIC_WIDTH            256
> > -#define W5_MIN_ENC_PIC_HEIGHT           128
> > -#define W5_MAX_ENC_PIC_WIDTH            8192
> > -#define W5_MAX_ENC_PIC_HEIGHT           8192
> > +#define W5_DEF_DEC_PIC_WIDTH            720U
> > +#define W5_DEF_DEC_PIC_HEIGHT           480U
> > +#define W5_MIN_DEC_PIC_8_WIDTH          8U
> > +#define W5_MIN_DEC_PIC_8_HEIGHT         8U
> > +#define W5_MIN_DEC_PIC_32_WIDTH         32U
> > +#define W5_MIN_DEC_PIC_32_HEIGHT        32U
> > +#define W5_MAX_DEC_PIC_WIDTH            8192U
> > +#define W5_MAX_DEC_PIC_HEIGHT           4320U
> > +#define W5_DEC_CODEC_STEP_WIDTH         1U
> > +#define W5_DEC_CODEC_STEP_HEIGHT        1U
> > +#define W5_DEC_RAW_STEP_WIDTH           32U
> > +#define W5_DEC_RAW_STEP_HEIGHT          16U
> > +
> > +#define W5_DEF_ENC_PIC_WIDTH            416U
> > +#define W5_DEF_ENC_PIC_HEIGHT           240U
> > +#define W5_MIN_ENC_PIC_WIDTH            256U
> > +#define W5_MIN_ENC_PIC_HEIGHT           128U
> > +#define W5_MAX_ENC_PIC_WIDTH            8192U
> > +#define W5_MAX_ENC_PIC_HEIGHT           8192U
> > +#define W5_ENC_CODEC_STEP_WIDTH         8U
> > +#define W5_ENC_CODEC_STEP_HEIGHT        8U
> > +#define W5_ENC_RAW_STEP_WIDTH           32U
> > +#define W5_ENC_RAW_STEP_HEIGHT          16U
> >
> >  //  application specific configuration
> >  #define VPU_ENC_TIMEOUT                 60000

[RESEND PATCH v7 4/4] media: chips-media: wave5: Support YUV422 raw pixel-formats on the encoder.

[Jackson.lee]

Add support for the YUV422P, NV16, NV61, YUV422M, NV16M,
NV61M raw pixel-formats to the Wave5 encoder.

All these formats have a chroma subsampling ratio of 4:2:2 and
therefore require a new image size calculation as the driver
previously only handled a ratio of 4:2:0.

Signed-off-by: Jackson.lee <jackson.lee@chipsnmedia.com>
Signed-off-by: Nas Chung <nas.chung@chipsnmedia.com>
Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
---
 .../chips-media/wave5/wave5-vpu-enc.c         | 89 +++++++++++++++----
 1 file changed, 74 insertions(+), 15 deletions(-)

diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
index ef9aa1562346..4e1c8a4e7272 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
@@ -66,6 +66,30 @@ static const struct vpu_format enc_fmt_list[FMT_TYPES][MAX_FMTS] = {
 			.v4l2_pix_fmt = V4L2_PIX_FMT_NV21M,
 			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
 		},
+		{
+			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV422P,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
+		},
+		{
+			.v4l2_pix_fmt = V4L2_PIX_FMT_NV16,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
+		},
+		{
+			.v4l2_pix_fmt = V4L2_PIX_FMT_NV61,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
+		},
+		{
+			.v4l2_pix_fmt = V4L2_PIX_FMT_YUV422M,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
+		},
+		{
+			.v4l2_pix_fmt = V4L2_PIX_FMT_NV16M,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
+		},
+		{
+			.v4l2_pix_fmt = V4L2_PIX_FMT_NV61M,
+			.v4l2_frmsize = &enc_frmsize[VPU_FMT_TYPE_RAW],
+		},
 	}
 };
 
@@ -109,13 +133,26 @@ static int start_encode(struct vpu_instance *inst, u32 *fail_res)
 	struct vb2_v4l2_buffer *dst_buf;
 	struct frame_buffer frame_buf;
 	struct enc_param pic_param;
-	u32 stride = ALIGN(inst->dst_fmt.width, 32);
-	u32 luma_size = (stride * inst->dst_fmt.height);
-	u32 chroma_size = ((stride / 2) * (inst->dst_fmt.height / 2));
+	const struct v4l2_format_info *info;
+	u32 stride = inst->src_fmt.plane_fmt[0].bytesperline;
+	u32 luma_size = 0;
+	u32 chroma_size = 0;
 
 	memset(&pic_param, 0, sizeof(struct enc_param));
 	memset(&frame_buf, 0, sizeof(struct frame_buffer));
 
+	info = v4l2_format_info(inst->src_fmt.pixelformat);
+	if (!info)
+		return -EINVAL;
+
+	if (info->mem_planes == 1) {
+		luma_size = stride * inst->dst_fmt.height;
+		chroma_size = luma_size / (info->hdiv * info->vdiv);
+	} else {
+		luma_size = inst->src_fmt.plane_fmt[0].sizeimage;
+		chroma_size = inst->src_fmt.plane_fmt[1].sizeimage;
+	}
+
 	dst_buf = v4l2_m2m_next_dst_buf(m2m_ctx);
 	if (!dst_buf) {
 		dev_dbg(inst->dev->dev, "%s: No destination buffer found\n", __func__);
@@ -480,6 +517,7 @@ static int wave5_vpu_enc_s_fmt_out(struct file *file, void *fh, struct v4l2_form
 {
 	struct vpu_instance *inst = wave5_to_vpu_inst(fh);
 	const struct vpu_format *vpu_fmt;
+	const struct v4l2_format_info *info;
 	int i, ret;
 
 	dev_dbg(inst->dev->dev, "%s: fourcc: %u width: %u height: %u num_planes: %u field: %u\n",
@@ -501,16 +539,20 @@ static int wave5_vpu_enc_s_fmt_out(struct file *file, void *fh, struct v4l2_form
 		inst->src_fmt.plane_fmt[i].sizeimage = f->fmt.pix_mp.plane_fmt[i].sizeimage;
 	}
 
-	if (inst->src_fmt.pixelformat == V4L2_PIX_FMT_NV12 ||
-	    inst->src_fmt.pixelformat == V4L2_PIX_FMT_NV12M) {
-		inst->cbcr_interleave = true;
-		inst->nv21 = false;
-	} else if (inst->src_fmt.pixelformat == V4L2_PIX_FMT_NV21 ||
-		   inst->src_fmt.pixelformat == V4L2_PIX_FMT_NV21M) {
-		inst->cbcr_interleave = true;
+	info = v4l2_format_info(inst->src_fmt.pixelformat);
+	if (!info)
+		return -EINVAL;
+
+	inst->cbcr_interleave = (info->comp_planes == 2) ? true : false;
+
+	switch (inst->src_fmt.pixelformat) {
+	case V4L2_PIX_FMT_NV21:
+	case V4L2_PIX_FMT_NV21M:
+	case V4L2_PIX_FMT_NV61:
+	case V4L2_PIX_FMT_NV61M:
 		inst->nv21 = true;
-	} else {
-		inst->cbcr_interleave = false;
+		break;
+	default:
 		inst->nv21 = false;
 	}
 
@@ -1095,13 +1137,23 @@ static void wave5_vpu_enc_buf_queue(struct vb2_buffer *vb)
 	v4l2_m2m_buf_queue(m2m_ctx, vbuf);
 }
 
-static void wave5_set_enc_openparam(struct enc_open_param *open_param,
-				    struct vpu_instance *inst)
+static int wave5_set_enc_openparam(struct enc_open_param *open_param,
+				   struct vpu_instance *inst)
 {
 	struct enc_wave_param input = inst->enc_param;
+	const struct v4l2_format_info *info;
 	u32 num_ctu_row = ALIGN(inst->dst_fmt.height, 64) / 64;
 	u32 num_mb_row = ALIGN(inst->dst_fmt.height, 16) / 16;
 
+	info = v4l2_format_info(inst->src_fmt.pixelformat);
+	if (!info)
+		return -EINVAL;
+
+	if (info->hdiv == 2 && info->vdiv == 1)
+		open_param->src_format = FORMAT_422;
+	else
+		open_param->src_format = FORMAT_420;
+
 	open_param->wave_param.gop_preset_idx = PRESET_IDX_IPP_SINGLE;
 	open_param->wave_param.hvs_qp_scale = 2;
 	open_param->wave_param.hvs_max_delta_qp = 10;
@@ -1190,6 +1242,8 @@ static void wave5_set_enc_openparam(struct enc_open_param *open_param,
 			open_param->wave_param.intra_refresh_arg = num_ctu_row;
 	}
 	open_param->wave_param.forced_idr_header_enable = input.forced_idr_header_enable;
+
+	return 0;
 }
 
 static int initialize_sequence(struct vpu_instance *inst)
@@ -1285,7 +1339,12 @@ static int wave5_vpu_enc_start_streaming(struct vb2_queue *q, unsigned int count
 
 		memset(&open_param, 0, sizeof(struct enc_open_param));
 
-		wave5_set_enc_openparam(&open_param, inst);
+		ret = wave5_set_enc_openparam(&open_param, inst);
+		if (ret) {
+			dev_dbg(inst->dev->dev, "%s: wave5_set_enc_openparam, fail: %d\n",
+				__func__, ret);
+			goto return_buffers;
+		}
 
 		ret = wave5_vpu_enc_open(inst, &open_param);
 		if (ret) {
-- 
2.43.0

RE: [RESEND PATCH v7 0/4] Add features to an existing driver

[jackson.lee]

Hi nicolas and Sebastian

I sent v7 patch for upstreaming , can you review the following ?

https://patchwork.linuxtv.org/project/linux-media/cover/20240812070823.125-1-jackson.lee@chipsnmedia.com/

thanks


> -----Original Message-----
> From: jackson.lee
> Sent: Monday, August 12, 2024 4:09 PM
> To: mchehab@kernel.org; nicolas@ndufresne.ca; sebastian.fricke@collabora.com
> Cc: linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
> hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com; jackson.lee
> <jackson.lee@chipsnmedia.com>
> Subject: [RESEND PATCH v7 0/4] Add features to an existing driver
> 
> The wave5 codec driver is a stateful encoder/decoder.
> The following patches is for supporting yuv422 inpuy format, supporting
> runtime suspend/resume feature and extra things.
> 
> v4l2-compliance results:
> ========================
> 
> v4l2-compliance 1.24.1, 64 bits, 64-bit time_t
> 
> Buffer ioctls:
>        warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported
>        warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported
>     test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>     test VIDIOC_EXPBUF: OK
>     test Requests: OK (Not Supported)
> 
> Total for wave5-dec device /dev/video0: 45, Succeeded: 45, Failed: 0,
> Warnings: 2 Total for wave5-enc device /dev/video1: 45, Succeeded: 45, Failed:
> 0, Warnings: 0
> 
> Fluster test results:
> =====================
> 
> Running test suite JCT-VC-HEVC_V1 with decoder GStreamer-H.265-V4L2-Gst1.0
> Using 1 parallel job(s)
> Ran 132/147 tests successfully               in 89.870 secs
> 
> (1 test fails because of not supporting to parse multi frames, 1 test fails
> because of a missing frame and slight corruption,
>  2 tests fail because of sizes which are incompatible with the IP, 11 tests
> fail because of unsupported 10 bit format)
> 
> Running test suite JVT-AVC_V1 with decoder GStreamer-H.264-V4L2-Gst1.0 Using
> 1 parallel job(s)
> Ran 77/135 tests successfully               in 30.364 secs
> 
> (58 fail because the hardware is unable to decode  MBAFF / FMO / Field /
> Extended profile streams.)
> 
> Change since v6:
> ================
> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime
> suspend/resume
>  - Change autosuspend delay to 100ms
>  - Add to enable/disable hrtimer in the runtime suspend/resume function for
> hw not using irq
> 
> Change since v5:
> ================
> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to calculate
> bytesperline and sizeimage.
>  - Fix v4l2-compliance error for the vidioc_enum_framesizes
> 
> * For [PATCH v4 1/4] media: chips-media: wave5: Support SPS/PPS generation
> for each IDR
>  - Remove warning messages for the checkpatch.pl script
> 
> Change since v4:
> ================
> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime
> suspend/resume
>  - Fix warning message
> 
> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to calculate
> bytesperline and sizeimage.
>  - Fix warning message
>  - add Reviewed-By tag
> 
> * For [PATCH v4 4/4] media: chips-media: wave5: Support YUV422 raw pixel-
> formats on the encoder
>  - add Reviewed-By tag
> 
> Change since v3:
> =================
> 
> * For [PATCH v4 1/4] media: chips-media: wave5: Support SPS/PPS generation
> for each IDR
>  - add Reviewed-By tag
> 
> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime
> suspend/resume
>  - add Reviewed-By tag
> 
> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to calculate
> bytesperline and sizeimage.
>  - modify the commit message
>  - define three framesize structures for decoder
> 
> * For [PATCH v4 4/4] media: chips-media: wave5: Support YUV422 raw pixel-
> formats on the encoder
>  - modify the commit message
>  - use the v4l2_format_info to calculate luma, chroma size
> 
> Change since v2:
> =================
> 
> * For [PATCH v3 0/4] media: chips-media: wave5: Support SPS/PPS generation
> for each IDR
>  - add the suggested _SHIFT suffix
> 
> * For [PATCH v3 1/4] media: chips-media: wave5: Support runtime
> suspend/resume
>  - change a commit message
> 
> * For [PATCH v3 2/4] media: chips-media: wave5: Use helpers to calculate
> bytesperline and sizeimage
>  - add pix_fmt_type parameter into wave5_update_pix_fmt function
>  - add min/max width/height values into dec_fmt_list
> 
> Change since v1:
> =================
> 
> * For [PATCH v2 0/4] media: chips-media: wave5: Support SPS/PPS generation
> for each IDR
>  - define a macro for register addresses
> 
> * For [PATCH v2 1/4] media: chips-media: wave5: Support runtime
> suspend/resume
>  - add auto suspend/resume
> 
> * For [PATCH v2 2/4] media: chips-media: wave5: Use helpers to calculate
> bytesperline and sizeimage
>  - use helper functions to calculate bytesperline and sizeimage
> 
> * For [PATCH v2 3/4] media: chips-media: wave5: Support YUV422 raw pixel-
> formats on the encoder
>  - remove unnecessary codes
> 
> Change since v0:
> =================
> The DEFAULT_SRC_SIZE macro was defined using multiple lines, To make a simple
> define, tab and multiple lines has been removed, The macro is defined using
> one line.
> 
> Jackson.lee (4):
>   media: chips-media: wave5: Support SPS/PPS generation for each IDR
>   media: chips-media: wave5: Support runtime suspend/resume
>   media: chips-media: wave5: Use helpers to calculate bytesperline and
>     sizeimage.
>   media: chips-media: wave5: Support YUV422 raw pixel-formats on the
>     encoder.
> 
>  .../platform/chips-media/wave5/wave5-helper.c |  37 +-
>  .../platform/chips-media/wave5/wave5-helper.h |   5 +
>  .../platform/chips-media/wave5/wave5-hw.c     |  30 +-
>  .../chips-media/wave5/wave5-vpu-dec.c         | 321 +++++++-----------
>  .../chips-media/wave5/wave5-vpu-enc.c         | 313 +++++++++--------
>  .../platform/chips-media/wave5/wave5-vpu.c    |  50 +++
>  .../platform/chips-media/wave5/wave5-vpu.h    |   5 +-
>  .../platform/chips-media/wave5/wave5-vpuapi.c |  33 +-
>  .../platform/chips-media/wave5/wave5-vpuapi.h |   1 +
>  .../chips-media/wave5/wave5-vpuconfig.h       |  27 +-
>  .../media/platform/chips-media/wave5/wave5.h  |   3 +
>  11 files changed, 460 insertions(+), 365 deletions(-)
> 
> --
> 2.43.0

[RESEND PATCH v7 0/4] Add features to an existing driver

[jackson.lee]

Hi nicolas and Sebastian

I sent v7 patch for upstreaming , can you review the following ?

https://patchwork.linuxtv.org/project/linux-media/cover/20240812070823.125-1-jackson.lee@chipsnmedia.com/

thanks


> -----Original Message-----
> From: jackson.lee
> Sent: Monday, August 12, 2024 4:09 PM
> To: mchehab@kernel.org; nicolas@ndufresne.ca; 
> sebastian.fricke@collabora.com
> Cc: linux-media@vger.kernel.org; linux-kernel@vger.kernel.org; 
> hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim 
> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com; jackson.lee 
> <jackson.lee@chipsnmedia.com>
> Subject: [RESEND PATCH v7 0/4] Add features to an existing driver
> 
> The wave5 codec driver is a stateful encoder/decoder.
> The following patches is for supporting yuv422 inpuy format, 
> supporting runtime suspend/resume feature and extra things.
> 
> v4l2-compliance results:
> ========================
> 
> v4l2-compliance 1.24.1, 64 bits, 64-bit time_t
> 
> Buffer ioctls:
>        warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported
>        warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported
>     test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>     test VIDIOC_EXPBUF: OK
>     test Requests: OK (Not Supported)
> 
> Total for wave5-dec device /dev/video0: 45, Succeeded: 45, Failed: 0,
> Warnings: 2 Total for wave5-enc device /dev/video1: 45, Succeeded: 45, Failed:
> 0, Warnings: 0
> 
> Fluster test results:
> =====================
> 
> Running test suite JCT-VC-HEVC_V1 with decoder 
> GStreamer-H.265-V4L2-Gst1.0 Using 1 parallel job(s)
> Ran 132/147 tests successfully               in 89.870 secs
> 
> (1 test fails because of not supporting to parse multi frames, 1 test 
> fails because of a missing frame and slight corruption,
>  2 tests fail because of sizes which are incompatible with the IP, 11 
> tests fail because of unsupported 10 bit format)
> 
> Running test suite JVT-AVC_V1 with decoder GStreamer-H.264-V4L2-Gst1.0 
> Using
> 1 parallel job(s)
> Ran 77/135 tests successfully               in 30.364 secs
> 
> (58 fail because the hardware is unable to decode  MBAFF / FMO / Field 
> / Extended profile streams.)
> 
> Change since v6:
> ================
> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime 
> suspend/resume
>  - Change autosuspend delay to 100ms
>  - Add to enable/disable hrtimer in the runtime suspend/resume 
> function for hw not using irq
> 
> Change since v5:
> ================
> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to 
> calculate bytesperline and sizeimage.
>  - Fix v4l2-compliance error for the vidioc_enum_framesizes
> 
> * For [PATCH v4 1/4] media: chips-media: wave5: Support SPS/PPS 
> generation for each IDR
>  - Remove warning messages for the checkpatch.pl script
> 
> Change since v4:
> ================
> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime 
> suspend/resume
>  - Fix warning message
> 
> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to 
> calculate bytesperline and sizeimage.
>  - Fix warning message
>  - add Reviewed-By tag
> 
> * For [PATCH v4 4/4] media: chips-media: wave5: Support YUV422 raw 
> pixel- formats on the encoder
>  - add Reviewed-By tag
> 
> Change since v3:
> =================
> 
> * For [PATCH v4 1/4] media: chips-media: wave5: Support SPS/PPS 
> generation for each IDR
>  - add Reviewed-By tag
> 
> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime 
> suspend/resume
>  - add Reviewed-By tag
> 
> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to 
> calculate bytesperline and sizeimage.
>  - modify the commit message
>  - define three framesize structures for decoder
> 
> * For [PATCH v4 4/4] media: chips-media: wave5: Support YUV422 raw 
> pixel- formats on the encoder
>  - modify the commit message
>  - use the v4l2_format_info to calculate luma, chroma size
> 
> Change since v2:
> =================
> 
> * For [PATCH v3 0/4] media: chips-media: wave5: Support SPS/PPS 
> generation for each IDR
>  - add the suggested _SHIFT suffix
> 
> * For [PATCH v3 1/4] media: chips-media: wave5: Support runtime 
> suspend/resume
>  - change a commit message
> 
> * For [PATCH v3 2/4] media: chips-media: wave5: Use helpers to 
> calculate bytesperline and sizeimage
>  - add pix_fmt_type parameter into wave5_update_pix_fmt function
>  - add min/max width/height values into dec_fmt_list
> 
> Change since v1:
> =================
> 
> * For [PATCH v2 0/4] media: chips-media: wave5: Support SPS/PPS 
> generation for each IDR
>  - define a macro for register addresses
> 
> * For [PATCH v2 1/4] media: chips-media: wave5: Support runtime 
> suspend/resume
>  - add auto suspend/resume
> 
> * For [PATCH v2 2/4] media: chips-media: wave5: Use helpers to 
> calculate bytesperline and sizeimage
>  - use helper functions to calculate bytesperline and sizeimage
> 
> * For [PATCH v2 3/4] media: chips-media: wave5: Support YUV422 raw 
> pixel- formats on the encoder
>  - remove unnecessary codes
> 
> Change since v0:
> =================
> The DEFAULT_SRC_SIZE macro was defined using multiple lines, To make a 
> simple define, tab and multiple lines has been removed, The macro is 
> defined using one line.
> 
> Jackson.lee (4):
>   media: chips-media: wave5: Support SPS/PPS generation for each IDR
>   media: chips-media: wave5: Support runtime suspend/resume
>   media: chips-media: wave5: Use helpers to calculate bytesperline and
>     sizeimage.
>   media: chips-media: wave5: Support YUV422 raw pixel-formats on the
>     encoder.
> 
>  .../platform/chips-media/wave5/wave5-helper.c |  37 +-
>  .../platform/chips-media/wave5/wave5-helper.h |   5 +
>  .../platform/chips-media/wave5/wave5-hw.c     |  30 +-
>  .../chips-media/wave5/wave5-vpu-dec.c         | 321 +++++++-----------
>  .../chips-media/wave5/wave5-vpu-enc.c         | 313 +++++++++--------
>  .../platform/chips-media/wave5/wave5-vpu.c    |  50 +++
>  .../platform/chips-media/wave5/wave5-vpu.h    |   5 +-
>  .../platform/chips-media/wave5/wave5-vpuapi.c |  33 +-
>  .../platform/chips-media/wave5/wave5-vpuapi.h |   1 +
>  .../chips-media/wave5/wave5-vpuconfig.h       |  27 +-
>  .../media/platform/chips-media/wave5/wave5.h  |   3 +
>  11 files changed, 460 insertions(+), 365 deletions(-)
> 
> --
> 2.43.0

Re: [RESEND PATCH v7 0/4] Add features to an existing driver

[Sebastian Fricke]

Hey Jackson,

On 04.09.2024 00:30, jackson.lee wrote:
>
>Hi nicolas and Sebastian
>
>I sent v7 patch for upstreaming , can you review the following ?

Yes we have that on our list and look into that as soon as possible.

>
>https://patchwork.linuxtv.org/project/linux-media/cover/20240812070823.125-1-jackson.lee@chipsnmedia.com/
>
>thanks

Regards,
Sebastian

>
>
>> -----Original Message-----
>> From: jackson.lee
>> Sent: Monday, August 12, 2024 4:09 PM
>> To: mchehab@kernel.org; nicolas@ndufresne.ca;
>> sebastian.fricke@collabora.com
>> Cc: linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
>> hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
>> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com; jackson.lee
>> <jackson.lee@chipsnmedia.com>
>> Subject: [RESEND PATCH v7 0/4] Add features to an existing driver
>>
>> The wave5 codec driver is a stateful encoder/decoder.
>> The following patches is for supporting yuv422 inpuy format,
>> supporting runtime suspend/resume feature and extra things.
>>
>> v4l2-compliance results:
>> ========================
>>
>> v4l2-compliance 1.24.1, 64 bits, 64-bit time_t
>>
>> Buffer ioctls:
>>        warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported
>>        warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported
>>     test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>>     test VIDIOC_EXPBUF: OK
>>     test Requests: OK (Not Supported)
>>
>> Total for wave5-dec device /dev/video0: 45, Succeeded: 45, Failed: 0,
>> Warnings: 2 Total for wave5-enc device /dev/video1: 45, Succeeded: 45, Failed:
>> 0, Warnings: 0
>>
>> Fluster test results:
>> =====================
>>
>> Running test suite JCT-VC-HEVC_V1 with decoder
>> GStreamer-H.265-V4L2-Gst1.0 Using 1 parallel job(s)
>> Ran 132/147 tests successfully               in 89.870 secs
>>
>> (1 test fails because of not supporting to parse multi frames, 1 test
>> fails because of a missing frame and slight corruption,
>>  2 tests fail because of sizes which are incompatible with the IP, 11
>> tests fail because of unsupported 10 bit format)
>>
>> Running test suite JVT-AVC_V1 with decoder GStreamer-H.264-V4L2-Gst1.0
>> Using
>> 1 parallel job(s)
>> Ran 77/135 tests successfully               in 30.364 secs
>>
>> (58 fail because the hardware is unable to decode  MBAFF / FMO / Field
>> / Extended profile streams.)
>>
>> Change since v6:
>> ================
>> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime
>> suspend/resume
>>  - Change autosuspend delay to 100ms
>>  - Add to enable/disable hrtimer in the runtime suspend/resume
>> function for hw not using irq
>>
>> Change since v5:
>> ================
>> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to
>> calculate bytesperline and sizeimage.
>>  - Fix v4l2-compliance error for the vidioc_enum_framesizes
>>
>> * For [PATCH v4 1/4] media: chips-media: wave5: Support SPS/PPS
>> generation for each IDR
>>  - Remove warning messages for the checkpatch.pl script
>>
>> Change since v4:
>> ================
>> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime
>> suspend/resume
>>  - Fix warning message
>>
>> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to
>> calculate bytesperline and sizeimage.
>>  - Fix warning message
>>  - add Reviewed-By tag
>>
>> * For [PATCH v4 4/4] media: chips-media: wave5: Support YUV422 raw
>> pixel- formats on the encoder
>>  - add Reviewed-By tag
>>
>> Change since v3:
>> =================
>>
>> * For [PATCH v4 1/4] media: chips-media: wave5: Support SPS/PPS
>> generation for each IDR
>>  - add Reviewed-By tag
>>
>> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime
>> suspend/resume
>>  - add Reviewed-By tag
>>
>> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to
>> calculate bytesperline and sizeimage.
>>  - modify the commit message
>>  - define three framesize structures for decoder
>>
>> * For [PATCH v4 4/4] media: chips-media: wave5: Support YUV422 raw
>> pixel- formats on the encoder
>>  - modify the commit message
>>  - use the v4l2_format_info to calculate luma, chroma size
>>
>> Change since v2:
>> =================
>>
>> * For [PATCH v3 0/4] media: chips-media: wave5: Support SPS/PPS
>> generation for each IDR
>>  - add the suggested _SHIFT suffix
>>
>> * For [PATCH v3 1/4] media: chips-media: wave5: Support runtime
>> suspend/resume
>>  - change a commit message
>>
>> * For [PATCH v3 2/4] media: chips-media: wave5: Use helpers to
>> calculate bytesperline and sizeimage
>>  - add pix_fmt_type parameter into wave5_update_pix_fmt function
>>  - add min/max width/height values into dec_fmt_list
>>
>> Change since v1:
>> =================
>>
>> * For [PATCH v2 0/4] media: chips-media: wave5: Support SPS/PPS
>> generation for each IDR
>>  - define a macro for register addresses
>>
>> * For [PATCH v2 1/4] media: chips-media: wave5: Support runtime
>> suspend/resume
>>  - add auto suspend/resume
>>
>> * For [PATCH v2 2/4] media: chips-media: wave5: Use helpers to
>> calculate bytesperline and sizeimage
>>  - use helper functions to calculate bytesperline and sizeimage
>>
>> * For [PATCH v2 3/4] media: chips-media: wave5: Support YUV422 raw
>> pixel- formats on the encoder
>>  - remove unnecessary codes
>>
>> Change since v0:
>> =================
>> The DEFAULT_SRC_SIZE macro was defined using multiple lines, To make a
>> simple define, tab and multiple lines has been removed, The macro is
>> defined using one line.
>>
>> Jackson.lee (4):
>>   media: chips-media: wave5: Support SPS/PPS generation for each IDR
>>   media: chips-media: wave5: Support runtime suspend/resume
>>   media: chips-media: wave5: Use helpers to calculate bytesperline and
>>     sizeimage.
>>   media: chips-media: wave5: Support YUV422 raw pixel-formats on the
>>     encoder.
>>
>>  .../platform/chips-media/wave5/wave5-helper.c |  37 +-
>>  .../platform/chips-media/wave5/wave5-helper.h |   5 +
>>  .../platform/chips-media/wave5/wave5-hw.c     |  30 +-
>>  .../chips-media/wave5/wave5-vpu-dec.c         | 321 +++++++-----------
>>  .../chips-media/wave5/wave5-vpu-enc.c         | 313 +++++++++--------
>>  .../platform/chips-media/wave5/wave5-vpu.c    |  50 +++
>>  .../platform/chips-media/wave5/wave5-vpu.h    |   5 +-
>>  .../platform/chips-media/wave5/wave5-vpuapi.c |  33 +-
>>  .../platform/chips-media/wave5/wave5-vpuapi.h |   1 +
>>  .../chips-media/wave5/wave5-vpuconfig.h       |  27 +-
>>  .../media/platform/chips-media/wave5/wave5.h  |   3 +
>>  11 files changed, 460 insertions(+), 365 deletions(-)
>>
>> --
>> 2.43.0
>
>
Sebastian Fricke
Consultant Software Engineer

Collabora Ltd
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales no 5513718.

RE: [RESEND PATCH v7 0/4] Add features to an existing driver

[jackson.lee]

Thanks Sebastian.

> -----Original Message-----
> From: Sebastian Fricke <sebastian.fricke@collabora.com>
> Sent: Wednesday, September 4, 2024 5:21 PM
> To: jackson.lee <jackson.lee@chipsnmedia.com>
> Cc: Mauro Carvalho Chehab <mchehab@kernel.org>; Nicolas Dufresne
> <nicolas@ndufresne.ca>; linux-media@vger.kernel.org; linux-
> kernel@vger.kernel.org; Hans Verkuil <hverkuil@xs4all.nl>; Nas Chung
> <nas.chung@chipsnmedia.com>; lafley.kim <lafley.kim@chipsnmedia.com>;
> Brnich, Brandon <b-brnich@ti.com>
> Subject: Re: [RESEND PATCH v7 0/4] Add features to an existing driver
> 
> Hey Jackson,
> 
> On 04.09.2024 00:30, jackson.lee wrote:
> >
> >Hi nicolas and Sebastian
> >
> >I sent v7 patch for upstreaming , can you review the following ?
> 
> Yes we have that on our list and look into that as soon as possible.
> 
> >
> >https://patchwork.linuxtv.org/project/linux-media/cover/20240812070823.
> >125-1-jackson.lee@chipsnmedia.com/
> >
> >thanks
> 
> Regards,
> Sebastian
> 
> >
> >
> >> -----Original Message-----
> >> From: jackson.lee
> >> Sent: Monday, August 12, 2024 4:09 PM
> >> To: mchehab@kernel.org; nicolas@ndufresne.ca;
> >> sebastian.fricke@collabora.com
> >> Cc: linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
> >> hverkuil@xs4all.nl; Nas Chung <nas.chung@chipsnmedia.com>; lafley.kim
> >> <lafley.kim@chipsnmedia.com>; b-brnich@ti.com; jackson.lee
> >> <jackson.lee@chipsnmedia.com>
> >> Subject: [RESEND PATCH v7 0/4] Add features to an existing driver
> >>
> >> The wave5 codec driver is a stateful encoder/decoder.
> >> The following patches is for supporting yuv422 inpuy format,
> >> supporting runtime suspend/resume feature and extra things.
> >>
> >> v4l2-compliance results:
> >> ========================
> >>
> >> v4l2-compliance 1.24.1, 64 bits, 64-bit time_t
> >>
> >> Buffer ioctls:
> >>        warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported
> >>        warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported
> >>     test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> >>     test VIDIOC_EXPBUF: OK
> >>     test Requests: OK (Not Supported)
> >>
> >> Total for wave5-dec device /dev/video0: 45, Succeeded: 45, Failed: 0,
> >> Warnings: 2 Total for wave5-enc device /dev/video1: 45, Succeeded: 45,
> Failed:
> >> 0, Warnings: 0
> >>
> >> Fluster test results:
> >> =====================
> >>
> >> Running test suite JCT-VC-HEVC_V1 with decoder
> >> GStreamer-H.265-V4L2-Gst1.0 Using 1 parallel job(s)
> >> Ran 132/147 tests successfully               in 89.870 secs
> >>
> >> (1 test fails because of not supporting to parse multi frames, 1 test
> >> fails because of a missing frame and slight corruption,
> >>  2 tests fail because of sizes which are incompatible with the IP, 11
> >> tests fail because of unsupported 10 bit format)
> >>
> >> Running test suite JVT-AVC_V1 with decoder
> >> GStreamer-H.264-V4L2-Gst1.0 Using
> >> 1 parallel job(s)
> >> Ran 77/135 tests successfully               in 30.364 secs
> >>
> >> (58 fail because the hardware is unable to decode  MBAFF / FMO /
> >> Field / Extended profile streams.)
> >>
> >> Change since v6:
> >> ================
> >> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime
> >> suspend/resume
> >>  - Change autosuspend delay to 100ms
> >>  - Add to enable/disable hrtimer in the runtime suspend/resume
> >> function for hw not using irq
> >>
> >> Change since v5:
> >> ================
> >> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to
> >> calculate bytesperline and sizeimage.
> >>  - Fix v4l2-compliance error for the vidioc_enum_framesizes
> >>
> >> * For [PATCH v4 1/4] media: chips-media: wave5: Support SPS/PPS
> >> generation for each IDR
> >>  - Remove warning messages for the checkpatch.pl script
> >>
> >> Change since v4:
> >> ================
> >> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime
> >> suspend/resume
> >>  - Fix warning message
> >>
> >> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to
> >> calculate bytesperline and sizeimage.
> >>  - Fix warning message
> >>  - add Reviewed-By tag
> >>
> >> * For [PATCH v4 4/4] media: chips-media: wave5: Support YUV422 raw
> >> pixel- formats on the encoder
> >>  - add Reviewed-By tag
> >>
> >> Change since v3:
> >> =================
> >>
> >> * For [PATCH v4 1/4] media: chips-media: wave5: Support SPS/PPS
> >> generation for each IDR
> >>  - add Reviewed-By tag
> >>
> >> * For [PATCH v4 2/4] media: chips-media: wave5: Support runtime
> >> suspend/resume
> >>  - add Reviewed-By tag
> >>
> >> * For [PATCH v4 3/4] media: chips-media: wave5: Use helpers to
> >> calculate bytesperline and sizeimage.
> >>  - modify the commit message
> >>  - define three framesize structures for decoder
> >>
> >> * For [PATCH v4 4/4] media: chips-media: wave5: Support YUV422 raw
> >> pixel- formats on the encoder
> >>  - modify the commit message
> >>  - use the v4l2_format_info to calculate luma, chroma size
> >>
> >> Change since v2:
> >> =================
> >>
> >> * For [PATCH v3 0/4] media: chips-media: wave5: Support SPS/PPS
> >> generation for each IDR
> >>  - add the suggested _SHIFT suffix
> >>
> >> * For [PATCH v3 1/4] media: chips-media: wave5: Support runtime
> >> suspend/resume
> >>  - change a commit message
> >>
> >> * For [PATCH v3 2/4] media: chips-media: wave5: Use helpers to
> >> calculate bytesperline and sizeimage
> >>  - add pix_fmt_type parameter into wave5_update_pix_fmt function
> >>  - add min/max width/height values into dec_fmt_list
> >>
> >> Change since v1:
> >> =================
> >>
> >> * For [PATCH v2 0/4] media: chips-media: wave5: Support SPS/PPS
> >> generation for each IDR
> >>  - define a macro for register addresses
> >>
> >> * For [PATCH v2 1/4] media: chips-media: wave5: Support runtime
> >> suspend/resume
> >>  - add auto suspend/resume
> >>
> >> * For [PATCH v2 2/4] media: chips-media: wave5: Use helpers to
> >> calculate bytesperline and sizeimage
> >>  - use helper functions to calculate bytesperline and sizeimage
> >>
> >> * For [PATCH v2 3/4] media: chips-media: wave5: Support YUV422 raw
> >> pixel- formats on the encoder
> >>  - remove unnecessary codes
> >>
> >> Change since v0:
> >> =================
> >> The DEFAULT_SRC_SIZE macro was defined using multiple lines, To make
> >> a simple define, tab and multiple lines has been removed, The macro
> >> is defined using one line.
> >>
> >> Jackson.lee (4):
> >>   media: chips-media: wave5: Support SPS/PPS generation for each IDR
> >>   media: chips-media: wave5: Support runtime suspend/resume
> >>   media: chips-media: wave5: Use helpers to calculate bytesperline and
> >>     sizeimage.
> >>   media: chips-media: wave5: Support YUV422 raw pixel-formats on the
> >>     encoder.
> >>
> >>  .../platform/chips-media/wave5/wave5-helper.c |  37 +-
> >>  .../platform/chips-media/wave5/wave5-helper.h |   5 +
> >>  .../platform/chips-media/wave5/wave5-hw.c     |  30 +-
> >>  .../chips-media/wave5/wave5-vpu-dec.c         | 321 +++++++-----------
> >>  .../chips-media/wave5/wave5-vpu-enc.c         | 313 +++++++++--------
> >>  .../platform/chips-media/wave5/wave5-vpu.c    |  50 +++
> >>  .../platform/chips-media/wave5/wave5-vpu.h    |   5 +-
> >>  .../platform/chips-media/wave5/wave5-vpuapi.c |  33 +-
> >>  .../platform/chips-media/wave5/wave5-vpuapi.h |   1 +
> >>  .../chips-media/wave5/wave5-vpuconfig.h       |  27 +-
> >>  .../media/platform/chips-media/wave5/wave5.h  |   3 +
> >>  11 files changed, 460 insertions(+), 365 deletions(-)
> >>
> >> --
> >> 2.43.0
> >
> >
> Sebastian Fricke
> Consultant Software Engineer
> 
> Collabora Ltd
> Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
> Registered in England & Wales no 5513718.