Re: [PATCH v2 00/24] Enable H.264/H.265 encoder support and fixes in common code

[Dikshita Agarwal <quic_dikshita@quicinc.com>]

On 8/13/2025 8:57 PM, Neil Armstrong wrote:
> On 13/08/2025 11:37, Dikshita Agarwal wrote:
>> Hi All,
>>
>> This patch series adds support for H.264 and H.265 encoder in iris
>> driver and includes a few fixes and cleanup in the common code that were
>> identified during encoder bring-up process.
>>
>> The changes include:
>> - Enabling support for H.264 and H.265 encoding.
>> - Fixes and improvements in shared componenets used by both encoder and
>> decoder paths.
>> - Ensuring compatibility and stability with the existing decoder flow.
>>
>> Changes in v2:
>> - Fixed sparse/coccinnelle issues.
>> - Fixed the kernel doc warning.
>> - Removed unsupported PEAK_BITRATE property from SM8250.
>> - Dropped patch 04/25 to fix quality issue with encoder.
>> - Enhanced command handling for encoder to allow start/stop commands.
>> - Expanded rate control condition checks to include additional rate
>>    control types for HFI Gen2.
>> - Updated default value to MAX_QP for all caps related to max QP settings.
>> - Add support for INPUT/OUTPUT_BUF_HOST_MAX_COUNT caps for encoder.
>> - Link to v1:
>> https://lore.kernel.org/r/20250704-iris-video-encoder-v1-0-b6ce24e273cf@quicinc.com
>>
>> All patches have been tested with v4l2-compliance, v4l2-ctl and
>> Gstreamer on SM8250 and SM8550 for encoder, at the same time ensured
>> that the existing decoder functionality remains uneffected.
>>
>> Commands used for V4l2-ctl validation:
>>
>> v4l2-ctl --verbose
>> --set-fmt-video-out=width=1280,height=720,pixelformat=NV12
>> --set-selection-output target=crop,top=0,left=0,width=1280,height=720
>> --set-fmt-video=pixelformat=H264 --stream-mmap --stream-out-mmap
>> --stream-from=/media/cyclists_1280x720_92frames.yuv
>> --stream-to=/tmp/cyclists_1280x720_92frames.h264 -d /dev/video1
>>
>> v4l2-ctl --verbose
>> --set-fmt-video-out=width=1280,height=720,pixelformat=NV12
>> --set-selection-output target=crop,top=0,left=0,width=1280,height=720
>> --set-fmt-video=pixelformat=HEVC --stream-mmap --stream-out-mmap
>> --stream-from=/media/cyclists_1280x720_92frames.yuv
>> --stream-to=/tmp/cyclists_1280x720_92frames.hevc -d /dev/video1
>>
>> Commands used for GST validation:
>>
>> gst-launch-1.0 -v filesrc location=/media/cyclists_1280x720_92frames.yuv !
>> rawvideoparse format=nv12 width=1280 height=720 framerate=30/1 ! v4l2h264enc
>> capture-io-mode=4 output-io-mode=4 ! filesink sync=true
>> location=/tmp/gst_cyclists_1280x720_92frames.h264
>>
>> gst-launch-1.0 -v filesrc location=/media/cyclists_1280x720_92frames.yuv !
>> rawvideoparse format=nv12 width=1280 height=720 framerate=30/1 ! v4l2h265enc
>> capture-io-mode=4 output-io-mode=4 ! filesink sync=true
>> location=/tmp/gst_cyclists_1280x720_92frames.hevc
>>
>> The result of v4l2-compliance on SM8550:
>> v4l2-compliance 1.29.0-5270, 64 bits, 64-bit time_t
>> v4l2-compliance SHA: dc947661089e 2024-11-11 10:25:38
>>
>> Compliance test for iris_driver device /dev/video1:
>>
>> Driver Info:
>>          Driver name      : iris_driver
>>          Card type        : Iris Encoder
>>          Bus info         : platform:aa00000.video-codec
>>          Driver version   : 6.16.0
>>          Capabilities     : 0x84204000
>>                  Video Memory-to-Memory Multiplanar
>>                  Streaming
>>                  Extended Pix Format
>>                  Device Capabilities
>>          Device Caps      : 0x04204000
>>                  Video Memory-to-Memory Multiplanar
>>                  Streaming
>>                  Extended Pix Format
>>          Detected Stateful Encoder
>>
>> Required ioctls:
>>          test VIDIOC_QUERYCAP: OK
>>          test invalid ioctls: OK
>>
>> Allow for multiple opens:
>>          test second /dev/video1 open: OK
>>          test VIDIOC_QUERYCAP: OK
>>          test VIDIOC_G/S_PRIORITY: OK
>>          test for unlimited opens: OK
>>
>> Debug ioctls:
>>          test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
>>          test VIDIOC_LOG_STATUS: OK (Not Supported)
>>
>> Input ioctls:
>>          test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
>>          test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>>          test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
>>          test VIDIOC_ENUMAUDIO: OK (Not Supported)
>>          test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
>>          test VIDIOC_G/S_AUDIO: OK (Not Supported)
>>          Inputs: 0 Audio Inputs: 0 Tuners: 0
>>
>> Output ioctls:
>>          test VIDIOC_G/S_MODULATOR: OK (Not Supported)
>>          test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>>          test VIDIOC_ENUMAUDOUT: OK (Not Supported)
>>          test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
>>          test VIDIOC_G/S_AUDOUT: OK (Not Supported)
>>          Outputs: 0 Audio Outputs: 0 Modulators: 0
>>
>> Input/Output configuration ioctls:
>>          test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
>>          test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
>>          test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
>>          test VIDIOC_G/S_EDID: OK (Not Supported)
>>
>> Control ioctls:
>>          test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
>>          test VIDIOC_QUERYCTRL: OK
>>          test VIDIOC_G/S_CTRL: OK
>>          test VIDIOC_G/S/TRY_EXT_CTRLS: OK
>>          test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
>>          test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
>>          Standard Controls: 37 Private Controls: 0
>>
>> Format ioctls:
>>          test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
>>          test VIDIOC_G/S_PARM: OK
>>          test VIDIOC_G_FBUF: OK (Not Supported)
>>          test VIDIOC_G_FMT: OK
>>          test VIDIOC_TRY_FMT: OK
>>          test VIDIOC_S_FMT: OK
>>          test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
>>          test Cropping: OK
>>          test Composing: OK (Not Supported)
>>          test Scaling: OK (Not Supported)
>>
>> Codec ioctls:
>>          test VIDIOC_(TRY_)ENCODER_CMD: OK
>>          test VIDIOC_G_ENC_INDEX: OK (Not Supported)
>>          test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>>
>> Buffer ioctls:
>>          test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>>          test CREATE_BUFS maximum buffers: OK
>>          test VIDIOC_REMOVE_BUFS: OK
>>          test VIDIOC_EXPBUF: OK
>>          test Requests: OK (Not Supported)
>>          test blocking wait: OK
>>
>> Test input 0:
>>
>> Streaming ioctls:
>>          test read/write: OK (Not Supported)
>>          Video Capture Multiplanar: Captured 61 buffers
>>          test MMAP (select, REQBUFS): OK
>>          Video Capture Multiplanar: Captured 61 buffers
>>          test MMAP (epoll, REQBUFS): OK
>>          Video Capture Multiplanar: Captured 61 buffers
>>          test MMAP (select, CREATE_BUFS): OK
>>          Video Capture Multiplanar: Captured 61 buffers
>>          test MMAP (epoll, CREATE_BUFS): OK
>>          test USERPTR (select): OK (Not Supported)
>>          test DMABUF: Cannot test, specify --expbuf-device
>>
>> Total for iris_driver device /dev/video1: 54, Succeeded: 54, Failed: 0,
>> Warnings: 0
>>
>> The result of v4l2-compliance on SM8250:
>> v4l2-compliance 1.29.0-5270, 64 bits, 64-bit time_t
>> v4l2-compliance SHA: dc947661089e 2024-11-11 10:25:38
>>
>> Compliance test for iris_driver device /dev/video1:
>>
>> Driver Info:
>>          Driver name      : iris_driver
>>          Card type        : Iris Encoder
>>          Bus info         : platform:aa00000.video-codec
>>          Driver version   : 6.16.0
>>          Capabilities     : 0x84204000
>>                  Video Memory-to-Memory Multiplanar
>>                  Streaming
>>                  Extended Pix Format
>>                  Device Capabilities
>>          Device Caps      : 0x04204000
>>                  Video Memory-to-Memory Multiplanar
>>                  Streaming
>>                  Extended Pix Format
>>          Detected Stateful Encoder
>>
>> Required ioctls:
>>          test VIDIOC_QUERYCAP: OK
>>          test invalid ioctls: OK
>>
>> Allow for multiple opens:
>>          test second /dev/video1 open: OK
>>          test VIDIOC_QUERYCAP: OK
>>          test VIDIOC_G/S_PRIORITY: OK
>>          test for unlimited opens: OK
>>
>> Debug ioctls:
>>          test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
>>          test VIDIOC_LOG_STATUS: OK (Not Supported)
>>
>> Input ioctls:
>>          test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
>>          test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>>          test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
>>          test VIDIOC_ENUMAUDIO: OK (Not Supported)
>>          test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
>>          test VIDIOC_G/S_AUDIO: OK (Not Supported)
>>          Inputs: 0 Audio Inputs: 0 Tuners: 0
>>
>> Output ioctls:
>>          test VIDIOC_G/S_MODULATOR: OK (Not Supported)
>>          test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>>          test VIDIOC_ENUMAUDOUT: OK (Not Supported)
>>          test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
>>          test VIDIOC_G/S_AUDOUT: OK (Not Supported)
>>          Outputs: 0 Audio Outputs: 0 Modulators: 0
>>
>> Input/Output configuration ioctls:
>>          test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
>>          test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
>>          test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
>>          test VIDIOC_G/S_EDID: OK (Not Supported)
>>
>> Control ioctls:
>>          test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
>>          test VIDIOC_QUERYCTRL: OK
>>          test VIDIOC_G/S_CTRL: OK
>>          test VIDIOC_G/S/TRY_EXT_CTRLS: OK
>>          test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
>>          test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
>>          Standard Controls: 19 Private Controls: 0
>>
>> Format ioctls:
>>          test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
>>          test VIDIOC_G/S_PARM: OK
>>          test VIDIOC_G_FBUF: OK (Not Supported)
>>          test VIDIOC_G_FMT: OK
>>          test VIDIOC_TRY_FMT: OK
>>          test VIDIOC_S_FMT: OK
>>          test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
>>          test Cropping: OK
>>          test Composing: OK (Not Supported)
>>          test Scaling: OK (Not Supported)
>>
>> Codec ioctls:
>>          test VIDIOC_(TRY_)ENCODER_CMD: OK
>>          test VIDIOC_G_ENC_INDEX: OK (Not Supported)
>>          test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>>
>> Buffer ioctls:
>>          test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>>          test CREATE_BUFS maximum buffers: OK
>>          test VIDIOC_REMOVE_BUFS: OK
>>          test VIDIOC_EXPBUF: OK
>>          test Requests: OK (Not Supported)
>>          test blocking wait: OK
>>
>> Test input 0:
>>
>> Streaming ioctls:
>>          test read/write: OK (Not Supported)
>>          Video Capture Multiplanar: Captured 61 buffers
>>          test MMAP (select, REQBUFS): OK
>>          Video Capture Multiplanar: Captured 61 buffers
>>          test MMAP (epoll, REQBUFS): OK
>>          Video Capture Multiplanar: Captured 61 buffers
>>          test MMAP (select, CREATE_BUFS): OK
>>          Video Capture Multiplanar: Captured 61 buffers
>>          test MMAP (epoll, CREATE_BUFS): OK
>>          test USERPTR (select): OK (Not Supported)
>>          test DMABUF: Cannot test, specify --expbuf-device
>>
>> Total for iris_driver device /dev/video1: 54, Succeeded: 54, Failed: 0,
>> Warnings: 0
>>
>> Looking forward to your review and feedback.
>>
>> Thanks,
>> Dikshita
>>
>> Signed-off-by: Dikshita Agarwal <quic_dikshita@quicinc.com>
>> ---
>> Dikshita Agarwal (24):
>>        media: iris: Fix buffer count reporting in internal buffer check
>>        media: iris: Report unreleased PERSIST buffers on session close
>>        media: iris: Fix memory leak by freeing untracked persist buffer
>>        media: iris: Fix port streaming handling
>>        media: iris: Allow substate transition to load resources during
>> output streaming
>>        media: iris: Always destroy internal buffers on firmware release
>> response
>>        media: iris: Update vbuf flags before v4l2_m2m_buf_done
>>        media: iris: Allow stop on firmware only if start was issued.
>>        media: iris: Send dummy buffer address for all codecs during drain
>>        media: iris: Fix missing LAST flag handling during drain
>>        media: iris: Add support for video encoder device
>>        media: iris: Initialize and deinitialize encoder instance structure
>>        media: iris: Add support for ENUM_FMT, S/G/TRY_FMT encoder
>>        media: iris: Add support for ENUM_FRAMESIZES/FRAMEINTERVALS for
>> encoder
>>        media: iris: Add support for VIDIOC_QUERYCAP for encoder video device
>>        media: iris: Add encoder support for V4L2 event subscription
>>        media: iris: Add support for G/S_SELECTION for encoder video device
>>        media: iris: Add support for G/S_PARM for encoder video device
>>        media: iris: Add platform-specific capabilities for encoder video
>> device
>>        media: iris: Add V4L2 streaming support for encoder video device
>>        media: iris: Set platform capabilities to firmware for encoder
>> video device
>>        media: iris: Allocate and queue internal buffers for encoder video
>> device
>>        media: iris: Add support for buffer management ioctls for encoder
>> device
>>        media: iris: Add support for drain sequence in encoder video device
>>
>>   drivers/media/platform/qcom/iris/Makefile          |   5 +-
>>   drivers/media/platform/qcom/iris/iris_buffer.c     | 220 ++++--
>>   drivers/media/platform/qcom/iris/iris_buffer.h     |   7 +-
>>   drivers/media/platform/qcom/iris/iris_common.c     | 232 ++++++
>>   drivers/media/platform/qcom/iris/iris_common.h     |  18 +
>>   drivers/media/platform/qcom/iris/iris_core.h       |  20 +-
>>   drivers/media/platform/qcom/iris/iris_ctrls.c      | 675 +++++++++++++++-
>>   drivers/media/platform/qcom/iris/iris_ctrls.h      |  15 +
>>   drivers/media/platform/qcom/iris/iris_hfi_common.h |   2 +-
>>   .../platform/qcom/iris/iris_hfi_gen1_command.c     | 485 +++++++++---
>>   .../platform/qcom/iris/iris_hfi_gen1_defines.h     | 112 ++-
>>   .../platform/qcom/iris/iris_hfi_gen1_response.c    |  60 +-
>>   .../platform/qcom/iris/iris_hfi_gen2_command.c     | 357 ++++++---
>>   .../platform/qcom/iris/iris_hfi_gen2_defines.h     |  44 +-
>>   .../platform/qcom/iris/iris_hfi_gen2_response.c    |  46 +-
>>   drivers/media/platform/qcom/iris/iris_instance.h   |  24 +
>>   .../platform/qcom/iris/iris_platform_common.h      |  74 +-
>>   .../media/platform/qcom/iris/iris_platform_gen2.c  | 522 ++++++++++++-
>>   .../platform/qcom/iris/iris_platform_qcs8300.h     | 352 ++++++++-
>>   .../platform/qcom/iris/iris_platform_sm8250.c      | 234 +++++-
>>   drivers/media/platform/qcom/iris/iris_probe.c      |  39 +-
>>   drivers/media/platform/qcom/iris/iris_state.c      |   9 +-
>>   drivers/media/platform/qcom/iris/iris_state.h      |   1 +
>>   drivers/media/platform/qcom/iris/iris_utils.c      |  36 +
>>   drivers/media/platform/qcom/iris/iris_utils.h      |   2 +
>>   drivers/media/platform/qcom/iris/iris_vb2.c        |  58 +-
>>   drivers/media/platform/qcom/iris/iris_vdec.c       | 249 +-----
>>   drivers/media/platform/qcom/iris/iris_vdec.h       |  13 +-
>>   drivers/media/platform/qcom/iris/iris_venc.c       | 579 ++++++++++++++
>>   drivers/media/platform/qcom/iris/iris_venc.h       |  27 +
>>   drivers/media/platform/qcom/iris/iris_vidc.c       | 299 +++++++-
>>   drivers/media/platform/qcom/iris/iris_vpu_buffer.c | 847
>> ++++++++++++++++++++-
>>   drivers/media/platform/qcom/iris/iris_vpu_buffer.h |  20 +
>>   33 files changed, 4970 insertions(+), 713 deletions(-)
>> ---
>> base-commit: c26e8dcd9d4e86d788c5bf7a5dd0ea70a95ab067
>> change-id: 20250704-iris-video-encoder-b193350b487a
>>
>> Best regards,
> 
> I have a successfull encoding on SM8650-HDK with the following commands
> (ffmpegs segfaults en encoding):
> 
> wget
> https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/1080/Big_Buck_Bunny_1080_10s_30MB.mp4 -O Big_Buck_Bunny_1080_10s_30MB_h264.mp4
> ffmpeg -c:v h264_v4l2m2m -i Big_Buck_Bunny_1080_10s_30MB_h264.mp4 -pix_fmt
> nv12 Big_Buck_Bunny_1080_10s.yuv
> v4l2-ctl --verbose
> --set-fmt-video-out=width=1920,height=1080,pixelformat=NV12
> --set-selection-output target=crop,top=0,left=0,width=1920,height=1080
> --set-fmt-video=pixelformat=H264 --stream-mmap --stream-out-mmap
> --stream-from=Big_Buck_Bunny_1080_10s.yuv
> --stream-to=Big_Buck_Bunny_1080_10s.h264 -d /dev/video1
> v4l2-ctl --verbose
> --set-fmt-video-out=width=1920,height=1080,pixelformat=NV12
> --set-selection-output target=crop,top=0,left=0,width=1920,height=1080
> --set-fmt-video=pixelformat=HEVC --stream-mmap --stream-out-mmap
> --stream-from=Big_Buck_Bunny_1080_10s.yuv
> --stream-to=Big_Buck_Bunny_1080_10s.hevc -d /dev/video1
> 
> But the video is 1280x720, if I drop the `--set-selection-output

Seems like a bug, checking and will fix in next revision

> target=crop,top=0,left=0,width=1920,height=1080` I correctly get 1920x1080
> but the color keeps moving like if the colors stride is wrong.

For input resolutions that are not 16x16 aligned, require the client to set
the crop rectangle using the s_selection API as per v4l2 specification.
Pls see
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-encoder.html
4.5.2.5.6

Thanks,
Dikshita

> 
> Neil
> 
>