[PATCH v2 00/17] media: apple: add avd driver

[Sofus Forstreuter <sofus.c@icloud.com>]

Hi,

This series enables hardware video decoding on Apple Silicon M1, M2 and
M3 SoCs. It adds support for all the codecs AVD (Apple Video Decoder)
supports, which is H264, H265, VP9 and AV1 (only on M3+).

The AVD is a bit unique both as a video decoder and an IP block on Apple
Silicon SoCs. Instead of following the normal "a register for each
parameter", AVD is programmed through one single register. This means
the order of the writes are important, but its also important to ensure
we dont write faster than the hardware can process. To solve this, we
use the included Cortex-M3 co-processor. The CM3 accepts unsigned code,
which we take advantage off [1].
The driver prepares the instructions and writes them into segments,
which are submitted to the CM3. The CM3 then programs the hardware and
notifies the AP when the job is done or if an error occurred.

The driver implements the V4L2 M2M stateless decoder API. The driver's
framework is based very heavy on rkvdecs driver. It therefore comes as
no suprise that `v4l2-compliance` reports no errors:
Total for avd device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

The fluster scores are:
JVT-AVC_V1: 77/135
JVT-FR-EXT: 42/69
VP9-TEST-VECTORS: 216/305
VP9-TEST-VECTORS-HIGH: 1/6
JCT-VC-HEVC_V1: 143/147
AV1-TEST-VECTORS: 238/242

Since not all unsupported H264 streams are rejected, the scores
depend on the reset working, this only works with a patch to the
apple-dart driver so that the IOMMU attach/detach trick works.

All codecs support 4:2:2, 4:2:0, in 10 and 8 bit. By default, AVD
outputs in NV12, NV16, P010 and P210. Support for 4:4:4 and 12 bit
formats has not been added yet.

Internally the AVD uses an Apple specific compressed and tiled format
called Interchange to store reference image data. This is stored after
image data across all codecs. Apple Interchange is used for sharing
buffers with other hardware blocks (GPU, display controller), support
for this is on its way [2].
The V4L2 pixel formats have been added to help with calculating sizes.

This driver has been tested on most Apple Silicon SoCs in our downstream
tree [3]. Its expected this driver will work on M4, M5, M6 and the Neo,
with firmware already being written for all except the M6 SoCs [4].

The groundwork for this was laid many years ago by Jamie, R and Eileen,
who through a combined effort managed to reverse engineer many core
parts of the AVD block.
This driver would not have been possible without the work done by
Eileen, who wrote initial support for H264, H265, VP9 and excellent
tooling [5], allowing me to reverse engineer AV1 and make the driver
conformant.

[1] https://github.com/sofus13/avd-fw/tree/avd-next-fw
[2] https://patch.msgid.link/20260917-apple-interchange-modifier-v1-1-875294689c48@gmail.com
[3] https://github.com/AsahiLinux/linux
[4] https://github.com/AsahiLinux/avd-fw
[5] https://github.com/eiln/avd/tree/main

Signed-off-by: Sofus Forstreuter <sofus.c@icloud.com>
---
Changes in v2:
- Patch order
- Fixed dt-bindings by constraining all properties
- Fixed dts, add interrupts, iommus and regs. Drop labels, fix names
- Add Apple Interchange format
- Add more av1 and vp9 validation to v4l2-ctrls
- Validate more codec parameters
- Use the new CM3 firmware (drop avd-hw, drop submission loops)
- Updated cover to reflect changes
- Link to v1: https://patch.msgid.link/20260918-avd-v1-0-49977931f455@icloud.com

To: Sven Peter <sven@kernel.org>
To: Janne Grunau <j@jannau.net>
To: Neal Gompa <neal@gompa.dev>
To: Mauro Carvalho Chehab <mchehab@kernel.org>
To: Rob Herring <robh@kernel.org>
To: Krzysztof Kozlowski <krzk+dt@kernel.org>
To: Conor Dooley <conor+dt@kernel.org>
To: Sofus Forstreuter <sofus.c@icloud.com>
To: Philipp Zabel <p.zabel@pengutronix.de>
To: Heiko Stuebner <heiko@sntech.de>
Cc: asahi@lists.linux.dev
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-media@vger.kernel.org
Cc: devicetree@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-rockchip@lists.infradead.org

---
Sofus Forstreuter (17):
      dt-bindings: media: add apple,avd
      media: v4l2: Add P210 pixel format
      media: v4l2: Add Apple interchange pixel formats
      media: v4l2-ctrls: validate av1 tile info
      media: v4l2-ctrls: validate vp9 tile_rows_log2
      media: apple: add avd driver
      media: apple: avd: add h264 support
      media: apple: avd: add vp9 support
      media: apple: avd: add hevc support
      media: apple: avd: add av1 support
      arm64: dts: apple: t8103: add avd nodes
      arm64: dts: apple: t8112: add avd nodes
      arm64: dts: apple: t8122: add avd nodes
      arm64: dts: apple: t600x: add avd nodes
      arm64: dts: apple: t602x: add avd nodes
      arm64: dts: apple: t6030: add avd nodes
      arm64: dts: apple: t6031: add avd nodes

 .../devicetree/bindings/media/apple,avd.yaml       |  127 +
 .../userspace-api/media/v4l/pixfmt-yuv-planar.rst  |   14 +-
 MAINTAINERS                                        |    2 +
 arch/arm64/boot/dts/apple/t600x-dieX.dtsi          |   32 +
 arch/arm64/boot/dts/apple/t602x-dieX.dtsi          |   32 +
 arch/arm64/boot/dts/apple/t6030.dtsi               |   32 +
 arch/arm64/boot/dts/apple/t6031-dieX.dtsi          |   32 +
 arch/arm64/boot/dts/apple/t8103.dtsi               |   32 +
 arch/arm64/boot/dts/apple/t8112.dtsi               |   32 +
 arch/arm64/boot/dts/apple/t8122.dtsi               |   32 +
 drivers/media/platform/Kconfig                     |    1 +
 drivers/media/platform/Makefile                    |    1 +
 drivers/media/platform/apple/Kconfig               |    5 +
 drivers/media/platform/apple/Makefile              |    3 +
 drivers/media/platform/apple/avd/Kconfig           |   18 +
 drivers/media/platform/apple/avd/Makefile          |    5 +
 .../media/platform/apple/avd/avd-av1-entropymode.c | 4698 ++++++++++++++++++++
 .../media/platform/apple/avd/avd-av1-entropymode.h |  284 ++
 drivers/media/platform/apple/avd/avd-av1.c         | 1432 ++++++
 drivers/media/platform/apple/avd/avd-drv.c         |  776 ++++
 drivers/media/platform/apple/avd/avd-h264.c        |  862 ++++
 drivers/media/platform/apple/avd/avd-hevc.c        | 1340 ++++++
 drivers/media/platform/apple/avd/avd-inst.h        |  210 +
 drivers/media/platform/apple/avd/avd-v4l2.c        |  992 +++++
 drivers/media/platform/apple/avd/avd-vp9.c         | 1054 +++++
 drivers/media/platform/apple/avd/avd.h             |  299 ++
 drivers/media/v4l2-core/v4l2-common.c              |    8 +
 drivers/media/v4l2-core/v4l2-ctrls-core.c          |   17 +-
 drivers/media/v4l2-core/v4l2-ioctl.c               |    7 +
 include/uapi/linux/v4l2-controls.h                 |    2 +-
 include/uapi/linux/videodev2.h                     |    9 +
 31 files changed, 12386 insertions(+), 4 deletions(-)
---
base-commit: 4efa625593e1e9c76346232413997556dea1d89f
change-id: 20260917-avd-7ee5474b468c

Best regards,
--  
Sofus Forstreuter <sofus.c@icloud.com>