Hardware-accelerated video decoders used through a firmware instead of hardware registers

[Paul Kocialkowski <paul.kocialkowski@bootlin.com>]

Hi,

With the work done on the media request API and the cedrus driver for
Allwinner ARM SoCs, we now have a kernel interface for exposing fixed-
hardware video decoding pipelines (currently MPEG-2 and H.264, with
H.265 on the way). Some work remains on the per-format interface and we
are looking to improve latency-related aspects, but we are all set to
have a nice interface here, that plays well with e.g. ffmpeg.

A specific situation came to my interest, which is apparently quite
common: some platforms have general-purpose microcontrollers embedded,
which can help with video decoding. They are however rarely to never
used to do the decoding itself (since they are general-purpose, not
DSPs) and just coordinate the decoding with the fixed-pipeline decoding
hardware block. The advantage is that the interface is just a simple
mailbox and the raw video bitstream from the file can be passed
directly without the need for userspace to do any parsing that the
codec requires.

One side-effect from this setup is that the actual hardware register
layout of the decoder is hidden away in a non-free piece of
microcontroller firmware, that's usually loaded at run-time.

With the recent developments on the media interface, we could interface
with these hardware decoders directly, which offers various advantages:
- we no longer need a 3rd party external non-free firmware, which just
  makes distribution easier for everyone and allows support in fully-
  free setups;
- all the usual advantages of having free code that can be fixed and
  updated instead of an obscure binary that many not always be doing
  the right thing;
- parsing of the slices is probably best done in userspace, and I
  heard that ffmpeg does this threaded, so there could be a latency
  advantage there as well, not to mention that it avoids the drag of
  a mailbox interface altogether;
- the general-purpose micro-controller can then be reused for something
  useful where it could actually make a performance difference.

As far as I understand, it seems that the video decoder for MT8173
fails in that category, where a MD32 general-purpose micro-controller
is used to only do the parsing. We even have device-tree nodes about
the decoder and encoder, but no register layout.

So I was wondering if the linux-media community should set some
boundaries here and push towards native implementations instead of
firmware-based ones. My opininon is that it definitely should.

It seems that other platforms (e.g. Tegra K1 and onwards) are in the
same situation, and I think the ChromiumOS downstream kernel uses an
obscure firmware on a general-purpose auxiliary ARM core (that's also
used at boot time IIRC).

What do you think?

Cheers,

Paul

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com