mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Nicolas Dufresne <nicolas@ndufresne.ca>
To: Dave Stevenson <dave.stevenson@raspberrypi.com>
Cc: Ming Qian <ming.qian@nxp.com>,
	Mauro Carvalho Chehab <mchehab@kernel.org>,
	Hans Verkuil <hverkuil-cisco@xs4all.nl>,
	Shawn Guo <shawnguo@kernel.org>, Rob Herring <robh+dt@kernel.org>,
	Sascha Hauer <s.hauer@pengutronix.de>,
	Pengutronix Kernel Team <kernel@pengutronix.de>,
	Fabio Estevam <festevam@gmail.com>,
	NXP Linux Team <linux-imx@nxp.com>,
	Linux Media Mailing List <linux-media@vger.kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>
Subject: Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag
Date: Tue, 05 Jul 2022 09:25:43 -0400	[thread overview]
Message-ID: <61e26a184b87dea6b17d3ba7c4437da29486e167.camel@ndufresne.ca> (raw)
In-Reply-To: <CAPY8ntC8-d7zupr=mNHdc053RV1Z1yjnmqbV=13AaT2gmMSrKw@mail.gmail.com>

Le mardi 05 juillet 2022 à 13:34 +0100, Dave Stevenson a écrit :
> Hi Ming and Nicolas
> 
> On Mon, 4 Jul 2022 at 16:53, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> > 
> > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
> > > user-space should be able to hint decoder
> > > the vb2 only contains codec config header,
> > > but does not contain any frame data.
> > > It's only used for parsing header, and can't be decoded.
> > 
> > This is copied from OMX specification. I think we we import this, we should at
> > least refer to the original.
> > 
> > > 
> > > Current, it's usually used by android.
> > > 
> > > Signed-off-by: Ming Qian <ming.qian@nxp.com>
> > > ---
> > >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> > >  include/uapi/linux/videodev2.h                   | 2 ++
> > >  2 files changed, 11 insertions(+)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> > > index 4638ec64db00..acdc4556f4f4 100644
> > > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > > @@ -607,6 +607,15 @@ Buffer Flags
> > >       the format. Any subsequent call to the
> > >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> > >       but return an ``EPIPE`` error code.
> > > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > > +
> > > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> > > +      - 0x00200000
> > > +      - This flag may be set when the buffer only contains codec config
> > > +    header, but does not contain any frame data. Usually the codec config
> > > +    header is merged to the next idr frame, with the flag
> > > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> > > +    split the header and queue it separately.
> > 
> > I think the documentation is clear. Now, if a driver uses this, will existing
> > userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ?
> > So we need existing driver to do this when flagged to, and just copy/append when
> > the userland didn't opt-in that feature ?
> 
> The commit text says it is for userspace feeding data into a video
> decoder, so it's a userspace choice instead of driver.

I see, the spec needs to be more clear then.

> 
> For encoders there is already V4L2_CID_MPEG_VIDEO_HEADER_MODE [1]
> which allows for V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE or
> V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. FFmpeg selects
> _SEPARATE by default [2], whilst the default is normally
> _JOINED_WITH_1ST_FRAME.

I did miss the addition of this API, thanks for the reminder. The problem right
now is that things are being added with the needed cross-reference.

> 
> It does raise the question as to whether all decoders will support
> header byte only buffers, and does there need to be a capabilities
> flag to denote that it is supported.
> And should encoders in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE mode set
> it on the headers only buffers?

What about:
- Document better that in absence of V4L2_CID_MPEG_VIDEO_HEADER_MODE in a
driver, V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME is to be assumed
- Document that in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE, driver should signal
the header buffer with the newly added buffer flag (I would suggest
V4L2_BUF_FLAG_HEADERS_ONLY)
- Document for each support CODECs if V4L2_CID_MPEG_VIDEO_HEADER_MODE can be
supported, and the expected headers and the packing of these 
- Document that decoders that didn't implement V4L2_CID_MPEG_VIDEO_HEADER_MODE
should be assumed to only support 
V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME decoding (even though it
allowed to support any alignment, even random).
- Cross-reference V4L2_CID_MPEG_VIDEO_HEADER_MODE into the new flag
documentation (whatever this new flag will be called).

> 
> A number of undefined elements of how this should be implemented/used :-(
> 
>   Dave
> 
> [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/ext-ctrls-codec.html
> [2] https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/v4l2_m2m_enc.c#L196

> 
> > >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> > > 
> > >        - ``V4L2_BUF_FLAG_REQUEST_FD``
> > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > index 5311ac4fde35..8708ef257710 100644
> > > --- a/include/uapi/linux/videodev2.h
> > > +++ b/include/uapi/linux/videodev2.h
> > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
> > >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
> > >  /* mem2mem encoder/decoder */
> > >  #define V4L2_BUF_FLAG_LAST                   0x00100000
> > > +/* Buffer only contains codec header */
> > > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
> > >  /* request_fd is valid */
> > >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
> > > 
> > 


  reply	other threads:[~2022-07-05 13:51 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-06-28  2:19 Ming Qian
2022-07-04 15:52 ` Nicolas Dufresne
2022-07-05  1:52   ` [EXT] " Ming Qian
2022-07-05 11:34     ` Ming Qian
2022-07-05 13:12       ` Nicolas Dufresne
2022-07-06  8:46         ` Ming Qian
2022-07-05 12:34   ` Dave Stevenson
2022-07-05 13:25     ` Nicolas Dufresne [this message]
2022-07-05 15:23 ` kernel test robot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=61e26a184b87dea6b17d3ba7c4437da29486e167.camel@ndufresne.ca \
    --to=nicolas@ndufresne.ca \
    --cc=dave.stevenson@raspberrypi.com \
    --cc=festevam@gmail.com \
    --cc=hverkuil-cisco@xs4all.nl \
    --cc=kernel@pengutronix.de \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-imx@nxp.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-media@vger.kernel.org \
    --cc=mchehab@kernel.org \
    --cc=ming.qian@nxp.com \
    --cc=robh+dt@kernel.org \
    --cc=s.hauer@pengutronix.de \
    --cc=shawnguo@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®