From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752821AbdI1Jvq (ORCPT ); Thu, 28 Sep 2017 05:51:46 -0400 Received: from mail-pg0-f46.google.com ([74.125.83.46]:51384 "EHLO mail-pg0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752776AbdI1Jv2 (ORCPT ); Thu, 28 Sep 2017 05:51:28 -0400 X-Google-Smtp-Source: AOwi7QCpMX9+sYA4RVmDwrr7CFbjG82MpY+Yv/XY8+Mk+A/jopGiMoqlRfGXgRnnaVg74e8mJIcSZA== From: Alexandre Courbot To: Mauro Carvalho Chehab , Hans Verkuil , Laurent Pinchart , Pawel Osciak , Marek Szyprowski , Tomasz Figa , Sakari Ailus , Gustavo Padovan Cc: linux-media@vger.kernel.org, linux-kernel@vger.kernel.org, Alexandre Courbot Subject: [RFC PATCH 9/9] [media] document jobs API Date: Thu, 28 Sep 2017 18:50:27 +0900 Message-Id: <20170928095027.127173-10-acourbot@chromium.org> X-Mailer: git-send-email 2.14.2.822.g60be5d43e6-goog In-Reply-To: <20170928095027.127173-1-acourbot@chromium.org> References: <20170928095027.127173-1-acourbot@chromium.org> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Still a work-in-progress, but hopefully conveys the general idea. Signed-off-by: Alexandre Courbot --- Documentation/media/intro.rst | 2 + Documentation/media/media_uapi.rst | 1 + Documentation/media/uapi/jobs/jobs-api.rst | 23 +++ Documentation/media/uapi/jobs/jobs-example.rst | 69 ++++++++ Documentation/media/uapi/jobs/jobs-intro.rst | 61 +++++++ Documentation/media/uapi/jobs/jobs-queue.rst | 73 ++++++++ Documentation/media/uapi/jobs/jobs-queue.svg | 192 +++++++++++++++++++++ .../media/uapi/v4l/vidioc-g-ext-ctrls.rst | 6 + 8 files changed, 427 insertions(+) create mode 100644 Documentation/media/uapi/jobs/jobs-api.rst create mode 100644 Documentation/media/uapi/jobs/jobs-example.rst create mode 100644 Documentation/media/uapi/jobs/jobs-intro.rst create mode 100644 Documentation/media/uapi/jobs/jobs-queue.rst create mode 100644 Documentation/media/uapi/jobs/jobs-queue.svg diff --git a/Documentation/media/intro.rst b/Documentation/media/intro.rst index 9ce2e23a0236..e39a9dd3444a 100644 --- a/Documentation/media/intro.rst +++ b/Documentation/media/intro.rst @@ -38,6 +38,8 @@ divided into five parts. 5. The :ref:`fifth part ` covers the CEC (Consumer Electronics Control) API. +6. The :ref:`sixth part ` covers the jobs API. + It should also be noted that a media device may also have audio components, like mixers, PCM capture, PCM playback, etc, which are controlled via ALSA API. For additional information and for the latest development code, see: diff --git a/Documentation/media/media_uapi.rst b/Documentation/media/media_uapi.rst index fd8ebe002cd2..254e3d085abc 100644 --- a/Documentation/media/media_uapi.rst +++ b/Documentation/media/media_uapi.rst @@ -27,5 +27,6 @@ License". uapi/rc/remote_controllers uapi/mediactl/media-controller uapi/cec/cec-api + uapi/jobs/jobs-api uapi/gen-errors uapi/fdl-appendix diff --git a/Documentation/media/uapi/jobs/jobs-api.rst b/Documentation/media/uapi/jobs/jobs-api.rst new file mode 100644 index 000000000000..3a7aa8568e93 --- /dev/null +++ b/Documentation/media/uapi/jobs/jobs-api.rst @@ -0,0 +1,23 @@ +.. -*- coding: utf-8; mode: rst -*- + +.. include:: + +.. _jobsapi: + +################## +Part VI - Jobs API +################## + +This part describes the V4L2 Jobs API. + +.. class:: toc-title + + Table of Contents + +.. toctree:: + :maxdepth: 5 + :numbered: + + jobs-intro + jobs-queue + jobs-example diff --git a/Documentation/media/uapi/jobs/jobs-example.rst b/Documentation/media/uapi/jobs/jobs-example.rst new file mode 100644 index 000000000000..0b725dfb58bd --- /dev/null +++ b/Documentation/media/uapi/jobs/jobs-example.rst @@ -0,0 +1,69 @@ +.. -*- coding: utf-8; mode: rst -*- + +.. _jobs-example: + +=========================================== +Example: Using the Jobs API for HDR capture +=========================================== + +HDR capturing involves taking two shots of the same image with different +exposure settings. When using the V4L2 API, one must wait for the first shot to +be completed before updating the exposure and taking the second one, introducing +synchronization with user-space and potential delays for the second image. + +The Jobs API allows us to simply submit two jobs with different exposure +parameters, and to dequeue their buffers to get the result. + +The following example shows how to do this on a capture device with the exposure +control. It assumes that the capture device has already been opened and its +format set. To keep the code simple this code does not handle errors. + +.. code-block:: c + + struct v4l2_ext_control ctrl[1]; + struct v4l2_ext_controls ctrls; + struct v4l2_jobqueue_init jq_init; + struct v4l2_buffer buf[2]; + int jq_fd; + + /* FD for the jobs queue */ + jq_fd = open("/dev/v4l2_jobqueue"); + + /* Initialize the jobs queue */ + jq_init.nb_devs = 1; + jq_init.fd[0] = capture_fd; + ioctl(jq_fd, VIDIOC_JOBQUEUE_INIT, &jq_init); + + /* Prepare and submit the first capture job, low exposure */ + ctrl[0].id = V4L2_CID_EXPOSURE; + ctrl[0].value = 32; + ctrls.which = V4L2_CTRL_WHICH_CURJOB_VAL; + ctrls.count = 1; + ctrsl.controls = ctrl; + ioctl(capture_fd, VICIOC_S_EXT_CTRLS, &ctrls); + ioctl(capture_fd, VIDIOC_QBUF, &buf[0]); + ioctl(jq_fd, VIDIOC_JOBQUEUE_QJOB, NULL); + + /* Prepare and submit the second capture job, high exposure */ + ctrl[0].id = V4L2_CID_EXPOSURE; + ctrl[0].value = 192; + ctrls.which = V4L2_CTRL_WHICH_CURJOB_VAL; + ctrls.count = 1; + ctrsl.controls = ctrl; + ioctl(capture_fd, VICIOC_S_EXT_CTRLS, &ctrls); + ioctl(capture_fd, VIDIOC_QBUF, &buf[1]); + ioctl(jq_fd, VIDIOC_JOBQUEUE_QJOB, NULL); + + /* Dequeue buffers with the captured data */ + ioctl(capture_fd, VIDIOC_DQBUF, &buf[0]); + ioctl(capture_fd, VIDIOC_DQBUF, &buf[1]); + + /* Dequeue jobs and confirm exposure parameters */ + ioctl(jq_fd, VIDIOC_JOBQUEUE_DQJOB, NULL); + ctrls.which = V4L2_CTRL_WHICH_DEQJOB_VAL + ioctl(fd, VIDIOC_G_EXT_CTRLS, &ctrls); + printf("Exposure for first capture: %d\n", ctrl[0].value); + + ioctl(jq_fd, VIDIOC_JOBQUEUE_DQJOB, NULL); + ioctl(fd, VIDIOC_G_EXT_CTRLS, &ctrls); + printf("Exposure for second capture: %d\n", ctrl[0].value); diff --git a/Documentation/media/uapi/jobs/jobs-intro.rst b/Documentation/media/uapi/jobs/jobs-intro.rst new file mode 100644 index 000000000000..34d7efe5bcfb --- /dev/null +++ b/Documentation/media/uapi/jobs/jobs-intro.rst @@ -0,0 +1,61 @@ +.. -*- coding: utf-8; mode: rst -*- + +.. _jobs-intro: + +============ +Introduction +============ + +The Jobs API allows to submit units of work ("jobs") to be processed +cooperatively by a set of opened devices. The state of jobs can be set and +queried independently from the actual hardware state, which means that +user-space does not need to explicitly wait for a job to finish before +submitting a new one with different parameters. + +The Jobs API is suitable for several workflows: + +- A set of devices need to cooperate on a given work, e.g. MIPI capture devices. +In this case the Jobs API ensures that all devices are properly set up for +processing the work. + +- Asynchronous submission of works with different parameters. Especially useful +for stateless codecs, but can also facilitate capture use-cases like HDR where +two frames with different exposures need to be captured as quickly as possible. + +- Quick switch between several states without having to explicitly reprogram +them. A job can be kept in memory and be reapplied at a future time. + +A job's will usually be used as follows: + +1. The state of the job is defined using the standard V4L2 API: controls, +formats, and other parameters are set, and buffers are queued on all devices +that take part in the job. + +2. The job is queued. All set parameters are applied, and the queued buffers are +processed. + +3. The job is dequeued. Output controls at the time of job completion can be +read back. If another job is queued, it is processed. + +Jobs are submitted to a jobs queue which processes them in sequential order. +Similarly to buffers, jobs must be dequeued and the state of the devices at the +time of job completion can then be read back. + +The Jobs API extends the existing V4L2 API and slightly changes the behavior of +some existing commands when in use. This part will describe the new APIs and how +to use them. + +WARNING: This is a work-in-progress. Comments are welcome. Many pieces are +missing, while some are subject to change. The list of missing pieces and open +questions include notably: + +- No format or crop setting +- No support for non-integer controls (this requires some rework of the control +framework) +- No support for media controller devices yet +- No error reporting when something unexpected happens while a job is being +processed +- It is probably not desirable to have a /dev/v4l2_jobsqueue device node for +managing jobs, but this makes testing easier at the moment. +- Jobs should have identifiers to make them easier to identify by user-space +- Dequeued buffers should carry the id of the job they were associated to diff --git a/Documentation/media/uapi/jobs/jobs-queue.rst b/Documentation/media/uapi/jobs/jobs-queue.rst new file mode 100644 index 000000000000..ecfc8dd46302 --- /dev/null +++ b/Documentation/media/uapi/jobs/jobs-queue.rst @@ -0,0 +1,73 @@ +.. -*- coding: utf-8; mode: rst -*- + +.. _jobs-queue: + +============== +The Jobs Queue +============== + +A jobs queue is instanciated from a set of already opened V4L2 devices, and +controls their behavior with atomic (from the user perspective) units of work +called jobs. User-space prepares a job by setting its desired state through +controls and formats, queuing the buffers to be processed for the job, and +finally queuing the job itself once it is ready. The hardware state will not be +affected until the job is effectively processed. + +Figure :ref:`jobs-queue` shows the life cycle of jobs within the queue: + +.. kernel-figure:: jobs-queue.svg + :alt: Life of a job inside the jobs queue + :align: center + +User-space creates a jobs queue by opening the jobs queue device and invoking +the :c:func:`VIDIOC_JOBQUEUE_INIT` ioctl with the list of opened file +descriptors of the devices being part of the new queue. After this, all the +passed devices are ready to perform according to the jobs API semantics, and the +opened jobs queue can accept queue and dequeue ioctls. + +Upon job queue creation, the first job is instanciated and ready to be prepared +and queued (we refer to this as the "current job"). User-space prepares the job +using variants of the regular V4L2 ioctls: for instance, in order to set +controls on a job, one will invoke :c:func:`VIDIOC_S_EXT_CTRLS` with +V4L2_CTRL_WHICH_CURJOB_VAL. + +The current job can be submitted using the :c:func:`VIDIOC_JOBQUEUE_QJOB` ioctl +when it is ready. A new job is immediately created and becomes the new active +job. + +Queued jobs are processed sequentially. As a job is processed, its parameters +for every device that is part of the queue are applied, and all its buffers are +processed. Once completed, the job is ready to be dequeued. + +Buffers can be dequeued independently of jobs. This allows a buffer that is +produced in the middle of the pipeline to be dequeued and used before the entire +job completes. However jobs **must** be dequeued at some point to avoid +accumulation of completed jobs. This is done using the +:c:func:`VIDIOC_JOBQUEUE_DQJOB` ioctl. User-space can then read the state of the +dequeued job at the time of its completion. For instance, in order to read the +values of controls at the time the dequeued job completed, one will invoke +:c:func:`VIDIOC_QUERY_EXT_CTRL` with V4L2_CTRL_WHICH_DEQJOB_VAL. + +Note that regular, unmodified ioctls will continue working as usual: for +instance, invoking :c:func:`VIDIOC_S_EXT_CTRLS` with V4L2_CTRL_WHICH_CUR_VAL +will immediately change the value of submitted controls. The only difference +when the jobs API is used is that buffers queued on devices that are part of a +jobs queue are not processed until the current job of that queue is submitted. + +By default, jobs are purely sequential: a new job is created as soon as the +current job is queued, and becomes the new current job. This new current job +initially has an empty state, meaning that its expected state is the state of +the previously queued job after it completes. Once a dequeued job is replaced by +the next one, it is deleted. + +For some use-cases however, it may make sense to reapply the exact state of a +previous job. The jobs API allows to do so using the +:c:func:`VIDIOC_JOBQUEUE_EXPORT_JOB` and :c:func:`VIDIOC_JOBQUEUE_IMPORT_JOB` +ioctls. + +:c:func:`VIDIOC_JOBQUEUE_EXPORT_JOB` returns a file descriptor that represents +the current job, which will be preserved in memory until that file descriptor is +closed. By passing that file descriptor to the +:c:func:`VIDIOC_JOBQUEUE_IMPORT_JOB`, the current job is replaced by the job +represented by the file descriptor. User-space can then tune the state of the +job, queue new buffers, and submit the job to be processed again. diff --git a/Documentation/media/uapi/jobs/jobs-queue.svg b/Documentation/media/uapi/jobs/jobs-queue.svg new file mode 100644 index 000000000000..e6d9408fd1df --- /dev/null +++ b/Documentation/media/uapi/jobs/jobs-queue.svg @@ -0,0 +1,192 @@ + + + + + + + + + + + Current + + + + + + + + + + + + + + + + + + + + + + + + + + + Queued + + + + + + + + + + + + + + + Dequeued + + + + + + + + + + + + + + + + + + + + + + + + + + + Completed + + + + + + + + + + + + + S_EXT_CTRL(WHICH_JOB_CUR_VAL) + + + G_EXT_CTRL(WHICH_DEQJOB_VAL) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Active + + + + + + + + + + + + + + + + + Devices + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst index 5ab8d2ac27b9..035d5e29bd2e 100644 --- a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst +++ b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst @@ -216,6 +216,12 @@ still cause this situation. You can only get the default value of the control, you cannot set or try it. + When the Jobs API is active ``V4L2_CTRL_WHICH_CURJOB_VAL`` will return or set the value of the current job, while ``V4L2_CTRL_WHICH_DEQJOB_VAL`` returns the value of the currently dequeued job. + + .. note:: + + It is invalid to try and set the value of the dequeued job. + For backwards compatibility you can also use a control class here (see :ref:`ctrl-class`). In that case all controls have to belong to that control class. This usage is deprecated, instead -- 2.14.2.822.g60be5d43e6-goog