Re: [RFC 0/2] Add standard exposure and gain controls for multiple captures

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

On Tue, Jul 22, 2025 at 10:46:16AM +0200, Julien Vuillaumier wrote:
> On 16/07/2025 02:12, Laurent Pinchart wrote:
> > On Wed, Jul 16, 2025 at 02:59:54AM +0300, Laurent Pinchart wrote:
> >> On Fri, Jul 11, 2025 at 01:05:42AM +0300, Mirela Rabulea wrote:
> >>> Add new standard controls as U32 arrays, for sensors with multiple
> >>> captures: V4L2_CID_EXPOSURE_MULTI, V4L2_CID_AGAIN_MULTI and
> >>> V4L2_CID_DGAIN_MULTI. These will be particularly useful for sensors
> >>> that have multiple captures, but the HDR merge is done inside the sensor,
> >>> in the end exposing a single stream, but still requiring AEC control
> >>> for all captures.
> >>
> >> It's also useful for sensors supporting DOL or DCG with HDR merge being
> >> performed outside of the sensor.
> > 
> > Regarless of where HDR merge is implemented, we will also need controls
> > to select the HDR mode. We have V4L2_CID_HDR_SENSOR_MODE, which doesn't
> > standardize the values, and that's not good enough. At least for DOL and
> > DCG with HDR merge implemented outside of the sensor, we need to
> > standardize the modes.
> 
> For the HDR-capable sensors with the HDR merge implemented outside, the 
> short capture(s) are likely implemented as separate streams, in order to 
> match the raw camera sensor model.

Yes, that's my expectation. They should use a different data type or a
different virtual channel (I expect most sensors to support both
options).

> In that case, the SDR/HDR mode switch, when supported, can be done by 
> configuring the sensor device internal route for the short capture stream.

That's an option too, but it won't allow us to select between different
HDR modes. For instance, the AR0830 supports both DOL (2 exposures) and
DCG (2 gains). We would need a way to select between those two modes.

> You mentioned the need to be able to select the HDR mode in a standard 
> way. Could you elaborate on the foreseen usage: would it be to select 
> SDR/HDR operation, to select between different HDR sub-modes, to inform 
> user space about HDR capability... ?

Both. From a libcamera perspective, I want standardized controls for
this, to avoid sensor-specific code as much as possible.

> > Can you tell which sensor(s) you're working with ?
> > 
> >>> All controls are in the same class, so they could all be set
> >>> atomically via VIDIOC_S_EXT_CTRLS, this could turn out to be
> >>> useful in case of sensors with context switching.
> >>
> >> Agreed, we should be able to set them all. Are we still unable to set
> >> controls from multiple classes atomatically ? I thought that limitation
> >> has been lifted.
> >>
> >>> Each element of the array will hold an u32 value (exposure or gain)
> >>> for one capture. The size of the array is up to the sensor driver which
> >>> will implement the controls and initialize them via v4l2_ctrl_new_custom().
> >>> With this approach, the user-space will have to set valid values
> >>> for all the captures represented in the array.
> >>
> >> I'll comment on the controls themselves in patch 2/2.
> >>
> >>> The v4l2-core only supports one scalar min/max/step value for the
> >>> entire array, and each element is validated and adjusted to be within
> >>> these bounds in v4l2_ctrl_type_op_validate(). The significance for the
> >>> maximum value for the exposure control could be "the max value for the
> >>> long exposure" or "the max value for the sum of all exposures". If none
> >>> of these is ok, the sensor driver can adjust the values as supported and
> >>> the user space can use the TRY operation to query the sensor for the
> >>> minimum or maximum values.
> >>
> >> Hmmmm... I wonder if we would need the ability to report different
> >> limits for different array elements. There may be over-engineering
> >> though, my experience with libcamera is that userspace really needs
> >> detailed information about those controls, and attempting to convey the
> >> precise information through the kernel-userspace API is bound to fail.
> >> That's why we implement a sensor database in libcamera, with information
> >> about how to convert control values to real gain and exposure time.
> >> Exposing (close to) raw register values and letting userspace handle the
> >> rest may be better.
> >>
> >>> Mirela Rabulea (2):
> >>>    LF-15161-6: media: Add exposure and gain controls for multiple
> >>>      captures
> >>>    LF-15161-7: Documentation: media: Describe exposure and gain controls
> >>>      for multiple captures
> >>
> >> Did you forget to remove the LF-* identifiers ? :-)
> >>
> >>>
> >>>   .../media/v4l/ext-ctrls-image-source.rst             | 12 ++++++++++++
> >>>   drivers/media/v4l2-core/v4l2-ctrls-defs.c            |  8 ++++++++
> >>>   include/uapi/linux/v4l2-controls.h                   |  3 +++
> >>>   3 files changed, 23 insertions(+)

-- 
Regards,

Laurent Pinchart