mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Hans Verkuil <hverkuil@xs4all.nl>
To: Shengjiu Wang <shengjiu.wang@gmail.com>
Cc: Shengjiu Wang <shengjiu.wang@nxp.com>,
	sakari.ailus@iki.fi, tfiga@chromium.org,
	m.szyprowski@samsung.com, mchehab@kernel.org,
	linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
	Xiubo.Lee@gmail.com, festevam@gmail.com, nicoleotsuka@gmail.com,
	lgirdwood@gmail.com, broonie@kernel.org, perex@perex.cz,
	tiwai@suse.com, alsa-devel@alsa-project.org,
	linuxppc-dev@lists.ozlabs.org
Subject: Re: [RFC PATCH v4 09/11] media: uapi: Add V4L2_CID_USER_IMX_ASRC_RATIO_MOD control
Date: Thu, 21 Sep 2023 16:09:48 +0200	[thread overview]
Message-ID: <2d44d574-08e5-4db3-87d9-5d12657f8935@xs4all.nl> (raw)
In-Reply-To: <CAA+D8AMZN59uTRs2sOrSeVb5AGopTzurNVCTNwJOVPahfEXd+w@mail.gmail.com>

On 21/09/2023 13:13, Shengjiu Wang wrote:
> On Thu, Sep 21, 2023 at 3:11 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>
>> On 21/09/2023 08:55, Shengjiu Wang wrote:
>>> On Wed, Sep 20, 2023 at 6:19 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>>>
>>>> On 20/09/2023 11:32, Shengjiu Wang wrote:
>>>>> The input clock and output clock may not be the accurate
>>>>> rate as the sample rate, there is some drift, so the convert
>>>>> ratio of i.MX ASRC module need to be changed according to
>>>>> actual clock rate.
>>>>>
>>>>> Add V4L2_CID_USER_IMX_ASRC_RATIO_MOD control for user to
>>>>> adjust the ratio.
>>>>>
>>>>> Signed-off-by: Shengjiu Wang <shengjiu.wang@nxp.com>
>>>>> ---
>>>>>  Documentation/userspace-api/media/v4l/control.rst | 5 +++++
>>>>>  drivers/media/v4l2-core/v4l2-ctrls-defs.c         | 1 +
>>>>>  include/uapi/linux/v4l2-controls.h                | 1 +
>>>>>  3 files changed, 7 insertions(+)
>>>>>
>>>>> diff --git a/Documentation/userspace-api/media/v4l/control.rst b/Documentation/userspace-api/media/v4l/control.rst
>>>>> index 4463fce694b0..2bc175900a34 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/control.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/control.rst
>>>>> @@ -318,6 +318,11 @@ Control IDs
>>>>>      depending on particular custom controls should check the driver name
>>>>>      and version, see :ref:`querycap`.
>>>>>
>>>>> +.. _v4l2-audio-imx:
>>>>> +
>>>>> +``V4L2_CID_USER_IMX_ASRC_RATIO_MOD``
>>>>> +    sets the rasampler ratio modifier of i.MX asrc module.
>>>>
>>>> rasampler -> resampler (I think?)
>>>>
>>>> This doesn't document at all what the type of the control is or how to interpret it.
>>>>
>>>>> +
>>>>>  Applications can enumerate the available controls with the
>>>>>  :ref:`VIDIOC_QUERYCTRL` and
>>>>>  :ref:`VIDIOC_QUERYMENU <VIDIOC_QUERYCTRL>` ioctls, get and set a
>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
>>>>> index 8696eb1cdd61..16f66f66198c 100644
>>>>> --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c
>>>>> +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
>>>>> @@ -1242,6 +1242,7 @@ const char *v4l2_ctrl_get_name(u32 id)
>>>>>       case V4L2_CID_COLORIMETRY_CLASS:        return "Colorimetry Controls";
>>>>>       case V4L2_CID_COLORIMETRY_HDR10_CLL_INFO:               return "HDR10 Content Light Info";
>>>>>       case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY:      return "HDR10 Mastering Display";
>>>>> +     case V4L2_CID_USER_IMX_ASRC_RATIO_MOD:                  return "ASRC RATIO MOD";
>>>>
>>>> Let's stay consistent with the other control names:
>>>>
>>>> "ASRC Ratio Modifier"
>>>>
>>>> But if this is a driver specific control, then this doesn't belong here.
>>>>
>>>> Driver specific controls are defined in the driver itself, including this
>>>> description.
>>>>
>>>> Same for the control documentation: if it is driver specific, then that
>>>> typically is documented either in a driver-specific public header, or
>>>> possibly in driver-specific documentation (Documentation/admin-guide/media/).
>>>>
>>>> But is this imx specific? Wouldn't other similar devices need this?
>>>
>>> It is imx specific.
>>
>> Why? I'm not opposed to this, but I wonder if you looked at datasheets of
>> similar devices from other vendors: would they use something similar?
> 
> I tried to find some datasheets for other vendors, but failed to find them.
> So I don't know how they implement this part.
> 
> Ratio modification on i.MX is to modify the configured ratio.
> For example, the input rate is 44.1kHz,  output rate is 48kHz,
> configured ratio = 441/480,   the ratio modification is to modify
> the fractional part of (441/480) with small steps.  because the
> input clock or output clock has drift in the real hardware.
> The ratio modification is signed value, it is added to configured
> ratio.
> 
> In our case, we have some sysfs interface for user to get the
> clock from input audio device and output audio device, user
> need to calculate the ratio dynamically , then configure the
> modification to driver

So this ratio modifier comes into play when either the audio input
or audio output (or both) are realtime audio inputs/outputs where
the sample rate is not a perfect 44.1 or 48 kHz, but slightly different?

If you would use this resampler to do offline resampling (i.e. resample
a 44.1 kHz wav file to a 48 kHz wav file), then this wouldn't be needed,
correct?

When dealing with realtime audio, userspace will know how to get the
precise sample rate, but that is out-of-scope of this driver. Here
you just need a knob to slightly tweak the resampling ratio.

If my understanding is correct, then I wonder if it is such a good
idea to put the rate into the v4l2_audio_format: it really has nothing
to do with the audio format as it is stored in memory.

What if you would drop that 'rate' field and instead create just a single
control for the resampling ratio. This can use struct v4l2_fract to represent
a fraction. It would be more work since v4l2_fract is currently not supported
for controls, but it is not hard to add support for that (just a bit tedious)
and I actually think this might be a perfect solution.

That way userspace can quite precisely tweak the ratio on the fly, and
it is a generic solution as well instead of mediatek specific.

Regards,

	Hans

> 
> May be other vendors has similar implementation. or make
> the definition be generic is an option.
> 
> best regards
> wang shengjiu
> 
>>
>> And the very short description you gave in the commit log refers to input
>> and output clock: how would userspace know those clock frequencies? In
>> other words, what information does userspace need in order to set this
>> control correctly? And is that information actually available? How would
>> you use this control?
>>
>> I don't really understand how this is supposed to be used.
>>
>>>
>>> Does this mean that I need to create a header file in include/uapi/linux
>>> folder to put this definition?  I just hesitate if this is necessary.
>>
>> Yes, put it there. There are some examples of this already:
>>
>> include/uapi/linux/aspeed-video.h
>> include/uapi/linux/max2175.h
>>
>>>
>>> There is folder Documentation/userspace-api/media/drivers/ for drivers
>>> Should this document in this folder, not in the
>>> Documentation/admin-guide/media/?
>>
>> Yes, you are correct. For the headers above, the corresponding documentation
>> is in:
>>
>> Documentation/userspace-api/media/drivers/aspeed-video.rst
>> Documentation/userspace-api/media/drivers/max2175.rst
>>
>> So you have some examples as reference.
>>
>> Frankly, what is in admin-guide and in userspace-api is a bit random, it
>> probably could use a cleanup.
>>
>> Regards,
>>
>>         Hans
>>
>>>
>>> Best regards
>>> Wang shengjiu
>>>>
>>>>>       default:
>>>>>               return NULL;
>>>>>       }
>>>>> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
>>>>> index c3604a0a3e30..b1c319906d12 100644
>>>>> --- a/include/uapi/linux/v4l2-controls.h
>>>>> +++ b/include/uapi/linux/v4l2-controls.h
>>>>> @@ -162,6 +162,7 @@ enum v4l2_colorfx {
>>>>>  /* The base for the imx driver controls.
>>>>>   * We reserve 16 controls for this driver. */
>>>>>  #define V4L2_CID_USER_IMX_BASE                       (V4L2_CID_USER_BASE + 0x10b0)
>>>>> +#define V4L2_CID_USER_IMX_ASRC_RATIO_MOD     (V4L2_CID_USER_IMX_BASE + 0)
>>>>>
>>>>>  /*
>>>>>   * The base for the atmel isc driver controls.
>>>>
>>>> Regards,
>>>>
>>>>         Hans
>>


  parent reply	other threads:[~2023-09-21 20:59 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-09-20  9:32 [RFC PATCH v4 00/11] Add audio support in v4l2 framework Shengjiu Wang
2023-09-20  9:32 ` [RFC PATCH v4 01/11] ASoC: fsl_asrc: define functions for memory to memory usage Shengjiu Wang
2023-09-20 10:31   ` Hans Verkuil
2023-09-20  9:32 ` [RFC PATCH v4 02/11] ASoC: fsl_easrc: " Shengjiu Wang
2023-09-20  9:32 ` [RFC PATCH v4 03/11] ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound Shengjiu Wang
2023-09-20  9:32 ` [RFC PATCH v4 04/11] ASoC: fsl_asrc: register m2m platform device Shengjiu Wang
2023-09-20  9:32 ` [RFC PATCH v4 05/11] ASoC: fsl_easrc: " Shengjiu Wang
2023-09-20  9:32 ` [RFC PATCH v4 06/11] media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag Shengjiu Wang
2023-09-20  9:32 ` [RFC PATCH v4 07/11] media: uapi: Add V4L2_AUDIO_FMT_LPCM fourcc format Shengjiu Wang
2023-09-20  9:32 ` [RFC PATCH v4 08/11] media: v4l2: Add audio capture and output support Shengjiu Wang
2023-09-20  9:32 ` [RFC PATCH v4 09/11] media: uapi: Add V4L2_CID_USER_IMX_ASRC_RATIO_MOD control Shengjiu Wang
2023-09-20 10:19   ` Hans Verkuil
2023-09-21  6:55     ` Shengjiu Wang
2023-09-21  7:11       ` Hans Verkuil
2023-09-21 11:13         ` Shengjiu Wang
2023-09-21 11:45           ` Mark Brown
2023-09-21 14:09           ` Hans Verkuil [this message]
2023-09-22  2:51             ` Shengjiu Wang
2023-09-22  8:54               ` Hans Verkuil
2023-09-22 10:52                 ` Shengjiu Wang
2023-09-22 11:18                   ` Hans Verkuil
2023-09-20  9:32 ` [RFC PATCH v4 10/11] media: audm2m: add virtual driver for audio memory to memory Shengjiu Wang
2023-09-20  9:32 ` [RFC PATCH v4 11/11] media: imx-asrc: Add memory to memory driver Shengjiu Wang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2d44d574-08e5-4db3-87d9-5d12657f8935@xs4all.nl \
    --to=hverkuil@xs4all.nl \
    --cc=Xiubo.Lee@gmail.com \
    --cc=alsa-devel@alsa-project.org \
    --cc=broonie@kernel.org \
    --cc=festevam@gmail.com \
    --cc=lgirdwood@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-media@vger.kernel.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=m.szyprowski@samsung.com \
    --cc=mchehab@kernel.org \
    --cc=nicoleotsuka@gmail.com \
    --cc=perex@perex.cz \
    --cc=sakari.ailus@iki.fi \
    --cc=shengjiu.wang@gmail.com \
    --cc=shengjiu.wang@nxp.com \
    --cc=tfiga@chromium.org \
    --cc=tiwai@suse.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®