From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F857233134 for ; Fri, 19 Dec 2025 15:18:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766157520; cv=none; b=mkJtvf4XF6YIf2K7aeBhBbdYZ7NjLU5899AC0Q7bMob6RFWGYhfbgqr+/ZluxKNOlpI3jqyGINsI4+tiiX7hBHIdkwCKwzIQXwEzWXOu7WywbXs//Bt2e/wsaJYMlvwZ2qBTRNJdTkGw8XcXO/oE11xoSYDNOndYNpv8JbDbvzE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766157520; c=relaxed/simple; bh=ePOl5CV57JQrRE0OHVBW+xEjLRbfQJdu22LM78xE+zM=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=FhrmXzIXBFLtTrBE1C7jPwlrz1N3AbG3PpkNGv8pNuDAoXBLKc9elrR4kW4UbkHgSrXYp7JSZQYlYTREosDbIXHh465Aqcv/XxHj3fa+OnkSETyoyAjYkeUpwuNC2Y2zfoZ2hbsasAOWX1jRq1aWE2bJmriACNrBHZLjJ9sQ1VU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ndufresne.ca; spf=pass smtp.mailfrom=ndufresne.ca; dkim=pass (2048-bit key) header.d=ndufresne-ca.20230601.gappssmtp.com header.i=@ndufresne-ca.20230601.gappssmtp.com header.b=2/YWPEO5; arc=none smtp.client-ip=209.85.222.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ndufresne.ca Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ndufresne.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ndufresne-ca.20230601.gappssmtp.com header.i=@ndufresne-ca.20230601.gappssmtp.com header.b="2/YWPEO5" Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-8b2ec756de0so185106585a.3 for ; Fri, 19 Dec 2025 07:18:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ndufresne-ca.20230601.gappssmtp.com; s=20230601; t=1766157517; x=1766762317; darn=vger.kernel.org; h=mime-version:user-agent:autocrypt:references:in-reply-to:date:cc:to :from:subject:message-id:from:to:cc:subject:date:message-id:reply-to; bh=ePOl5CV57JQrRE0OHVBW+xEjLRbfQJdu22LM78xE+zM=; b=2/YWPEO5AkEXKVGW8gJCkou3qNR5D/kSW5rBv9qEvQZv4c9a2hFm8/F7OhzG6Mm9oH EWR5fUjhoL/RNOVo0BsUA1WhC02bnweLMXCuQI7APNe8Ih/7uvQcQr9M5O6LY6brW9yb giU03T81unkxtaudaf7xO8WZEs4t5xe1pGXDUXIUDJyhBHWD1krEPp0rLNRhQvtY0M03 YDPDk5/MgSuJhfOX3RdSj2usDwX4J8w/ajIBhZojudMuWavUP1KD2Q/Loli2/tKditYa +oJS1Z7rKP/yZl1TtZVc/uwbVdB3m5rPzwf07+Of9D7K+zKAVjuXctjdyhSgSBARh4xe jyxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766157517; x=1766762317; h=mime-version:user-agent:autocrypt:references:in-reply-to:date:cc:to :from:subject:message-id:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ePOl5CV57JQrRE0OHVBW+xEjLRbfQJdu22LM78xE+zM=; b=A8ZePxWJw9tQftNA3MERvxAls+4x1xFCQ5Wx7/QpXBqwOmGrdG6V40yVlOrwvu8+K2 mYe56EC8APJkzVLa7mWTiugine8qdFz1gaN4BJ4FHVMfgt1Sj+9cTLvgCCFjIta/xsWj ksqRVTCyvh4TA3IGY1V1O7o6oWvsljU+ltOjcPBkevKzi4Qy8GYmX8aY++WI5JbIQ8m3 yb+uGqtzMTwlfipsytnT3V9L1W89oy4aHuQxGk3dx0JzLhRa8lKfabKJcZUBB803frF3 PgEF8TgysQT/2ogP5+vvj04JqTaseff6OQpgxNXXD2ou+nzvPU6J2VvfESxmQLHniy8O JxMg== X-Forwarded-Encrypted: i=1; AJvYcCWMuVDhRXyjmw2IkVo09MPgu7Hh/VDOtu9JrFRn3H44lCBWejTzNztXEzaZnzvsQEOzpvOMp6ocIrE84Nk=@vger.kernel.org X-Gm-Message-State: AOJu0YxhMw8SjIOp98WOkwGnVHV23gg1hrqWdNlHknzm1yg96xwVR89X fYusY7fTDkUm0BcAF59o8oCtQ3w7Nafyc7pLOMsJK2wyukOjg9tFn2jWBjcPtP0/xGM= X-Gm-Gg: AY/fxX7PhRXlQRyYUWUr6FB4TDynHPxSOVjSoNcyCCEEE9+Fx3xjchKLJLalRkbB/Wz NHFPJzCzQgiQgChgKfGW2/fKzCQGT9djgsYEVgMGcxs6V/v77jt+qRBQ9CdPo418KBhJAr9zYV1 OsBFHPuSmqUQeGXXkE+Tdmq70EMYtkqc50GqopkBreMEdzZcGGtTxKeBEAW7QFXpVRAOuNaEwTd XJqui9+tNDcflgVBGlJ7bz6xONb2LzHGSY9sX3Kft9oFEyHPKHi8BJ0VUqDvuiVWSGbug9VMqz/ KALiWTx+t39zi7YZsmftAwM+0LigFWYfzCy0kRWBNjE1fykIYZsmrpPILZWZ0WzF6sd/C+H7eLj /hpH8NkNPjFBLTYuKFxe6UnVSTTpBLPxP4KHmYt13zLqBwsTRV0MZQtpI1BZ1aEVmv/jRFFTDOI qYC5Igy5Mn9DRq7jNL X-Google-Smtp-Source: AGHT+IGau10k1eP0J6ZgC191UIoT5zL6u4IAchl+ERoj9e6cKwo0XprjejYKXEIsulZQEnpyM9HZrw== X-Received: by 2002:a05:620a:1789:b0:8b9:7a1a:8c73 with SMTP id af79cd13be357-8c08fa99ac2mr523096585a.46.1766157517165; Fri, 19 Dec 2025 07:18:37 -0800 (PST) Received: from ?IPv6:2606:6d00:17:7b4b::c41? ([2606:6d00:17:7b4b::c41]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8c095b79b32sm200473985a.0.2025.12.19.07.18.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Dec 2025 07:18:36 -0800 (PST) Message-ID: Subject: Re: [PATCH v1] media: videobuf2: Allow applications customize data offsets of capture buffers From: Nicolas Dufresne To: Hans Verkuil , Hirokazu Honda Cc: Dmitry Osipenko , Mauro Carvalho Chehab , Tomasz Figa , Benjamin Gaignard , Daniel Almeida , linux-media@vger.kernel.org, linux-kernel@vger.kernel.org Date: Fri, 19 Dec 2025 10:18:34 -0500 In-Reply-To: <221d3f70-b418-4f89-b352-773c544ed428@kernel.org> References: <20220322132329.6527-1-dmitry.osipenko@collabora.com> <5d367bbb-551f-4164-8473-a6c4b68793c0@kernel.org> <221d3f70-b418-4f89-b352-773c544ed428@kernel.org> Autocrypt: addr=nicolas@ndufresne.ca; prefer-encrypt=mutual; keydata=mDMEaCN2ixYJKwYBBAHaRw8BAQdAM0EHepTful3JOIzcPv6ekHOenE1u0vDG1gdHFrChD /e0J05pY29sYXMgRHVmcmVzbmUgPG5pY29sYXNAbmR1ZnJlc25lLmNhPoicBBMWCgBEAhsDBQsJCA cCAiICBhUKCQgLAgQWAgMBAh4HAheABQkJZfd1FiEE7w1SgRXEw8IaBG8S2UGUUSlgcvQFAmibrjo CGQEACgkQ2UGUUSlgcvQlQwD/RjpU1SZYcKG6pnfnQ8ivgtTkGDRUJ8gP3fK7+XUjRNIA/iXfhXMN abIWxO2oCXKf3TdD7aQ4070KO6zSxIcxgNQFtDFOaWNvbGFzIER1ZnJlc25lIDxuaWNvbGFzLmR1Z nJlc25lQGNvbGxhYm9yYS5jb20+iJkEExYKAEECGwMFCwkIBwICIgIGFQoJCAsCBBYCAwECHgcCF4 AWIQTvDVKBFcTDwhoEbxLZQZRRKWBy9AUCaCyyxgUJCWX3dQAKCRDZQZRRKWBy9ARJAP96pFmLffZ smBUpkyVBfFAf+zq6BJt769R0al3kHvUKdgD9G7KAHuioxD2v6SX7idpIazjzx8b8rfzwTWyOQWHC AAS0LU5pY29sYXMgRHVmcmVzbmUgPG5pY29sYXMuZHVmcmVzbmVAZ21haWwuY29tPoiZBBMWCgBBF iEE7w1SgRXEw8IaBG8S2UGUUSlgcvQFAmibrGYCGwMFCQll93UFCwkIBwICIgIGFQoJCAsCBBYCAw ECHgcCF4AACgkQ2UGUUSlgcvRObgD/YnQjfi4+L8f4fI7p1pPMTwRTcaRdy6aqkKEmKsCArzQBAK8 bRLv9QjuqsE6oQZra/RB4widZPvphs78H0P6NmpIJ Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-KSkR1bVw/bmN+66WFj60" User-Agent: Evolution 3.58.2 (3.58.2-1.fc43) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 --=-KSkR1bVw/bmN+66WFj60 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Hans, Le mercredi 17 d=C3=A9cembre 2025 =C3=A0 11:02 +0100, Hans Verkuil a =C3=A9= crit=C2=A0: > > For me, the most central issue in V4L2 is that the memory allocation/im= portation > > is bound to the operation queues. That brings all sort of issues such= =C2=A0 > >=20 > > - We can't queue twice the same frame > > - We can't mix external buffer with device allocated buffer > > - All buffers must have the exact same stride >=20 > The three limitations above are all technically possible to implement wit= h the > current vb2 framework/streaming uAPI, it's just that nobody was ever moti= vated > enough to add support for it. I don't see how technically it is possible without new uAPI to support heterogeneous strides, nor queuing twice the same frame while running in MM= AP, or mix device memory and externally imported memory. Please feel free to enlighten me if you have some spare time. Perhaps worth to mention that this is about doing this without creating gli= tches or jumps cause by expensive drain and streamoff/on sequences. >=20 > > - Application is responsible for caching which memory goes to which v4l= 2_buffer >=20 > True, but is this really a big deal? Maybe a lesser deal, but its extra complexity for both sides. The current b= ug being that if you use import mode only to workaround "queuing twice" issue,= you will endup with two mapping (which in some stateful codec firmware is not allowed by firmwares). So on top of the lookup userspace is doing to match buffer ids to their memory (have to cache pointers and fd and all), the dri= ver (or vb2) should also implement caching. My proposal imply solving that dual mapping issue for both current and future stream mode. In userspace, there is also cases, where the video buffers comes from other process, and you don't really know if two FD values points to the same dmab= uf. This is the kind of scenarios the DRM subsystem had to deal in compositors,= in our case that would be something such as pipewire. This caching is either m= icro optimization or simply to support firmware limitation, but a guarantee to h= ave 1 memory object for one chunk is in my opinion achievable and allow reducing complexity. >=20 > > - Attempting to import a buffer requires a free spot in the queue >=20 > True. >=20 > >=20 > > This adds on top of the v4l2_buffer structure limitation we have been t= argeting > > so far. With the growth of modern standard API (think Vulkan Video nota= bly), it > > becomes apparent that the model is too inflexible. This inability to se= parate > > memory allocation and importation from operations creates a lot of comp= lexity in > > user-space, leading to complicated bugs. > >=20 > > I've been quite about it, since until now I didn't have a solution in m= ind, but > > I recently come with some ideas. I'll will try develop these ideas, at = least in > > prose for now and come up with an RFC, hopefully somewhere beginning of= January > > 2026. That my proposal is accepted or not isn't quite relevant. But hop= efully it > > will be a starter to go go beyond just fixing what we see. In fact, thi= s next > > step is for me doing to be quite decisive if I continue doing codecs in= V4L2 or > > not in the long run. But I'm sure this is not just about video codecs. >=20 > I'm looking forward to your RFC! >=20 > What is important for me is that whatever we come up with, it is somethin= g that > existing drivers can easily support. A new streaming/buffer allocation uA= PI that > can only be supported by new drivers will need very, very good reasons fo= r it to > be accepted. >=20 > The nice thing about the v4l2_buffer_ext proposal was that it can easily = be > supported by all drivers. >=20 > From a technical perspective both struct v4l2_buffer and struct v4l2_form= at have > reached end-of-life: they are full of historical cruft, they are ineffici= ent, the > 32-bit/time32 compatibility code is awful and hard to maintain. I want to get a lot further in the proposal writing before going into exten= ded debate, but there is absolutely nothing set in stone. I already mention tha= t my proposal it to create traction, and I will have no problem moving direction= base on feedback. I can reassure you that part of the plan is to be able to implement that fo= r all drivers. In fact, I think its a good requirement for any proposal here. My proposal will not replace VB2 or the format. Perhaps few bits of the format= will be replicated somewhere else, but not all of it. Perhaps we'll get into bre= aking down the gigantic "format" structure into manageable part in the future, bu= t this aspect of the _ext proposal, with DRM formats and modifiers is orthogo= nal to what I have in mind. Also, the vast majority of what I want to proposal can be implemented insid= e vb2, without changing any drivers or userspace. But the full gain, I believ= e, will requires a new v4l2_memory type. Feel free to think of your own ideas toward these goals, our best bet for t= he future is if we can combine best ideas together. regards, Nicolas --=-KSkR1bVw/bmN+66WFj60 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iHUEABYKAB0WIQTvDVKBFcTDwhoEbxLZQZRRKWBy9AUCaUVsygAKCRDZQZRRKWBy 9L8/AP4zspaJwxAYdYw94fmT8EYEdyaJmWADL/NG13Gk6LJtEwD+NbMiev4MPZEx Cgy49EWQurDh3yBGZZwVVCkgUJZx3gg= =mUCg -----END PGP SIGNATURE----- --=-KSkR1bVw/bmN+66WFj60--