From: Mattijs Korpershoek <mkorpershoek@kernel.org>
To: Hans Verkuil <hverkuil+cisco@kernel.org>,
Laurent Pinchart <laurent.pinchart@ideasonboard.com>,
Kieran Bingham <kieran.bingham@ideasonboard.com>,
Sakari Ailus <sakari.ailus@linux.intel.com>,
Mauro Carvalho Chehab <mchehab@kernel.org>,
Andy Walls <awalls@md.metrocast.net>,
Kees Cook <kees@kernel.org>,
"Gustavo A. R. Silva" <gustavoars@kernel.org>
Cc: Mattijs Korpershoek <mkorpershoek@kernel.org>,
Nicolas Dufresne <nicolas.dufres@collabora.com>,
Hans de Goede <hansg@kernel.org>,
Michael Riesch <michael.riesch@collabora.com>,
Dave Stevenson <dave.stevenson@raspberrypi.com>,
Maxime Ripard <mripard@kernel.org>,
linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-hardening@vger.kernel.org,
Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Subject: [PATCH RFC v2 00/16] media: Fault-Tolerant V4L2
Date: Wed, 18 Nov 2026 13:43:44 +0100 [thread overview]
Message-ID: <20261118-v4l2-sensor-detect-v2-0-1bb1a445b556@kernel.org> (raw)
On media-centric camera pipelines, every entity is known at boot time
and described statically (for example via device-tree).
The V4L2 framework assumes that every entity of the camera
pipeline is always present. This is a reasonable assumption for
integrated cameras in laptops or phones. However, in automotive systems,
cameras (e.g. rear-view or surround-view) can be damaged or disconnected.
When the framework assumes all sensors are present, a single missing
camera can prevent the entire pipeline from operating.
This is a known problem and has been discussed at various conferences:
* https://www.linuxtv.org/downloads/presentations/media_summit_2026/Michael%20-%20The%20Butterfly%20Effect.pdf
* https://www.linuxtv.org/news.php?entry=2022-11-14-0.hverkuil
* https://lore.kernel.org/linux-media/20261021224119.GA383235@killaraus.ideasonboard.com/T/#u
For example, in the following pipeline:
"""
imx219 9-0010 -> ds90ub953 7-0044 -> ds90ub960 pad0 -+
|-- pad4 -> cdns_csi2rx -> ticsi2rx
imx219 10-0010 -> ds90ub953 7-0045 -> ds90ub960 pad1 -+
"""
If imx219 9-0010 fails to probe, media-ctl -p will show:
"""
ds90ub953 7-0044 -> ds90ub960 pad0 -+
|-- pad4 -> cdns_csi2rx -> ticsi2rx
imx219 10-0010 -> ds90ub953 7-0045 -> ds90ub960 pad1 -+
"""
Now, if ds90ub953 7-0044 fails to probe(), media-ctl -p will only show the
following topology:
"""
cdns_csi2rx -> ticsi2rx
"""
To solve this, we chose to go for a simple solution inspired by
DRM connectors.
We created an ioctl for media entitities to report their current
connection status.
This allows applications to discover an entity state when discovering
the graph (using media-ctl -p, for example)
The entity status is then cached. When applications start streaming,
the status is polled on a regular basis and reported as a media event
every time it changes.
This mandates that the driver can probe even when the device is
not connected.
This has the benefits of being fairly simple to implement, understand
and support in drivers.
This new behaviour is opt-in. It is enabled via a module parameter
for the Media Controler: mc.fault_tolerant.
Patch 1-9 Add event support at media device level
Patch 10-14 Add entity status and ioctls to interact with it
Patch 15-16 implement this pattern switch for the imx219 driver.
With this, we can hot-plug the sensor and get notified via the media
device of the status:
[root@fedora]# ./media-event-monitor
Subscribed to ENTITY_REGISTRATION and ENTITY_STATUS_CHANGE on /dev/media0
Waiting for events (Ctrl-C to quit)...
event: type=2 (ENTITY_STATUS_CHANGE) id=0 seq=0 ts=403045107120 pending=0
entity_id=40 status=DISCONNECTED
event: type=2 (ENTITY_STATUS_CHANGE) id=0 seq=1 ts=415292362587 pending=0
entity_id=40 status=CONNECTED
The code for this demo is available at:
https://gitlab.com/-/snippets/6077614
The media device event code is mostly Sakari's work which can be found at:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=partial-reg
Current known limitations:
* Only leaf entities (typically sensor nodes) are covered. We should
also address entities that are in the middle of the media graph
This has been tested on a SK-AM69 board on top of the following series:
* https://lore.kernel.org/all/20260824121451.3348583-1-sakari.ailus@linux.intel.com/
* https://lore.kernel.org/all/20261104-j721e-csi2rx-crash-sd-enable-streams-v1-1-b77ee38c8dcf@kernel.org/
Thanks,
Mattijs
---
Changes in v2:
- Notifications are now done at media device level (Laurent, Nicolas)
- Connection status is tracked at entity level (Laurent)
- imx219: Fail the probe if CHIP_ID does not match (Dave)
- Dropped patch 'media: imx219: Move LP-11 state switch to power_on()'
as we did not get any feedback from cadence on this
- Replaced VIDIOC_SUBDEV_G_CONNECTION_STATUS with a media controller
ioctl instead: MEDIA_IOC_G_ENTITY_STATUS
- Do status polling in generic framework (Dave)
- Increase polling interval to 10s (Dave)
- Polling/Fault tolerance is only enabled when mc.faul_tolerance=1
module param is set (Dave, Hans V, Sakari)
- Reworked .detect() polling to be only done when streaming is on. This
should address the privacy led blinking concerns (Dave, Hans D)
- Link to v1: https://lore.kernel.org/r/20261001-v4l2-sensor-detect-v1-0-a45993be17b8@kernel.org
---
Laurent Pinchart (1):
media: mc: Add per-file-handle data support
Mattijs Korpershoek (7):
media: media.h: Add entity connection status
media: mc: Add .detect() to media_entity_operations
media: mc: Add MEDIA_IOC_G_ENTITY_STATUS ioctl
media: mc: Add MEDIA_EVENT_ENTITY_STATUS_CHANGE event
media: mc: Add detect polling helpers for drivers
media: imx219: Allow driver probe with missing sensor
media: imx219: Implement .detect() sensor operation
Sakari Ailus (8):
media: mc: Add struct media_device_fh pointer to media device IOCTL op
v4l: event: Separate event related information from struct v4l2_fh
v4l: event: Rework data structures for separating events and V4L2
v4l: event: Split the event framework into generic and V4L2 bits
media: mc: Maintain a list of open file handles in a media device
media: mc: Add MEDIA_IOC_DQEVENT
media: mc: Add MEDIA_EVENT_ENTITY_REGISTRATION
Documentation: media: Add Media controller event documentation
.../userspace-api/media/mediactl/media-funcs.rst | 3 +
.../media/mediactl/media-ioc-dqevent.rst | 130 +++++++++
.../media/mediactl/media-ioc-g-entity-status.rst | 92 ++++++
.../media/mediactl/media-ioc-subscribe-event.rst | 112 +++++++
drivers/media/Kconfig | 3 +
drivers/media/i2c/imx219.c | 117 +++++---
drivers/media/mc/Kconfig | 3 +
drivers/media/mc/Makefile | 3 +-
drivers/media/mc/mc-device.c | 236 ++++++++++++++-
drivers/media/mc/mc-devnode.c | 19 +-
drivers/media/mc/mc-entity.c | 59 ++++
drivers/media/mc/mc-evbase.c | 260 +++++++++++++++++
drivers/media/mc/mc-event.c | 244 ++++++++++++++++
drivers/media/pci/ivtv/ivtv-fileops.c | 2 +-
drivers/media/usb/uvc/uvc_ctrl.c | 36 ++-
drivers/media/v4l2-core/v4l2-ctrls-api.c | 16 +-
drivers/media/v4l2-core/v4l2-ctrls-core.c | 20 +-
drivers/media/v4l2-core/v4l2-event.c | 324 +++++++++------------
drivers/media/v4l2-core/v4l2-fh.c | 12 +-
include/media/media-device.h | 24 ++
include/media/media-devnode.h | 18 +-
include/media/media-entity.h | 77 +++++
include/media/media-evbase.h | 200 +++++++++++++
include/media/media-event.h | 181 ++++++++++++
include/media/media-fh.h | 37 +++
include/media/v4l2-event.h | 106 ++++---
include/media/v4l2-fh.h | 32 +-
include/uapi/linux/media.h | 51 +++-
28 files changed, 2079 insertions(+), 338 deletions(-)
---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
base-tree: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
change-id: 20260923-v4l2-sensor-detect-32a0b2936cc1
Best regards,
--
Mattijs Korpershoek <mkorpershoek@kernel.org>
next reply other threads:[~2026-11-18 12:43 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 12:43 Mattijs Korpershoek [this message]
2026-11-18 12:43 ` [PATCH RFC v2 01/16] media: mc: Add struct media_device_fh pointer to media device IOCTL op Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 02/16] media: mc: Add per-file-handle data support Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 03/16] v4l: event: Separate event related information from struct v4l2_fh Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 04/16] v4l: event: Rework data structures for separating events and V4L2 Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 05/16] v4l: event: Split the event framework into generic and V4L2 bits Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 06/16] media: mc: Maintain a list of open file handles in a media device Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 07/16] media: mc: Add MEDIA_IOC_DQEVENT Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 08/16] media: mc: Add MEDIA_EVENT_ENTITY_REGISTRATION Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 09/16] Documentation: media: Add Media controller event documentation Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 10/16] media: media.h: Add entity connection status Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 11/16] media: mc: Add .detect() to media_entity_operations Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 12/16] media: mc: Add MEDIA_IOC_G_ENTITY_STATUS ioctl Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 13/16] media: mc: Add MEDIA_EVENT_ENTITY_STATUS_CHANGE event Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 14/16] media: mc: Add detect polling helpers for drivers Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 15/16] media: imx219: Allow driver probe with missing sensor Mattijs Korpershoek
2026-11-18 12:44 ` [PATCH RFC v2 16/16] media: imx219: Implement .detect() sensor operation Mattijs Korpershoek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118-v4l2-sensor-detect-v2-0-1bb1a445b556@kernel.org \
--to=mkorpershoek@kernel.org \
--cc=awalls@md.metrocast.net \
--cc=dave.stevenson@raspberrypi.com \
--cc=gustavoars@kernel.org \
--cc=hansg@kernel.org \
--cc=hverkuil+cisco@kernel.org \
--cc=kees@kernel.org \
--cc=kieran.bingham@ideasonboard.com \
--cc=laurent.pinchart+renesas@ideasonboard.com \
--cc=laurent.pinchart@ideasonboard.com \
--cc=linux-hardening@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=mchehab@kernel.org \
--cc=michael.riesch@collabora.com \
--cc=mripard@kernel.org \
--cc=nicolas.dufres@collabora.com \
--cc=sakari.ailus@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®