mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Mattijs Korpershoek <mkorpershoek@kernel.org>
To: Hans Verkuil <hverkuil+cisco@kernel.org>,
	 Laurent Pinchart <laurent.pinchart@ideasonboard.com>,
	 Kieran Bingham <kieran.bingham@ideasonboard.com>,
	 Sakari Ailus <sakari.ailus@linux.intel.com>,
	 Mauro Carvalho Chehab <mchehab@kernel.org>,
	 Andy Walls <awalls@md.metrocast.net>,
	Kees Cook <kees@kernel.org>,
	 "Gustavo A. R. Silva" <gustavoars@kernel.org>
Cc: Mattijs Korpershoek <mkorpershoek@kernel.org>,
	 Nicolas Dufresne <nicolas.dufres@collabora.com>,
	 Hans de Goede <hansg@kernel.org>,
	 Michael Riesch <michael.riesch@collabora.com>,
	 Dave Stevenson <dave.stevenson@raspberrypi.com>,
	 Maxime Ripard <mripard@kernel.org>,
	linux-media@vger.kernel.org,  linux-kernel@vger.kernel.org,
	linux-hardening@vger.kernel.org,
	 Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Subject: [PATCH RFC v2 00/16] media: Fault-Tolerant V4L2
Date: Wed, 18 Nov 2026 13:43:44 +0100	[thread overview]
Message-ID: <20261118-v4l2-sensor-detect-v2-0-1bb1a445b556@kernel.org> (raw)

On media-centric camera pipelines, every entity is known at boot time
and described statically (for example via device-tree).

The V4L2 framework assumes that every entity of the camera
pipeline is always present. This is a reasonable assumption for
integrated cameras in laptops or phones. However, in automotive systems,
cameras (e.g. rear-view or surround-view) can be damaged or disconnected.
When the framework assumes all sensors are present, a single missing
camera can prevent the entire pipeline from operating.

This is a known problem and has been discussed at various conferences:
* https://www.linuxtv.org/downloads/presentations/media_summit_2026/Michael%20-%20The%20Butterfly%20Effect.pdf
* https://www.linuxtv.org/news.php?entry=2022-11-14-0.hverkuil
* https://lore.kernel.org/linux-media/20261021224119.GA383235@killaraus.ideasonboard.com/T/#u

For example, in the following pipeline:
"""
  imx219  9-0010 -> ds90ub953 7-0044 -> ds90ub960 pad0 -+
                                                        |-- pad4 -> cdns_csi2rx -> ticsi2rx
  imx219 10-0010 -> ds90ub953 7-0045 -> ds90ub960 pad1 -+
"""

If imx219 9-0010 fails to probe, media-ctl -p will show:
"""
                    ds90ub953 7-0044 -> ds90ub960 pad0 -+
                                                        |-- pad4 -> cdns_csi2rx -> ticsi2rx
  imx219 10-0010 -> ds90ub953 7-0045 -> ds90ub960 pad1 -+
"""

Now, if ds90ub953 7-0044 fails to probe(), media-ctl -p will only show the
following topology:
"""
                                                                    cdns_csi2rx -> ticsi2rx
"""

To solve this, we chose to go for a simple solution inspired by
DRM connectors.

We created an ioctl for media entitities to report their current
connection status.
This allows applications to discover an entity state when discovering
the graph (using media-ctl -p, for example)
The entity status is then cached. When applications start streaming,
the status is polled on a regular basis and reported as a media event
every time it changes.
This mandates that the driver can probe even when the device is
not connected.

This has the benefits of being fairly simple to implement, understand
and support in drivers.

This new behaviour is opt-in. It is enabled via a module parameter
for the Media Controler: mc.fault_tolerant.

Patch 1-9 Add event support at media device level
Patch 10-14 Add entity status and ioctls to interact with it
Patch 15-16 implement this pattern switch for the imx219 driver.

With this, we can hot-plug the sensor and get notified via the media
device of the status:

[root@fedora]# ./media-event-monitor
Subscribed to ENTITY_REGISTRATION and ENTITY_STATUS_CHANGE on /dev/media0
Waiting for events (Ctrl-C to quit)...
event: type=2 (ENTITY_STATUS_CHANGE) id=0 seq=0 ts=403045107120 pending=0
  entity_id=40 status=DISCONNECTED
event: type=2 (ENTITY_STATUS_CHANGE) id=0 seq=1 ts=415292362587 pending=0
  entity_id=40 status=CONNECTED

The code for this demo is available at:
https://gitlab.com/-/snippets/6077614

The media device event code is mostly Sakari's work which can be found at:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=partial-reg

Current known limitations:
* Only leaf entities (typically sensor nodes) are covered. We should
  also address entities that are in the middle of the media graph

This has been tested on a SK-AM69 board on top of the following series:
* https://lore.kernel.org/all/20260824121451.3348583-1-sakari.ailus@linux.intel.com/
* https://lore.kernel.org/all/20261104-j721e-csi2rx-crash-sd-enable-streams-v1-1-b77ee38c8dcf@kernel.org/

Thanks,
Mattijs

---
Changes in v2:
- Notifications are now done at media device level (Laurent, Nicolas)
- Connection status is tracked at entity level (Laurent)
- imx219: Fail the probe if CHIP_ID does not match (Dave)
- Dropped patch 'media: imx219: Move LP-11 state switch to power_on()'
  as we did not get any feedback from cadence on this
- Replaced VIDIOC_SUBDEV_G_CONNECTION_STATUS with a media controller
  ioctl instead: MEDIA_IOC_G_ENTITY_STATUS
- Do status polling in generic framework (Dave)
- Increase polling interval to 10s (Dave)
- Polling/Fault tolerance is only enabled when mc.faul_tolerance=1
  module param is set (Dave, Hans V, Sakari)
- Reworked .detect() polling to be only done when streaming is on. This
  should address the privacy led blinking concerns (Dave, Hans D)
- Link to v1: https://lore.kernel.org/r/20261001-v4l2-sensor-detect-v1-0-a45993be17b8@kernel.org

---
Laurent Pinchart (1):
      media: mc: Add per-file-handle data support

Mattijs Korpershoek (7):
      media: media.h: Add entity connection status
      media: mc: Add .detect() to media_entity_operations
      media: mc: Add MEDIA_IOC_G_ENTITY_STATUS ioctl
      media: mc: Add MEDIA_EVENT_ENTITY_STATUS_CHANGE event
      media: mc: Add detect polling helpers for drivers
      media: imx219: Allow driver probe with missing sensor
      media: imx219: Implement .detect() sensor operation

Sakari Ailus (8):
      media: mc: Add struct media_device_fh pointer to media device IOCTL op
      v4l: event: Separate event related information from struct v4l2_fh
      v4l: event: Rework data structures for separating events and V4L2
      v4l: event: Split the event framework into generic and V4L2 bits
      media: mc: Maintain a list of open file handles in a media device
      media: mc: Add MEDIA_IOC_DQEVENT
      media: mc: Add MEDIA_EVENT_ENTITY_REGISTRATION
      Documentation: media: Add Media controller event documentation

 .../userspace-api/media/mediactl/media-funcs.rst   |   3 +
 .../media/mediactl/media-ioc-dqevent.rst           | 130 +++++++++
 .../media/mediactl/media-ioc-g-entity-status.rst   |  92 ++++++
 .../media/mediactl/media-ioc-subscribe-event.rst   | 112 +++++++
 drivers/media/Kconfig                              |   3 +
 drivers/media/i2c/imx219.c                         | 117 +++++---
 drivers/media/mc/Kconfig                           |   3 +
 drivers/media/mc/Makefile                          |   3 +-
 drivers/media/mc/mc-device.c                       | 236 ++++++++++++++-
 drivers/media/mc/mc-devnode.c                      |  19 +-
 drivers/media/mc/mc-entity.c                       |  59 ++++
 drivers/media/mc/mc-evbase.c                       | 260 +++++++++++++++++
 drivers/media/mc/mc-event.c                        | 244 ++++++++++++++++
 drivers/media/pci/ivtv/ivtv-fileops.c              |   2 +-
 drivers/media/usb/uvc/uvc_ctrl.c                   |  36 ++-
 drivers/media/v4l2-core/v4l2-ctrls-api.c           |  16 +-
 drivers/media/v4l2-core/v4l2-ctrls-core.c          |  20 +-
 drivers/media/v4l2-core/v4l2-event.c               | 324 +++++++++------------
 drivers/media/v4l2-core/v4l2-fh.c                  |  12 +-
 include/media/media-device.h                       |  24 ++
 include/media/media-devnode.h                      |  18 +-
 include/media/media-entity.h                       |  77 +++++
 include/media/media-evbase.h                       | 200 +++++++++++++
 include/media/media-event.h                        | 181 ++++++++++++
 include/media/media-fh.h                           |  37 +++
 include/media/v4l2-event.h                         | 106 ++++---
 include/media/v4l2-fh.h                            |  32 +-
 include/uapi/linux/media.h                         |  51 +++-
 28 files changed, 2079 insertions(+), 338 deletions(-)
---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
base-tree: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
change-id: 20260923-v4l2-sensor-detect-32a0b2936cc1

Best regards,
--  
Mattijs Korpershoek <mkorpershoek@kernel.org>


             reply	other threads:[~2026-11-18 12:43 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 12:43 Mattijs Korpershoek [this message]
2026-11-18 12:43 ` [PATCH RFC v2 01/16] media: mc: Add struct media_device_fh pointer to media device IOCTL op Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 02/16] media: mc: Add per-file-handle data support Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 03/16] v4l: event: Separate event related information from struct v4l2_fh Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 04/16] v4l: event: Rework data structures for separating events and V4L2 Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 05/16] v4l: event: Split the event framework into generic and V4L2 bits Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 06/16] media: mc: Maintain a list of open file handles in a media device Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 07/16] media: mc: Add MEDIA_IOC_DQEVENT Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 08/16] media: mc: Add MEDIA_EVENT_ENTITY_REGISTRATION Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 09/16] Documentation: media: Add Media controller event documentation Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 10/16] media: media.h: Add entity connection status Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 11/16] media: mc: Add .detect() to media_entity_operations Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 12/16] media: mc: Add MEDIA_IOC_G_ENTITY_STATUS ioctl Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 13/16] media: mc: Add MEDIA_EVENT_ENTITY_STATUS_CHANGE event Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 14/16] media: mc: Add detect polling helpers for drivers Mattijs Korpershoek
2026-11-18 12:43 ` [PATCH RFC v2 15/16] media: imx219: Allow driver probe with missing sensor Mattijs Korpershoek
2026-11-18 12:44 ` [PATCH RFC v2 16/16] media: imx219: Implement .detect() sensor operation Mattijs Korpershoek

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118-v4l2-sensor-detect-v2-0-1bb1a445b556@kernel.org \
    --to=mkorpershoek@kernel.org \
    --cc=awalls@md.metrocast.net \
    --cc=dave.stevenson@raspberrypi.com \
    --cc=gustavoars@kernel.org \
    --cc=hansg@kernel.org \
    --cc=hverkuil+cisco@kernel.org \
    --cc=kees@kernel.org \
    --cc=kieran.bingham@ideasonboard.com \
    --cc=laurent.pinchart+renesas@ideasonboard.com \
    --cc=laurent.pinchart@ideasonboard.com \
    --cc=linux-hardening@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-media@vger.kernel.org \
    --cc=mchehab@kernel.org \
    --cc=michael.riesch@collabora.com \
    --cc=mripard@kernel.org \
    --cc=nicolas.dufres@collabora.com \
    --cc=sakari.ailus@linux.intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®