From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
To: Wentong Wu <wentong.wu@intel.com>
Cc: mchehab@kernel.org, sakari.ailus@linux.intel.com,
linux-media@vger.kernel.org, srinivas.pandruvada@intel.com,
pierre-louis.bossart@linux.intel.com, zhifeng.wang@intel.com,
xiang.ye@intel.com, tian.shu.qiu@intel.com, bingbu.cao@intel.com,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)
Date: Wed, 15 Feb 2023 11:43:05 +0200 [thread overview]
Message-ID: <Y+ypKYI4c6fHL4Eu@pendragon.ideasonboard.com> (raw)
In-Reply-To: <20230213022347.2480307-1-wentong.wu@intel.com>
Hello Wentong,
On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
> companion chip designed to provide secure and low power vision capability
> to IA platforms. IVSC is available in existing commercial platforms from
> multiple OEMs.
>
> The primary use case of IVSC is to bring in context awareness. IVSC
> interfaces directly with the platform main camera sensor via a CSI-2 link
> and processes the image data with the embedded AI engine. The detected
> events are sent over I2C to ISH (Intel Sensor Hub) for additional data
> fusion from multiple sensors. The fusion results are used to implement
> advanced use cases like:
> - Face detection to unlock screen
> - Detect user presence to manage backlight setting or waking up system
Do you have plan to support these features in the ivsc driver in the
future ?
> Since the Image Processing Unit(IPU) used on the host processor needs to
> configure the CSI-2 link in normal camera usages, the CSI-2 link and
> camera sensor can only be used in mutually-exclusive ways by host IPU and
> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
> driver can take ownership of the CSI-2 link and camera sensor using
> interfaces provided by this IVSC driver.
>
> Switching ownership requires an interface with two different hardware
> modules inside IVSC. The software interface to these modules is via Intel
> MEI (The Intel Management Engine) commands. These two hardware modules
> have two different MEI UUIDs to enumerate. These hardware modules are:
> - ACE (Algorithm Context Engine): This module is for algorithm computing
> when IVSC owns camera sensor. Also ACE module controls camera sensor's
> ownership. This hardware module is used to set ownership of camera sensor.
> - CSI (Camera Serial Interface): This module is used to route camera
> sensor data either to IVSC or to host for IPU driver and application.
>
> IVSC also provides a privacy mode. When privacy mode is turned on,
> camera sensor can't be used. This means that both ACE and host IPU can't
> get image data. And when this mode is turned on, host IPU driver is
> informed via a registered callback, so that user can be notified.
How does the privacy mode work, and how can the user trust that the
closed-source IVSC and IME firmwares will honour the privacy settings ?
> In summary, to acquire ownership of camera by IPU driver, first ACE
> module needs to be informed of ownership and then to setup MIPI CSI-2
> link for the camera sensor and IPU.
>
> Implementation:
> There are two different drivers to handle ACE and CSI hardware modules
> inside IVSC.
> - mei_csi: MEI client driver to send commands and receive notifications
> from CSI module.
> - mei_ace: MEI client driver to send commands and get status from ACE
> module.
> Interface is exposed via ivsc.h to acquire and release camera sensor and
> CSI-2 link.
Do I understand correctly, from your diagram below, that the
communication between the IME and IVSC goes through SPI ?
> Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
> -----------------------------------------------------------------------------
> | Host Processor |
> | |
> | ----------------- ----------------- --------------- |
> | | | | | | | I2C |
> | | IPU | | ISH | |camera driver|--| |
> | | | | | | | | |
> | ----------------- ----------------- --------------- | |
> | | | | | |
> | | | --------------- | |
> | | | | | | |
> | | | | IVSC driver | | |
> | | | | | | |
> | | | --------------- | |
> | | | | | |
> ----------------|-----------------------|----------------------|---------|---
> | CSI | I2C |SPI |
> | | | |
> ----------------|-----------------------|---------------- | |
> | IVSC | | | |
> | | | | |
> | ----------------- ----------------- | | |
> | | | | | | | |
> | | CSI | | ACE | |------| |
> | | | | | | |
> | ----------------- ----------------- | |
> | | | I2C | |
> ----------------|-----------------------|---------------- |
> | CSI | |
> | | |
> -------------------------------- |
> | | I2C |
> | camera sensor |-----------------------------|
> | |
> --------------------------------
>
> Wentong Wu (3):
> media: pci: intel: ivsc: Add CSI submodule
> media: pci: intel: ivsc: Add ACE submodule
> media: pci: intel: ivsc: Add acquire/release API for ivsc
>
> drivers/media/pci/Kconfig | 1 +
> drivers/media/pci/intel/Makefile | 2 +
> drivers/media/pci/intel/ivsc/Kconfig | 12 +
> drivers/media/pci/intel/ivsc/Makefile | 7 +
> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
> drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
> drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++
> drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
> drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
> include/linux/ivsc.h | 74 ++++
> 10 files changed, 1090 insertions(+)
> create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
> create mode 100644 drivers/media/pci/intel/ivsc/Makefile
> create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
> create mode 100644 include/linux/ivsc.h
--
Regards,
Laurent Pinchart
next prev parent reply other threads:[~2023-02-15 9:44 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-13 2:23 Wentong Wu
2023-02-13 2:23 ` [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule Wentong Wu
2023-02-14 21:32 ` Sakari Ailus
2023-02-13 2:23 ` [PATCH v2 2/3] media: pci: intel: ivsc: Add ACE submodule Wentong Wu
2023-02-13 2:23 ` [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc Wentong Wu
2023-02-14 16:06 ` Sakari Ailus
2023-02-15 9:03 ` Hans de Goede
2023-02-15 9:45 ` Laurent Pinchart
2023-02-17 6:10 ` Wu, Wentong
2023-02-17 11:19 ` Laurent Pinchart
2023-02-20 3:50 ` Wu, Wentong
2023-02-20 8:36 ` Sakari Ailus
2023-02-20 8:57 ` Wu, Wentong
2023-02-20 9:03 ` Hans de Goede
2023-03-01 0:09 ` Laurent Pinchart
2023-02-28 6:42 ` Wu, Wentong
2023-02-28 6:35 ` Wu, Wentong
2023-02-28 8:23 ` Sakari Ailus
2023-03-01 7:26 ` Wu, Wentong
2023-03-01 11:11 ` Sakari Ailus
2023-02-13 3:20 ` [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC) Pandruvada, Srinivas
2023-02-14 6:34 ` Wu, Wentong
[not found] ` <20230213034202.2926-1-hdanton@sina.com>
2023-02-14 6:25 ` [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule Wu, Wentong
[not found] ` <20230214083533.3410-1-hdanton@sina.com>
2023-02-14 12:40 ` Wu, Wentong
[not found] ` <20230214235405.3587-1-hdanton@sina.com>
2023-02-17 5:52 ` Wu, Wentong
[not found] ` <20230217062815.1682-1-hdanton@sina.com>
2023-02-17 7:37 ` Wu, Wentong
2023-02-15 9:43 ` Laurent Pinchart [this message]
2023-02-15 12:09 ` [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC) Bingbu Cao
2023-02-16 13:12 ` Sakari Ailus
2023-02-17 1:43 ` Bingbu Cao
2023-02-17 6:28 ` Wu, Wentong
2023-02-17 10:49 ` Sakari Ailus
2023-02-20 3:45 ` Wu, Wentong
2023-02-17 6:20 ` Wu, Wentong
2023-02-17 11:12 ` Laurent Pinchart
2023-02-20 4:00 ` Wu, Wentong
2023-03-01 10:34 ` Sakari Ailus
2023-03-01 10:41 ` Hans de Goede
2023-03-07 8:17 ` Wu, Wentong
2023-03-07 8:30 ` Sakari Ailus
2023-03-07 8:40 ` Wu, Wentong
2023-03-07 9:10 ` Hans de Goede
2023-03-09 1:08 ` Wu, Wentong
2023-03-09 9:28 ` Hans de Goede
2023-03-09 13:21 ` Wu, Wentong
2023-03-09 15:24 ` Hans de Goede
2023-03-16 2:58 ` Wu, Wentong
2023-03-16 9:03 ` Hans de Goede
2023-03-17 7:30 ` Wu, Wentong
2023-03-17 8:58 ` Sakari Ailus
2023-03-19 13:09 ` Wu, Wentong
2023-03-16 8:11 ` Wu, Wentong
2023-03-16 8:37 ` Hans de Goede
2023-03-03 6:10 ` Wu, Wentong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y+ypKYI4c6fHL4Eu@pendragon.ideasonboard.com \
--to=laurent.pinchart@ideasonboard.com \
--cc=bingbu.cao@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=mchehab@kernel.org \
--cc=pierre-louis.bossart@linux.intel.com \
--cc=sakari.ailus@linux.intel.com \
--cc=srinivas.pandruvada@intel.com \
--cc=tian.shu.qiu@intel.com \
--cc=wentong.wu@intel.com \
--cc=xiang.ye@intel.com \
--cc=zhifeng.wang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®