From: Wentong Wu <wentong.wu@intel.com>
To: mchehab@kernel.org, sakari.ailus@linux.intel.com,
linux-media@vger.kernel.org
Cc: srinivas.pandruvada@intel.com,
pierre-louis.bossart@linux.intel.com, zhifeng.wang@intel.com,
xiang.ye@intel.com, tian.shu.qiu@intel.com, bingbu.cao@intel.com,
linux-kernel@vger.kernel.org, Wentong Wu <wentong.wu@intel.com>
Subject: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)
Date: Mon, 13 Feb 2023 10:23:44 +0800 [thread overview]
Message-ID: <20230213022347.2480307-1-wentong.wu@intel.com> (raw)
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
companion chip designed to provide secure and low power vision capability
to IA platforms. IVSC is available in existing commercial platforms from
multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC
interfaces directly with the platform main camera sensor via a CSI-2 link
and processes the image data with the embedded AI engine. The detected
events are sent over I2C to ISH (Intel Sensor Hub) for additional data
fusion from multiple sensors. The fusion results are used to implement
advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system
Since the Image Processing Unit(IPU) used on the host processor needs to
configure the CSI-2 link in normal camera usages, the CSI-2 link and
camera sensor can only be used in mutually-exclusive ways by host IPU and
IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera sensor using
interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware
modules inside IVSC. The software interface to these modules is via Intel
MEI (The Intel Management Engine) commands. These two hardware modules
have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's
ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.
IVSC also provides a privacy mode. When privacy mode is turned on,
camera sensor can't be used. This means that both ACE and host IPU can't
get image data. And when this mode is turned on, host IPU driver is
informed via a registered callback, so that user can be notified.
In summary, to acquire ownership of camera by IPU driver, first ACE
module needs to be informed of ownership and then to setup MIPI CSI-2
link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules
inside IVSC.
- mei_csi: MEI client driver to send commands and receive notifications
from CSI module.
- mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via ivsc.h to acquire and release camera sensor and
CSI-2 link.
Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------
| Host Processor |
| |
| ----------------- ----------------- --------------- |
| | | | | | | I2C |
| | IPU | | ISH | |camera driver|--| |
| | | | | | | | |
| ----------------- ----------------- --------------- | |
| | | | | |
| | | --------------- | |
| | | | | | |
| | | | IVSC driver | | |
| | | | | | |
| | | --------------- | |
| | | | | |
----------------|-----------------------|----------------------|---------|---
| CSI | I2C |SPI |
| | | |
----------------|-----------------------|---------------- | |
| IVSC | | | |
| | | | |
| ----------------- ----------------- | | |
| | | | | | | |
| | CSI | | ACE | |------| |
| | | | | | |
| ----------------- ----------------- | |
| | | I2C | |
----------------|-----------------------|---------------- |
| CSI | |
| | |
-------------------------------- |
| | I2C |
| camera sensor |-----------------------------|
| |
--------------------------------
Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivsc
drivers/media/pci/Kconfig | 1 +
drivers/media/pci/intel/Makefile | 2 +
drivers/media/pci/intel/ivsc/Kconfig | 12 +
drivers/media/pci/intel/ivsc/Makefile | 7 +
drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++
drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
include/linux/ivsc.h | 74 ++++
10 files changed, 1090 insertions(+)
create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
create mode 100644 drivers/media/pci/intel/ivsc/Makefile
create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
create mode 100644 include/linux/ivsc.h
--
2.25.1
next reply other threads:[~2023-02-13 2:21 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-13 2:23 Wentong Wu [this message]
2023-02-13 2:23 ` [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule Wentong Wu
2023-02-14 21:32 ` Sakari Ailus
2023-02-13 2:23 ` [PATCH v2 2/3] media: pci: intel: ivsc: Add ACE submodule Wentong Wu
2023-02-13 2:23 ` [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc Wentong Wu
2023-02-14 16:06 ` Sakari Ailus
2023-02-15 9:03 ` Hans de Goede
2023-02-15 9:45 ` Laurent Pinchart
2023-02-17 6:10 ` Wu, Wentong
2023-02-17 11:19 ` Laurent Pinchart
2023-02-20 3:50 ` Wu, Wentong
2023-02-20 8:36 ` Sakari Ailus
2023-02-20 8:57 ` Wu, Wentong
2023-02-20 9:03 ` Hans de Goede
2023-03-01 0:09 ` Laurent Pinchart
2023-02-28 6:42 ` Wu, Wentong
2023-02-28 6:35 ` Wu, Wentong
2023-02-28 8:23 ` Sakari Ailus
2023-03-01 7:26 ` Wu, Wentong
2023-03-01 11:11 ` Sakari Ailus
2023-02-13 3:20 ` [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC) Pandruvada, Srinivas
2023-02-14 6:34 ` Wu, Wentong
[not found] ` <20230213034202.2926-1-hdanton@sina.com>
2023-02-14 6:25 ` [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule Wu, Wentong
[not found] ` <20230214083533.3410-1-hdanton@sina.com>
2023-02-14 12:40 ` Wu, Wentong
[not found] ` <20230214235405.3587-1-hdanton@sina.com>
2023-02-17 5:52 ` Wu, Wentong
[not found] ` <20230217062815.1682-1-hdanton@sina.com>
2023-02-17 7:37 ` Wu, Wentong
2023-02-15 9:43 ` [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC) Laurent Pinchart
2023-02-15 12:09 ` Bingbu Cao
2023-02-16 13:12 ` Sakari Ailus
2023-02-17 1:43 ` Bingbu Cao
2023-02-17 6:28 ` Wu, Wentong
2023-02-17 10:49 ` Sakari Ailus
2023-02-20 3:45 ` Wu, Wentong
2023-02-17 6:20 ` Wu, Wentong
2023-02-17 11:12 ` Laurent Pinchart
2023-02-20 4:00 ` Wu, Wentong
2023-03-01 10:34 ` Sakari Ailus
2023-03-01 10:41 ` Hans de Goede
2023-03-07 8:17 ` Wu, Wentong
2023-03-07 8:30 ` Sakari Ailus
2023-03-07 8:40 ` Wu, Wentong
2023-03-07 9:10 ` Hans de Goede
2023-03-09 1:08 ` Wu, Wentong
2023-03-09 9:28 ` Hans de Goede
2023-03-09 13:21 ` Wu, Wentong
2023-03-09 15:24 ` Hans de Goede
2023-03-16 2:58 ` Wu, Wentong
2023-03-16 9:03 ` Hans de Goede
2023-03-17 7:30 ` Wu, Wentong
2023-03-17 8:58 ` Sakari Ailus
2023-03-19 13:09 ` Wu, Wentong
2023-03-16 8:11 ` Wu, Wentong
2023-03-16 8:37 ` Hans de Goede
2023-03-03 6:10 ` Wu, Wentong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230213022347.2480307-1-wentong.wu@intel.com \
--to=wentong.wu@intel.com \
--cc=bingbu.cao@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=mchehab@kernel.org \
--cc=pierre-louis.bossart@linux.intel.com \
--cc=sakari.ailus@linux.intel.com \
--cc=srinivas.pandruvada@intel.com \
--cc=tian.shu.qiu@intel.com \
--cc=xiang.ye@intel.com \
--cc=zhifeng.wang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®