Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

["Pandruvada, Srinivas" <srinivas.pandruvada@intel.com>]

On Mon, 2023-02-13 at 10:23 +0800, Wentong Wu wrote:
> 

v2 without change log, why?

> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is
> a
> companion chip designed to provide secure and low power vision
> capability
> to IA platforms. IVSC is available in existing commercial platforms
> from
> multiple OEMs.
> 
> The primary use case of IVSC is to bring in context awareness. IVSC
> interfaces directly with the platform main camera sensor via a CSI-2
> link
> and processes the image data with the embedded AI engine. The
> detected
> events are sent over I2C to ISH (Intel Sensor Hub) for additional
> data
> fusion from multiple sensors. The fusion results are used to
> implement
> advanced use cases like:
>  - Face detection to unlock screen
>  - Detect user presence to manage backlight setting or waking up
> system
> 
> Since the Image Processing Unit(IPU) used on the host processor needs
> to
> configure the CSI-2 link in normal camera usages, the CSI-2 link and
> camera sensor can only be used in mutually-exclusive ways by host IPU
> and
> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The
> IPU
> driver can take ownership of the CSI-2 link and camera sensor using
> interfaces provided by this IVSC driver.
> 
> Switching ownership requires an interface with two different hardware
> modules inside IVSC. The software interface to these modules is via
> Intel
> MEI (The Intel Management Engine) commands. These two hardware
> modules
> have two different MEI UUIDs to enumerate. These hardware modules
> are:
>  - ACE (Algorithm Context Engine): This module is for algorithm
> computing
> when IVSC owns camera sensor. Also ACE module controls camera
> sensor's
> ownership. This hardware module is used to set ownership of camera
> sensor.
>  - CSI (Camera Serial Interface): This module is used to route camera
> sensor data either to IVSC or to host for IPU driver and application.
> 
> IVSC also provides a privacy mode. When privacy mode is turned on,
> camera sensor can't be used. This means that both ACE and host IPU
> can't
> get image data. And when this mode is turned on, host IPU driver is
> informed via a registered callback, so that user can be notified.
> 
> In summary, to acquire ownership of camera by IPU driver, first ACE
> module needs to be informed of ownership and then to setup MIPI CSI-2
> link for the camera sensor and IPU.
> 
> Implementation:
> There are two different drivers to handle ACE and CSI hardware
> modules
> inside IVSC.
>  - mei_csi: MEI client driver to send commands and receive
> notifications
> from CSI module.
>  - mei_ace: MEI client driver to send commands and get status from
> ACE
> module.
> Interface is exposed via ivsc.h to acquire and release camera sensor
> and
> CSI-2 link.
> 
> Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
> ---------------------------------------------------------------------
> --------
> > Host
> > Processor                                                          
> >   |
> >                                                                    
> >        |
> >       -----------------       -----------------       -------------
> > --     |
> >       |               |       |               |       |            
> > | I2C |
> >       |      IPU      |       |      ISH      |       |camera
> > driver|--|  |
> >       |               |       |               |       |            
> > |  |  |
> >       -----------------       -----------------       -------------
> > --  |  |
> >               |                       |                     
> > |         |  |
> >               |                       |               -------------
> > --  |  |
> >               |                       |               |            
> > |  |  |
> >               |                       |               | IVSC driver
> > |  |  |
> >               |                       |               |            
> > |  |  |
> >               |                       |               -------------
> > --  |  |
> >               |                       |                     
> > |         |  |
> ----------------|-----------------------|----------------------|-----
> ----|---
>                 | CSI                   | I2C                 
> |SPI      |
>                 |                       |                     
> |         |
> ----------------|-----------------------|----------------     
> |         |
> > IVSC          |                                       |     
> > |         |
> >               |                                       |     
> > |         |
> >       -----------------       -----------------       |     
> > |         |
> >       |               |       |               |       |     
> > |         |
> >       |      CSI      |       |      ACE      |       |------
> > |         |
> >       |               |       |               |      
> > |                |
> >       -----------------       -----------------      
> > |                |
> >               |                       | I2C          
> > |                |
> ----------------|-----------------------|----------------
>                 |
>                 | CSI                  
> |                                |
>                 |                      
> |                                |
>             --------------------------------
>                              |
>             |                              |
> I2C                         |
>             |         camera sensor        |-------------------------
> ----|
>             |                              |
>             --------------------------------
> 
> Wentong Wu (3):
>   media: pci: intel: ivsc: Add CSI submodule
>   media: pci: intel: ivsc: Add ACE submodule
>   media: pci: intel: ivsc: Add acquire/release API for ivsc
> 
>  drivers/media/pci/Kconfig              |   1 +
>  drivers/media/pci/intel/Makefile       |   2 +
>  drivers/media/pci/intel/ivsc/Kconfig   |  12 +
>  drivers/media/pci/intel/ivsc/Makefile  |   7 +
>  drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++
>  drivers/media/pci/intel/ivsc/mei_ace.c | 472
> +++++++++++++++++++++++++
>  drivers/media/pci/intel/ivsc/mei_ace.h |  36 ++
>  drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
>  drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
>  include/linux/ivsc.h                   |  74 ++++
>  10 files changed, 1090 insertions(+)
>  create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
>  create mode 100644 drivers/media/pci/intel/ivsc/Makefile
>  create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
>  create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
>  create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
>  create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
>  create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
>  create mode 100644 include/linux/ivsc.h
>