Re: [PATCH v4 00/14] iommufd: Add vIOMMU infrastructure (Part-2: vDEVICE)

[Alexey Kardashevskiy <aik@amd.com>]

On 22/10/24 11:20, Nicolin Chen wrote:
> Following the previous vIOMMU series, this adds another vDEVICE structure,
> representing the association from an iommufd_device to an iommufd_viommu.
> This gives the whole architecture a new "v" layer:
>    _______________________________________________________________________
>   |                      iommufd (with vIOMMU/vDEVICE)                    |
>   |                        _____________      _____________               |
>   |                       |             |    |             |              |
>   |      |----------------|    vIOMMU   |<---|   vDEVICE   |<------|      |
>   |      |                |             |    |_____________|       |      |
>   |      |     ______     |             |     _____________     ___|____  |
>   |      |    |      |    |             |    |             |   |        | |
>   |      |    | IOAS |<---|(HWPT_PAGING)|<---| HWPT_NESTED |<--| DEVICE | |
>   |      |    |______|    |_____________|    |_____________|   |________| |
>   |______|________|______________|__________________|_______________|_____|
>          |        |              |                  |               |
>    ______v_____   |        ______v_____       ______v_____       ___v__
>   |   struct   |  |  PFN  |  (paging)  |     |  (nested)  |     |struct|
>   |iommu_device|  |------>|iommu_domain|<----|iommu_domain|<----|device|
>   |____________|   storage|____________|     |____________|     |______|
> 
> This vDEVICE object is used to collect and store all vIOMMU-related device
> information/attributes in a VM. As an initial series for vDEVICE, add only
> the virt_id to the vDEVICE, which is a vIOMMU specific device ID in a VM:
> e.g. vSID of ARM SMMUv3, vDeviceID of AMD IOMMU, and vID of Intel VT-d to
> a Context Table. This virt_id helps IOMMU drivers to link the vID to a pID
> of the device against the physical IOMMU instance. This is essential for a
> vIOMMU-based invalidation, where the request contains a device's vID for a
> device cache flush, e.g. ATC invalidation.
> 
> Therefore, with this vDEVICE object, support a vIOMMU-based invalidation,
> by reusing IOMMUFD_CMD_HWPT_INVALIDATE for a vIOMMU object to flush cache
> with a given driver data.
> 
> As for the implementation of the series, add driver support in ARM SMMUv3
> for a real world use case.
> 
> This series is on Github:
> https://github.com/nicolinc/iommufd/commits/iommufd_viommu_p2-v4
> 
> For testing, try this "with-rmr" branch:
> https://github.com/nicolinc/iommufd/commits/iommufd_viommu_p2-v4-with-rmr

Is there any real example of a .vdevice_alloc hook, besides the 
selftests? It is not in iommufd_viommu_p2-v4-with-rmr, hence the 
question. I am trying to sketch something with this new machinery and 
less guessing would be nice. Thanks,


> Paring QEMU branch for testing:
> https://github.com/nicolinc/qemu/commits/wip/for_iommufd_viommu_p2-v4
> 
> Changelog
> v4
>   * Added missing brackets in switch-case
>   * Fixed the unreleased idev refcount issue
>   * Reworked the iommufd_vdevice_alloc allocator
>   * Dropped support for IOMMU_VIOMMU_TYPE_DEFAULT
>   * Added missing TEST_LENGTH and fail_nth coverages
>   * Added a verification to the driver-allocated vDEVICE object
>   * Added an iommufd_vdevice_abort for a missing mutex protection
>   * Added a u64 structure arm_vsmmu_invalidation_cmd for user command
>     conversion
> v3
>   https://lore.kernel.org/all/cover.1728491532.git.nicolinc@nvidia.com/
>   * Added Jason's Reviewed-by
>   * Split this invalidation part out of the part-1 series
>   * Repurposed VDEV_ID ioctl to a wider vDEVICE structure and ioctl
>   * Reduced viommu_api functions by allowing drivers to access viommu
>     and vdevice structure directly
>   * Dropped vdevs_rwsem by using xa_lock instead
>   * Dropped arm_smmu_cache_invalidate_user
> v2
>   https://lore.kernel.org/all/cover.1724776335.git.nicolinc@nvidia.com/
>   * Limited vdev_id to one per idev
>   * Added a rw_sem to protect the vdev_id list
>   * Reworked driver-level APIs with proper lockings
>   * Added a new viommu_api file for IOMMUFD_DRIVER config
>   * Dropped useless iommu_dev point from the viommu structure
>   * Added missing index numnbers to new types in the uAPI header
>   * Dropped IOMMU_VIOMMU_INVALIDATE uAPI; Instead, reuse the HWPT one
>   * Reworked mock_viommu_cache_invalidate() using the new iommu helper
>   * Reordered details of set/unset_vdev_id handlers for proper lockings
> v1
>   https://lore.kernel.org/all/cover.1723061377.git.nicolinc@nvidia.com/
> 
> Thanks!
> Nicolin
> 
> Jason Gunthorpe (2):
>    iommu: Add iommu_copy_struct_from_full_user_array helper
>    iommu/arm-smmu-v3: Allow ATS for IOMMU_DOMAIN_NESTED
> 
> Nicolin Chen (12):
>    iommufd/viommu: Introduce IOMMUFD_OBJ_VDEVICE and its related struct
>    iommufd/viommu: Add IOMMU_VDEVICE_ALLOC ioctl
>    iommufd/selftest: Add IOMMU_VDEVICE_ALLOC test coverage
>    iommu/viommu: Add cache_invalidate to iommufd_viommu_ops
>    iommufd/hw_pagetable: Enforce cache invalidation op on vIOMMU-based
>      hwpt_nested
>    iommufd: Allow hwpt_id to carry viommu_id for IOMMU_HWPT_INVALIDATE
>    iommufd/viommu: Add vdev_to_dev helper
>    iommufd/selftest: Add mock_viommu_cache_invalidate
>    iommufd/selftest: Add IOMMU_TEST_OP_DEV_CHECK_CACHE test command
>    iommufd/selftest: Add vIOMMU coverage for IOMMU_HWPT_INVALIDATE ioctl
>    Documentation: userspace-api: iommufd: Update vDEVICE
>    iommu/arm-smmu-v3: Add arm_vsmmu_cache_invalidate
> 
>   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   9 +-
>   drivers/iommu/iommufd/iommufd_private.h       |  12 ++
>   drivers/iommu/iommufd/iommufd_test.h          |  30 +++
>   include/linux/iommu.h                         |  49 ++++-
>   include/linux/iommufd.h                       |  50 +++++
>   include/uapi/linux/iommufd.h                  |  61 +++++-
>   tools/testing/selftests/iommu/iommufd_utils.h |  83 +++++++
>   .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c     | 162 +++++++++++++-
>   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  32 ++-
>   drivers/iommu/iommufd/device.c                |  11 +
>   drivers/iommu/iommufd/driver.c                |   7 +
>   drivers/iommu/iommufd/hw_pagetable.c          |  36 +++-
>   drivers/iommu/iommufd/main.c                  |   7 +
>   drivers/iommu/iommufd/selftest.c              | 115 +++++++++-
>   drivers/iommu/iommufd/viommu.c                | 108 ++++++++++
>   tools/testing/selftests/iommu/iommufd.c       | 204 +++++++++++++++++-
>   .../selftests/iommu/iommufd_fail_nth.c        |   4 +
>   Documentation/userspace-api/iommufd.rst       |  41 +++-
>   18 files changed, 983 insertions(+), 38 deletions(-)
> 

-- 
Alexey