[PATCH v2 0/7] iommu/arm-smmu-v3: Support guest-level Realm VSMMU (Part-1)

[PATCH v2 0/7] iommu/arm-smmu-v3: Support guest-level Realm VSMMU (Part-1)

[Nicolin Chen]

This series prepares Realm guest IOMMU and DMA subsystems for devices that
are present but not yet inside the Realm trust boundary.

Inside a confidential guest, we expect that all the iommus are going to be
T=1 aware; there seems to be no interest in supporting IOMMUs that handle
T=0 only. This means they only work on devices that have been put into the
run state and issue T=1 transactions.

Today an iommu driver assumes it translates for every device behind it: it
probes the device, takes a translating default domain, and puts the DMA API
on its page tables. That assumption is wrong for T=0 devices, whose DMA the
hypervisor translates instead, so the mappings made by the guest are never
consulted and the device silently stops working.

To solve this problem, make TDISP T=0 the default for devices associated
with a confidential IOMMU. The tdisp_t1 flag records the future T=1 state.

While T=0, the device remains outside the Realm trust boundary. Its DMA is
translated by the host or hypervisor:

    [device: T=0] ===> [hypervisor] ===> [memory]
        |
        +-- (guest association) --> [confidential IOMMU: blocking DMA]

         Legend: ===> DMA path; --> guest topology and control

After reaching T=1, the device is inside the Realm trust boundary. Its DMA
may use the guest VSMMU after RMM binds the device to its virtual stream:

    [device] ===> [VSMMU: S1] ===> [RMM/PSMMU: S2 + DPT] ===> [memory]

As part 1 of Realm VSMMU support, implement enough core infrastructure to
manage a confidential vIOMMU:
 - Detect and mark it as confidential in the core iommu structures
 - Support devices in T=0 mode by keeping the DMA API in direct mode. The
   iommu is up and running, but parks such a device in its blocking domain
   and translates for nothing, as the core takes over what the device still
   needs, which today is ATS
 - Keep it parked: refuse a translating domain by every route in, whether
   through the default domain, sysfs, group attach or replace, or PASID

Part 1 only adds the helpers required for T=0 setup and teardown. It does
not add iommu_tdisp_enter_t1() until part 2 has a transition to perform.

Then implement SMMUv3 support:
 - Let a CPU-integrated IOMMU declare that its DMA can reach private
   memory, so coherent allocations and streaming mappings avoid the SWIOTLB
   shared pool
 - Identify a T=1 VSMMU using trusted RSI because firmware does not
   distinguish it from a normal T=0 SMMU
 - Validate the firmware MMIO range against RSI before activating the VSMMU
   and allocating its queues and tables from private memory

The series is on github:
https://github.com/nicolinc/iommufd/commits/smmuv3_realm_guest_p1-v2

Changelog
v2:
 * Rebase on v7.4-rc1
 * Drop Assisted-by tags
 * Drop Aneesh's firmware patch that was merged
 * Skip Tegra CMDQV implementation in the realm world
 * Use arm_smccc_1_2_smc() to zero inputs as RMM spec demands
 * Add missing iommu_using_t0_stream() check in iommu_dma_prepare_msi()
v1:
 https://lore.kernel.org/all/cover.1789010941.git.nicolinc@nvidia.com/

Future work
===========

Part 2 will add the T=1 transition after validation through:

  - TSM, whose guest-side interface remains under discussion
  - RMM, using the Arm-specific RSI VDEV command

For a multi-device IOMMU group, the first device accepted into T=1 claims
the group. Part 2 will reject T=1 for every other member.

The current RSI binding validation returns one VSID, so part 2 must reject
a firmware specification containing multiple SIDs upon the T=1 transition.

Nicolin Chen (7):
  firmware: arm_rmm: Add VSMMU commands and fields
  dma-mapping: Let a device declare that it reaches private memory
  dma-mapping: Keep DMA memory private for capable devices
  iommu: Let a driver mark an IOMMU as confidential
  iommu: Introduce TDISP T=0 state for confidential IOMMUs
  iommu: Park TDISP T=0 devices in the blocking domain
  iommu/arm-smmu-v3: Probe a guest-level Realm VSMMU via RSI commands

 drivers/iommu/Makefile                      |  3 +
 drivers/iommu/iommu-priv.h                  | 19 +++++
 include/linux/arm-rsi-cmds.h                | 29 +++++++
 include/linux/arm-smccc-rsi.h               | 20 +++++
 include/linux/device.h                      |  4 +
 include/linux/dma-direct.h                  |  7 +-
 include/linux/dma-mapping.h                 |  4 +
 include/linux/iommu.h                       | 14 ++++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 64 ++++++++++++++-
 drivers/iommu/iommu-cc.c                    | 57 +++++++++++++
 drivers/iommu/iommu.c                       | 91 +++++++++++++++++++--
 kernel/dma/direct.c                         | 16 ++--
 kernel/dma/mapping.c                        | 23 +++++-
 kernel/dma/swiotlb.c                        |  2 +-
 14 files changed, 335 insertions(+), 18 deletions(-)
 create mode 100644 drivers/iommu/iommu-cc.c


base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.43.0

[PATCH v2 1/7] firmware: arm_rmm: Add VSMMU commands and fields

[Nicolin Chen]

DEN0137 2.0-bet3 adds the RSI_VSMMU_GET_INFO command and replaces the old
RSI_VSMMU_ACTIVATE command with RSI_ARCH_DEV_ACTIVATE.

Add the command IDs, return codes, SMMUv3 device type, and helpers for both
commands.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 include/linux/arm-rsi-cmds.h  | 29 +++++++++++++++++++++++++++++
 include/linux/arm-smccc-rsi.h | 20 ++++++++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/include/linux/arm-rsi-cmds.h b/include/linux/arm-rsi-cmds.h
index 3f7a6a833993b..a1836b19c1556 100644
--- a/include/linux/arm-rsi-cmds.h
+++ b/include/linux/arm-rsi-cmds.h
@@ -236,4 +236,33 @@ static inline unsigned long rsi_attestation_token_continue(phys_addr_t granule,
 	return res.a0;
 }
 
+static inline unsigned long rsi_vsmmu_get_info(phys_addr_t addr,
+					       phys_addr_t *top)
+{
+	struct arm_smccc_1_2_regs regs = {
+		.a0 = SMC_RSI_VSMMU_GET_INFO,
+		.a1 = addr,
+	};
+
+	arm_smccc_1_2_smc(&regs, &regs);
+	if (regs.a0 == RSI_SUCCESS && top)
+		*top = regs.a1;
+
+	return regs.a0;
+}
+
+static inline unsigned long rsi_arch_dev_activate(phys_addr_t base,
+						  unsigned long dev_type)
+{
+	struct arm_smccc_1_2_regs regs = {
+		.a0 = SMC_RSI_ARCH_DEV_ACTIVATE,
+		.a1 = base,
+		.a2 = dev_type,
+	};
+
+	arm_smccc_1_2_smc(&regs, &regs);
+
+	return regs.a0;
+}
+
 #endif /* __LINUX_ARM_RSI_CMDS_H_ */
diff --git a/include/linux/arm-smccc-rsi.h b/include/linux/arm-smccc-rsi.h
index cca881ddfdbab..fff0073622153 100644
--- a/include/linux/arm-smccc-rsi.h
+++ b/include/linux/arm-smccc-rsi.h
@@ -37,6 +37,7 @@
 #define RSI_ERROR_STATE		UL(2)
 #define RSI_INCOMPLETE		UL(3)
 #define RSI_ERROR_UNKNOWN	UL(4)
+#define RSI_ERROR_DEVICE	UL(5)
 
 #define SMC_RSI_FID(n)		ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL,      \
 						   ARM_SMCCC_SMC_64,         \
@@ -190,4 +191,23 @@ struct realm_config {
  */
 #define SMC_RSI_HOST_CALL			SMC_RSI_FID(0x199)
 
+/*
+ * Get information about a VSMMU.
+ *
+ * arg1 == Base IPA of the VSMMU
+ * ret0 == Status / error
+ * ret1 == Top IPA of the VSMMU
+ */
+#define SMC_RSI_VSMMU_GET_INFO			SMC_RSI_FID(0x19A)
+
+#define RSI_ARCH_DEV_SMMUV3			UL(0)
+
+/*
+ * Activate an architectural device.
+ *
+ * arg1 == Base IPA of the architectural device
+ * arg2 == Architectural device type
+ * ret0 == Status / error
+ */
+#define SMC_RSI_ARCH_DEV_ACTIVATE		SMC_RSI_FID(0x19B)
 #endif /* __LINUX_ARM_SMCCC_RSI_H_ */
-- 
2.43.0

[PATCH v2 2/7] dma-mapping: Let a device declare that it reaches private memory

[Nicolin Chen]

DMA devices in a confidential guest can access only shared memory, or both
private and shared memory. Some method outside the DMA API will confirm the
device is able to access private memory through DMA.

Add DEV_FLAG_DMA_CC_PRIVATE and dma_set_cc_private(dev, bool).

The flag must be stable while any DMA mapping exists.

Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 include/linux/device.h      |  4 ++++
 include/linux/dma-mapping.h |  4 ++++
 kernel/dma/mapping.c        | 21 +++++++++++++++++++++
 3 files changed, 29 insertions(+)

diff --git a/include/linux/device.h b/include/linux/device.h
index 429e39f647cf9..01d7877d0b571 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -604,6 +604,8 @@ struct device_physical_location {
  *		ancestor device.
  * @DEV_FLAG_OFFLINE_DISABLED: If set, the device is permanently online.
  * @DEV_FLAG_OFFLINE: Set after successful invocation of bus type's .offline().
+ * @DEV_FLAG_DMA_CC_PRIVATE: The device is able to access private (encrypted)
+ *		memory, no shared memory bouncing is required from the DMA API.
  * @DEV_FLAG_COUNT: Number of defined struct_device_flags.
  */
 enum struct_device_flags {
@@ -617,6 +619,7 @@ enum struct_device_flags {
 	DEV_FLAG_OF_NODE_REUSED = 7,
 	DEV_FLAG_OFFLINE_DISABLED = 8,
 	DEV_FLAG_OFFLINE = 9,
+	DEV_FLAG_DMA_CC_PRIVATE = 10,
 
 	DEV_FLAG_COUNT
 };
@@ -834,6 +837,7 @@ __create_dev_flag_accessors(dma_coherent, DEV_FLAG_DMA_COHERENT);
 __create_dev_flag_accessors(of_node_reused, DEV_FLAG_OF_NODE_REUSED);
 __create_dev_flag_accessors(offline_disabled, DEV_FLAG_OFFLINE_DISABLED);
 __create_dev_flag_accessors(offline, DEV_FLAG_OFFLINE);
+__create_dev_flag_accessors(dma_cc_private, DEV_FLAG_DMA_CC_PRIVATE);
 
 #undef __create_dev_flag_accessors
 
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index efde0e5a58dc2..3cc38db4a7490 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -218,6 +218,7 @@ void *dma_vmap_noncontiguous(struct device *dev, size_t size,
 void dma_vunmap_noncontiguous(struct device *dev, void *vaddr);
 int dma_mmap_noncontiguous(struct device *dev, struct vm_area_struct *vma,
 		size_t size, struct sg_table *sgt);
+void dma_set_cc_private(struct device *dev, bool private);
 #else /* CONFIG_HAS_DMA */
 static inline dma_addr_t dma_map_page_attrs(struct device *dev,
 		struct page *page, size_t offset, size_t size,
@@ -357,6 +358,9 @@ static inline int dma_mmap_noncontiguous(struct device *dev,
 {
 	return -EINVAL;
 }
+static inline void dma_set_cc_private(struct device *dev, bool private)
+{
+}
 #endif /* CONFIG_HAS_DMA */
 
 #ifdef CONFIG_IOMMU_DMA
diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c
index 82e701648d604..ded585b39fb92 100644
--- a/kernel/dma/mapping.c
+++ b/kernel/dma/mapping.c
@@ -501,6 +501,27 @@ static void dma_setup_need_sync(struct device *dev)
 static inline void dma_setup_need_sync(struct device *dev) { }
 #endif /* !CONFIG_DMA_NEED_SYNC */
 
+/**
+ * dma_set_cc_private - Set whether @dev can DMA to private memory
+ * @dev: device whose confidential-computing DMA access is being updated
+ * @private: whether the device can DMA to private memory
+ *
+ * Called when a device is known to be able to DMA to private memory.
+ * For architected CPU integrated devices their kernel drivers will
+ * self-accept using a driver specific validation. Other devices will
+ * have a userspace managed process.
+ *
+ * DMA mapping must not be active when this flag is changed. For CPU
+ * integrated devices the driver should self accept early during probe
+ * after validation. Other devices have this flag set automatically
+ * before probing.
+ */
+void dma_set_cc_private(struct device *dev, bool private)
+{
+	dev_assign_dma_cc_private(dev, private);
+}
+EXPORT_SYMBOL_GPL(dma_set_cc_private);
+
 /*
  * The whole dma_get_sgtable() idea is fundamentally unsafe - it seems
  * that the intention is to allow exporting memory allocated via the
-- 
2.43.0

[PATCH v2 3/7] dma-mapping: Keep DMA memory private for capable devices

[Nicolin Chen]

Architecture code can require devices to use unencrypted memory. This makes
the DMA API use shared memory for coherent allocations and bounce streaming
mappings through SWIOTLB.

A device marked with DEV_FLAG_DMA_CC_PRIVATE can reach private memory
directly. Add dma_require_decrypted() to combine force_dma_unencrypted()
with the new device flag, and use it throughout the generic DMA code.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 include/linux/dma-direct.h |  7 ++++++-
 kernel/dma/direct.c        | 16 ++++++++--------
 kernel/dma/mapping.c       |  2 +-
 kernel/dma/swiotlb.c       |  2 +-
 4 files changed, 16 insertions(+), 11 deletions(-)

diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h
index daa31a1adf7b0..fb35ca27941cc 100644
--- a/include/linux/dma-direct.h
+++ b/include/linux/dma-direct.h
@@ -134,6 +134,11 @@ static inline bool force_dma_unencrypted(struct device *dev)
 }
 #endif /* CONFIG_ARCH_HAS_FORCE_DMA_UNENCRYPTED */
 
+static inline bool dma_require_decrypted(struct device *dev)
+{
+	return !dev_dma_cc_private(dev) && force_dma_unencrypted(dev);
+}
+
 static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size,
 		bool is_ram, unsigned long attrs)
 {
@@ -146,7 +151,7 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size,
 	 * requires unencrypted DMA addresses. Treat it as not DMA-capable
 	 * so the caller can fall back to a suitable SWIOTLB pool.
 	 */
-	if (!(attrs & DMA_ATTR_CC_SHARED) && force_dma_unencrypted(dev))
+	if (!(attrs & DMA_ATTR_CC_SHARED) && dma_require_decrypted(dev))
 		return false;
 
 	if (is_ram && !IS_ENABLED(CONFIG_ARCH_DMA_ADDR_T_64BIT) &&
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index 25e9a86835fd3..b3b9ff090c590 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -41,7 +41,7 @@ static inline struct page *dma_direct_to_page(struct device *dev,
 
 u64 dma_direct_get_required_mask(struct device *dev)
 {
-	bool require_decrypted = force_dma_unencrypted(dev);
+	bool require_decrypted = dma_require_decrypted(dev);
 	phys_addr_t phys = ((phys_addr_t)max_pfn << PAGE_SHIFT) - 1;
 	u64 max_dma = phys_to_dma_direct(dev, phys, require_decrypted);
 
@@ -72,7 +72,7 @@ static gfp_t dma_direct_optimal_gfp_mask(struct device *dev, u64 *phys_limit)
 
 bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)
 {
-	bool require_decrypted = force_dma_unencrypted(dev);
+	bool require_decrypted = dma_require_decrypted(dev);
 	dma_addr_t dma_addr = phys_to_dma_direct(dev, phys, require_decrypted);
 
 	if (dma_addr == DMA_MAPPING_ERROR)
@@ -226,7 +226,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,
 	struct page *page;
 	void *cpu_addr;
 
-	if (force_dma_unencrypted(dev))
+	if (dma_require_decrypted(dev))
 		attrs |= __DMA_ATTR_ALLOC_CC_SHARED;
 
 	if (attrs & __DMA_ATTR_ALLOC_CC_SHARED) {
@@ -385,7 +385,7 @@ void dma_direct_free(struct device *dev, size_t size,
 	 * If the allocation used decrypted/shared backing pages, restore
 	 * the encryption state on free.
 	 */
-	if (force_dma_unencrypted(dev))
+	if (dma_require_decrypted(dev))
 		attrs |= __DMA_ATTR_ALLOC_CC_SHARED;
 
 	if (attrs & __DMA_ATTR_ALLOC_CC_SHARED)
@@ -452,7 +452,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size,
 	struct page *page;
 	void *cpu_addr;
 
-	if (force_dma_unencrypted(dev))
+	if (dma_require_decrypted(dev))
 		attrs |= __DMA_ATTR_ALLOC_CC_SHARED;
 
 	if ((attrs & __DMA_ATTR_ALLOC_CC_SHARED) && dma_direct_use_pool(dev, gfp))
@@ -496,7 +496,7 @@ void dma_direct_free_pages(struct device *dev, size_t size,
 	 * if the device had requested for an unencrypted buffer,
 	 * convert it to encrypted on free
 	 */
-	bool mark_mem_encrypted = force_dma_unencrypted(dev);
+	bool mark_mem_encrypted = dma_require_decrypted(dev);
 
 	/* If page is not from an atomic pool, dma_free_from_pool_page() fails */
 	if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) &&
@@ -665,7 +665,7 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma,
 	const pgoff_t pgoff_end = vma_end_pgoff(vma);
 	int ret = -ENXIO;
 
-	if (force_dma_unencrypted(dev))
+	if (dma_require_decrypted(dev))
 		attrs |= DMA_ATTR_CC_SHARED;
 
 	vma->vm_page_prot = dma_pgprot(dev, vma->vm_page_prot, attrs);
@@ -814,7 +814,7 @@ size_t dma_direct_max_mapping_size(struct device *dev)
 	/* If SWIOTLB is active, use its maximum mapping size */
 	if (is_swiotlb_active(dev) &&
 	    (dma_addressing_limited(dev) || is_swiotlb_force_bounce(dev) ||
-	     force_dma_unencrypted(dev)))
+	     dma_require_decrypted(dev)))
 		return swiotlb_max_mapping_size(dev);
 
 	return SIZE_MAX;
diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c
index ded585b39fb92..0252473edd0b5 100644
--- a/kernel/dma/mapping.c
+++ b/kernel/dma/mapping.c
@@ -673,7 +673,7 @@ void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle,
 		return NULL;
 	}
 
-	if (force_dma_unencrypted(dev))
+	if (dma_require_decrypted(dev))
 		attrs |= __DMA_ATTR_ALLOC_CC_SHARED;
 
 	if (dma_alloc_from_dev_coherent(dev, size, dma_handle, &cpu_addr)) {
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index 2c57b49ac7c06..ddfc4533934a2 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -1550,7 +1550,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr,
 	if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
 
 		/* swiotlb pool is incorrect for this device */
-		if (unlikely(mem->cc_shared != force_dma_unencrypted(dev)))
+		if (unlikely(mem->cc_shared != dma_require_decrypted(dev)))
 			return (phys_addr_t)DMA_MAPPING_ERROR;
 
 	} else if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) {
-- 
2.43.0

[PATCH v2 4/7] iommu: Let a driver mark an IOMMU as confidential

[Nicolin Chen]

An IOMMU inside a confidential VM needs private memory for its queues and
tables. Later changes also need to identify it when handling TDISP devices.

Add a confidential marker and a helper for IOMMU drivers. The helper also
marks the IOMMU device as able to access private memory. Drivers must call
it before iommu_device_register().

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 include/linux/iommu.h | 10 ++++++++++
 drivers/iommu/iommu.c | 20 ++++++++++++++++++++
 2 files changed, 30 insertions(+)

diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index ef5043752a365..e8fb5ee59883e 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -824,6 +824,8 @@ struct iommu_domain_ops {
  * @singleton_group: Used internally for drivers that have only one group
  * @max_pasids: number of supported PASIDs
  * @ready: set once iommu_device_register() has completed successfully
+ * @confidential: this instance runs inside a confidential VM. Set by the driver
+ *                before any dma allocation.
  */
 struct iommu_device {
 	struct list_head list;
@@ -833,6 +835,7 @@ struct iommu_device {
 	struct iommu_group *singleton_group;
 	u32 max_pasids;
 	bool ready;
+	bool confidential;
 };
 
 /**
@@ -892,6 +895,8 @@ struct dev_iommu {
 int iommu_device_register(struct iommu_device *iommu,
 			  const struct iommu_ops *ops,
 			  struct device *hwdev);
+void iommu_device_set_confidential(struct iommu_device *iommu,
+				   struct device *hwdev);
 void iommu_device_unregister(struct iommu_device *iommu);
 int  iommu_device_sysfs_add(struct iommu_device *iommu,
 			    struct device *parent,
@@ -1423,6 +1428,11 @@ static inline int iommu_device_register(struct iommu_device *iommu,
 	return -ENODEV;
 }
 
+static inline void iommu_device_set_confidential(struct iommu_device *iommu,
+						 struct device *hwdev)
+{
+}
+
 static inline struct iommu_device *dev_to_iommu_device(struct device *dev)
 {
 	return NULL;
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index b486b8bbd1fcd..ba98a0bb800a5 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -17,6 +17,7 @@
 #include <linux/slab.h>
 #include <linux/errno.h>
 #include <linux/host1x_context_bus.h>
+#include <linux/dma-mapping.h>
 #include <linux/iommu.h>
 #include <linux/iommufd.h>
 #include <linux/idr.h>
@@ -313,6 +314,25 @@ int iommu_device_register(struct iommu_device *iommu,
 }
 EXPORT_SYMBOL_GPL(iommu_device_register);
 
+/**
+ * iommu_device_set_confidential - Mark an IOMMU instance as confidential
+ * @iommu: the IOMMU instance
+ * @hwdev: the struct device of @iommu, whose own DMA reaches private memory
+ *
+ * Declare that @iommu runs inside a confidential VM
+ *
+ * Call this before @iommu makes any DMA allocation of its own, and not merely
+ * before iommu_device_register(). Queues and tables allocated any earlier land
+ * in shared memory that the hypervisor can read.
+ */
+void iommu_device_set_confidential(struct iommu_device *iommu,
+				   struct device *hwdev)
+{
+	iommu->confidential = true;
+	dma_set_cc_private(hwdev, true);
+}
+EXPORT_SYMBOL_GPL(iommu_device_set_confidential);
+
 void iommu_device_unregister(struct iommu_device *iommu)
 {
 	for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++)
-- 
2.43.0

[PATCH v2 5/7] iommu: Introduce TDISP T=0 state for confidential IOMMUs

[Nicolin Chen]

TDISP devices have two independent DMA streams, T=1 and T=0. In several
platforms these streams terminate into different IOMMU environments with
different translation tables. For such platforms the IOMMU driver in Linux
handles exactly one of the streams. Typically we expect the T=1 stream to
be linked to the vIOMMU and the T=0 stream to simply be left as a no-iommu
identity configuration.

As the HW has these two different DMA translation environments, the iommu
subsystem must participate in switching between them. When the device uses
T=0, its T=1 vIOMMU path should be left in BLOCKING to disable it, and a
simplified "T=0 IDENTITY" mode is setup within the DMA API.

    [device: T=0] ===> [hypervisor] ===> [memory]
          |
          +-- (guest association) --> [confidential IOMMU: blocking DMA]

Setting the vIOMMU to BLOCKING also gives up its control of ATS, which is
now managed by the core code as part of simplified T=0 mode.

Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/Makefile     |  3 ++
 drivers/iommu/iommu-priv.h | 19 +++++++++++++
 include/linux/iommu.h      |  4 +++
 drivers/iommu/iommu-cc.c   | 57 ++++++++++++++++++++++++++++++++++++++
 drivers/iommu/iommu.c      | 17 ++++++++++--
 5 files changed, 98 insertions(+), 2 deletions(-)
 create mode 100644 drivers/iommu/iommu-cc.c

diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 29a26d2a3af12..0fc48b5d3c406 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -6,6 +6,9 @@ obj-$(CONFIG_RISCV_IOMMU) += riscv/
 obj-$(CONFIG_GENERIC_PT) += generic_pt/fmt/
 obj-$(CONFIG_HYPERV) += hyperv/
 obj-$(CONFIG_IOMMU_API) += iommu.o
+ifdef CONFIG_ARCH_HAS_CC_PLATFORM
+obj-$(CONFIG_IOMMU_API) += iommu-cc.o
+endif
 obj-$(CONFIG_IOMMU_SUPPORT) += iommu-pages.o
 obj-$(CONFIG_IOMMU_API) += iommu-traces.o
 obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
diff --git a/drivers/iommu/iommu-priv.h b/drivers/iommu/iommu-priv.h
index 1f9f886b2c107..624466384b6e3 100644
--- a/drivers/iommu/iommu-priv.h
+++ b/drivers/iommu/iommu-priv.h
@@ -8,6 +8,25 @@
 #include <linux/iommu-debug-pagealloc.h>
 #include <linux/msi.h>
 
+#ifdef CONFIG_ARCH_HAS_CC_PLATFORM
+void iommu_tdisp_enter_t0(struct device *dev);
+void iommu_tdisp_exit_t0(struct device *dev);
+#else
+static inline void iommu_tdisp_enter_t0(struct device *dev)
+{
+}
+
+static inline void iommu_tdisp_exit_t0(struct device *dev)
+{
+}
+#endif
+
+static inline bool iommu_using_t0_stream(struct device *dev)
+{
+	return dev->iommu->iommu_dev->confidential &&
+	       !dev->iommu->tdisp_t1;
+}
+
 static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
 {
 	/*
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index e8fb5ee59883e..1b5817a7d0be9 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -873,6 +873,9 @@ struct iommu_fault_param {
  * @max_pasids:  number of PASIDs this device can consume
  * @attach_deferred: the dma domain attachment is deferred
  * @pci_32bit_workaround: Limit DMA allocations to 32-bit IOVAs
+ * @tdisp_t1: device uses its confidential IOMMU's T=1 stream. A zero value
+ *            means T=0, where the T=1 stream remains blocked while the core
+ *            owns required DMA operations such as ATS.
  * @require_direct: device requires IOMMU_RESV_DIRECT regions
  * @shadow_on_flush: IOTLB flushes are used to sync shadow tables
  *
@@ -888,6 +891,7 @@ struct dev_iommu {
 	u32				max_pasids;
 	u32				attach_deferred:1;
 	u32				pci_32bit_workaround:1;
+	u32				tdisp_t1:1;
 	u32				require_direct:1;
 	u32				shadow_on_flush:1;
 };
diff --git a/drivers/iommu/iommu-cc.c b/drivers/iommu/iommu-cc.c
new file mode 100644
index 0000000000000..d107b39696a08
--- /dev/null
+++ b/drivers/iommu/iommu-cc.c
@@ -0,0 +1,57 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * TDISP devices can have separate T=0 and T=1 DMA streams that terminate in
+ * different IOMMU environments. The Linux IOMMU driver manages the T=1 stream
+ * while the T=0 stream uses the physical path.
+ *
+ *   [device: T=0] ===> [hypervisor] ===> [memory]
+ *       |
+ *       +-- (guest association) --> [confidential IOMMU: blocking DMA]
+ */
+#include <linux/pci.h>
+#include <linux/pci-ats.h>
+
+#include "iommu-priv.h"
+
+/**
+ * iommu_tdisp_enter_t0 - Prepare @dev while TDISP is T=0
+ * @dev: device entering TDISP T=0
+ *
+ * The IOMMU core invokes this helper for a device on a confidential IOMMU
+ * while TDISP is T=0. The device remains associated with its IOMMU while the
+ * core owns DMA operations required in this state. Enable ATS here when the
+ * device requires it, since the IOMMU driver does not operate ATS while in
+ * BLOCKED.
+ */
+void iommu_tdisp_enter_t0(struct device *dev)
+{
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return;
+
+	pdev = to_pci_dev(dev);
+	if (!pci_ats_required(pdev))
+		return;
+
+	ret = pci_enable_ats(pdev, PCI_ATS_MIN_STU);
+	if (ret)
+		dev_warn(dev, "cannot enable ATS while TDISP is T=0\n");
+}
+
+/**
+ * iommu_tdisp_exit_t0 - Undo iommu_tdisp_enter_t0()
+ * @dev: device leaving TDISP T=0
+ */
+void iommu_tdisp_exit_t0(struct device *dev)
+{
+	struct pci_dev *pdev;
+
+	if (!dev_is_pci(dev))
+		return;
+
+	pdev = to_pci_dev(dev);
+	if (pci_ats_required(pdev) && pdev->ats_enabled)
+		pci_disable_ats(pdev);
+}
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index ba98a0bb800a5..38d0158045f57 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -535,9 +535,17 @@ static int iommu_init_device(struct device *dev)
 	}
 	dev->iommu->iommu_dev = iommu_dev;
 
+	if (iommu_using_t0_stream(dev)) {
+		/*
+		 * After probe the iommu has to leave the T=1 stream in BLOCKING,
+		 * while the T=0 stream will go through the physical path.
+		 */
+		iommu_tdisp_enter_t0(dev);
+	}
+
 	ret = iommu_device_link(iommu_dev, dev);
 	if (ret)
-		goto err_release;
+		goto err_tdisp_t0;
 
 	group = ops->device_group(dev);
 	if (WARN_ON_ONCE(group == NULL))
@@ -555,7 +563,9 @@ static int iommu_init_device(struct device *dev)
 
 err_unlink:
 	iommu_device_unlink(iommu_dev, dev);
-err_release:
+err_tdisp_t0:
+	if (iommu_using_t0_stream(dev))
+		iommu_tdisp_exit_t0(dev);
 	if (ops->release_device)
 		ops->release_device(dev);
 err_module_put:
@@ -573,6 +583,9 @@ static void iommu_deinit_device(struct device *dev)
 
 	lockdep_assert_held(&group->mutex);
 
+	if (iommu_using_t0_stream(dev))
+		iommu_tdisp_exit_t0(dev);
+
 	iommu_device_unlink(dev->iommu->iommu_dev, dev);
 
 	/*
-- 
2.43.0

[PATCH v2 6/7] iommu: Park TDISP T=0 devices in the blocking domain

[Nicolin Chen]

A TDISP T=0 device stays associated with its guest T=1 vIOMMU, yet doesn't
use that IOMMU for DMA translation. Instead, park its guest-visible stream
in the driver's blocking domain.

Linux's model requires a device to either use T=0 or T=1. Aside from MSI,
it cannot use T=0 for normal DMA once moved into T=1 mode, since the device
will have IOVA's that are not valid in T=0.

Select the blocking domain before normal default-domain policy is applied.

Direct-map reserved regions are still fine, because the T=0 stream always
goes to an identity mapping. Skip them here, as recording the request would
only refuse the device its blocking domain.

Reject sysfs domain changes, driver-managed group attachment, and any PASID
attachments that would install a translating domain. The state is uniform
across each group because all devices on a confidential IOMMU start in T=0.

Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/iommu.c | 54 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 51 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 38d0158045f57..6414c4d8828d3 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -1238,6 +1238,14 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain,
 	unsigned long pg_size;
 	int ret = 0;
 
+	/*
+	 * The direct-map reserved regions are still fine because the T=0 stream
+	 * always goes to an identity mapping. Skip them: recording the request
+	 * would only refuse the device its blocking domain.
+	 */
+	if (iommu_using_t0_stream(dev))
+		return 0;
+
 	pg_size = domain->pgsize_bitmap ? 1UL << __ffs(domain->pgsize_bitmap) : 0;
 
 	if (WARN_ON_ONCE(iommu_is_dma_domain(domain) && !pg_size))
@@ -1790,11 +1798,22 @@ __iommu_group_alloc_default_domain(struct iommu_group *group, int req_type)
 static struct iommu_domain *
 iommu_group_alloc_default_domain(struct iommu_group *group, int req_type)
 {
-	const struct iommu_ops *ops = dev_iommu_ops(iommu_group_first_dev(group));
+	struct device *first = iommu_group_first_dev(group);
+	const struct iommu_ops *ops = dev_iommu_ops(first);
 	struct iommu_domain *dom;
 
 	lockdep_assert_held(&group->mutex);
 
+	/*
+	 * When in T=0 mode the T=1 vIOMMU must always be set to BLOCKED to
+	 * release control of ATS.
+	 */
+	if (iommu_using_t0_stream(first)) {
+		if (!ops->blocked_domain)
+			return ERR_PTR(-EINVAL);
+		return ops->blocked_domain;
+	}
+
 	/*
 	 * Allow legacy drivers to specify the domain that will be the default
 	 * domain. This should always be either an IDENTITY/BLOCKED/PLATFORM
@@ -2399,6 +2418,10 @@ static int __iommu_attach_group(struct iommu_domain *domain,
 	    !domain_iommu_ops_compatible(dev_iommu_ops(dev), domain))
 		return -EINVAL;
 
+	/* A domain cannot be attached while using the T=0 stream. */
+	if (iommu_using_t0_stream(dev))
+		return -EINVAL;
+
 	return __iommu_group_set_domain(group, domain);
 }
 
@@ -3325,6 +3348,12 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,
 		goto out_unlock;
 	}
 
+	/* Default domain changes are not allowed while using the T=0 stream. */
+	if (iommu_using_t0_stream(iommu_group_first_dev(group))) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
 	ret = iommu_setup_default_domain(group, req_type);
 	if (ret)
 		goto out_unlock;
@@ -3690,6 +3719,12 @@ int iommu_attach_device_pasid(struct iommu_domain *domain,
 		goto out_unlock;
 	}
 
+	/* PASID cannot be used on the T=0 stream. */
+	if (iommu_using_t0_stream(dev)) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
 	for_each_group_device(group, device) {
 		/*
 		 * Skip PASID validation for devices without PASID support
@@ -3783,6 +3818,12 @@ int iommu_replace_device_pasid(struct iommu_domain *domain,
 		goto out_unlock;
 	}
 
+	/* PASID cannot be used on the T=0 stream. */
+	if (iommu_using_t0_stream(dev)) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
 	entry = iommu_make_pasid_array_entry(domain, handle);
 	curr = xa_cmpxchg(&group->pasid_array, pasid, NULL,
 			  XA_ZERO_ENTRY, GFP_KERNEL);
@@ -4017,6 +4058,12 @@ int iommu_replace_group_handle(struct iommu_group *group,
 		return -EINVAL;
 
 	mutex_lock(&group->mutex);
+	/* Like attach, replacement is not allowed when using the T=0 stream. */
+	if (iommu_using_t0_stream(iommu_group_first_dev(group))) {
+		ret = -EINVAL;
+		goto err_unlock;
+	}
+
 	entry = iommu_make_pasid_array_entry(new_domain, handle);
 	ret = xa_reserve(&group->pasid_array, IOMMU_NO_PASID, GFP_KERNEL);
 	if (ret)
@@ -4281,8 +4328,9 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr)
 		return ret;
 
 	mutex_lock(&group->mutex);
-	/* An IDENTITY domain must pass through */
-	if (group->domain && group->domain->type != IOMMU_DOMAIN_IDENTITY) {
+	/* An IDENTITY domain or a T=0 device must pass through */
+	if (group->domain && group->domain->type != IOMMU_DOMAIN_IDENTITY &&
+	    !iommu_using_t0_stream(dev)) {
 		switch (group->domain->cookie_type) {
 		case IOMMU_COOKIE_DMA_MSI:
 		case IOMMU_COOKIE_DMA_IOVA:
-- 
2.43.0

[PATCH v2 7/7] iommu/arm-smmu-v3: Probe a guest-level Realm VSMMU via RSI commands

[Nicolin Chen]

In CCA, the ACPI/etc doesn't tell the OS whether the SMMU is a confidential
T=1 instance or a normal T=0 one. Instead, it has to be learned by issuing
a trusted RSI. Before the kernel can operate the trusted T=1 IOMMU, it also
has to validate all the information that it gets from ACPI against the true
information that it gets from the RSI call, to ensure the ACPI is correct
and prevent substitution attacks.

Use RSI_VSMMU_GET_INFO during probe and require the returned register range
to match firmware. Then use RSI_ARCH_DEV_ACTIVATE before mapping registers.

Currently the driver doesn't support a T=0 SMMU inside a realm. For example
it doesn't make the page table allocations into shared memory. If we are in
a realm reject any SMMU that is not T=1.

Once the SMMU is activated, its DMA will follow the DEV_FLAG_DMA_CC_PRIVATE
rules. It must set that flag to ensure its coherent allocations for its own
structures are allocated from private memory, not the SWIOTLB shared pool.

Also skip Tegra241 CMDQV implementation in the realm world, since realm has
the support only for a standard SMMU for now.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 64 ++++++++++++++++++++-
 1 file changed, 62 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 6d4520c473086..f3a9436db15e4 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -11,9 +11,11 @@
 
 #include <linux/acpi.h>
 #include <linux/acpi_iort.h>
+#include <linux/arm-rsi-cmds.h>
 #include <linux/bitops.h>
 #include <linux/crash_dump.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/err.h>
 #include <linux/interrupt.h>
 #include <linux/io-pgtable.h>
@@ -5611,8 +5613,11 @@ static int acpi_smmu_iort_probe_model(struct acpi_iort_node *node,
 		/*
 		 * Tegra241 implementation stores its SMMU options and impl_dev
 		 * in DSDT. Thus, go through the ACPI tables unconditionally.
+		 *
+		 * Realm doesn't support Tegra CMDQV yet.
 		 */
-		acpi_smmu_dsdt_probe_tegra241_cmdqv(node, smmu);
+		if (!is_realm_world())
+			acpi_smmu_dsdt_probe_tegra241_cmdqv(node, smmu);
 		break;
 	}
 
@@ -5676,7 +5681,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 		smmu->features |= ARM_SMMU_FEAT_COHERENCY;
 
 	if (of_device_is_compatible(dev->of_node, "nvidia,tegra264-smmu")) {
-		tegra_cmdqv_dt_probe(dev->of_node, smmu);
+		/* Realm doesn't support Tegra CMDQV yet */
+		if (!is_realm_world())
+			tegra_cmdqv_dt_probe(dev->of_node, smmu);
 		static_branch_enable(&arm_smmu_erratum_repeat_tlbi_cfgi_key);
 	}
 
@@ -5779,6 +5786,53 @@ static struct arm_smmu_device *arm_smmu_impl_probe(struct arm_smmu_device *smmu)
 	return ERR_PTR(ret);
 }
 
+static void arm_smmu_clear_realm_private(void *dev)
+{
+	dma_set_cc_private(dev, false);
+}
+
+static int arm_smmu_probe_realm_vsmmu(struct arm_smmu_device *smmu,
+				      const struct resource *res)
+{
+	struct device *dev = smmu->dev;
+	unsigned long rsi_ret;
+	phys_addr_t top;
+
+	/*
+	 * Currently the driver does not support a T=0 SMMU inside a realm. For
+	 * instance it does not make the page table allocations into shared
+	 * memory. If we are in a realm reject any SMMU that is not T=1.
+	 */
+	rsi_ret = rsi_vsmmu_get_info(res->start, &top);
+	if (rsi_ret != RSI_SUCCESS) {
+		dev_err(dev, "RSI_VSMMU_GET_INFO failed for %pr: %lu\n", res,
+			rsi_ret);
+		return -ENODEV;
+	}
+
+	if (top != res->end + 1) {
+		dev_err(dev, "VSMMU range %pr ends at %pa\n", res, &top);
+		return -EINVAL;
+	}
+
+	rsi_ret = rsi_arch_dev_activate(res->start, RSI_ARCH_DEV_SMMUV3);
+	if (rsi_ret != RSI_SUCCESS) {
+		dev_err(dev, "RSI_ARCH_DEV_ACTIVATE failed for %pr: %lu\n", res,
+			rsi_ret);
+		return -EIO;
+	}
+
+	/*
+	 * Once activated, the SMMU DMA follows DEV_FLAG_DMA_CC_PRIVATE so its
+	 * queues and tables are allocated from private memory, not the SWIOTLB
+	 * shared pool. It only translates for a device we've requested the RMM
+	 * to put into T=1.
+	 */
+	iommu_device_set_confidential(&smmu->iommu, dev);
+
+	return devm_add_action_or_reset(dev, arm_smmu_clear_realm_private, dev);
+}
+
 static int arm_smmu_device_probe(struct platform_device *pdev)
 {
 	int irq, ret;
@@ -5814,6 +5868,12 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	}
 	ioaddr = res->start;
 
+	if (is_realm_world()) {
+		ret = arm_smmu_probe_realm_vsmmu(smmu, res);
+		if (ret)
+			return ret;
+	}
+
 	/*
 	 * Don't map the IMPLEMENTATION DEFINED regions, since they may contain
 	 * the PMCG registers which are reserved by the PMU driver.
-- 
2.43.0