[RFC PATCH kernel 00/17] PCI/TSM: coco/sev-guest: Implement SEV-TIO PCIe TDISP (phase2)

[RFC PATCH kernel 00/17] PCI/TSM: coco/sev-guest: Implement SEV-TIO PCIe TDISP (phase2)

[Alexey Kardashevskiy]

Here are some patches to continue enabling SEV-TIO on AMD.

SEV-TIO allows guests to establish trust in a device that supports TEE
Device Interface Security Protocol (TDISP, defined in PCIe r6.0+) and
then interact with the device via private memory.

In order to streamline upstreaming process, a common TSM infrastructure
is being developed in collaboration with Intel+ARM+RiscV. There is
Documentation/driver-api/pci/tsm.rst with proposed phases:
1. IDE: encrypt PCI, host only
2. TDISP in guest: lock + accept flow, interface report <= WE ARE HERE
3. Enable on host: secure MMIO + DMA, KVM changes
4. Device attestation: certificates, measurements


Acronyms:
TEE - Trusted Execution Environments, a concept of managing trust between the host and devices
TSM - TEE Security Manager (TSM), an entity which ensures security on the host
PSP - AMD platform secure processor (also "ASP", "AMD-SP"), acts as TSM on AMD.
SEV TIO - the TIO protocol implemented by the PSP and used by the host, extension to SEV-SNP
GHCB - guest/host communication block - a protocol for guest-to-host communication via a shared page
TDISP - TEE Device Interface Security Protocol (PCIe).



Flow:
- Boot guest OS, load sev-guest.ko which registers itself as a TSM
- PCI TSM creates sysfs nodes under "tsm" subdirectory in for all
  TDISP-capable devices
- lock the device via:
	echo tsm0 > "/sys/bus/pci/devices/0000:01:00.0/tsm/lock"
- accept the device via:
	echo 1 > "/sys/bus/pci/devices/0000:01:00.0/tsm/accept"
- load the device driver:
	- DMA to encrypted memory should work right away
	- Reported TEE MMIO regions will be mapped as encrypted


Patches 01/17..05/17 are fixes and can go in sooner.
Patches 06/17..17/17 are the minimum required by the VM to get encrypted MMIO and DMA.

Doing "io_tlb_default_mem.for_alloc = true" in swiotlb_init_remap() enabled T=1 DMA
to shared guest memory.


The previous conversation is here:
https://lore.kernel.org/r/20260225053806.3311234-1-aik@amd.com

This is based on the last Dan's patch series rebased on top of
v7.3-rc2 with the DMA SWIOTLB fixes from Aneesh.

The whole tree is here: https://github.com/AMDESE/linux-kvm/commits/tsm-next/
The host support is here: https://github.com/AMDESE/linux-kvm/commits/tsm
Some raw QEMU sketch is here: https://github.com/AMDESE/qemu/commits/tsm-next

Please comment. Thanks.


The SEV TIO spec:
https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/specifications/58271.pdf



Alexey Kardashevskiy (16):
  pci/dma/tsm: Call disable DMA bus hook on cleanup
  pci/tsm: Fix stale comment about TDI report range start
  tsm/core: Store range_id in pci_tsm_mmio_entry
  crypto/ccp/tsm: Use TSM API for DOE
  tsm-core: Register nevertheless
  x86/io/tsm: Allow mixed ioremap for shared+private BARs
  x86/dma: Revert "x86: Remove unnecessary architecture-specific
    <asm/device.h>"
  x86/dma: Add ARCH_HAS_PHYS_TO_DMA
  dma/swiotlb: Force shared DMA for allocatios from SWIOTLB
  tsm/core: Add TDI status
  coco/sev-guest: Allow multiple source files in the driver
  x86/sev: Pass HV features to sev-guest device via platform data
  x86/sev: Add GHCB calls for SEV-TIO
  x86/sev: Implement guest TSM driver for SEV-TIO (phase2, DMA)
  x86/sev: Enable secure MMIO (phase2)
  x86/sev: Flush IOMMU TLB for trusted devices

Dan Williams (1):
  x86, dma: Allow accepted devices to map private memory

 arch/x86/Kconfig                                    |   1 +
 drivers/virt/coco/sev-guest/Kconfig                 |   1 +
 drivers/virt/coco/sev-guest/Makefile                |   5 +-
 arch/x86/include/asm/device.h                       |  13 +
 arch/x86/include/asm/dma-direct.h                   |  77 +++
 arch/x86/include/asm/sev-common.h                   |   3 +
 arch/x86/include/asm/sev.h                          |  13 +
 arch/x86/include/uapi/asm/svm.h                     |  43 ++
 drivers/virt/coco/sev-guest/sev-guest.h             |  20 +
 include/linux/dma-direct.h                          |   2 +-
 include/linux/io.h                                  |   8 +
 include/linux/ioport.h                              |   3 +
 include/linux/pci-tsm.h                             |  71 ++
 include/uapi/linux/sev-guest.h                      |  12 +
 arch/x86/coco/sev/core.c                            | 148 +++-
 arch/x86/mm/ioremap.c                               |   2 +-
 arch/x86/mm/mem_encrypt.c                           |   5 +-
 drivers/crypto/ccp/sev-dev-tsm.c                    |   5 +-
 drivers/pci/pci-driver.c                            |   3 +
 drivers/pci/tsm/core.c                              |   3 +-
 drivers/virt/coco/sev-guest/{sev-guest.c => core.c} |  25 +-
 drivers/virt/coco/sev-guest/tio.c                   | 725 ++++++++++++++++++++
 drivers/virt/coco/tsm-core.c                        |  16 +-
 drivers/xen/swiotlb-xen.c                           |   2 +-
 kernel/dma/direct.c                                 |   3 +-
 kernel/dma/swiotlb.c                                |   2 +-
 kernel/resource.c                                   |  63 ++
 mm/ioremap.c                                        |   2 +-
 28 files changed, 1249 insertions(+), 27 deletions(-)
 create mode 100644 arch/x86/include/asm/device.h
 create mode 100644 arch/x86/include/asm/dma-direct.h
 create mode 100644 drivers/virt/coco/sev-guest/sev-guest.h
 rename drivers/virt/coco/sev-guest/{sev-guest.c => core.c} (97%)
 create mode 100644 drivers/virt/coco/sev-guest/tio.c

-- 
2.55.0

[RFC PATCH kernel 01/17] pci/dma/tsm: Call disable DMA bus hook on cleanup

[Alexey Kardashevskiy]

Since XXXX ("PCI, device core: Add private memory access for DEVICE_TRUST_TCB")
the TSM subsystem relies on the PCI subsystem to ask the TSM to enable
DMA.

That misses teardown path to disable DMA, add it now.

Fixes: XXXX ("PCI, device core: Add private memory access for DEVICE_TRUST_TCB")
Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---

Squash into "PCI, device core: Add private memory access for
DEVICE_TRUST_TCB"?
---
 drivers/pci/pci-driver.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
index a036f3d7e6d5..633926b55da6 100644
--- a/drivers/pci/pci-driver.c
+++ b/drivers/pci/pci-driver.c
@@ -1739,6 +1739,9 @@ static void pci_dma_cleanup(struct device *dev)
 
 	if (!driver->driver_managed_dma)
 		iommu_device_unuse_default_domain(dev);
+
+	if (device_tcb_trusted(dev))
+		pci_tsm_disable_dma(to_pci_dev(dev));
 }
 
 /*
-- 
2.55.0

[RFC PATCH kernel 02/17] pci/tsm: Fix stale comment about TDI report range start

[Alexey Kardashevskiy]

The PCIe spec r6 and later defines the MMIO range start in 4K units which
was not the intention. The upcoming change makes it a byte address.
The structure is already fixed to match the new definition but the comment
is stale, fix it.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 drivers/pci/tsm/core.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/pci/tsm/core.c b/drivers/pci/tsm/core.c
index 9ac216ad896d..2a18be5be56e 100644
--- a/drivers/pci/tsm/core.c
+++ b/drivers/pci/tsm/core.c
@@ -623,7 +623,6 @@ EXPORT_SYMBOL_GPL(pci_tsm_mmio_teardown);
 #define PCI_TSM_DEVIF_REPORT_MMIO_ATTR_IS_UPDATABLE BIT(3)
 #define PCI_TSM_DEVIF_REPORT_MMIO_ATTR_RANGE_ID GENMASK(31, 16)
 
-/* An interface report 'pfn' is 4K in size */
 struct pci_tsm_devif_mmio {
 	__le64 phys;
 	__le32 nr_pfns;
-- 
2.55.0

[RFC PATCH kernel 03/17] tsm/core: Store range_id in pci_tsm_mmio_entry

[Alexey Kardashevskiy]

The pci_tsm_mmio struct represent a parsed TDI report which platform
TSM driver will apply at the TDI accept step.

On top of what the existing structure stores already, the AMD TSM driver
is going to need a range ID for the PSP communication.

Instead of calculating it in the platform TSM driver, just pass it along.

While at this, set the flag as resource_contains() checks for the type
when pci_tsm_mmio_setup() inserts the range.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---

Not quite sure if IORESOURCE_MEM deserves a separate fix;
also if it is just me who hit this.
---
 include/linux/pci-tsm.h | 2 ++
 drivers/pci/tsm/core.c  | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/include/linux/pci-tsm.h b/include/linux/pci-tsm.h
index 6d5fadd79360..9b34304d9a1f 100644
--- a/include/linux/pci-tsm.h
+++ b/include/linux/pci-tsm.h
@@ -161,10 +161,12 @@ struct pci_tsm_pf0 {
  * @res: MMIO address range (typically Guest Physical Address, GPA)
  * @tsm_offset: Host Physical Address, HPA obfuscation offset added by the TSM.
  *		Translates report addresses to GPA.
+ * @range_id: PCI BAR index for this range
  */
 struct pci_tsm_mmio_entry {
 	struct resource res;
 	u64 tsm_offset;
+	unsigned char range_id;
 };
 
 struct pci_tsm_mmio {
diff --git a/drivers/pci/tsm/core.c b/drivers/pci/tsm/core.c
index 2a18be5be56e..a197d56edbfb 100644
--- a/drivers/pci/tsm/core.c
+++ b/drivers/pci/tsm/core.c
@@ -746,6 +746,8 @@ struct pci_tsm_mmio *pci_tsm_mmio_alloc(struct pci_dev *pdev)
 
 		entry->res.start = range.start;
 		entry->res.end = range.end;
+		entry->res.flags = IORESOURCE_MEM;
+		entry->range_id = bar;
 		entry->tsm_offset = tsm_offset;
 		mmio->nr++;
 	}
-- 
2.55.0

[RFC PATCH kernel 04/17] crypto/ccp/tsm: Use TSM API for DOE

[Alexey Kardashevskiy]

Compared to the currently used pci_doe(), the pci_tsm_doe_transfer() helper
performs a few more sanity checks, use that helper now.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 drivers/crypto/ccp/sev-dev-tsm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/ccp/sev-dev-tsm.c b/drivers/crypto/ccp/sev-dev-tsm.c
index 46f2539d2d5a..6e704ad239af 100644
--- a/drivers/crypto/ccp/sev-dev-tsm.c
+++ b/drivers/crypto/ccp/sev-dev-tsm.c
@@ -40,8 +40,9 @@ static int sev_tio_spdm_cmd(struct tio_dsm *dsm, int ret)
 
 	/* ret > 0 means "SPDM requested" */
 	while (ret == PCI_DOE_FEATURE_CMA || ret == PCI_DOE_FEATURE_SSESSION) {
-		ret = pci_doe(dsm->tsm.doe_mb, PCI_VENDOR_ID_PCI_SIG, ret,
-			      spdm->req, spdm->req_len, spdm->rsp, spdm->rsp_len);
+		ret = pci_tsm_doe_transfer(dsm->tsm.base_tsm.pdev, ret,
+					   spdm->req, spdm->req_len,
+					   spdm->rsp, spdm->rsp_len);
 		if (ret < 0)
 			break;
 
-- 
2.55.0

[RFC PATCH kernel 05/17] tsm-core: Register nevertheless

[Alexey Kardashevskiy]

When this and platform TSM are built-in, there is no way to control
the init order other than alphabetical.

Force the class init when a TSM platform driver registers itself.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---

The alternative is making it a subsys_init() and drop tsm_exit().
---
 drivers/virt/coco/tsm-core.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/virt/coco/tsm-core.c b/drivers/virt/coco/tsm-core.c
index 0843b77c6549..accee6a5cba8 100644
--- a/drivers/virt/coco/tsm-core.c
+++ b/drivers/virt/coco/tsm-core.c
@@ -51,6 +51,7 @@ static const struct attribute_group *tsm_pci_groups[] = {
 };
 
 static void tsm_release(struct device *);
+static bool tsm_class_registered;
 static const struct class tsm_class = {
 	.name		= "tsm",
 	.dev_release	= tsm_release,
@@ -58,6 +59,14 @@ static const struct class tsm_class = {
 };
 static DEFINE_IDA(tsm_ida);
 
+static int tsm_class_init(void)
+{
+	int ret = class_register(&tsm_class);
+
+	tsm_class_registered = ret == 0;
+	return ret;
+}
+
 static int match_id(struct device *dev, const void *data)
 {
 	struct tsm_dev *tsm_dev = container_of(dev, struct tsm_dev, dev);
@@ -82,6 +91,9 @@ static struct tsm_dev *alloc_tsm_dev(struct device *parent)
 
 	struct tsm_dev *tsm_dev __free(kfree) =
 		kzalloc_obj(*tsm_dev);
+
+	tsm_class_init();
+
 	if (!tsm_dev)
 		return ERR_PTR(-ENOMEM);
 
@@ -254,12 +266,14 @@ static void tsm_release(struct device *dev)
 
 static int __init tsm_init(void)
 {
-	return class_register(&tsm_class);
+	return tsm_class_init();
 }
 module_init(tsm_init)
 
 static void __exit tsm_exit(void)
 {
+	if (!tsm_class_registered)
+		return;
 	class_unregister(&tsm_class);
 	xa_destroy(&tsm_ide_streams);
 }
-- 
2.55.0

[RFC PATCH kernel 06/17] x86/io/tsm: Allow mixed ioremap for shared+private BARs

[Alexey Kardashevskiy]

A TDISP device advertises private MMIO ranges via a TDI report.
Upon transitioning to RUN, accesses to those must be made with
the encrypted bit set in a PTE.

A device can allow private access to a part of a BAR, for example,
MSIX BAR (when MSIX config is not locked or MSIX is emulated by the HV).

The existing __ioremap_caller() fails if not every single page has
the same protection in PTE and this breaks device drivers (usually
TDISP-unaware) calling pci_iomap() and such.

Adjust __ioremap_caller() to map such MMIO BAR in chunks.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 include/linux/io.h     |  8 +++
 include/linux/ioport.h |  3 +
 arch/x86/mm/ioremap.c  |  2 +-
 kernel/resource.c      | 63 ++++++++++++++++++++
 mm/ioremap.c           |  2 +-
 5 files changed, 76 insertions(+), 2 deletions(-)

diff --git a/include/linux/io.h b/include/linux/io.h
index 0642c7ee41db..9f5d5928f30c 100644
--- a/include/linux/io.h
+++ b/include/linux/io.h
@@ -27,6 +27,8 @@ void __iowrite64_copy(void __iomem *to, const void *from, size_t count);
 #ifdef CONFIG_MMU
 int ioremap_page_range(unsigned long addr, unsigned long end,
 		       phys_addr_t phys_addr, pgprot_t prot);
+int ioremap_map_page_range(unsigned long vaddr, phys_addr_t phys_addr,
+			   unsigned long size, pgprot_t prot);
 int vmap_page_range(unsigned long addr, unsigned long end,
 		    phys_addr_t phys_addr, pgprot_t prot);
 #else
@@ -35,6 +37,12 @@ static inline int ioremap_page_range(unsigned long addr, unsigned long end,
 {
 	return 0;
 }
+static inline int ioremap_map_page_range(unsigned long vaddr,
+					 phys_addr_t phys_addr,
+					 unsigned long size, pgprot_t prot)
+{
+	return 0;
+}
 static inline int vmap_page_range(unsigned long addr, unsigned long end,
 				  phys_addr_t phys_addr, pgprot_t prot)
 {
diff --git a/include/linux/ioport.h b/include/linux/ioport.h
index 122f1eefb4b9..49d35b31f22d 100644
--- a/include/linux/ioport.h
+++ b/include/linux/ioport.h
@@ -444,8 +444,11 @@ walk_iomem_res_desc(unsigned long desc, unsigned long flags, u64 start, u64 end,
 		    void *arg, int (*func)(struct resource *, void *));
 extern int walk_soft_reserve_res(u64 start, u64 end, void *arg,
 				 int (*func)(struct resource *, void *));
+extern int walk_encrypted_mem_res(u64 start, u64 end, void *arg,
+				  int (*func)(struct resource *, void *));
 extern int
 region_intersects_soft_reserve(resource_size_t start, size_t size);
+extern int region_intersects_encrypted(resource_size_t start, size_t size);
 
 struct resource *devm_request_free_mem_region(struct device *dev,
 		struct resource *base, unsigned long size);
diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c
index 12c8180ca1ba..976ce6d5cf70 100644
--- a/arch/x86/mm/ioremap.c
+++ b/arch/x86/mm/ioremap.c
@@ -298,7 +298,7 @@ __ioremap_caller(resource_size_t phys_addr, unsigned long size,
 	if (memtype_kernel_map_sync(phys_addr, size, pcm))
 		goto err_free_area;
 
-	if (ioremap_page_range(vaddr, vaddr + size, phys_addr, prot))
+	if (ioremap_map_page_range(vaddr, phys_addr, size, prot))
 		goto err_free_area;
 
 	ret_addr = (void __iomem *) (vaddr + offset);
diff --git a/kernel/resource.c b/kernel/resource.c
index 1b9cf2244b3f..c575ed87d67a 100644
--- a/kernel/resource.c
+++ b/kernel/resource.c
@@ -26,6 +26,8 @@
 #include <linux/mm.h>
 #include <linux/mount.h>
 #include <linux/resource_ext.h>
+#include <linux/cc_platform.h>
+#include <linux/io.h>
 #include <uapi/linux/magic.h>
 #include <linux/string.h>
 #include <linux/vmalloc.h>
@@ -713,6 +715,67 @@ int region_intersects_soft_reserve(resource_size_t start, size_t size)
 }
 EXPORT_SYMBOL_GPL(region_intersects_soft_reserve);
 
+/*
+ * Walk encrypted MMIO ranges registered for TDISP/TSM private device
+ * regions (see encrypted_iomem_resource).
+ */
+int walk_encrypted_mem_res(u64 start, u64 end, void *arg,
+			   int (*func)(struct resource *, void *))
+{
+	return walk_res_desc(&encrypted_iomem_resource, start, end,
+			     IORESOURCE_MEM, IORES_DESC_ENCRYPTED, arg, func);
+}
+EXPORT_SYMBOL_GPL(walk_encrypted_mem_res);
+
+int region_intersects_encrypted(resource_size_t start, size_t size)
+{
+	guard(read_lock)(&resource_lock);
+	return __region_intersects(&encrypted_iomem_resource, start, size,
+				   IORESOURCE_MEM, IORES_DESC_ENCRYPTED);
+}
+EXPORT_SYMBOL_GPL(region_intersects_encrypted);
+
+#ifdef CONFIG_MMU
+/*
+ * Map @size bytes at @vaddr to @phys_addr using @prot, splitting the mapping
+ * when TDISP/TSM has registered only part of the range in
+ * encrypted_iomem_resource and per-page encryption attributes are required.
+ */
+int ioremap_map_page_range(unsigned long vaddr, phys_addr_t phys_addr,
+			   unsigned long size, pgprot_t prot)
+{
+	if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) ||
+	    region_intersects_encrypted(phys_addr, size) == REGION_DISJOINT)
+		return ioremap_page_range(vaddr, vaddr + size, phys_addr, prot);
+
+	while (size) {
+		bool enc = region_intersects_encrypted(phys_addr, PAGE_SIZE) ==
+			   REGION_INTERSECTS;
+		unsigned long run = PAGE_SIZE;
+		pgprot_t run_prot = enc ? pgprot_encrypted(prot) : prot;
+
+		phys_addr += PAGE_SIZE;
+		vaddr += PAGE_SIZE;
+		size -= PAGE_SIZE;
+
+		while (size &&
+		       (region_intersects_encrypted(phys_addr, PAGE_SIZE) ==
+			REGION_INTERSECTS) == enc) {
+			run += PAGE_SIZE;
+			phys_addr += PAGE_SIZE;
+			vaddr += PAGE_SIZE;
+			size -= PAGE_SIZE;
+		}
+
+		if (vmap_page_range(vaddr - run, vaddr, phys_addr - run, run_prot))
+			return -EINVAL;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ioremap_map_page_range);
+#endif /* CONFIG_MMU */
+
 void __weak arch_remove_reservations(struct resource *avail)
 {
 }
diff --git a/mm/ioremap.c b/mm/ioremap.c
index c36dd9f62fd5..ccf938628e55 100644
--- a/mm/ioremap.c
+++ b/mm/ioremap.c
@@ -40,7 +40,7 @@ void __iomem *generic_ioremap_prot(phys_addr_t phys_addr, size_t size,
 	vaddr = (unsigned long)area->addr;
 	area->phys_addr = phys_addr;
 
-	if (ioremap_page_range(vaddr, vaddr + size, phys_addr, prot)) {
+	if (ioremap_map_page_range(vaddr, phys_addr, size, prot)) {
 		free_vm_area(area);
 		return NULL;
 	}
-- 
2.55.0

[RFC PATCH kernel 07/17] x86/dma: Revert "x86: Remove unnecessary architecture-specific <asm/device.h>"

[Alexey Kardashevskiy]

The trusted guest device DMA setup is going to distinguish
encrypted and decrypted memory and dev_archdata is a logical place for
it.

This reverts commit c256d2a8adf2f5670ca262979c451ec4c1108e38.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 arch/x86/include/asm/device.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/x86/include/asm/device.h b/arch/x86/include/asm/device.h
new file mode 100644
index 000000000000..7c0a52ca2f4d
--- /dev/null
+++ b/arch/x86/include/asm/device.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_DEVICE_H
+#define _ASM_X86_DEVICE_H
+
+struct dev_archdata {
+};
+
+struct pdev_archdata {
+};
+
+#endif /* _ASM_X86_DEVICE_H */
-- 
2.55.0

[RFC PATCH kernel 08/17] x86/dma: Add ARCH_HAS_PHYS_TO_DMA

[Alexey Kardashevskiy]

Confidential VMs work with both trusted and legacy (untrusted devices).
The trusted devices allow DMA to/from encrypted guest memory while legacy
devices have to use SWIOTLB or share pages for DMA.

In case of P2P between trusted and legacy devices, the trusted device has
to perform DMA with T=1 (a PCIe IDE TLP bit saying "encrypt") but the target
may be shared.

One way of allowing the above on AMD SEV is using IOMMU sDTE vTOM feature
which is a bus address:
- below that address all accesses target private guest memory;
- above - shared memory.

Effectively this adds some high bit in a DMA handle to mark "shared" (on
AMD).

Copy the generic phys_to_dma / dma_to_phys helpers from
include/linux/dma-direct.h into arch/x86/include/asm/dma-direct.h and
select ARCH_HAS_PHYS_TO_DMA for x86.

The x86 implementation is the same as the generic code for
__phys_to_dma, dma_range_map handling, and SME encryption, with two
additions for confidential devices:
 - cc_shared_dma_offset and cc_private_dma_offset in dev_archdata
 - when TDI is accepted is true, phys_to_dma() adds the shared or
   private offset (selected by DMA_ATTR_CC_SHARED), and dma_to_phys()
   subtracts it on the way back

phys_to_dma() also gains an attrs argument so dma_capable() and swiotlb can
pass mapping attributes through. Update the call sites accordingly.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---

DMA_ATTR_CC_SHARED or __DMA_ATTR_ALLOC_CC_SHARED?
---
 arch/x86/Kconfig                  |  1 +
 arch/x86/include/asm/device.h     |  2 +
 arch/x86/include/asm/dma-direct.h | 77 ++++++++++++++++++++
 include/linux/dma-direct.h        |  2 +-
 drivers/xen/swiotlb-xen.c         |  2 +-
 kernel/dma/swiotlb.c              |  2 +-
 6 files changed, 83 insertions(+), 3 deletions(-)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 15fd9ec5ecac..c4df431e74b2 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -112,6 +112,7 @@ config X86
 	select ARCH_HAS_UBSAN
 	select ARCH_HAS_DEBUG_WX
 	select ARCH_HAS_ZONE_DMA_SET if EXPERT
+	select ARCH_HAS_PHYS_TO_DMA
 	select ARCH_HAVE_NMI_SAFE_CMPXCHG
 	select ARCH_HAVE_EXTRA_ELF_NOTES
 	select ARCH_MEMORY_ORDER_TSO
diff --git a/arch/x86/include/asm/device.h b/arch/x86/include/asm/device.h
index 7c0a52ca2f4d..0dbc2c125522 100644
--- a/arch/x86/include/asm/device.h
+++ b/arch/x86/include/asm/device.h
@@ -3,6 +3,8 @@
 #define _ASM_X86_DEVICE_H
 
 struct dev_archdata {
+	dma_addr_t cc_shared_dma_offset;
+	dma_addr_t cc_private_dma_offset;
 };
 
 struct pdev_archdata {
diff --git a/arch/x86/include/asm/dma-direct.h b/arch/x86/include/asm/dma-direct.h
new file mode 100644
index 000000000000..37074c87bbec
--- /dev/null
+++ b/arch/x86/include/asm/dma-direct.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef ASM_X86_DMA_DIRECT_H
+#define ASM_X86_DMA_DIRECT_H 1
+
+#include <linux/pci-tsm.h>
+
+static inline dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t paddr)
+{
+	if (dev->dma_range_map)
+		return translate_phys_to_dma(dev, paddr);
+
+	return paddr;
+}
+
+static inline bool __device_cc_accepted(struct device *dev)
+{
+	if (!dev || !dev_is_pci(dev) || !to_pci_dev(dev)->tsm)
+		return false;
+
+	return test_bit(PCI_TSM_F_ACCEPT, &to_pci_dev(dev)->tsm->flags);
+}
+
+static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr, unsigned long attrs)
+{
+	if (__device_cc_accepted(dev)) {
+		if (attrs & DMA_ATTR_CC_SHARED)
+			return __phys_to_dma(dev, paddr) + dev->archdata.cc_shared_dma_offset;
+
+		return __phys_to_dma(dev, paddr) + dev->archdata.cc_private_dma_offset;
+	}
+
+	return dma_addr_encrypted(__phys_to_dma(dev, paddr));
+}
+
+static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr)
+{
+	phys_addr_t paddr;
+
+	if (__device_cc_accepted(dev)) {
+		if (dev->archdata.cc_shared_dma_offset &&
+		    daddr >= dev->archdata.cc_shared_dma_offset)
+			return daddr - dev->archdata.cc_shared_dma_offset;
+
+		if (dev->archdata.cc_private_dma_offset &&
+		    daddr >= dev->archdata.cc_private_dma_offset)
+			return daddr - dev->archdata.cc_private_dma_offset;
+	}
+
+	daddr = dma_addr_canonical(daddr);
+	if (dev->dma_range_map)
+		paddr = translate_dma_to_phys(dev, daddr);
+	else
+		paddr = daddr;
+
+	return paddr;
+}
+
+static inline dma_addr_t phys_to_dma_unencrypted(struct device *dev, phys_addr_t paddr)
+{
+	if (__device_cc_accepted(dev))
+		return __phys_to_dma(dev, paddr) + dev->archdata.cc_shared_dma_offset;
+
+	return dma_addr_unencrypted(__phys_to_dma(dev, paddr));
+}
+
+static inline dma_addr_t phys_to_dma_encrypted(struct device *dev, phys_addr_t paddr)
+{
+	if (__device_cc_accepted(dev))
+		return __phys_to_dma(dev, paddr) + dev->archdata.cc_private_dma_offset;
+
+	return dma_addr_encrypted(__phys_to_dma(dev, paddr));
+}
+
+#define phys_to_dma_unencrypted phys_to_dma_unencrypted
+#define phys_to_dma_encrypted phys_to_dma_encrypted
+
+#endif /* ASM_X86_DMA_DIRECT_H */
diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h
index daa31a1adf7b..616b66ccf322 100644
--- a/include/linux/dma-direct.h
+++ b/include/linux/dma-direct.h
@@ -150,7 +150,7 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size,
 		return false;
 
 	if (is_ram && !IS_ENABLED(CONFIG_ARCH_DMA_ADDR_T_64BIT) &&
-	    min(addr, end) < phys_to_dma(dev, PFN_PHYS(min_low_pfn)))
+	    min(addr, end) < phys_to_dma(dev, PFN_PHYS(min_low_pfn), attrs))
 		return false;
 
 	return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit);
diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index e2538824ef52..915c174f8b56 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -55,7 +55,7 @@ static inline phys_addr_t xen_phys_to_bus(struct device *dev, phys_addr_t paddr)
 
 static inline dma_addr_t xen_phys_to_dma(struct device *dev, phys_addr_t paddr)
 {
-	return phys_to_dma(dev, xen_phys_to_bus(dev, paddr));
+	return phys_to_dma(dev, xen_phys_to_bus(dev, paddr), 0);
 }
 
 static inline phys_addr_t xen_bus_to_phys(struct device *dev,
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index ded7016a46a7..d7c7d15740ae 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -1764,7 +1764,7 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_t paddr, size_t size,
 	phys_addr_t swiotlb_addr;
 	dma_addr_t dma_addr;
 
-	trace_swiotlb_bounced(dev, phys_to_dma(dev, paddr), size);
+	trace_swiotlb_bounced(dev, phys_to_dma(dev, paddr, attrs), size);
 
 	swiotlb_addr = swiotlb_tbl_map_single(dev, paddr, size, 0, dir, &attrs);
 	if (swiotlb_addr == (phys_addr_t)DMA_MAPPING_ERROR)
-- 
2.55.0

[RFC PATCH kernel 09/17] dma/swiotlb: Force shared DMA for allocatios from SWIOTLB

[Alexey Kardashevskiy]

SWIOTLB allocations are always decrypted but the DMA layer is unaware
of those and will miss to adjust the DMA handle.

Force __DMA_ATTR_ALLOC_CC_SHARED if SWIOTLB is used for memory allocations.

This will be exploited by AMD SEV guest vTOM feature where DMA handles
of shared memory have some high bit set (not to be confused with the
Cbit).

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 kernel/dma/direct.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index da665ca22d5c..dacf513d45fd 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -273,13 +273,14 @@ void *dma_direct_alloc(struct device *dev, size_t size,
 	}
 
 	if (is_swiotlb_for_alloc(dev)) {
-		page = dma_direct_alloc_swiotlb(dev, size, attrs);
+		page = dma_direct_alloc_swiotlb(dev, size, attrs | __DMA_ATTR_ALLOC_CC_SHARED);
 		if (page) {
 			/*
 			 * swiotlb allocations comes from pool already marked
 			 * decrypted
 			 */
 			mark_mem_decrypt = false;
+			attrs |= __DMA_ATTR_ALLOC_CC_SHARED;
 			goto setup_page;
 		}
 		return NULL;
-- 
2.55.0

[RFC PATCH kernel 10/17] x86, dma: Allow accepted devices to map private memory

[Alexey Kardashevskiy]

From: Dan Williams <dan.j.williams@intel.com>

With the arrival of "accepted" devices, devices that have been enabled to
DMA to private encrypted memory, coherent DMA allocation no longer requires
page conversion. Update force_dma_unencrypted() to skip accepted devices.

Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: x86@kernel.org
Cc: "H. Peter Anvin" <hpa@zytor.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
[aik: re-added device_cc_accepted()]
Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 arch/x86/mm/mem_encrypt.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 95bae74fdab2..805800ddbd79 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -20,10 +20,11 @@
 bool force_dma_unencrypted(struct device *dev)
 {
 	/*
-	 * For SEV, all DMA must be to unencrypted addresses.
+	 * Require unencrypted DMA unless the device has been "accepted",
+	 * enabled by a TSM driver to DMA to private encrypted memory.
 	 */
 	if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
-		return true;
+		return !__device_cc_accepted(dev);
 
 	/*
 	 * For SME, all DMA must be to unencrypted addresses if the
-- 
2.55.0

[RFC PATCH kernel 11/17] tsm/core: Add TDI status

[Alexey Kardashevskiy]

Define a structure with all info about a TDI such as TDISP status,
bind state, used START_INTERFACE options and the report digest.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 include/linux/pci-tsm.h | 69 ++++++++++++++++++++
 1 file changed, 69 insertions(+)

diff --git a/include/linux/pci-tsm.h b/include/linux/pci-tsm.h
index 9b34304d9a1f..2ca577e2a37e 100644
--- a/include/linux/pci-tsm.h
+++ b/include/linux/pci-tsm.h
@@ -11,6 +11,7 @@
 struct pci_tsm;
 struct tsm_dev;
 struct kvm;
+struct tsm_tdi_status;
 enum pci_tsm_req_scope;
 
 /*
@@ -88,6 +89,7 @@ struct pci_tsm_ops {
 
 	int (*refresh_evidence)(struct pci_tsm *tsm, const void *nonce,
 				size_t nonce_len);
+	int (*tdi_status)(struct pci_dev *pdev, struct tsm_tdi_status *ts);
 };
 
 /**
@@ -331,4 +333,71 @@ static inline ssize_t pci_tsm_guest_req(struct pci_dev *pdev,
 	return -ENXIO;
 }
 #endif
+
+/* private: */
+
+enum tsm_tdisp_state {
+	TDISP_STATE_CONFIG_UNLOCKED = 0,
+	TDISP_STATE_CONFIG_LOCKED = 1,
+	TDISP_STATE_RUN = 2,
+	TDISP_STATE_ERROR = 3,
+};
+
+enum tsm_tdisp_status {
+	TDISP_STATE_BOUND = 0,
+	TDISP_STATE_INVALID = 1,
+	TDISP_STATE_UNBOUND = 2,
+};
+
+/*
+ * struct tdisp_interface_id - TDISP INTERFACE_ID Definition
+ *
+ * @function_id: Identifies the function of the device hosting the TDI
+ *   15:0: @rid: Requester ID
+ *   23:16: @rseg: Requester Segment (Reserved if Requester Segment Valid is Clear)
+ *   24: @rseg_valid: Requester Segment Valid
+ *   31:25 – Reserved
+ * 8B - Reserved
+ */
+#define TSM_TDISP_IID_REQUESTER_ID      GENMASK(15, 0)
+#define TSM_TDISP_IID_RSEG              GENMASK(23, 16)
+#define TSM_TDISP_IID_RSEG_VALID        BIT(24)
+
+struct tdisp_interface_id {
+	__u32 function_id; /* TSM_TDISP_IID_xxxx */
+	__u8 reserved[8];
+} __packed;
+
+struct tsm_tdi_status {
+	__u8 status; /* enum tsm_tdisp_status */
+	__u8 state; /* enum tsm_tdisp_state */
+	__u8 meas_digest_fresh;
+	__u8 meas_digest_valid;
+	__u8 all_request_redirect;
+	__u8 bind_p2p;
+	__u8 lock_msix;
+	__u8 no_fw_update;
+	__u16 cache_line_size;
+	__u64 spdm_algos; /* Bitmask of TSM_SPDM_ALGOS */
+	__u8 certs_digest[48];
+	__u8 meas_digest[48];
+	__u8 interface_report_digest[48];
+	__u64 intf_report_counter;
+	struct tdisp_interface_id id;
+	__u64 tdi_id;
+} __packed;
+
+enum tsm_spdm_algos {
+	TSM_SPDM_ALGOS_DHE_SECP256R1,
+	TSM_SPDM_ALGOS_DHE_SECP384R1,
+	TSM_SPDM_ALGOS_AEAD_AES_128_GCM,
+	TSM_SPDM_ALGOS_AEAD_AES_256_GCM,
+	TSM_SPDM_ALGOS_ASYM_TPM_ALG_RSASSA_3072,
+	TSM_SPDM_ALGOS_ASYM_TPM_ALG_ECDSA_ECC_NIST_P256,
+	TSM_SPDM_ALGOS_ASYM_TPM_ALG_ECDSA_ECC_NIST_P384,
+	TSM_SPDM_ALGOS_HASH_TPM_ALG_SHA_256,
+	TSM_SPDM_ALGOS_HASH_TPM_ALG_SHA_384,
+	TSM_SPDM_ALGOS_KEY_SCHED_SPDM_KEY_SCHEDULE,
+};
+
 #endif /*__PCI_TSM_H */
-- 
2.55.0

[RFC PATCH kernel 12/17] coco/sev-guest: Allow multiple source files in the driver

[Alexey Kardashevskiy]

Prepare for SEV-TIO support as it is going to equal or bigger
than the existing sev_guest.c which is already 700 lines and
keeps growing.

No behavioural change expected.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 drivers/virt/coco/sev-guest/Makefile                |  2 +-
 drivers/virt/coco/sev-guest/sev-guest.h             | 16 ++++++++++++++++
 drivers/virt/coco/sev-guest/{sev-guest.c => core.c} | 10 ++--------
 3 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/drivers/virt/coco/sev-guest/Makefile b/drivers/virt/coco/sev-guest/Makefile
index 63d67c27723a..a8eb28b6736e 100644
--- a/drivers/virt/coco/sev-guest/Makefile
+++ b/drivers/virt/coco/sev-guest/Makefile
@@ -1,2 +1,2 @@
 # SPDX-License-Identifier: GPL-2.0-only
-obj-$(CONFIG_SEV_GUEST) += sev-guest.o
+obj-$(CONFIG_SEV_GUEST) += core.o
diff --git a/drivers/virt/coco/sev-guest/sev-guest.h b/drivers/virt/coco/sev-guest/sev-guest.h
new file mode 100644
index 000000000000..b2a97778e635
--- /dev/null
+++ b/drivers/virt/coco/sev-guest/sev-guest.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef __SEV_GUEST_H__
+#define __SEV_GUEST_H__
+
+#include <linux/miscdevice.h>
+#include <asm/sev.h>
+
+struct snp_guest_dev {
+	struct device *dev;
+	struct miscdevice misc;
+
+	struct snp_msg_desc *msg_desc;
+};
+
+#endif /* __SEV_GUEST_H__ */
diff --git a/drivers/virt/coco/sev-guest/sev-guest.c b/drivers/virt/coco/sev-guest/core.c
similarity index 99%
rename from drivers/virt/coco/sev-guest/sev-guest.c
rename to drivers/virt/coco/sev-guest/core.c
index 935537a41469..98f698809720 100644
--- a/drivers/virt/coco/sev-guest/sev-guest.c
+++ b/drivers/virt/coco/sev-guest/core.c
@@ -27,19 +27,13 @@
 #include <uapi/linux/psp-sev.h>
 
 #include <asm/svm.h>
-#include <asm/sev.h>
+
+#include "sev-guest.h"
 
 #define DEVICE_NAME	"sev-guest"
 
 #define SVSM_MAX_RETRIES		3
 
-struct snp_guest_dev {
-	struct device *dev;
-	struct miscdevice misc;
-
-	struct snp_msg_desc *msg_desc;
-};
-
 /*
  * The VMPCK ID represents the key used by the SNP guest to communicate with the
  * SEV firmware in the AMD Secure Processor (ASP, aka PSP). By default, the key
-- 
2.55.0

[RFC PATCH kernel 13/17] x86/sev: Pass HV features to sev-guest device via platform data

[Alexey Kardashevskiy]

The SEV HV advertises supported options via GHCB, the guest stores
bitmask in the SEV core.

The sev-guest device is a user visible platform device (/dev/sev-guest)
which provides the interface to communicate with the HV via GHCB.
It may choose to limit the functionality depending on what is available.
The first user is going to be SEV-TIO which is going to trigger creation
of "tsm" folders in the PCI sysfs but only if GHCB advertised such
support.

This should cause no behavioural change.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 arch/x86/coco/sev/core.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c
index cc292d7c6fd1..5b912f42f493 100644
--- a/arch/x86/coco/sev/core.c
+++ b/arch/x86/coco/sev/core.c
@@ -1381,11 +1381,6 @@ static int snp_issue_guest_request(struct snp_guest_req *req)
 	return ret;
 }
 
-static struct platform_device sev_guest_device = {
-	.name		= "sev-guest",
-	.id		= -1,
-};
-
 static struct platform_device tpm_svsm_device = {
 	.name		= "tpm-svsm",
 	.id		= -1,
@@ -1393,10 +1388,17 @@ static struct platform_device tpm_svsm_device = {
 
 static int __init snp_init_platform_device(void)
 {
+	struct platform_device *dev;
+
 	if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP))
 		return -ENODEV;
 
-	if (platform_device_register(&sev_guest_device))
+	dev = platform_device_register_data(NULL, "sev-guest", -1,
+					    &sev_hv_features,
+					    sizeof(sev_hv_features));
+	if (IS_ERR(dev))
+		return PTR_ERR(dev);
+	if (!dev)
 		return -ENODEV;
 
 	if (snp_svsm_vtpm_probe() &&
-- 
2.55.0

[RFC PATCH kernel 14/17] x86/sev: Add GHCB calls for SEV-TIO

[Alexey Kardashevskiy]

SEV-TIO is a PSP protocol allowing PCI pass through of trusted devices
(TDI == TEE Device Interface) to an SNP VM. The VMM advertises
the support via HV_FEATURES and implements new calls:

1) TDI operation: the host executes on the guest behalf:
- TDI bind/unbind to/from the SNP VM in the PSP;
- TDI start/stop which translates to TDISP interface start/stop.

2) TIO guest request for an encrypted communication channel between
the SNP VM and the PSP, it follows the existing Extended Guest Request
pattern to read device evidence:
- get TDI status;
- validate MMIO ranges (a variant of the PVALIDATE instruction for
  MMIO RMPs);
- configure sDTE (a secure IOMMU descriptor).

Extend snp_req_data to pass more parameters.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 arch/x86/include/asm/sev-common.h |  1 +
 arch/x86/include/asm/sev.h        |  5 +++
 arch/x86/include/uapi/asm/svm.h   | 40 +++++++++++++++++++
 arch/x86/coco/sev/core.c          | 42 ++++++++++++++++++++
 4 files changed, 88 insertions(+)

diff --git a/arch/x86/include/asm/sev-common.h b/arch/x86/include/asm/sev-common.h
index 01a6e4dbe423..ff763c3c5d63 100644
--- a/arch/x86/include/asm/sev-common.h
+++ b/arch/x86/include/asm/sev-common.h
@@ -137,6 +137,7 @@ enum psc_op {
 #define GHCB_HV_FT_SNP			BIT_ULL(0)
 #define GHCB_HV_FT_SNP_AP_CREATION	BIT_ULL(1)
 #define GHCB_HV_FT_SNP_MULTI_VMPL	BIT_ULL(5)
+#define GHCB_HV_FT_SNP_SEV_TIO		BIT_ULL(7)
 
 /*
  * SNP Page State Change NAE event
diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index 9e7a077c445d..89aaccb053ba 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -149,6 +149,9 @@ struct snp_req_data {
 	unsigned long resp_gpa;
 	unsigned long data_gpa;
 	unsigned int data_npages;
+	unsigned int guest_rid;
+	unsigned long npages;
+	unsigned long param;
 };
 
 #define MAX_AUTHTAG_LEN		32
@@ -597,6 +600,8 @@ static inline void sev_evict_cache(void *va, int npages)
 	}
 }
 
+int sev_tio_op(u32 guest_rid, unsigned int op, u64 *fw_err, u64 *tdi_id);
+
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
 #define snp_vmpl 0
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 010a45c9f614..93597ad492bf 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -122,6 +122,44 @@
 #define SVM_VMGEXIT_SAVIC_REGISTER_GPA		0
 #define SVM_VMGEXIT_SAVIC_UNREGISTER_GPA	1
 #define SVM_VMGEXIT_SAVIC_SELF_GPA		~0ULL
+#define SVM_VMGEXIT_SEV_TIO_GR			0x80000020ull
+#define SVM_VMGEXIT_SEV_TIO_GR_INFO_STATE	BIT(0)
+#define SVM_VMGEXIT_SEV_TIO_GR_INFO_CERTS	BIT(1)
+#define SVM_VMGEXIT_SEV_TIO_GR_INFO_MEAS	BIT(2)
+#define SVM_VMGEXIT_SEV_TIO_GR_INFO_REPORT	BIT(3)
+
+#define SVM_VMGEXIT_SEV_TIO_GR_SDTE_VALIDATE	BIT(0)
+#define SVM_VMGEXIT_SEV_TIO_GR_SDTE_VTOM	GENMASK_ULL(51, 21)
+
+/*
+ * TIO_GUEST_REQUEST's MMIO_VALIDATE_REQ/MMIO_CONFIG_REQ encoding for MMIO in RDX:
+ *
+ * T....... ....GGGG GGGGGGGG GGGGGGGG GGGGGGGG GGGGGGGG GGGG.... ........
+ * Where:
+ *	G - guest physical address
+ *	T - TEE
+ */
+#define SVM_VMGEXIT_SEV_TIO_GR_MMIO_GFN(r)      (((r) & 0x000FFFFFFFFFF000ULL) >> PAGE_SHIFT)
+#define SVM_VMGEXIT_SEV_TIO_GR_MMIO_RESERVED(r) ((r) & 0x7FF0000000000FFFULL)
+#define SVM_VMGEXIT_SEV_TIO_GR_MMIO_PRIVATE(r)  (!!((r) & BIT(63)))
+
+#define SVM_VMGEXIT_SEV_TIO_GR_MMIO_NUM(r)      ((uint32_t)((r) >> 32))
+#define SVM_VMGEXIT_SEV_TIO_GR_MMIO_ADDR(r)     ((uint32_t)((r) & 0xFFFFFFFF))
+
+#define SVM_VMGEXIT_SEV_TIO_GR_MMIO_MK_VALIDATE(start, private) \
+	((SVM_VMGEXIT_SEV_TIO_GR_MMIO_GFN(start) << PAGE_SHIFT) | \
+	((private) ? BIT(63) : 0))
+
+#define SVM_VMGEXIT_SEV_TIO_OP			0x80000021ull
+#define SVM_VMGEXIT_SEV_TIO_GR_MMIO_MK_NUM_BDFN(n, bdfn) (((uint64_t)(n) << 32) | (bdfn))
+
+#define SVM_VMGEXIT_SEV_TIO_OP_PARAM(guest_id, action)	((u64)(action)<<32|(guest_id))
+#define SVM_VMGEXIT_SEV_TIO_OP_ACTION(exitinfo1)	((exitinfo1)>>32)
+#define SVM_VMGEXIT_SEV_TIO_OP_GUEST_ID(exitinfo1)	((exitinfo1) & 0xFFFFFFFF)
+#define SVM_VMGEXIT_SEV_TIO_OP_BIND	0
+#define SVM_VMGEXIT_SEV_TIO_OP_UNBIND	1
+#define SVM_VMGEXIT_SEV_TIO_OP_RUN	2
+#define SVM_VMGEXIT_SEV_TIO_OP_STOP	3
 #define SVM_VMGEXIT_HV_FEATURES			0x8000fffdull
 #define SVM_VMGEXIT_TERM_REQUEST		0x8000fffeull
 #define SVM_VMGEXIT_TERM_REASON(reason_set, reason_code)	\
@@ -245,6 +283,8 @@
 	{ SVM_VMGEXIT_GUEST_REQUEST,	"vmgexit_guest_request" }, \
 	{ SVM_VMGEXIT_EXT_GUEST_REQUEST, "vmgexit_ext_guest_request" }, \
 	{ SVM_VMGEXIT_AP_CREATION,	"vmgexit_ap_creation" }, \
+	{ SVM_VMGEXIT_SEV_TIO_GR,	"vmgexit_sev_tio_guest_request" }, \
+	{ SVM_VMGEXIT_SEV_TIO_OP,	"vmgexit_sev_tio_op" }, \
 	{ SVM_VMGEXIT_HV_FEATURES,	"vmgexit_hypervisor_feature" }, \
 	{ SVM_EXIT_ERR,         "invalid_guest_state" }
 
diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c
index 5b912f42f493..ed0e4546d5e5 100644
--- a/arch/x86/coco/sev/core.c
+++ b/arch/x86/coco/sev/core.c
@@ -104,6 +104,37 @@ static unsigned long snp_tsc_freq_khz __ro_after_init;
 DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
 DEFINE_PER_CPU(struct sev_es_save_area *, sev_vmsa);
 
+int sev_tio_op(u32 guest_rid, unsigned int op, u64 *fw_err, u64 *tdi_id)
+{
+	struct ghcb_state state;
+	struct es_em_ctxt ctxt;
+	struct ghcb *ghcb;
+	int ret;
+
+	/* __sev_get_ghcb() needs IRQs disabled because it uses per-CPU GHCB. */
+	guard(irqsave)();
+
+	ghcb = __sev_get_ghcb(&state);
+	if (!ghcb)
+		return -EIO;
+
+	vc_ghcb_invalidate(ghcb);
+	ret = sev_es_ghcb_hv_call(ghcb, &ctxt, SVM_VMGEXIT_SEV_TIO_OP,
+				  SVM_VMGEXIT_SEV_TIO_OP_PARAM(guest_rid, op), 0);
+
+	*fw_err = ghcb->save.sw_exit_info_2;
+	if (*fw_err)
+		ret = -EIO;
+
+	if (!ret && op == SVM_VMGEXIT_SEV_TIO_OP_BIND && tdi_id)
+		*tdi_id = ghcb_get_rcx(ghcb);
+
+	__sev_put_ghcb(&state);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(sev_tio_op);
+
 /*
  * SVSM related information:
  *   When running under an SVSM, the VMPL that Linux is executing at must be
@@ -1345,6 +1376,11 @@ static int snp_issue_guest_request(struct snp_guest_req *req)
 	if (req->exit_code == SVM_VMGEXIT_EXT_GUEST_REQUEST) {
 		ghcb_set_rax(ghcb, input->data_gpa);
 		ghcb_set_rbx(ghcb, input->data_npages);
+	} else if (req->exit_code == SVM_VMGEXIT_SEV_TIO_GR) {
+		ghcb_set_rax(ghcb, input->data_gpa);
+		ghcb_set_rbx(ghcb, input->data_npages);
+		ghcb_set_rcx(ghcb, ((uint64_t)input->npages << 32) | input->guest_rid);
+		ghcb_set_rdx(ghcb, input->param);
 	}
 
 	ret = sev_es_ghcb_hv_call(ghcb, &ctxt, req->exit_code, input->req_gpa, input->resp_gpa);
@@ -1354,6 +1390,8 @@ static int snp_issue_guest_request(struct snp_guest_req *req)
 	req->exitinfo2 = ghcb->save.sw_exit_info_2;
 	switch (req->exitinfo2) {
 	case 0:
+		if (req->exit_code == SVM_VMGEXIT_SEV_TIO_GR)
+			input->param = ghcb_get_rdx(ghcb);
 		break;
 
 	case SNP_GUEST_VMM_ERR(SNP_GUEST_VMM_ERR_BUSY):
@@ -1366,6 +1404,10 @@ static int snp_issue_guest_request(struct snp_guest_req *req)
 			input->data_npages = ghcb_get_rbx(ghcb);
 			ret = -ENOSPC;
 			break;
+		} else if (req->exit_code == SVM_VMGEXIT_SEV_TIO_GR) {
+			input->data_npages = ghcb_get_rbx(ghcb);
+			ret = -ENOSPC;
+			break;
 		}
 		fallthrough;
 	default:
-- 
2.55.0

[RFC PATCH kernel 15/17] x86/sev: Implement guest TSM driver for SEV-TIO (phase2, DMA)

[Alexey Kardashevskiy]

Add the guest-side PCI TSM backend for AMD SEV-TIO, wiring the generic
pci_tsm framework to GHCB TIO operations and PSP guest requests.

Register pci_tsm_ops from sev-guest on probe when TIO GHCB support is
present (gated by the tsm_enable module parameter).
lock() binds a TDI, constructs a devsec TSM object and transitions to
CONFIG_LOCKED; accept() transitions the TDI to RUN and programs the SDTE
with vTOM and guest access permissions; unlock() invalidates the SDTE,
unbinds the TDI, quiesces DMA and transitions to CONFIG_UNLOCKED.

Introduce tio.c with TIO guest-request handling (TIO_MSG_SDTE_WRITE_REQ
and related message layouts) and set cc_shared_dma_offset from the
configurable tsm_vtom so shared DMA addresses are translated for the
hypervisor.  Extend asm/sev.h with TIO message types and TDISP SPDM
algorithm constants.

Build tio.o when PCI_TSM is enabled and select PCI_TSM from SEV_GUEST
Kconfig.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 drivers/virt/coco/sev-guest/Kconfig     |   1 +
 drivers/virt/coco/sev-guest/Makefile    |   3 +
 arch/x86/include/asm/sev.h              |   8 +
 drivers/virt/coco/sev-guest/sev-guest.h |   4 +
 drivers/virt/coco/sev-guest/core.c      |  14 +
 drivers/virt/coco/sev-guest/tio.c       | 347 ++++++++++++++++++++
 6 files changed, 377 insertions(+)

diff --git a/drivers/virt/coco/sev-guest/Kconfig b/drivers/virt/coco/sev-guest/Kconfig
index a6405ab6c2c3..4255072dfa1a 100644
--- a/drivers/virt/coco/sev-guest/Kconfig
+++ b/drivers/virt/coco/sev-guest/Kconfig
@@ -3,6 +3,7 @@ config SEV_GUEST
 	default m
 	depends on AMD_MEM_ENCRYPT
 	select TSM_REPORTS
+	select PCI_TSM if PCI
 	help
 	  SEV-SNP firmware provides the guest a mechanism to communicate with
 	  the PSP without risk from a malicious hypervisor who wishes to read,
diff --git a/drivers/virt/coco/sev-guest/Makefile b/drivers/virt/coco/sev-guest/Makefile
index a8eb28b6736e..84a8b0dd62cd 100644
--- a/drivers/virt/coco/sev-guest/Makefile
+++ b/drivers/virt/coco/sev-guest/Makefile
@@ -1,2 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-$(CONFIG_SEV_GUEST) += core.o
+ifeq ($(CONFIG_PCI_TSM),y)
+obj-$(CONFIG_SEV_GUEST) += tio.o
+endif
diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index 89aaccb053ba..375cb5346ec3 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -182,6 +182,14 @@ enum msg_type {
 
 	SNP_MSG_TSC_INFO_REQ = 17,
 	SNP_MSG_TSC_INFO_RSP,
+	TIO_MSG_TDI_INFO_REQ = 19,
+	TIO_MSG_TDI_INFO_RSP = 20,
+	TIO_MSG_MMIO_VALIDATE_REQ = 21,
+	TIO_MSG_MMIO_VALIDATE_RSP = 22,
+	TIO_MSG_MMIO_CONFIG_REQ = 23,
+	TIO_MSG_MMIO_CONFIG_RSP = 24,
+	TIO_MSG_SDTE_WRITE_REQ = 25,
+	TIO_MSG_SDTE_WRITE_RSP = 26,
 
 	SNP_MSG_TYPE_MAX
 };
diff --git a/drivers/virt/coco/sev-guest/sev-guest.h b/drivers/virt/coco/sev-guest/sev-guest.h
index b2a97778e635..c823a782739f 100644
--- a/drivers/virt/coco/sev-guest/sev-guest.h
+++ b/drivers/virt/coco/sev-guest/sev-guest.h
@@ -11,6 +11,10 @@ struct snp_guest_dev {
 	struct miscdevice misc;
 
 	struct snp_msg_desc *msg_desc;
+
+	struct tsm_dev *tsmdev;
 };
 
+void sev_guest_tsm_set_ops(bool set, struct snp_guest_dev *snp_dev);
+
 #endif /* __SEV_GUEST_H__ */
diff --git a/drivers/virt/coco/sev-guest/core.c b/drivers/virt/coco/sev-guest/core.c
index 98f698809720..8448c123ab1e 100644
--- a/drivers/virt/coco/sev-guest/core.c
+++ b/drivers/virt/coco/sev-guest/core.c
@@ -45,6 +45,10 @@ static int vmpck_id = -1;
 module_param(vmpck_id, int, 0444);
 MODULE_PARM_DESC(vmpck_id, "The VMPCK ID to use when communicating with the PSP.");
 
+static bool tsm_enable = true;
+module_param(tsm_enable, bool, 0644);
+MODULE_PARM_DESC(tsm_enable, "Enable SEV TIO");
+
 static inline struct snp_guest_dev *to_snp_dev(struct file *file)
 {
 	struct miscdevice *dev = file->private_data;
@@ -668,6 +672,14 @@ static int __init sev_guest_probe(struct platform_device *pdev)
 	snp_dev->msg_desc = mdesc;
 	dev_info(dev, "Initialized SEV guest driver (using VMPCK%d communication key)\n",
 		 mdesc->vmpck_id);
+
+	u64 *sev_hv_features_ptr = dev_get_platdata(&pdev->dev);
+	if (!sev_hv_features_ptr || !(*sev_hv_features_ptr & GHCB_HV_FT_SNP_SEV_TIO))
+		tsm_enable = false;
+
+	if (tsm_enable)
+		sev_guest_tsm_set_ops(true, snp_dev);
+
 	return 0;
 
 e_msg_init:
@@ -681,6 +693,8 @@ static void __exit sev_guest_remove(struct platform_device *pdev)
 	struct snp_guest_dev *snp_dev = platform_get_drvdata(pdev);
 
 	snp_msg_free(snp_dev->msg_desc);
+	if (tsm_enable)
+		sev_guest_tsm_set_ops(false, snp_dev);
 	misc_deregister(&snp_dev->misc);
 }
 
diff --git a/drivers/virt/coco/sev-guest/tio.c b/drivers/virt/coco/sev-guest/tio.c
new file mode 100644
index 000000000000..99ff2b9c872a
--- /dev/null
+++ b/drivers/virt/coco/sev-guest/tio.c
@@ -0,0 +1,347 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/bitops.h>
+#include <linux/minmax.h>
+#include <linux/pci.h>
+#include <linux/psp-sev.h>
+#include <linux/tsm.h>
+#include <linux/pci-tsm.h>
+#include <crypto/gcm.h>
+#include <uapi/linux/sev-guest.h>
+
+#include <asm/svm.h>
+#include <asm/sev.h>
+
+#include "sev-guest.h"
+
+ulong tsm_vtom = (2ULL << 40);
+module_param(tsm_vtom, ulong, 0644);
+MODULE_PARM_DESC(tsm_vtom, "SEV TIO vTOM value");
+
+#define tsm_dev_to_snp_dev(t)	((struct snp_guest_dev *)dev_get_drvdata((t)->dev.parent))
+#define pdev_to_tdi(p)		container_of((p)->tsm, struct tio_guest_tdi, ds.base_tsm)
+#define ghcb_tio_sbdfn(pdev)	((pci_domain_nr((pdev)->bus) << 16) | pci_dev_id(pdev))
+
+struct tio_guest_tdi {
+	struct pci_tsm_devsec ds;
+	struct snp_guest_dev *snp_dev;
+	u64 tdi_id; /* Runtime FW generated TDI id */
+};
+
+static int handle_tio_guest_request(struct snp_guest_dev *snp_dev, u8 type,
+				    void *req_buf, size_t req_sz, void *resp_buf, u32 resp_sz,
+				    u64 *bdfn, u64 *param, u64 *fw_err)
+{
+	struct snp_msg_desc *mdesc = snp_dev->msg_desc;
+	struct snp_guest_req req = {
+		.msg_version = 2,
+		.msg_type = type,
+		.vmpck_id = mdesc->vmpck_id,
+		.req_buf = kmemdup(req_buf, req_sz, GFP_KERNEL),
+		.req_sz = req_sz,
+		.resp_buf = kmalloc(resp_sz, GFP_KERNEL),
+		.resp_sz = resp_sz,
+		.exit_code = SVM_VMGEXIT_SEV_TIO_GR,
+		.input.guest_rid = 0,
+		.input.param = 0,
+	};
+	int ret;
+
+	if (!req.req_buf || !req.resp_buf) {
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	if (bdfn) {
+		req.input.guest_rid = *bdfn & 0xFFFFFFFF;
+		req.input.npages = *bdfn >> 32;
+	}
+	req.input.param = *param;
+
+	ret = snp_send_guest_request(mdesc, &req);
+
+	memcpy(resp_buf, req.resp_buf, resp_sz);
+	*param = req.input.param;
+	*fw_err = req.exitinfo2;
+
+error_exit:
+	kfree(req.resp_buf);
+	kfree(req.req_buf);
+
+	return ret;
+}
+
+struct tio_msg_tdi_info_req {
+	u64 tdi_id;
+	u8 reserved[8];
+} __packed;
+
+enum {
+	TIO_MSG_TDI_INFO_RSP_STATUS_BOUND = 0,
+	TIO_MSG_TDI_INFO_RSP_STATUS_INVALID = 1,
+	TIO_MSG_TDI_INFO_RSP_STATUS_UNBOUND = 2,
+};
+
+struct tio_msg_tdi_info_rsp {
+	u64 tdi_id;
+	u16 status; /* TIO_MSG_TDI_INFO_RSP_STATUS_xxx */
+	u8 reserved1[6];
+
+	u32 meas_digest_valid:1;
+	u32 meas_digest_fresh:1;
+	u32 reserved2:30;
+
+	/* These are TDISP's LOCK_INTERFACE_REQUEST flags */
+	u32 no_fw_update:1;
+	u32 cache_line_size:1;
+	u32 lock_msix:1;
+	u32 bind_p2p:1;
+	u32 all_request_redirect:1;
+	u32 reserved3:27;
+
+	u64 spdm_algos;
+	u8 certs_digest[48];
+	u8 meas_digest[48];
+	u8 interface_report_digest[48];
+	u64 tdi_report_count;
+	u64 reserved4;
+} __packed;
+
+struct sdte {
+	u64 v                  : 1;
+	u64 reserved           : 3;
+	u64 cxlio              : 3;
+	u64 reserved1          : 45;
+	u64 ppr                : 1;
+	u64 reserved2          : 1;
+	u64 giov               : 1;
+	u64 gv                 : 1;
+	u64 glx                : 2;
+	u64 gcr3_tbl_rp0       : 3;
+	u64 ir                 : 1;
+	u64 iw                 : 1;
+	u64 reserved3          : 1;
+	u16 domain_id;
+	u16 gcr3_tbl_rp1;
+	u32 interrupt          : 1;
+	u32 reserved4          : 5;
+	u32 ex                 : 1;
+	u32 sd                 : 1;
+	u32 reserved5          : 2;
+	u32 sats               : 1;
+	u32 gcr3_tbl_rp2       : 21;
+	u64 giv                : 1;
+	u64 gint_tbl_len       : 4;
+	u64 reserved6          : 1;
+	u64 gint_tbl           : 46;
+	u64 reserved7          : 2;
+	u64 gpm                : 2;
+	u64 reserved8          : 3;
+	u64 hpt_mode           : 1;
+	u64 reserved9          : 4;
+	u32 asid               : 12;
+	u32 reserved10         : 3;
+	u32 viommu_en          : 1;
+	u32 guest_device_id    : 16;
+	u32 guest_id           : 15;
+	u32 guest_id_mbo       : 1;
+	u32 reserved11         : 1;
+	u32 vmpl               : 2;
+	u32 reserved12         : 3;
+	u32 attrv              : 1;
+	u32 reserved13         : 1;
+	u32 sa                 : 8;
+	u8 ide_stream_id[8];
+	u32 vtom_en            : 1;
+	u32 vtom               : 31;
+	u32 rp_id              : 5;
+	u32 reserved14         : 27;
+	u8  reserved15[0x40-0x30];
+} __packed;
+
+struct tio_msg_sdte_write_req {
+	u64 tdi_id;
+	u8 reserved[8];
+	struct sdte sdte;
+} __packed;
+
+/*
+ * Status codes from TIO_MSG_SDTE_WRITE_REQ
+ */
+enum sdte_write_status {
+	SDTE_WRITE_SUCCESS = 0,
+	SDTE_WRITE_INVALID_TDI = 1,
+	SDTE_WRITE_TDI_NOT_BOUND = 2,
+	SDTE_WRITE_RESERVED = 3,
+};
+
+struct tio_msg_sdte_write_rsp {
+	u64 tdi_id;
+	u16 status; /* SDTE_WRITE_xxx */
+	u8 reserved[6];
+} __packed;
+
+static int tio_tdi_sdte_write(struct pci_dev *pdev, struct snp_guest_dev *snp_dev,
+			      uint64_t tdi_id, u64 vtom, bool invalidate)
+{
+	size_t resp_len = sizeof(struct tio_msg_sdte_write_rsp) + AUTHTAG_LEN;
+	struct tio_msg_sdte_write_rsp *rsp __free(kfree_sensitive) = kzalloc(resp_len, GFP_KERNEL);
+	struct tio_msg_sdte_write_req req;
+	u64 flags = vtom | (invalidate ? 0 : SVM_VMGEXIT_SEV_TIO_GR_SDTE_VALIDATE);
+	u64 bdfn = ghcb_tio_sbdfn(pdev);
+	u64 fw_err = 0;
+	int rc;
+
+	BUILD_BUG_ON(sizeof(struct sdte) * 8 != 512);
+
+	pci_notice(pdev, "SDTE write vTOM=%llx", flags);
+
+	if (!invalidate)
+		req = (struct tio_msg_sdte_write_req) {
+			.tdi_id = tdi_id,
+			.sdte.vmpl = 0,
+			.sdte.vtom = vtom >> 21,
+			.sdte.vtom_en = 1,
+			.sdte.iw = 1,
+			.sdte.ir = 1,
+			.sdte.v = 1,
+		};
+	else
+		req = (struct tio_msg_sdte_write_req) {
+			.tdi_id = tdi_id,
+		};
+
+	if (!rsp)
+		return -ENOMEM;
+
+	rc = handle_tio_guest_request(snp_dev, TIO_MSG_SDTE_WRITE_REQ,
+				      &req, sizeof(req), rsp, resp_len,
+				      &bdfn, &flags, &fw_err);
+	if (rc || fw_err || rsp->status != SDTE_WRITE_SUCCESS) {
+		pci_err(pdev, "SDTE write failed with rc=%d, fwerr=0x%llx, status=%x\n",
+			rc, fw_err, rsp->status);
+		if (!rc)
+			return -EFAULT;
+		return rc;
+	}
+
+	pdev->dev.archdata.cc_shared_dma_offset = invalidate ? 0 : vtom;
+
+	return 0;
+}
+
+static struct pci_tsm *sev_guest_lock(struct tsm_dev *tsmdev, struct pci_dev *pdev)
+{
+	struct tio_guest_tdi *gtdi __free(kfree) = kzalloc(sizeof(*gtdi), GFP_KERNEL);
+	u64 fw_err = 0, tdi_id = 0;
+	int rc;
+
+	if (!gtdi)
+		return ERR_PTR(-ENOMEM);
+
+	/* Enabling device tells the HV to register MMIO as memory slots */
+	rc = pci_enable_device_mem(pdev);
+	if (rc)
+		return ERR_PTR(rc);
+
+	rc = pci_tsm_devsec_constructor(pdev, &gtdi->ds, tsmdev);
+	if (rc)
+		return ERR_PTR(rc);
+
+	gtdi->snp_dev = tsm_dev_to_snp_dev(tsmdev);
+
+	rc = sev_tio_op(ghcb_tio_sbdfn(pdev), SVM_VMGEXIT_SEV_TIO_OP_BIND, &fw_err, &tdi_id);
+	if (rc) {
+		pci_err(pdev, "TDI bind CONFIG_LOCKED failed rc=%d fw=0x%llx\n",
+			rc, fw_err);
+		return ERR_PTR(rc);
+	}
+	pci_dbg(pdev, "New TDI ID=%llx\n", tdi_id);
+
+	struct device_evidence *evidence = device_evidence_create(0, HASH_ALGO_SHA384);
+	if (!evidence)
+		return ERR_PTR(-ENOMEM);
+	gtdi->ds.base_tsm.evidence = evidence;
+
+	gtdi->tdi_id = tdi_id;
+
+	return &no_free_ptr(gtdi)->ds.base_tsm;
+}
+
+static void sev_guest_unlock(struct pci_tsm *tsm)
+{
+	struct pci_dev *pdev = tsm->pdev;
+	u64 fw_err = 0;
+
+	sev_tio_op(ghcb_tio_sbdfn(pdev), SVM_VMGEXIT_SEV_TIO_OP_UNBIND, &fw_err, NULL);
+
+	/* Quiesce DMA */
+	sev_tio_op(ghcb_tio_sbdfn(pdev), SVM_VMGEXIT_SEV_TIO_OP_STOP, &fw_err, NULL);
+
+	tsm->pdev->tsm = NULL;
+	kvfree(tsm);
+}
+
+static int sev_guest_accept(struct pci_dev *pdev)
+{
+	struct pci_tsm *tsm = pdev->tsm;
+	u64 fw_err = 0;
+
+	if (!tsm->evidence->obj[DEVICE_EVIDENCE_TYPE_REPORT].data) {
+		pci_warn_once(pdev, "Cannot accept without the report");
+		return -ENODEV;
+	}
+
+	int ret = tio_tdi_sdte_write(pdev, snp_dev, gtdi->tdi_id, 0, false); // Mark everything "shared"
+	if (ret)
+		return ret;
+
+	return sev_tio_op(ghcb_tio_sbdfn(pdev), SVM_VMGEXIT_SEV_TIO_OP_RUN, &fw_err, NULL);
+}
+
+static int sev_guest_enable_dma(struct pci_dev *pdev)
+{
+	struct tio_guest_tdi *gtdi = pdev_to_tdi(pdev);
+	struct snp_guest_dev *snp_dev = gtdi->snp_dev;
+	int ret;
+
+	ret = tio_tdi_sdte_write(pdev, snp_dev, gtdi->tdi_id, tsm_vtom, false);
+
+	return ret;
+}
+
+static void sev_guest_disable_dma(struct pci_dev *pdev)
+{
+	struct tio_guest_tdi *gtdi = pdev_to_tdi(pdev);
+	struct snp_guest_dev *snp_dev = gtdi->snp_dev;
+	int rc;
+
+	rc = tio_tdi_sdte_write(pdev, snp_dev, gtdi->tdi_id, tsm_vtom, true);
+	if (rc)
+		pr_err("SDTE_WRITE failed, ret=%d\n", rc);
+}
+
+struct pci_tsm_ops sev_guest_tsm_ops = {
+	.lock = sev_guest_lock,
+	.unlock = sev_guest_unlock,
+	.run = sev_guest_accept,
+	.enable_dma = sev_guest_enable_dma,
+	.disable_dma = sev_guest_disable_dma,
+};
+
+void sev_guest_tsm_set_ops(bool set, struct snp_guest_dev *snp_dev)
+{
+	if (set) {
+		struct tsm_dev *tsmdev;
+
+		tsmdev = tsm_register(snp_dev->dev, &sev_guest_tsm_ops);
+		if (!IS_ERR_OR_NULL(tsmdev))
+			snp_dev->tsmdev = tsmdev;
+		return;
+	}
+
+	if (snp_dev->tsmdev) {
+		tsm_unregister(snp_dev->tsmdev);
+		snp_dev->tsmdev = NULL;
+	}
+}
-- 
2.55.0

[RFC PATCH kernel 16/17] x86/sev: Enable secure MMIO (phase2)

[Alexey Kardashevskiy]

In order to enable secure MMIO, RMP needs to be updated. Unlike RAM
(where PVALIDATE validate the RMP entry), MMIO RMP entries require
assistance from the PSP.

Add TDI_INFO TIO guest request command. Request the TDI interface report
from the PSP. Use the report to validate MMIO ranges reported as non-NonTEE
(i.e. encrypted). The TSM subsystem notifies the PCI subsystem to
automatically map validated MMIO ranges as encrypted.

Since for MMIO the TSM driver needs TDISP report, add shared memory buffers
for receiving the device evidence. Use DMA SWIOTLB for these buffers,
explicitly, by forcing 32bit DMA mask.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 include/uapi/linux/sev-guest.h     |  12 +
 drivers/virt/coco/sev-guest/core.c |   3 +-
 drivers/virt/coco/sev-guest/tio.c  | 394 +++++++++++++++++++-
 3 files changed, 400 insertions(+), 9 deletions(-)

diff --git a/include/uapi/linux/sev-guest.h b/include/uapi/linux/sev-guest.h
index fcdfea767fca..28db79886ca5 100644
--- a/include/uapi/linux/sev-guest.h
+++ b/include/uapi/linux/sev-guest.h
@@ -13,6 +13,7 @@
 #define __UAPI_LINUX_SEV_GUEST_H_
 
 #include <linux/types.h>
+#include <linux/uuid.h>
 
 #define SNP_REPORT_USER_DATA_SIZE 64
 
@@ -96,4 +97,15 @@ struct snp_ext_report_req {
 #define SNP_GUEST_VMM_ERR_INVALID_LEN	1
 #define SNP_GUEST_VMM_ERR_BUSY		2
 
+/* Optional Certificates/measurements/report data from TIO_GUEST_REQUEST */
+struct tio_blob_table_entry {
+	guid_t guid;
+	__u32 offset;
+	__u32 length;
+} __packed;
+
+/* Attestation report: 70dc5b0e-0cc0-4cd5-97bb-ff0ba25bf320 */
+#define TIO_GUID_REPORT \
+	GUID_INIT(0x70dc5b0e, 0x0cc0, 0x4cd5, 0x97, 0xbb, 0xff, 0x0b, 0xa2, 0x5b, 0xf3, 0x20)
+
 #endif /* __UAPI_LINUX_SEV_GUEST_H_ */
diff --git a/drivers/virt/coco/sev-guest/core.c b/drivers/virt/coco/sev-guest/core.c
index 8448c123ab1e..8710c41daed6 100644
--- a/drivers/virt/coco/sev-guest/core.c
+++ b/drivers/virt/coco/sev-guest/core.c
@@ -23,6 +23,7 @@
 #include <linux/uuid.h>
 #include <linux/configfs.h>
 #include <linux/mm.h>
+#include <linux/dma-mapping.h>
 #include <uapi/linux/sev-guest.h>
 #include <uapi/linux/psp-sev.h>
 
@@ -677,7 +678,7 @@ static int __init sev_guest_probe(struct platform_device *pdev)
 	if (!sev_hv_features_ptr || !(*sev_hv_features_ptr & GHCB_HV_FT_SNP_SEV_TIO))
 		tsm_enable = false;
 
-	if (tsm_enable)
+	if (tsm_enable && !dma_set_mask(&pdev->dev, DMA_BIT_MASK(32)))
 		sev_guest_tsm_set_ops(true, snp_dev);
 
 	return 0;
diff --git a/drivers/virt/coco/sev-guest/tio.c b/drivers/virt/coco/sev-guest/tio.c
index 99ff2b9c872a..06addbb3a9ad 100644
--- a/drivers/virt/coco/sev-guest/tio.c
+++ b/drivers/virt/coco/sev-guest/tio.c
@@ -22,15 +22,71 @@ MODULE_PARM_DESC(tsm_vtom, "SEV TIO vTOM value");
 #define pdev_to_tdi(p)		container_of((p)->tsm, struct tio_guest_tdi, ds.base_tsm)
 #define ghcb_tio_sbdfn(pdev)	((pci_domain_nr((pdev)->bus) << 16) | pci_dev_id(pdev))
 
+#define TIO_DATA_PAGES	(SZ_32K >> PAGE_SHIFT)
+#define SPDM_MEASUREMENTS_NONCE_LEN 32
+
+static void sev_free_shared_pages(struct device *dev, void *buf,
+				  unsigned long npages, dma_addr_t dma_handle)
+{
+	dma_free_coherent(dev, npages << PAGE_SHIFT, buf, dma_handle);
+}
+
+static void *sev_alloc_shared_pages(struct device *dev, unsigned long npages,
+				    dma_addr_t *dma_handle)
+{
+	return dma_alloc_coherent(dev, npages << PAGE_SHIFT, dma_handle, GFP_KERNEL);
+}
+
 struct tio_guest_tdi {
 	struct pci_tsm_devsec ds;
 	struct snp_guest_dev *snp_dev;
 	u64 tdi_id; /* Runtime FW generated TDI id */
 };
 
+static void device_evidence_object_clear(struct device_evidence_object *obj)
+{
+	if (!obj)
+		return;
+
+	kfree(obj->digest);
+	kfree(obj->data);
+	obj->data = NULL;
+	obj->len = 0;
+	obj->digest = NULL;
+}
+
+static int device_evidence_object_assign(struct device_evidence_object *obj,
+					 const void *src, size_t len)
+{
+	void *copy;
+
+	device_evidence_object_clear(obj);
+	if (!len || !src)
+		return 0;
+
+	copy = kmemdup(src, len, GFP_KERNEL);
+	if (!copy)
+		return -ENOMEM;
+
+	obj->data = copy;
+	obj->len = len;
+	return 0;
+}
+
+static void device_evidence_release(struct device_evidence *evidence)
+{
+	unsigned int i;
+
+	if (!evidence)
+		return;
+
+	for (i = 0; i <= DEVICE_EVIDENCE_TYPE_MAX; i++)
+		device_evidence_object_clear(&evidence->obj[i]);
+}
+
 static int handle_tio_guest_request(struct snp_guest_dev *snp_dev, u8 type,
 				    void *req_buf, size_t req_sz, void *resp_buf, u32 resp_sz,
-				    u64 *bdfn, u64 *param, u64 *fw_err)
+				    void *pt, u64 *npages, u64 *bdfn, u64 *param, u64 *fw_err)
 {
 	struct snp_msg_desc *mdesc = snp_dev->msg_desc;
 	struct snp_guest_req req = {
@@ -52,6 +108,10 @@ static int handle_tio_guest_request(struct snp_guest_dev *snp_dev, u8 type,
 		goto error_exit;
 	}
 
+	if (pt && npages) {
+		req.certs_data = pt;
+		req.input.data_npages = *npages;
+	}
 	if (bdfn) {
 		req.input.guest_rid = *bdfn & 0xFFFFFFFF;
 		req.input.npages = *bdfn >> 32;
@@ -71,6 +131,69 @@ static int handle_tio_guest_request(struct snp_guest_dev *snp_dev, u8 type,
 	return ret;
 }
 
+static int guest_request_tio_data(struct snp_guest_dev *snp_dev, u8 type,
+				  void *req_buf, size_t req_sz, void *resp_buf, u32 resp_sz,
+				  u64 bdfn, struct device_evidence_object *report,
+				  u64 *fw_err)
+{
+	u64 npages = TIO_DATA_PAGES, param = 0;
+	struct tio_blob_table_entry *pt;
+	dma_addr_t dh = 0;
+	int rc;
+
+	pt = sev_alloc_shared_pages(snp_dev->dev, TIO_DATA_PAGES, &dh);
+	if (!pt)
+		return -ENOMEM;
+
+	if (report)
+		param |= SVM_VMGEXIT_SEV_TIO_GR_INFO_REPORT;
+
+	rc = handle_tio_guest_request(snp_dev, type, req_buf, req_sz, resp_buf, resp_sz,
+				      pt, &npages, &bdfn, &param, fw_err);
+	if (npages > TIO_DATA_PAGES) {
+		sev_free_shared_pages(snp_dev->dev, pt, TIO_DATA_PAGES, dh);
+		pt = sev_alloc_shared_pages(snp_dev->dev, npages, &dh);
+		if (!pt)
+			return -ENOMEM;
+
+		rc = handle_tio_guest_request(snp_dev, type, req_buf, req_sz, resp_buf, resp_sz,
+					      pt, &npages, &bdfn, &param, fw_err);
+	}
+	if (rc)
+		goto out_free_pt;
+
+	if (report)
+		device_evidence_object_clear(report);
+
+	for (unsigned int i = 0; i < 3; ++i) {
+		u8 *ptr = ((u8 *)pt) + pt[i].offset;
+		size_t len = pt[i].length;
+
+		if (guid_is_null(&pt[i].guid))
+			break;
+
+		if (!len)
+			continue;
+
+		if (guid_equal(&pt[i].guid, &TIO_GUID_REPORT) && report)
+			rc = device_evidence_object_assign(report, ptr, len);
+		else
+			continue;
+		if (rc)
+			goto out_clear_blobs;
+	}
+	sev_free_shared_pages(snp_dev->dev, pt, npages, dh);
+
+	return 0;
+
+out_clear_blobs:
+	if (report)
+		device_evidence_object_clear(report);
+out_free_pt:
+	sev_free_shared_pages(snp_dev->dev, pt, npages, dh);
+	return rc;
+}
+
 struct tio_msg_tdi_info_req {
 	u64 tdi_id;
 	u8 reserved[8];
@@ -107,6 +230,224 @@ struct tio_msg_tdi_info_rsp {
 	u64 reserved4;
 } __packed;
 
+static int tio_tdi_status(struct pci_dev *pdev, struct snp_guest_dev *snp_dev,
+			  struct tsm_tdi_status *ts, uint64_t tdi_id,
+			  struct device_evidence_object *report)
+{
+	size_t resp_len = sizeof(struct tio_msg_tdi_info_rsp) + AUTHTAG_LEN;
+	struct tio_msg_tdi_info_rsp *rsp __free(kfree_sensitive) = kzalloc(resp_len, GFP_KERNEL);
+	struct tio_msg_tdi_info_req req = {
+		.tdi_id = tdi_id,
+	};
+	u64 fw_err = 0;
+	int rc;
+
+	pci_notice(pdev, "TDI info");
+	if (!rsp)
+		return -ENOMEM;
+
+	rc = guest_request_tio_data(snp_dev, TIO_MSG_TDI_INFO_REQ, &req,
+				    sizeof(req), rsp, resp_len,
+				    ghcb_tio_sbdfn(pdev), report, &fw_err);
+	if (rc)
+		return rc;
+
+	ts->tdi_id = rsp->tdi_id;
+
+	return 0;
+}
+
+struct tio_msg_mmio_validate_req {
+	u64 tdi_id;
+	u8 reserved2[8];
+	u64 subrange_base;
+	u32 subrange_page_count;
+	u32 range_offset;
+
+	u16 validated:1; /* Desired value to set RMP.Validated for the range */
+	/*
+	 * Force validated:
+	 * 0: If subrange does not have RMP.Validated set uniformly, fail.
+	 * 1: If subrange does not have RMP.Validated set uniformly, force
+	 *    to requested value
+	 */
+	u16 force_validated:1;
+	u16 reserved3:14;
+
+	u16 range_id;
+	u8 reserved4[12];
+} __packed;
+
+/* Status codes from TIO_MSG_MMIO_VALIDATE_REQ */
+enum mmio_validate_status {
+	MMIO_VALIDATE_SUCCESS = 0,
+	MMIO_VALIDATE_INVALID_TDI = 1,
+	MMIO_VALIDATE_TDI_UNBOUND = 2,
+	MMIO_VALIDATE_NOT_ASSIGNED = 3, /* At least one page is not assigned to the guest */
+	MMIO_VALIDATE_NOT_IO = 4,	/* At least one page is not an I/O page */
+	MMIO_VALIDATE_NOT_UNIFORM = 5,  /* Validated bit is not uniformly set for range */
+	MMIO_VALIDATE_NOT_IMMUTABLE = 6,/* >=1 page does not have immutable bit set */
+	MMIO_VALIDATE_NOT_MAPPED = 7,   /* At least one page is not mapped to the expected GPA */
+	MMIO_VALIDATE_NOT_REPORTED = 8, /* Range ID is not reported in TDI report */
+	MMIO_VALIDATE_OUT_OF_RANGE = 9, /* Subrange is out the MMIO range in TDI report */
+	MMIO_VALIDATE_NOT_4K = 10,	/* >=1 page is not 4K page size */
+};
+
+struct tio_msg_mmio_validate_rsp {
+	u64 tdi_id;
+	u16 status; /* MMIO_VALIDATE_xxx */
+	u8 reserved1[6];
+	u64 subrange_base;
+	u32 subrange_page_count;
+	u32 range_offset;
+
+	u16 changed:1; /* Validated bit has changed due to this operation */
+	u16 reserved2:15;
+
+	u16 range_id;
+	u8 reserved3[12];
+} __packed;
+
+static int mmio_validate_range(struct snp_guest_dev *snp_dev, struct pci_dev *pdev,
+			       uint64_t tdi_id, unsigned int range_id,
+			       resource_size_t start, resource_size_t size,
+			       bool invalidate, u64 *fw_err, u16 *status)
+{
+	size_t resp_len = sizeof(struct tio_msg_mmio_validate_rsp) + AUTHTAG_LEN;
+	struct tio_msg_mmio_validate_rsp *rsp __free(kfree_sensitive) =
+			kzalloc(resp_len, GFP_KERNEL);
+	struct tio_msg_mmio_validate_req req = {
+		.tdi_id = tdi_id,
+		.subrange_base = start >> 12,
+		.subrange_page_count = size >> 12,
+		.range_offset = 0,
+		.validated = !invalidate, /* Desired value to set RMP.Validated for the range */
+		.force_validated = 0,
+		.range_id = range_id,
+	};
+	u64 num_bdfn = SVM_VMGEXIT_SEV_TIO_GR_MMIO_MK_NUM_BDFN(size >> 12, ghcb_tio_sbdfn(pdev));
+	u64 mmio_val = SVM_VMGEXIT_SEV_TIO_GR_MMIO_MK_VALIDATE(start, !invalidate);
+	int rc;
+
+	if (!rsp)
+		return -ENOMEM;
+
+	rc = handle_tio_guest_request(snp_dev, TIO_MSG_MMIO_VALIDATE_REQ,
+				      &req, sizeof(req), rsp, resp_len,
+				      NULL, NULL, &num_bdfn, &mmio_val, fw_err);
+	if (rc || *fw_err || rsp->status != MMIO_VALIDATE_SUCCESS) {
+		pci_err(pdev, "MMIO validate failed with rc=%d, fwerr=0x%llx, status=%x\n",
+			rc, *fw_err, rsp->status);
+		if (!rc)
+			return -EFAULT;
+		return rc;
+	}
+
+	*status = rsp->status;
+
+	return 0;
+}
+
+static void tio_tdi_mmio_invalidate(struct pci_dev *pdev, struct snp_guest_dev *snp_dev,
+				    uint64_t tdi_id)
+{
+	struct pci_tsm *tsm = pdev->tsm;
+	u16 mmio_status;
+	u64 fw_err = 0;
+	int i = 0, rc = 0;
+	struct pci_tsm_devsec *devsec_tsm = to_pci_tsm_devsec(tsm);
+	struct pci_tsm_mmio *mmio = devsec_tsm->mmio;
+
+	if (!mmio)
+		return;
+
+	pci_notice(pdev, "MMIO invalidate");
+
+	for (i = 0; i < mmio->nr; ++i) {
+		struct pci_tsm_mmio_entry *entry = pci_tsm_mmio_entry(mmio, i);
+		struct resource *res = &entry->res;
+		unsigned int range_id = entry->range_id;
+
+		if (range_id >= PCI_NUM_RESOURCES ||
+		    !resource_contains(pci_resource_n(pdev, range_id), res)) {
+			pci_info(pdev, "Skipping MMIO [%d] %pr: no BAR %u window\n",
+				 i, res, range_id);
+			continue;
+		}
+
+		mmio_status = 0;
+		rc = mmio_validate_range(snp_dev, pdev, tdi_id, range_id,
+					 res->start, resource_size(res), true, &fw_err,
+					 &mmio_status);
+		if (rc || fw_err != SEV_RET_SUCCESS || mmio_status != MMIO_VALIDATE_SUCCESS) {
+			pci_err(pdev, "MMIO #%d %llx..%llx validation failed 0x%llx %d\n",
+				range_id, res->start, res->end, fw_err, mmio_status);
+			continue;
+		}
+
+		pci_notice(pdev, "MMIO #%d %llx..%llx invalidated\n",
+			   range_id, res->start, res->end);
+	}
+
+	pci_tsm_mmio_teardown(devsec_tsm->mmio);
+	kfree(devsec_tsm->mmio);
+	devsec_tsm->mmio = NULL;
+}
+
+static int tio_tdi_mmio_validate(struct pci_dev *pdev, struct snp_guest_dev *snp_dev,
+				 uint64_t tdi_id)
+{
+	struct pci_tsm *tsm = pdev->tsm;
+	u16 mmio_status;
+	u64 fw_err = 0;
+	int i, rc = 0;
+	struct pci_tsm_mmio *mmio __free(kfree) = pci_tsm_mmio_alloc(pdev);
+
+	if (!mmio)
+		return -ENOMEM;
+
+	pci_notice(pdev, "MMIO validate");
+
+	for (i = 0; i < mmio->nr; ++i) {
+		struct pci_tsm_mmio_entry *entry = pci_tsm_mmio_entry(mmio, i);
+		struct resource *res = &entry->res;
+		unsigned int range_id = entry->range_id;
+
+		if (range_id >= PCI_NUM_RESOURCES ||
+		    !resource_contains(pci_resource_n(pdev, range_id), res)) {
+			pci_info(pdev,
+				 "Skipping MMIO [%d] %pr: no BAR %u window\n",
+				 i, res, range_id);
+			continue;
+		}
+
+		mmio_status = 0;
+		rc = mmio_validate_range(snp_dev, pdev, tdi_id, range_id, res->start,
+					 resource_size(res), false, &fw_err, &mmio_status);
+		if (rc || fw_err != SEV_RET_SUCCESS || mmio_status != MMIO_VALIDATE_SUCCESS) {
+			pci_err(pdev, "MMIO #%d %llx..%llx validation failed 0x%llx %d\n",
+				range_id, res->start, res->end, fw_err, mmio_status);
+			continue;
+		}
+
+		pci_notice(pdev, "MMIO #%d %llx..%llx validated\n", range_id, res->start, res->end);
+	}
+
+	if (!rc) {
+		rc = pci_tsm_mmio_setup(pdev, mmio);
+		if (!rc) {
+			struct pci_tsm_devsec *devsec_tsm = to_pci_tsm_devsec(tsm);
+
+			devsec_tsm->mmio = no_free_ptr(mmio);
+		}
+	}
+
+	if (rc)
+		tio_tdi_mmio_invalidate(pdev, snp_dev, tdi_id);
+
+	return rc;
+}
+
 struct sdte {
 	u64 v                  : 1;
 	u64 reserved           : 3;
@@ -216,7 +557,7 @@ static int tio_tdi_sdte_write(struct pci_dev *pdev, struct snp_guest_dev *snp_de
 
 	rc = handle_tio_guest_request(snp_dev, TIO_MSG_SDTE_WRITE_REQ,
 				      &req, sizeof(req), rsp, resp_len,
-				      &bdfn, &flags, &fw_err);
+				      NULL, NULL, &bdfn, &flags, &fw_err);
 	if (rc || fw_err || rsp->status != SDTE_WRITE_SUCCESS) {
 		pci_err(pdev, "SDTE write failed with rc=%d, fwerr=0x%llx, status=%x\n",
 			rc, fw_err, rsp->status);
@@ -233,6 +574,7 @@ static int tio_tdi_sdte_write(struct pci_dev *pdev, struct snp_guest_dev *snp_de
 static struct pci_tsm *sev_guest_lock(struct tsm_dev *tsmdev, struct pci_dev *pdev)
 {
 	struct tio_guest_tdi *gtdi __free(kfree) = kzalloc(sizeof(*gtdi), GFP_KERNEL);
+	struct tsm_tdi_status ts = {};
 	u64 fw_err = 0, tdi_id = 0;
 	int rc;
 
@@ -258,11 +600,21 @@ static struct pci_tsm *sev_guest_lock(struct tsm_dev *tsmdev, struct pci_dev *pd
 	}
 	pci_dbg(pdev, "New TDI ID=%llx\n", tdi_id);
 
-	struct device_evidence *evidence = device_evidence_create(0, HASH_ALGO_SHA384);
-	if (!evidence)
+	struct device_evidence *ev = device_evidence_create(0, HASH_ALGO_SHA384);
+	if (!ev)
 		return ERR_PTR(-ENOMEM);
-	gtdi->ds.base_tsm.evidence = evidence;
 
+	rc = tio_tdi_status(pdev, gtdi->snp_dev, &ts, tdi_id,
+			    &ev->obj[DEVICE_EVIDENCE_TYPE_REPORT]);
+	if (rc)
+		return ERR_PTR(rc);
+
+	if (!ev->obj[DEVICE_EVIDENCE_TYPE_REPORT].data) {
+		device_evidence_release(ev);
+		return ERR_PTR(-ENODEV);
+	}
+
+	gtdi->ds.base_tsm.evidence = ev;
 	gtdi->tdi_id = tdi_id;
 
 	return &no_free_ptr(gtdi)->ds.base_tsm;
@@ -271,19 +623,33 @@ static struct pci_tsm *sev_guest_lock(struct tsm_dev *tsmdev, struct pci_dev *pd
 static void sev_guest_unlock(struct pci_tsm *tsm)
 {
 	struct pci_dev *pdev = tsm->pdev;
+	struct tio_guest_tdi *gtdi = pdev_to_tdi(pdev);
+	struct snp_guest_dev *snp_dev = gtdi->snp_dev;
 	u64 fw_err = 0;
 
-	sev_tio_op(ghcb_tio_sbdfn(pdev), SVM_VMGEXIT_SEV_TIO_OP_UNBIND, &fw_err, NULL);
+	tio_tdi_mmio_invalidate(pdev, snp_dev, gtdi->tdi_id);
 
 	/* Quiesce DMA */
 	sev_tio_op(ghcb_tio_sbdfn(pdev), SVM_VMGEXIT_SEV_TIO_OP_STOP, &fw_err, NULL);
 
-	tsm->pdev->tsm = NULL;
+	/*
+	 * Up until now the VMM has been blocking clearing of BME and the device may
+	 * not be able to recover without BME going via 0, do it now.
+	 * Note that the device reset is still needed, leave to the userspace to
+	 * decide on that.
+	 */
+	pci_disable_device(pdev);
+
+	sev_tio_op(ghcb_tio_sbdfn(pdev), SVM_VMGEXIT_SEV_TIO_OP_UNBIND, &fw_err, NULL);
+
+	device_evidence_release(tsm->evidence);
 	kvfree(tsm);
 }
 
 static int sev_guest_accept(struct pci_dev *pdev)
 {
+	struct tio_guest_tdi *gtdi = pdev_to_tdi(pdev);
+	struct snp_guest_dev *snp_dev = gtdi->snp_dev;
 	struct pci_tsm *tsm = pdev->tsm;
 	u64 fw_err = 0;
 
@@ -296,7 +662,19 @@ static int sev_guest_accept(struct pci_dev *pdev)
 	if (ret)
 		return ret;
 
-	return sev_tio_op(ghcb_tio_sbdfn(pdev), SVM_VMGEXIT_SEV_TIO_OP_RUN, &fw_err, NULL);
+	ret = sev_tio_op(ghcb_tio_sbdfn(pdev), SVM_VMGEXIT_SEV_TIO_OP_RUN, &fw_err, NULL);
+	if (ret)
+		return ret;
+
+	ret = tio_tdi_mmio_validate(pdev, snp_dev, gtdi->tdi_id);
+	if (ret)
+		goto stop_tdi;
+
+	return 0;
+
+stop_tdi:
+	sev_tio_op(ghcb_tio_sbdfn(pdev), SVM_VMGEXIT_SEV_TIO_OP_STOP, &fw_err, NULL);
+	return ret;
 }
 
 static int sev_guest_enable_dma(struct pci_dev *pdev)
-- 
2.55.0

[RFC PATCH kernel 17/17] x86/sev: Flush IOMMU TLB for trusted devices

[Alexey Kardashevskiy]

IOMMU performs RMP checks when SNP is enabled, the results are
cached along with the IOMMU translations. When a VM lowers permission
of a mapped page (moves to a lower VMPL level or from read+write to
read-only or private to shared), the cached RMP check results require
invalidation.

At the moment the only way to invalidate IOMMU cache is the RMPUPDATE
instruction which flushes all IOMMU TLBs. It is a host privileged
instruction so a VM needs a way to ensure the host has done it.
Note that the guest's RMPADJUST/PVALIDATE do not flush IOMMU TLBs.

The host implements a new "IOMMU TLB Flush" VMGEXIT code which is
advertised via bit#11 in the GHCB Hypervisor capabilities.

Use RMPUPDATE in the following way:
- allocate a page per VCPU (to allow lockless flushing);
- When invalidation is needed, copy two patterns (A and B) to the page;
- invalidate the page so the host can make it shared;
- use new GHCB call to request RMPUPDATE on the host;
- the host makes the page shared;
- the host clears pattern A;
- the host makes the page private again;
- the host returns to the guest;
- check if pattern A has changed and pattern B has not;
- if the above failed, panic().

The patterns are located far enough to not hit the same cache line to
work with the cipher text hiding feature.

The host can choose to not execute the request, WARN_ON if this
is the case. Further patches will attempt to handle this in other way.

Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
---
 arch/x86/include/asm/sev-common.h |  2 +
 arch/x86/include/uapi/asm/svm.h   |  3 +
 arch/x86/coco/sev/core.c          | 92 ++++++++++++++++++++
 3 files changed, 97 insertions(+)

diff --git a/arch/x86/include/asm/sev-common.h b/arch/x86/include/asm/sev-common.h
index ff763c3c5d63..51abf8d061fa 100644
--- a/arch/x86/include/asm/sev-common.h
+++ b/arch/x86/include/asm/sev-common.h
@@ -138,6 +138,7 @@ enum psc_op {
 #define GHCB_HV_FT_SNP_AP_CREATION	BIT_ULL(1)
 #define GHCB_HV_FT_SNP_MULTI_VMPL	BIT_ULL(5)
 #define GHCB_HV_FT_SNP_SEV_TIO		BIT_ULL(7)
+#define GHCB_HV_FT_SNP_IOMMU_TLB_FLUSH	BIT_ULL(11)
 
 /*
  * SNP Page State Change NAE event
@@ -210,6 +211,7 @@ struct snp_psc_desc {
 #define GHCB_TERM_SECURE_TSC		10	/* Secure TSC initialization failed */
 #define GHCB_TERM_SVSM_CA_REMAP_FAIL	11	/* SVSM is present but CA could not be remapped */
 #define GHCB_TERM_SAVIC_FAIL		12	/* Secure AVIC-specific failure */
+#define GHCB_TERM_IOMMUTLB_FLUSH	13	/* IOMMUTLB flush failed for SEV-TIO device */
 
 #define GHCB_RESP_CODE(v)		((v) & GHCB_MSR_INFO_MASK)
 
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 93597ad492bf..269050942c8e 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -160,6 +160,8 @@
 #define SVM_VMGEXIT_SEV_TIO_OP_UNBIND	1
 #define SVM_VMGEXIT_SEV_TIO_OP_RUN	2
 #define SVM_VMGEXIT_SEV_TIO_OP_STOP	3
+#define SVM_VMGEXIT_IOMMU_TLB_FLUSH		0x80000022ull
+#define SVM_VMGEXIT_IOMMU_TLB_FLUSH_NO_ACTION	1
 #define SVM_VMGEXIT_HV_FEATURES			0x8000fffdull
 #define SVM_VMGEXIT_TERM_REQUEST		0x8000fffeull
 #define SVM_VMGEXIT_TERM_REASON(reason_set, reason_code)	\
@@ -285,6 +287,7 @@
 	{ SVM_VMGEXIT_AP_CREATION,	"vmgexit_ap_creation" }, \
 	{ SVM_VMGEXIT_SEV_TIO_GR,	"vmgexit_sev_tio_guest_request" }, \
 	{ SVM_VMGEXIT_SEV_TIO_OP,	"vmgexit_sev_tio_op" }, \
+	{ SVM_VMGEXIT_IOMMU_TLB_FLUSH, "vmgexit_sev_tio_iommu_tlb_flush" }, \
 	{ SVM_VMGEXIT_HV_FEATURES,	"vmgexit_hypervisor_feature" }, \
 	{ SVM_EXIT_ERR,         "invalid_guest_state" }
 
diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c
index ed0e4546d5e5..aa5a3abb4796 100644
--- a/arch/x86/coco/sev/core.c
+++ b/arch/x86/coco/sev/core.c
@@ -44,6 +44,7 @@
 #include <asm/cpuid/api.h>
 #include <asm/cmdline.h>
 #include <asm/msr.h>
+#include <asm/archrandom.h>
 
 #include "internal.h"
 
@@ -103,6 +104,36 @@ static unsigned long snp_tsc_freq_khz __ro_after_init;
 
 DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
 DEFINE_PER_CPU(struct sev_es_save_area *, sev_vmsa);
+DEFINE_PER_CPU(u8 *, iommu_tlb_flush_ghcb_page);
+static atomic_t sev_tio_devices_num;
+
+static int alloc_iommu_tlb_flush_ghcb_pages(void)
+{
+	unsigned int cpu;
+	struct page *pg;
+	void *p;
+
+	/*
+	 * Allocate per CPU pages while encrypted DMA is not happening yet
+	 * and smashing is cheap.
+	 */
+	for_each_possible_cpu(cpu) {
+		if (per_cpu(iommu_tlb_flush_ghcb_page, cpu))
+			continue;
+
+		pg = alloc_pages_node(cpu_to_node(cpu), GFP_KERNEL, 0);
+		if (!pg)
+			return -ENOMEM;
+
+		p = page_to_virt(pg);
+		/* Trigger psmash in the host os now to avoid psmash race later */
+		snp_set_memory_shared((unsigned long)p, 1);
+		snp_set_memory_private((unsigned long)p, 1);
+		per_cpu(iommu_tlb_flush_ghcb_page, cpu) = p;
+	}
+
+	return 0;
+}
 
 int sev_tio_op(u32 guest_rid, unsigned int op, u64 *fw_err, u64 *tdi_id)
 {
@@ -111,6 +142,24 @@ int sev_tio_op(u32 guest_rid, unsigned int op, u64 *fw_err, u64 *tdi_id)
 	struct ghcb *ghcb;
 	int ret;
 
+	if (!(sev_hv_features & GHCB_HV_FT_SNP_SEV_TIO))
+		return -EPERM;
+
+	if (op == SVM_VMGEXIT_SEV_TIO_OP_RUN || op == SVM_VMGEXIT_SEV_TIO_OP_STOP) {
+		if (!(sev_hv_features & GHCB_HV_FT_SNP_IOMMU_TLB_FLUSH))
+			return -EPERM;
+
+		if (op == SVM_VMGEXIT_SEV_TIO_OP_RUN) {
+			if (atomic_inc_return(&sev_tio_devices_num) == 1) {
+				ret = alloc_iommu_tlb_flush_ghcb_pages();
+				if (ret)
+					return ret;
+			}
+		} else if (atomic_dec_return(&sev_tio_devices_num) == 0) {
+			/* Do cleanup or leave it like this? */
+		}
+	}
+
 	/* __sev_get_ghcb() needs IRQs disabled because it uses per-CPU GHCB. */
 	guard(irqsave)();
 
@@ -347,6 +396,42 @@ static int vmgexit_psc(struct ghcb *ghcb, struct snp_psc_desc *desc)
 	return ret;
 }
 
+static int ghcb_flush_iommu_tlb(struct ghcb *ghcb)
+{
+	/* AES encrypts with 16 byte blocks */
+	unsigned long s1[BITS_TO_LONGS(128)], s2[BITS_TO_LONGS(128)];
+	void *p = this_cpu_read(iommu_tlb_flush_ghcb_page), *p2;
+	struct es_em_ctxt ctxt;
+	int ret;
+
+	if (!p)
+		return -ENOMEM;
+
+	/* Keep patterns apart far enough to not share the same cache line */
+	p2 = (u8 *) p + 2048;
+
+	vc_ghcb_invalidate(ghcb);
+
+	BUILD_BUG_ON(ARRAY_SIZE(s1) != 2);
+	if (!rdrand_long(s1) || !rdrand_long(s1 + 1) ||
+	    !rdrand_long(s2) || !rdrand_long(s2 + 1))
+		return -EFAULT;
+
+	memcpy(p, s1, sizeof(s1));
+	memcpy(p2, s2, sizeof(s2));
+
+	pvalidate((unsigned long) p, RMP_PG_SIZE_4K, false);
+	ret = sev_es_ghcb_hv_call(ghcb, &ctxt, SVM_VMGEXIT_IOMMU_TLB_FLUSH, __pa(p), 0);
+	pvalidate((unsigned long) p, RMP_PG_SIZE_4K, true);
+
+	/* Ensure that the host change is visible */
+	smp_mb();
+
+	if (!memcmp(p, s1, sizeof(s1)) || memcmp(p2, s2, sizeof(s2)))
+		return -EFAULT;
+
+	return 0;
+}
 static unsigned long __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr,
 				       unsigned long vaddr_end, int op)
 {
@@ -404,6 +489,13 @@ static unsigned long __set_pages_state(struct snp_psc_desc *data, unsigned long
 	if (!ghcb || vmgexit_psc(ghcb, data))
 		sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_PSC);
 
+	if (atomic_read(&sev_tio_devices_num)) {
+		int ret = ghcb_flush_iommu_tlb(ghcb);
+
+		if (ret)
+			sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_IOMMUTLB_FLUSH);
+	}
+
 	__sev_put_ghcb(&state);
 
 	local_irq_restore(flags);
-- 
2.55.0

Re: [RFC PATCH kernel 10/17] x86, dma: Allow accepted devices to map private memory

[Jason Gunthorpe]

On Wed, Sep 16, 2026 at 09:51:50PM +1000, Alexey Kardashevskiy wrote:
>  bool force_dma_unencrypted(struct device *dev)
>  {
>  	/*
> -	 * For SEV, all DMA must be to unencrypted addresses.
> +	 * Require unencrypted DMA unless the device has been "accepted",
> +	 * enabled by a TSM driver to DMA to private encrypted memory.
>  	 */
>  	if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
> -		return true;
> +		return !__device_cc_accepted(dev);

I want this out of arch code, after Aneesh's recent rework Nicolin is trying
this approach:

https://lore.kernel.org/all/abddd433493324a6bfdf8ba9710e96d90f0d1ccc.1789010941.git.nicolinc@nvidia.com/

Jason

Re: [RFC PATCH kernel 01/17] pci/dma/tsm: Call disable DMA bus hook on cleanup

[Borislav Petkov]

On Wed, Sep 16, 2026 at 09:51:41PM +1000, Alexey Kardashevskiy wrote:
> +	if (device_tcb_trusted(dev))

Where is that function and the rest of the gunk needed for me to apply before
I review those?

The patches on that branch:

https://github.com/AMDESE/linux-kvm/commits/tsm-next

which come before those here are enough?

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette