[PATCH v3 00/13] KVM: ITS hardening for pKVM

[PATCH v3 00/13] KVM: ITS hardening for pKVM

[Fuad Tabba]

Hi folks,

Changes since v2 [1]:
  - Redesigned: the takeover, the register handling, the command checks,
    and how the tables and ITTs are taken. The new design is described
    below, and each patch from v2 lists its changes below its ---.
  - Restructured the patches, merging and redoing some of them.

This is v3 of the ITS hardening series [1], which I'm carrying while
Seb is on leave. Much of the design and the code were redone, so it's
described whole here.

The ITS writes to the memory its command queue and tables point to, and
the host owns all of it, so host control of the ITS is a write
primitive into hypervisor and protected-VM memory.

When the hypervisor initialises, it takes ownership of each ITS's
control frame. Inside the stop_machine() that deprivileges the host, it
takes the command queue and tables the ITS registers point to, and the
ITTs of the devices already mapped. If that fails, no CPU is
deprivileged and KVM_CREATE_VM fails.

From then on, every host access to the control frame traps
(GITS_TRANSLATER stays host-mapped). The hypervisor checks each command
before copying it into the hardware queue, takes each ITT for as long
as its device is mapped, and writes each level-1 device table entry
only once. A command or register write it rejects aborts the host's
access.

This is the ITS part. The redistributor part follows as its own series.

A few points for discussion:

  - GICv4 is off for the host in protected mode (patch 3): the vLPI
    commands aren't emulated yet, and VMAPP's VPT_base is a host-chosen
    address the ITS and the redistributors write to. The host loses
    direct injection for its non-protected guests. The ITS part stands
    on its own, and what the host gives up is a feature that is off by
    default (kvm-arm.vgic_v4_enable).

  - The hypervisor's device tracker lives in the eight pages the driver
    gives it at setup, which hold about a thousand devices per ITS on a
    4K host. Past that, its_create_device() fails as it would on running
    out of LPIs. I've left growing it for when a platform needs more.

  - KVM doesn't initialise in protected mode on a host with more than
    eight ITSes, as each ITS takes one of the eight MMIO region slots
    (patch 9).

Tested on QEMU with a reproducer that writes to the ITS through /dev/mem
from the host's userspace, as a hostile host would. The hypervisor
rejects forged level-1 entries, out-of-range MAPTIs, MAPTIs queued after
their device's unmap and the register writes it doesn't allow, and
still forwards the commands queued before a rejected one.

A PCIe device hot-added after deprivilege, then unbound and rebound,
exercises publishing a level-1 entry and taking and releasing the ITT,
and is clean under PROVE_LOCKING. Hosts with 16K or 64K pages, or with
GICv3 and no GICv4, boot and run a non-protected guest, and
vgic_lpi_stress passes with and without pKVM.

In a debug build that caps the tracker, the driver fails the extra
device. In one that forces the takeover to fail, the host boots and
KVM_CREATE_VM fails. Not tested: hardware, two ITSes, and suspend and
resume (QEMU's PSCI has no SYSTEM_SUSPEND).

The series is structured as follows:

  01:     MMIO donation.
  02:     Taking host-registered MMIO regions at finalisation.
  03-06:  Driver preparation: GICv4 off, page-aligned ITTs, the wait
          target under the ITS lock, the shared command accessors.
  07:     Deprivileging the host under stop_machine().
  08-09:  The takeover: the driver's entry points, then the hypervisor
          trapping the control frame, taking the command queue and
          emulating GITS_CTLR and GITS_CBASER.
  10-11:  The tables and GITS_BASER<n>, then the ITTs.
  12:     Bounding the IDs that select what the ITS writes.
  13:     Documentation.

Based on v7.4-rc1.

Cheers,
/fuad

[1] https://lore.kernel.org/all/20260807164322.2970811-2-sebastianene@google.com/

Bartłomiej Grzesik (2):
  irqchip/gic-v3: Disable GICv4 features in protected KVM mode
  irqchip/gic-v3-its: Page-align ITT allocations under protected KVM

Fuad Tabba (7):
  irqchip/gic-v3-its: Compute the wait target under the ITS lock
  KVM: arm64: Share the ITS command field accessors with the hypervisor
  KVM: arm64: Deprivilege the host under stop_machine()
  KVM: arm64: Take the ITS tables and restrict GITS_BASER<n>
  KVM: arm64: Take the ITT for the lifetime of a device mapping
  KVM: arm64: Reject ITS commands whose IDs are out of range
  Documentation: KVM: arm64: Describe the ITS hardening

Mostafa Saleh (1):
  KVM: arm64: Donate MMIO to the hypervisor

Sebastian Ene (3):
  KVM: arm64: Take host-registered MMIO regions at pKVM finalisation
  irqchip/gic-v3-its: Add the protected-KVM takeover entry points
  KVM: arm64: Emulate the ITS command queue

 .../admin-guide/kernel-parameters.txt         |   8 +-
 Documentation/virt/kvm/arm/pkvm.rst           |  32 +
 arch/arm64/include/asm/kvm_asm.h              |   2 +
 arch/arm64/include/asm/kvm_hcall.h            |   2 +
 arch/arm64/include/asm/kvm_pgtable.h          |   2 +
 arch/arm64/include/asm/kvm_pkvm.h             |  16 +
 arch/arm64/kvm/arm.c                          |   4 +
 arch/arm64/kvm/hyp/include/nvhe/its_emulate.h |  16 +
 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h |   5 +
 arch/arm64/kvm/hyp/nvhe/Makefile              |   3 +-
 arch/arm64/kvm/hyp/nvhe/hyp-main.c            |   8 +
 arch/arm64/kvm/hyp/nvhe/its_emulate.c         | 777 ++++++++++++++++++
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         | 130 ++-
 arch/arm64/kvm/hyp/nvhe/setup.c               |  22 +
 arch/arm64/kvm/hyp/pgtable.c                  |   8 +-
 arch/arm64/kvm/pkvm.c                         | 105 ++-
 arch/arm64/kvm/vgic/vgic-its.c                |  15 -
 drivers/irqchip/irq-gic-common.h              |  13 +
 drivers/irqchip/irq-gic-v3-its.c              | 256 +++++-
 drivers/irqchip/irq-gic-v3.c                  |   8 +
 include/linux/irqchip/arm-gic-v3.h            |  60 ++
 21 files changed, 1439 insertions(+), 53 deletions(-)
 create mode 100644 arch/arm64/kvm/hyp/include/nvhe/its_emulate.h
 create mode 100644 arch/arm64/kvm/hyp/nvhe/its_emulate.c


base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.47.3

[PATCH v3 01/13] KVM: arm64: Donate MMIO to the hypervisor

[Fuad Tabba]

From: Mostafa Saleh <smostafa@google.com>

Add __pkvm_host_donate_hyp_mmio() so IOMMU drivers in the hypervisor can
take ownership of the IOMMU MMIO. The range is mapped in the hypervisor
private range and annotated as owned by the hypervisor in the host
stage-2, so host accesses fault and further donations of the same range
are refused. MMIO has no struct hyp_page, so the annotation is the only
state, and a dedicated function is simpler than teaching the memory
donation path about MMIO. Donation is only allowed before the host is
de-privileged, which is when the hypervisor IOMMU drivers initialise.

Make kvm_pgtable_stage2_unmap() leave invalid counted PTEs (annotations)
in place. host_stage2_unmap_dev_all() unmaps all non-memory ranges to
recycle page-table pages under memory pressure, and would otherwise drop
the annotation and let the host fault the MMIO back in. No caller depends
on the old behaviour: host_stage2_unmap_dev_all() only covers non-memory
ranges, where no annotation existed until now, and the guest unmap paths
check that the PTE is valid first. Returning an error for counted PTEs
instead is not an option, as a non-zero visitor return stops the walk,
which would abort recycling at the first hypervisor-owned MMIO page and
inject an abort into the host for an unrelated address.

Link: https://lore.kernel.org/r/20261015200903.2735565-2-smostafa@google.com
Signed-off-by: Mostafa Saleh <smostafa@google.com>
[Fuad: pass kvm_pgtable_get_leaf() the walk flags ee26443f3dd70 added;
 drop __pkvm_hyp_donate_host_mmio(), which this series doesn't call]
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---

Notes:
    Changes since v2: the version from Mostafa's SMMUv3 v9, which maps
    the MMIO in the hypervisor's private range rather than at
    __hyp_va().

 arch/arm64/include/asm/kvm_pgtable.h          |  2 +
 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h |  5 ++
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         | 62 ++++++++++++++++++-
 arch/arm64/kvm/hyp/pgtable.c                  |  8 +--
 4 files changed, 70 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index 1c9c57cd6ae40..1a3f6e851c94f 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -721,6 +721,8 @@ int kvm_pgtable_stage2_annotate(struct kvm_pgtable *pgt, u64 addr, u64 size,
  * freed. Unmapping a cacheable page will ensure that it is clean to the PoC if
  * FWB is not supported by the CPU.
  *
+ * Invalid counted PTEs (annotations) are left in place.
+ *
  * Return: 0 on success, negative error code on failure.
  */
 int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size);
diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
index ec85a95471207..16e5864ab7f08 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
@@ -36,6 +36,11 @@ int __pkvm_guest_share_host(struct pkvm_hyp_vcpu *vcpu, u64 gfn);
 int __pkvm_guest_unshare_host(struct pkvm_hyp_vcpu *vcpu, u64 gfn);
 int __pkvm_host_unshare_hyp(u64 pfn);
 int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages);
+/*
+ * Donate MMIO range to the hypervisor, it will be mapped in the hypervisor's
+ * private range and unmapped from the host stage-2.
+ */
+int __pkvm_host_donate_hyp_mmio(u64 pfn, u64 nr_pages, unsigned long *haddr);
 int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages);
 int __pkvm_host_share_ffa(u64 pfn, u64 nr_pages);
 int __pkvm_host_unshare_ffa(u64 pfn, u64 nr_pages);
diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
index d4c332a918035..a6dacaca77610 100644
--- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c
+++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
@@ -385,7 +385,11 @@ static int host_stage2_unmap_dev_all(void)
 	u64 addr = 0;
 	int i, ret;
 
-	/* Unmap all non-memory regions to recycle the pages */
+	/*
+	 * Unmap all non-memory regions to recycle the pages.
+	 * That relies on kvm_pgtable_stage2_unmap() not clearing
+	 * counted PTEs which include hypervisor MMIO.
+	 */
 	for (i = 0; i < hyp_memblock_nr; i++, addr = reg->base + reg->size) {
 		reg = &hyp_memory[i];
 		ret = kvm_pgtable_stage2_unmap(pgt, addr, reg->base - addr);
@@ -1146,6 +1150,62 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages)
 	return ret;
 }
 
+int __pkvm_host_donate_hyp_mmio(u64 pfn, u64 nr_pages, unsigned long *haddr)
+{
+	u64 phys = hyp_pfn_to_phys(pfn);
+	u64 size = PAGE_SIZE * nr_pages;
+	unsigned long addr;
+	kvm_pte_t pte;
+	u64 offset;
+	int ret;
+
+	/* Only before de-privilege. */
+	if (static_branch_unlikely(&kvm_protected_mode_initialized))
+		return -EPERM;
+
+	if (!pfn_range_is_valid(pfn, nr_pages))
+		return -EINVAL;
+
+	host_lock_component();
+	for (offset = 0; offset < size; offset += PAGE_SIZE) {
+		if (addr_is_memory(phys + offset)) {
+			ret = -EINVAL;
+			goto unlock;
+		}
+		ret = kvm_pgtable_get_leaf(&host_mmu.pgt, phys + offset, &pte, NULL, 0);
+		if (ret)
+			goto unlock;
+		if (pte && !kvm_pte_valid(pte)) {
+			ret = -EPERM;
+			goto unlock;
+		}
+	}
+
+	ret = __pkvm_create_private_mapping(phys, size, PAGE_HYP_DEVICE, &addr);
+	if (ret)
+		goto unlock;
+
+	/*
+	 * Mark the range as owned by the hypervisor in the host stage-2.
+	 * The annotation makes host accesses fault, and is retained by
+	 * host_stage2_unmap_dev_all()
+	 */
+	ret = host_stage2_try(kvm_pgtable_stage2_annotate, &host_mmu.pgt,
+			      phys, size, &host_s2_pool,
+			      KVM_HOST_INVALID_PTE_TYPE_DONATION,
+			      FIELD_PREP(KVM_HOST_DONATION_PTE_OWNER_MASK, PKVM_ID_HYP));
+	if (ret) {
+		hyp_lock_component();
+		WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, addr, size) != size);
+		hyp_unlock_component();
+	} else {
+		*haddr = addr;
+	}
+unlock:
+	host_unlock_component();
+	return ret;
+}
+
 int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages)
 {
 	u64 phys = hyp_pfn_to_phys(pfn);
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 9c83837b184d3..041bd0d522f2c 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -1163,13 +1163,9 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 	kvm_pte_t *childp = NULL;
 	bool need_flush = false;
 
-	if (!kvm_pte_valid(ctx->old)) {
-		if (stage2_pte_is_counted(ctx->old)) {
-			kvm_clear_pte(ctx->ptep);
-			mm_ops->put_page(ctx->ptep);
-		}
+	/* Invalid PTEs are left in place, including annotations. */
+	if (!kvm_pte_valid(ctx->old))
 		return 0;
-	}
 
 	if (kvm_pte_table(ctx->old, ctx->level)) {
 		childp = kvm_pte_follow(ctx->old, mm_ops);
-- 
2.47.3

[PATCH v3 02/13] KVM: arm64: Take host-registered MMIO regions at pKVM finalisation

[Fuad Tabba]

From: Sebastian Ene <sebastianene@google.com>

To mediate a device, the protected KVM hypervisor has to own its
registers, so that a host access to them takes a stage-2 abort rather
than reaching the device.

Let the host register MMIO regions before the hypervisor initialises,
and have __pkvm_init_finalise() donate each one and record where it's
mapped in the hypervisor's private range. The donation runs before
pkvm_check_host_ownership(), which then covers those mappings too. A
failed donation fails the hypervisor's initialisation before the host
stage 2 is installed, so nothing is given back.

Nothing registers a region yet.

Signed-off-by: Sebastian Ene <sebastianene@google.com>
Co-developed-by: Fuad Tabba <fuad.tabba@linux.dev>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---

Notes:
    Changes since v2 (v2 patches 02 and 03, merged):
    - The trap handling moved to patch 9, which calls the ITS emulation
      directly (was a per-region callback)
    - Each region records its hypervisor VA
    - A failed donation is no longer unwound

 arch/arm64/include/asm/kvm_pkvm.h     | 12 ++++++++++++
 arch/arm64/kvm/hyp/nvhe/mem_protect.c |  3 +++
 arch/arm64/kvm/hyp/nvhe/setup.c       | 22 ++++++++++++++++++++++
 3 files changed, 37 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
index fd042d2b54624..4695c35b31e1a 100644
--- a/arch/arm64/include/asm/kvm_pkvm.h
+++ b/arch/arm64/include/asm/kvm_pkvm.h
@@ -17,6 +17,18 @@
 
 #define HYP_MEMBLOCK_REGIONS 128
 
+/* MMIO regions the host registers for the hypervisor to take at finalisation. */
+#define PKVM_PROTECTED_REGS_NUM	8
+
+struct pkvm_protected_reg {
+	u64			pfn;
+	u64			nr_pages;
+	void __iomem		*hyp_va;
+};
+
+extern struct pkvm_protected_reg kvm_nvhe_sym(pkvm_protected_regs)[];
+extern unsigned int kvm_nvhe_sym(num_protected_reg);
+
 int pkvm_init_host_vm(struct kvm *kvm);
 int pkvm_create_hyp_vm(struct kvm *kvm);
 bool pkvm_hyp_vm_is_created(struct kvm *kvm);
diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
index a6dacaca77610..8d45f45d30e6a 100644
--- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c
+++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
@@ -28,6 +28,9 @@ struct host_mmu host_mmu;
 
 static struct hyp_pool host_s2_pool;
 
+unsigned int num_protected_reg;
+struct pkvm_protected_reg pkvm_protected_regs[PKVM_PROTECTED_REGS_NUM];
+
 static DEFINE_PER_CPU(struct pkvm_hyp_vm *, __current_vm);
 #define current_vm (*this_cpu_ptr(&__current_vm))
 
diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c
index 7bbc26b28faa5..18f348d1331a8 100644
--- a/arch/arm64/kvm/hyp/nvhe/setup.c
+++ b/arch/arm64/kvm/hyp/nvhe/setup.c
@@ -295,6 +295,24 @@ static int fix_hyp_pgtable_refcnt(void)
 				&walker);
 }
 
+static int donate_protected_mmio_regions(void)
+{
+	struct pkvm_protected_reg *region;
+	unsigned long haddr;
+	int ret, i;
+
+	for (i = 0; i < num_protected_reg; i++) {
+		region = &pkvm_protected_regs[i];
+		ret = __pkvm_host_donate_hyp_mmio(region->pfn, region->nr_pages, &haddr);
+		if (ret)
+			return ret;
+
+		region->hyp_va = (void __iomem *)haddr;
+	}
+
+	return 0;
+}
+
 void __noreturn __pkvm_init_finalise(void)
 {
 	struct kvm_cpu_context *host_ctxt = host_data_ptr(host_ctxt);
@@ -335,6 +353,10 @@ void __noreturn __pkvm_init_finalise(void)
 	if (ret)
 		goto out;
 
+	ret = donate_protected_mmio_regions();
+	if (ret)
+		goto out;
+
 	ret = pkvm_check_host_ownership();
 	if (ret)
 		goto out;
-- 
2.47.3

[PATCH v3 03/13] irqchip/gic-v3: Disable GICv4 features in protected KVM mode

[Fuad Tabba]

From: Bartłomiej Grzesik <bgrzesik@google.com>

Later patches have the protected KVM hypervisor take the ITS from the
host and check each command the host queues. They don't emulate the
vLPI commands, as VMAPP's VPT_base, for one, is a host-chosen address
the ITS and the redistributors write to.

Clear has_vlpis and has_rvpeid at redistributor probe in protected mode,
so the host never issues a vLPI command, and print that GICv4 is off.
The host loses GICv4 direct injection for its non-protected guests,
which is off unless kvm-arm.vgic_v4_enable is set, and can't restore a
guest saved with GICD_TYPER2.nASSGIcap set.

The test is kvm_get_mode(), as the redistributors are probed before the
cpucaps are final. It lives in irq-gic-common.h as
gic_is_protected_kvm(), so that the ITS driver can use it too and
linux/kvm_host.h stays out of the rest of irqchip.

Signed-off-by: Bartłomiej Grzesik <bgrzesik@google.com>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 Documentation/admin-guide/kernel-parameters.txt |  8 +++++---
 drivers/irqchip/irq-gic-common.h                | 13 +++++++++++++
 drivers/irqchip/irq-gic-v3.c                    |  8 ++++++++
 3 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd94..3e63e44609522 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3275,8 +3275,9 @@ Kernel parameters
 				hardware (with FEAT_NV2).
 
 			Defaults to VHE/nVHE based on hardware support. Setting
-			mode to "protected" will disable kexec and hibernation
-			for the host. To force nVHE on VHE hardware, add
+			mode to "protected" will disable kexec, hibernation and
+			GICv4 direct LPI injection for the host. To force nVHE
+			on VHE hardware, add
 			"arm64_sw.hvhe=0 id_aa64mmfr1.vh=0" to the
 			command-line.
 			"nested" and "protected" are experimental and should be
@@ -3296,7 +3297,8 @@ Kernel parameters
 
 	kvm-arm.vgic_v4_enable=
 			[KVM,ARM,EARLY] Allow use of GICv4 for direct
-			injection of LPIs.
+			injection of LPIs. Has no effect when
+			kvm-arm.mode=protected.
 
 	kvm-arm.wfe_trap_policy=
 			[KVM,ARM] Control when to set WFE instruction trap for
diff --git a/drivers/irqchip/irq-gic-common.h b/drivers/irqchip/irq-gic-common.h
index 710cab61d9195..87b5c57dec0f1 100644
--- a/drivers/irqchip/irq-gic-common.h
+++ b/drivers/irqchip/irq-gic-common.h
@@ -10,6 +10,9 @@
 #include <linux/irqdomain.h>
 #include <linux/msi.h>
 #include <linux/irqchip/arm-gic-common.h>
+#ifdef CONFIG_ARM64
+#include <linux/kvm_host.h>
+#endif
 
 struct gic_quirk {
 	const char *desc;
@@ -29,6 +32,16 @@ void gic_enable_quirks(u32 iidr, const struct gic_quirk *quirks,
 void gic_enable_of_quirks(const struct device_node *np,
 			  const struct gic_quirk *quirks, void *data);
 
+#ifdef CONFIG_ARM64
+/* Not is_protected_kvm_enabled(): the GIC probes before the cpucaps are final. */
+static inline bool gic_is_protected_kvm(void)
+{
+	return kvm_get_mode() == KVM_MODE_PROTECTED;
+}
+#else
+static inline bool gic_is_protected_kvm(void) { return false; }
+#endif
+
 #define RDIST_FLAGS_PROPBASE_NEEDS_FLUSHING    (1 << 0)
 #define RDIST_FLAGS_RD_TABLES_PREALLOCATED     (1 << 1)
 #define RDIST_FLAGS_FORCE_NON_SHAREABLE        (1 << 2)
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 94873646d46fb..30ed17f26a208 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -1123,6 +1123,14 @@ static void gic_update_rdist_properties(void)
 	gic_iterate_rdists(__gic_update_rdist_properties);
 	if (WARN_ON(gic_data.ppi_nr == UINT_MAX))
 		gic_data.ppi_nr = 0;
+
+	/* Protected KVM doesn't emulate the vLPI commands. */
+	if (gic_is_protected_kvm() && gic_data.rdists.has_vlpis) {
+		pr_info("GICv4 features disabled in protected KVM mode\n");
+		gic_data.rdists.has_vlpis = false;
+		gic_data.rdists.has_rvpeid = false;
+	}
+
 	pr_info("GICv3 features: %d PPIs%s%s\n",
 		gic_data.ppi_nr,
 		gic_data.has_rss ? ", RSS" : "",
-- 
2.47.3

[PATCH v3 04/13] irqchip/gic-v3-its: Page-align ITT allocations under protected KVM

[Fuad Tabba]

From: Bartłomiej Grzesik <bgrzesik@google.com>

A later patch has the protected KVM hypervisor take each device's ITT
for as long as the device is mapped, by having the host donate its
pages. A page the hypervisor already owns can't be donated again, so the
MAPD of a device whose ITT shares a page with another device's would be
rejected.

Round each ITT up to whole pages in protected mode. This spends up to a
page per MSI device where a 256-byte ITT would have shared one: tens of
pages on a phone, thousands on a server with thousands of VFs.

Signed-off-by: Bartłomiej Grzesik <bgrzesik@google.com>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 drivers/irqchip/irq-gic-v3-its.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 282e4bafb084f..8cc1bd8e7835b 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3493,6 +3493,9 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	nr_ites = max(2, nvecs);
 	sz = nr_ites * (FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1);
 	sz = max(sz, ITS_ITT_ALIGN);
+	/* The hypervisor takes ITT pages per device. */
+	if (gic_is_protected_kvm())
+		sz = PAGE_ALIGN(sz);
 
 	itt = itt_alloc_pool(its->numa_node, sz);
 
-- 
2.47.3

[PATCH v3 05/13] irqchip/gic-v3-its: Compute the wait target under the ITS lock

[Fuad Tabba]

its_wait_for_range_completion() converts the posted command pointer into
a queue offset against its->cmd_base, after the caller has dropped
its->lock. That's fine while cmd_base never changes, but the protected
KVM takeover in a later patch switches cmd_base to a copy of the queue
under the lock, and a wait racing with it would compute its target
against the wrong base.

Compute the offset in its_post_commands(), under the lock, and pass it
to the wait instead of the pointer.

No functional change intended.

Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 drivers/irqchip/irq-gic-v3-its.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 8cc1bd8e7835b..f06da58cccf3d 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1144,13 +1144,13 @@ static struct its_cmd_block *its_allocate_entry(struct its_node *its)
 	return cmd;
 }
 
-static struct its_cmd_block *its_post_commands(struct its_node *its)
+static u64 its_post_commands(struct its_node *its)
 {
 	u64 wr = its_cmd_ptr_to_offset(its, its->cmd_write);
 
 	writel_relaxed(wr, its->base + GITS_CWRITER);
 
-	return its->cmd_write;
+	return wr;
 }
 
 static void its_flush_cmd(struct its_node *its, struct its_cmd_block *cmd)
@@ -1167,13 +1167,12 @@ static void its_flush_cmd(struct its_node *its, struct its_cmd_block *cmd)
 
 static int its_wait_for_range_completion(struct its_node *its,
 					 u64	prev_idx,
-					 struct its_cmd_block *to)
+					 u64	to_idx)
 {
-	u64 rd_idx, to_idx, linear_idx;
+	u64 rd_idx, linear_idx;
 	u32 count = 1000000;	/* 1s! */
 
 	/* Linearize to_idx if the command set has wrapped around */
-	to_idx = its_cmd_ptr_to_offset(its, to);
 	if (to_idx < prev_idx)
 		to_idx += ITS_CMD_QUEUE_SZ;
 
@@ -1216,10 +1215,10 @@ void name(struct its_node *its,						\
 	  buildtype builder,						\
 	  struct its_cmd_desc *desc)					\
 {									\
-	struct its_cmd_block *cmd, *sync_cmd, *next_cmd;		\
+	struct its_cmd_block *cmd, *sync_cmd;				\
 	synctype *sync_obj;						\
 	unsigned long flags;						\
-	u64 rd_idx;							\
+	u64 rd_idx, wr_idx;						\
 									\
 	raw_spin_lock_irqsave(&its->lock, flags);			\
 									\
@@ -1242,10 +1241,10 @@ void name(struct its_node *its,						\
 									\
 post:									\
 	rd_idx = readl_relaxed(its->base + GITS_CREADR);		\
-	next_cmd = its_post_commands(its);				\
+	wr_idx = its_post_commands(its);				\
 	raw_spin_unlock_irqrestore(&its->lock, flags);			\
 									\
-	if (its_wait_for_range_completion(its, rd_idx, next_cmd))	\
+	if (its_wait_for_range_completion(its, rd_idx, wr_idx))		\
 		pr_err_ratelimited("ITS cmd %ps failed\n", builder);	\
 }
 
-- 
2.47.3

[PATCH v3 06/13] KVM: arm64: Share the ITS command field accessors with the hypervisor

[Fuad Tabba]

The vgic's ITS emulation reads command fields with its_cmd_mask_field()
and the its_cmd_get_*() accessors. struct its_cmd_block, the block they
decode, is private to the ITS driver, and a later patch has the
protected KVM hypervisor parse the host's commands too.

Move the command block and the accessors to arm-gic-v3.h, which already
holds the GITS_CMD_* encodings, so there is one copy of each. Only KVM
reads commands, but the accessors decode the block the driver writes, so
they sit with it rather than in a KVM header.

No functional change intended.

Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 arch/arm64/kvm/vgic/vgic-its.c     | 15 ---------------
 drivers/irqchip/irq-gic-v3-its.c   | 10 ----------
 include/linux/irqchip/arm-gic-v3.h | 22 ++++++++++++++++++++++
 3 files changed, 22 insertions(+), 25 deletions(-)

diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c
index 7e9b265b1f521..78e480d112025 100644
--- a/arch/arm64/kvm/vgic/vgic-its.c
+++ b/arch/arm64/kvm/vgic/vgic-its.c
@@ -802,21 +802,6 @@ static bool vgic_its_map_lpi(struct kvm *kvm, struct its_ite *ite,
 	return false;
 }
 
-static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
-{
-	return (le64_to_cpu(its_cmd[word]) >> shift) & (BIT_ULL(size) - 1);
-}
-
-#define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
-#define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
-#define its_cmd_get_size(cmd)		(its_cmd_mask_field(cmd, 1,  0,  5) + 1)
-#define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
-#define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
-#define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
-#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 44) << 8)
-#define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
-#define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
-
 /*
  * The DISCARD command frees an Interrupt Translation Table Entry (ITTE).
  * Must be called with the its_lock mutex held.
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index f06da58cccf3d..3e27c7d58e024 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -528,16 +528,6 @@ struct its_cmd_desc {
 	};
 };
 
-/*
- * The ITS command block, which is what the ITS actually parses.
- */
-struct its_cmd_block {
-	union {
-		u64	raw_cmd[4];
-		__le64	raw_cmd_le[4];
-	};
-};
-
 #define ITS_CMD_QUEUE_SZ		SZ_64K
 #define ITS_CMD_QUEUE_NR_ENTRIES	(ITS_CMD_QUEUE_SZ / sizeof(struct its_cmd_block))
 
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index ea5fd2374ebe0..e0bec40adf2b7 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -657,6 +657,28 @@ static inline bool gic_enable_sre(void)
 	return !!(val & ICC_SRE_EL1_SRE);
 }
 
+struct its_cmd_block {
+	union {
+		u64	raw_cmd[4];
+		__le64	raw_cmd_le[4];
+	};
+};
+
+static inline u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
+{
+	return (le64_to_cpu(its_cmd[word]) >> shift) & (BIT_ULL(size) - 1);
+}
+
+#define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
+#define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
+#define its_cmd_get_size(cmd)		(its_cmd_mask_field(cmd, 1,  0,  5) + 1)
+#define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
+#define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
+#define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
+#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 44) << 8)
+#define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
+#define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
+
 #endif
 
 #endif
-- 
2.47.3

[PATCH v3 07/13] KVM: arm64: Deprivilege the host under stop_machine()

[Fuad Tabba]

pkvm_drop_host_privileges() installs the host stage 2 on every CPU with
on_each_cpu(), and each CPU goes back to running the host as soon as its
own __pkvm_prot_finalize() returns. The ITS takeover a later patch adds
has to run while every other CPU is stopped and still privileged, and
let them finalise only after it. Stopping them means no CPU can post an
ITS command while the driver switches to its copies of the queue and
tables. Keeping them privileged means a takeover that fails leaves the
host as it was.

Finalise from a stop_machine() callback instead: no CPU leaves it until
every CPU's callback has returned. An error from any CPU still reaches
finalize_pkvm(), which runs once at boot.

Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 arch/arm64/kvm/pkvm.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index 016997288fd23..16dfb249b4961 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -11,6 +11,7 @@
 #include <asm/kvm_mmu.h>
 #include <linux/memblock.h>
 #include <linux/mutex.h>
+#include <linux/stop_machine.h>
 
 #include <asm/kvm_pkvm.h>
 
@@ -312,25 +313,19 @@ int pkvm_init_host_vm(struct kvm *kvm)
 	return 0;
 }
 
-static void __init _kvm_host_prot_finalize(void *arg)
+static int __init __pkvm_drop_host_privileges(void *arg)
 {
-	int *err = arg;
-
-	if (WARN_ON(kvm_call_hyp_nvhe(__pkvm_prot_finalize)))
-		WRITE_ONCE(*err, -EINVAL);
+	return WARN_ON(kvm_call_hyp_nvhe(__pkvm_prot_finalize)) ? -EINVAL : 0;
 }
 
 static int __init pkvm_drop_host_privileges(void)
 {
-	int ret = 0;
-
 	/*
 	 * Flip the static key upfront as that may no longer be possible
 	 * once the host stage 2 is installed.
 	 */
 	static_branch_enable(&kvm_protected_mode_initialized);
-	on_each_cpu(_kvm_host_prot_finalize, &ret, 1);
-	return ret;
+	return stop_machine(__pkvm_drop_host_privileges, NULL, cpu_online_mask);
 }
 
 static void __init pkvm_selftests(void)
-- 
2.47.3

[PATCH v3 08/13] irqchip/gic-v3-its: Add the protected-KVM takeover entry points

[Fuad Tabba]

From: Sebastian Ene <sebastianene@google.com>

Under protected KVM the hypervisor takes the memory the ITS reads its
command queue and tables from. From the takeover on, the driver writes
its commands and level-1 device table entries into copies that the
hypervisor checks before the ITS sees them.

Add the driver's side of the takeover KVM runs at deprivilege, in three
steps. its_pkvm_setup_prepare() allocates the copies and the pages the
hypervisor keeps its state in, and lists the devices already mapped,
with their ITTs, as the ITS's own record of them is IMPLEMENTATION
DEFINED. It holds every ITS's dev_alloc_lock, which device creation and
teardown already take, until its_pkvm_setup_finish(), so the list stays
exact.

its_pkvm_setup_commit() runs with every CPU stopped, and switches the
driver to its copies only once the hypervisor has taken every ITS, so a
failure leaves the driver on the hardware queues. If the switch didn't
run, finish frees the copies and the state pages. After a failed commit,
that includes the state of an ITS the hypervisor took, which the host
can still use as no CPU is deprivileged.

its_pkvm_iter_its() gives KVM each ITS's base, to register the control
frames before the hypervisor initialises. Nothing calls these yet.

Signed-off-by: Sebastian Ene <sebastianene@google.com>
Co-developed-by: Bartłomiej Grzesik <bgrzesik@google.com>
Signed-off-by: Bartłomiej Grzesik <bgrzesik@google.com>
Co-developed-by: Fuad Tabba <fuad.tabba@linux.dev>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---

Notes:
    Changes since v2 (v2 patch 05):
    - Prepare, commit and finish entry points replace the lock pair KVM
      held across on_each_cpu() with IRQs off (sashiko)
    - Prepare does every allocation before stop_machine(), with
      GFP_KERNEL rather than GFP_ATOMIC under raw spinlocks (sashiko)
    - Commit runs in the deprivilege's stop_machine() callback, and the
      driver switches to its copies only once every ITS is taken
      (sashiko)
    - Every ITS's dev_alloc_lock is held from prepare to finish
    - The descriptor names memory only: the state pages, the queue copy
      and the devices already mapped (was register values and its_baser)

 drivers/irqchip/irq-gic-v3-its.c   | 186 +++++++++++++++++++++++++++++
 include/linux/irqchip/arm-gic-v3.h |  31 +++++
 2 files changed, 217 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 3e27c7d58e024..298e4ca1b0759 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -92,6 +92,7 @@ struct its_baser {
 };
 
 struct its_device;
+struct its_pkvm_state;
 
 /*
  * The ITS structure - contains most of the infrastructure, with the
@@ -126,6 +127,7 @@ struct its_node {
 	unsigned int		msi_domain_flags;
 	u32			pre_its_base; /* for Socionext Synquacer */
 	int			vlpi_redist_offset;
+	struct its_pkvm_state	*pkvm;
 };
 
 static DEFINE_PER_CPU(struct its_node *, local_4_1_its);
@@ -5243,6 +5245,190 @@ static int __init its_compute_its_list_map(struct its_node *its)
 	return its_number;
 }
 
+/*
+ * Protected KVM: from the takeover on, the driver writes commands into a copy
+ * of the command queue, and the hypervisor checks each before the ITS sees it.
+ */
+struct its_pkvm_state {
+	void			*priv;
+	void			*cmdq;
+	struct its_pkvm_desc	*desc;
+	size_t			desc_size;
+};
+
+/* Held, with every ITS's dev_alloc_lock, from prepare to finish. */
+static DEFINE_MUTEX(its_pkvm_lock);
+
+/* Pages for the hypervisor's per-ITS state. */
+#define ITS_PKVM_PRIV_ORDER	3
+
+static int __init its_pkvm_alloc_copies(struct its_node *its,
+					struct its_pkvm_state *st)
+{
+	struct page *page;
+
+	page = its_alloc_pages_node(its->numa_node, GFP_KERNEL, ITS_PKVM_PRIV_ORDER);
+	if (!page)
+		return -ENOMEM;
+	st->priv = page_address(page);
+
+	page = its_alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
+				    get_order(ITS_CMD_QUEUE_SZ));
+	if (!page)
+		return -ENOMEM;
+	st->cmdq = page_address(page);
+
+	return 0;
+}
+
+static int __init its_pkvm_prepare_one(struct its_node *its)
+{
+	struct its_pkvm_state *st;
+	struct its_pkvm_desc *desc;
+	struct its_device *dev;
+	u32 nr_devices = 0;
+	struct page *page;
+	int i, ret;
+
+	st = kzalloc_obj(*st);
+	if (!st)
+		return -ENOMEM;
+	its->pkvm = st;
+
+	mutex_lock_nest_lock(&its->dev_alloc_lock, &its_pkvm_lock);
+
+	ret = its_pkvm_alloc_copies(its, st);
+	if (ret)
+		return ret;
+
+	list_for_each_entry(dev, &its->its_device_list, entry)
+		nr_devices++;
+
+	st->desc_size = PAGE_ALIGN(struct_size(desc, devices, nr_devices));
+	page = its_alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
+				    get_order(st->desc_size));
+	if (!page)
+		return -ENOMEM;
+	desc = page_address(page);
+	st->desc = desc;
+
+	desc->priv_pa = virt_to_phys(st->priv);
+	desc->priv_pages = 1 << ITS_PKVM_PRIV_ORDER;
+	desc->cmdq_copy_pa = virt_to_phys(st->cmdq);
+
+	i = 0;
+	list_for_each_entry(dev, &its->its_device_list, entry) {
+		desc->devices[i].itt_pa = virt_to_phys(dev->itt);
+		desc->devices[i].device_id = dev->device_id;
+		desc->devices[i].event_bits = ilog2(dev->nr_ites);
+		i++;
+	}
+	desc->nr_devices = nr_devices;
+
+	return 0;
+}
+
+static int __init its_pkvm_hand_over(struct its_node *its,
+				     its_pkvm_setup_fn setup)
+{
+	struct its_pkvm_state *st = its->pkvm;
+
+	return setup(its->phys_base, st->desc, st->desc_size);
+}
+
+static void __init its_pkvm_switch(struct its_node *its)
+{
+	struct its_pkvm_state *st = its->pkvm;
+
+	its->cmd_write = (struct its_cmd_block *)st->cmdq + (its->cmd_write - its->cmd_base);
+	its->cmd_base = st->cmdq;
+	st->cmdq = NULL;
+	st->priv = NULL;
+}
+
+static void __init its_pkvm_finish_one(struct its_node *its)
+{
+	struct its_pkvm_state *st = its->pkvm;
+
+	if (st->priv)
+		its_free_pages(st->priv, ITS_PKVM_PRIV_ORDER);
+	if (st->cmdq)
+		its_free_pages(st->cmdq, get_order(ITS_CMD_QUEUE_SZ));
+
+	if (st->desc)
+		its_free_pages(st->desc, get_order(st->desc_size));
+
+	mutex_unlock(&its->dev_alloc_lock);
+	kfree(st);
+	its->pkvm = NULL;
+}
+
+/*
+ * The protected-KVM takeover: prepare builds each ITS's descriptor and holds
+ * off device creation, commit hands every ITS to the hypervisor with every
+ * other CPU parked in stop_machine(), and finish frees what the driver no
+ * longer uses. Call finish after prepare, whatever either returns.
+ */
+int __init its_pkvm_setup_prepare(void)
+{
+	struct its_node *its;
+	int ret;
+
+	mutex_lock(&its_pkvm_lock);
+
+	list_for_each_entry(its, &its_nodes, entry) {
+		ret = its_pkvm_prepare_one(its);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+int __init its_pkvm_setup_commit(its_pkvm_setup_fn setup)
+{
+	struct its_node *its;
+	int ret;
+
+	/* The driver switches to its copies only once every ITS is taken. */
+	list_for_each_entry(its, &its_nodes, entry) {
+		ret = its_pkvm_hand_over(its, setup);
+		if (ret)
+			return ret;
+	}
+
+	list_for_each_entry(its, &its_nodes, entry)
+		its_pkvm_switch(its);
+
+	return 0;
+}
+
+void __init its_pkvm_setup_finish(void)
+{
+	struct its_node *its;
+
+	list_for_each_entry(its, &its_nodes, entry) {
+		if (its->pkvm)
+			its_pkvm_finish_one(its);
+	}
+
+	mutex_unlock(&its_pkvm_lock);
+}
+
+int __init its_pkvm_iter_its(its_pkvm_its_fn fn)
+{
+	struct its_node *its;
+	int ret;
+
+	list_for_each_entry(its, &its_nodes, entry) {
+		ret = fn(its->phys_base);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
 static int __init its_probe_one(struct its_node *its)
 {
 	u64 baser, tmp;
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index e0bec40adf2b7..a708e188898ef 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -679,6 +679,37 @@ static inline u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size
 #define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
 #define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
 
+/* A device the driver mapped before the hypervisor took over the ITS. */
+struct its_pkvm_device {
+	u64		itt_pa;
+	u32		device_id;
+	u8		event_bits;
+};
+
+/*
+ * What the host hands the hypervisor at setup: pages for its per-ITS state,
+ * and the memory it cannot discover from the ITS itself. Physical addresses,
+ * page-aligned.
+ */
+struct its_pkvm_desc {
+	u64			priv_pa;
+	u64			priv_pages;
+	u64			cmdq_copy_pa;
+	u32			nr_devices;
+	struct its_pkvm_device	devices[];
+};
+
+/* Called with every CPU parked: no allocation. */
+typedef int (*its_pkvm_setup_fn)(phys_addr_t its_base,
+				 struct its_pkvm_desc *desc, size_t desc_size);
+
+typedef int (*its_pkvm_its_fn)(phys_addr_t its_base);
+
+int its_pkvm_setup_prepare(void);
+int its_pkvm_setup_commit(its_pkvm_setup_fn setup);
+void its_pkvm_setup_finish(void);
+int its_pkvm_iter_its(its_pkvm_its_fn fn);
+
 #endif
 
 #endif
-- 
2.47.3

[PATCH v3 09/13] KVM: arm64: Emulate the ITS command queue

[Fuad Tabba]

From: Sebastian Ene <sebastianene@google.com>

The ITS writes to the memory its commands point to, so a host that
queues commands freely can aim those writes at the hypervisor or a
protected guest.

Register each ITS's control frame with the hypervisor, which takes
ownership of it at initialisation, so that every host access to it traps
once the host is deprivileged. GITS_TRANSLATER is in the next frame,
which stays host-mapped. Each ITS takes one of the eight region slots,
so KVM doesn't initialise in protected mode with more than eight ITSes.
Inside the stop_machine() callback that deprivileges the host, the
hypervisor reads GITS_CBASER, GITS_CWRITER and GITS_CREADR itself, takes
the queue and pins the driver's copy. If that fails, no CPU is
deprivileged and KVM_CREATE_VM fails with -ENODEV.

On a GITS_CWRITER write, the hypervisor reads each new command once from
the driver's copy of the queue, checks it and copies it into the
hardware queue. A rejected command aborts the host's store, after the
commands before it have been forwarded. For now that's a vLPI command or
an unknown encoding, neither of which the driver queues.

An access without a valid syndrome aborts. Otherwise, reads go to the
hardware. Writes abort, except to GITS_CWRITER, to GITS_BASER<n> until
the next patch, and to GITS_CTLR and GITS_CBASER, which the driver
writes on suspend and resume. A GITS_CBASER write goes through only with
the ITS disabled and quiescent, and the hypervisor writes back the value
it read at setup. An enable aborts unless GITS_CBASER holds that value.
It's dropped rather than aborted while the ITS isn't quiescent (IHI
0069H.b 12.19.4), as the driver writes one from a syscore callback when
a quiesce times out, and an abort there would take the host down.

The takeover takes cpus_read_lock() before the driver's prepare, as
drivers take dev_alloc_lock through the MSI layer with it held.

Signed-off-by: Sebastian Ene <sebastianene@google.com>
Co-developed-by: Bartłomiej Grzesik <bgrzesik@google.com>
Signed-off-by: Bartłomiej Grzesik <bgrzesik@google.com>
Co-developed-by: Fuad Tabba <fuad.tabba@linux.dev>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---

Notes:
    Changes since v2 (v2 patches 04, 06, 10, 11 and 13, merged):
    - The takeover runs in the stop_machine() callback that
      deprivileges the host, while every CPU is still privileged (was
      after the stage-2 install)
    - A failed takeover deprivileges no CPU, and KVM_CREATE_VM then
      returns -ENODEV
    - The hypervisor reads GITS_CBASER, GITS_CWRITER and GITS_CREADR
      itself, rather than taking the host's queue and length
    - Every rejected access aborts, a rejected command once the commands
      before it are forwarded (v2 dropped it and skipped the
      instruction)
    - Commands are forwarded and never rolled back: the completion poll,
      the -EIO/-EBUSY split and the rollback are gone (sashiko)
    - The vLPI commands and unknown encodings are rejected (were
      forwarded)
    - A write to a register the emulation doesn't handle aborts, where
      v2 forwarded it (sashiko)
    - Accesses are decoded by offset and size, where v2's handlers
      ignored the offset within a register (sashiko)
    - A GITS_CBASER write stores the value read at setup, whole, where
      v2 let the host's cacheability and shareability through (sashiko)
    - A GITS_CBASER write while the ITS is enabled or not quiescent
      aborts, and one that goes through resets the hypervisor's side of
      the queue
    - An enable aborts unless GITS_CBASER still holds the value read at
      setup
    - GITS_CWRITER's offset is bounded by the queue size, and a write
      that would overrun GITS_CREADR aborts
    - Every forwarded command is cleaned to PoC, where v2's test for it
      ignored inner cacheability (sashiko)
    - The trap handler, from v2 patch 03, calls the ITS emulation
      directly (was a per-region callback)
    - The written value is masked to the access size, and a read is
      sign-extended per ESR_EL2.SSE and sized per SF (sashiko)
    - The trap path tests num_protected_reg first and no longer calls
      addr_is_memory(), and FAR_TO_FIPA_OFFSET() replaces v2's FAR_MASK
    - KVM registers each ITS from the driver's its_pkvm_iter_its(), at
      KVM init (was a DT walk)

 arch/arm64/include/asm/kvm_asm.h              |   2 +
 arch/arm64/include/asm/kvm_hcall.h            |   2 +
 arch/arm64/include/asm/kvm_pkvm.h             |   4 +
 arch/arm64/kvm/arm.c                          |   4 +
 arch/arm64/kvm/hyp/include/nvhe/its_emulate.h |  16 +
 arch/arm64/kvm/hyp/nvhe/Makefile              |   3 +-
 arch/arm64/kvm/hyp/nvhe/hyp-main.c            |   8 +
 arch/arm64/kvm/hyp/nvhe/its_emulate.c         | 311 ++++++++++++++++++
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         |  65 ++++
 arch/arm64/kvm/pkvm.c                         |  98 +++++-
 10 files changed, 510 insertions(+), 3 deletions(-)
 create mode 100644 arch/arm64/kvm/hyp/include/nvhe/its_emulate.h
 create mode 100644 arch/arm64/kvm/hyp/nvhe/its_emulate.c

diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index abda476e023de..339850e2702d0 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -122,6 +122,8 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_sync_state,
 	__KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
+	/* Called with kvm_protected_mode_initialized on, before __pkvm_prot_finalize. */
+	__KVM_HOST_SMCCC_FUNC___pkvm_its_emulate_setup,
 
 	MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
 };
diff --git a/arch/arm64/include/asm/kvm_hcall.h b/arch/arm64/include/asm/kvm_hcall.h
index 5f77a8aa29635..0748232c387ec 100644
--- a/arch/arm64/include/asm/kvm_hcall.h
+++ b/arch/arm64/include/asm/kvm_hcall.h
@@ -254,5 +254,7 @@ DECLARE_KVM_HOST_HCALL0(void, __pkvm_vcpu_put)
 DECLARE_KVM_HOST_HCALL0(void, __pkvm_vcpu_sync_state)
 DECLARE_KVM_HOST_HCALL(void, __pkvm_tlb_flush_vmid,
 	pkvm_handle_t, handle)
+DECLARE_KVM_HOST_HCALL(int, __pkvm_its_emulate_setup,
+	phys_addr_t, its_base, phys_addr_t, desc_pa, size_t, desc_size)
 
 #endif /* __ARM64_KVM_HCALL_H__ */
diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
index 4695c35b31e1a..e1203b06ec7b3 100644
--- a/arch/arm64/include/asm/kvm_pkvm.h
+++ b/arch/arm64/include/asm/kvm_pkvm.h
@@ -24,11 +24,15 @@ struct pkvm_protected_reg {
 	u64			pfn;
 	u64			nr_pages;
 	void __iomem		*hyp_va;
+	/* The ITS's state once it is taken. */
+	void			*priv;
 };
 
 extern struct pkvm_protected_reg kvm_nvhe_sym(pkvm_protected_regs)[];
 extern unsigned int kvm_nvhe_sym(num_protected_reg);
 
+int pkvm_register_its_regions(void);
+
 int pkvm_init_host_vm(struct kvm *kvm);
 int pkvm_create_hyp_vm(struct kvm *kvm);
 bool pkvm_hyp_vm_is_created(struct kvm *kvm);
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 157bf4b812020..789f6ca13ae12 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -3128,6 +3128,10 @@ static int __init init_hyp_mode(void)
 		if (err)
 			goto out_err;
 
+		err = pkvm_register_its_regions();
+		if (err)
+			goto out_err;
+
 		err = kvm_hyp_init_protection(hyp_va_bits);
 		if (err) {
 			kvm_err("Failed to init hyp memory protection\n");
diff --git a/arch/arm64/kvm/hyp/include/nvhe/its_emulate.h b/arch/arm64/kvm/hyp/include/nvhe/its_emulate.h
new file mode 100644
index 0000000000000..0b0d4aa25a5b7
--- /dev/null
+++ b/arch/arm64/kvm/hyp/include/nvhe/its_emulate.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 Google LLC
+ * Author: Sebastian Ene <sebastianene@google.com>
+ */
+#ifndef __ARM64_KVM_NVHE_ITS_EMULATE_H__
+#define __ARM64_KVM_NVHE_ITS_EMULATE_H__
+
+#include <asm/kvm_pkvm.h>
+
+int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
+			   size_t desc_size);
+bool pkvm_its_emulate_handler(struct pkvm_protected_reg *region, u64 offset,
+			      bool write, u64 *val, u8 size);
+
+#endif /* __ARM64_KVM_NVHE_ITS_EMULATE_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
index 56a7e2c05f73a..2c50940849782 100644
--- a/arch/arm64/kvm/hyp/nvhe/Makefile
+++ b/arch/arm64/kvm/hyp/nvhe/Makefile
@@ -24,7 +24,8 @@ CFLAGS_switch.nvhe.o += -Wno-override-init
 
 hyp-obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \
 	 hyp-main.o hyp-smp.o psci-relay.o alloc.o early_alloc.o page_alloc.o \
-	 cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o
+	 cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o \
+	 its_emulate.o
 hyp-obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
 	 ../hyp-entry.o ../exception.o ../pgtable.o ../vgic-v5-sr.o
 hyp-obj-y += ../../../kernel/smccc-call.o
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 179d7c8277b88..5efaa32037a5b 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -22,6 +22,7 @@
 
 #include <nvhe/alloc.h>
 #include <nvhe/ffa.h>
+#include <nvhe/its_emulate.h>
 #include <nvhe/mem_protect.h>
 #include <nvhe/mm.h>
 #include <nvhe/pkvm.h>
@@ -1169,6 +1170,12 @@ DEFINE_KVM_HOST_HCALL(void, __pkvm_tlb_flush_vmid,
 	put_pkvm_hyp_vm(hyp_vm);
 }
 
+DEFINE_KVM_HOST_HCALL(int, __pkvm_its_emulate_setup,
+	phys_addr_t, its_base, phys_addr_t, desc_pa, size_t, desc_size)
+{
+	return pkvm_its_emulate_setup(its_base, desc_pa, desc_size);
+}
+
 DEFINE_KVM_HOST_HCALL(void, __kvm_flush_cpu_context,
 	struct kvm_s2_mmu __kern *, mmu)
 {
@@ -1603,6 +1610,7 @@ static const hcall_t host_hcall[] = {
 	HANDLE_FUNC(__pkvm_vcpu_put),
 	HANDLE_FUNC(__pkvm_vcpu_sync_state),
 	HANDLE_FUNC(__pkvm_tlb_flush_vmid),
+	HANDLE_FUNC(__pkvm_its_emulate_setup),
 };
 
 static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
diff --git a/arch/arm64/kvm/hyp/nvhe/its_emulate.c b/arch/arm64/kvm/hyp/nvhe/its_emulate.c
new file mode 100644
index 0000000000000..b0181cb3fe9ba
--- /dev/null
+++ b/arch/arm64/kvm/hyp/nvhe/its_emulate.c
@@ -0,0 +1,311 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Emulation of the host's accesses to a GICv3 ITS in protected KVM.
+ *
+ * Copyright (C) 2026 Google LLC
+ * Author: Sebastian Ene <sebastianene@google.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/io.h>
+#include <linux/irqchip/arm-gic-v3.h>
+
+#include <asm/kvm_pkvm.h>
+
+#include <nvhe/its_emulate.h>
+#include <nvhe/mem_protect.h>
+#include <nvhe/memory.h>
+#include <nvhe/spinlock.h>
+
+#define GITS_CWRITER_OFFSET	GENMASK_ULL(19, 5)
+#define GITS_CWRITER_RETRY	BIT_ULL(0)
+#define GITS_CREADR_OFFSET	GENMASK_ULL(19, 5)
+#define GITS_CBASER_SIZE	GENMASK_ULL(7, 0)
+
+/* Per-ITS state, in the pages the host donated at setup. */
+struct its_priv {
+	hyp_spinlock_t		lock;
+	void __iomem		*base;
+	u64			cbaser;
+
+	/* cmdq_hw is hyp-owned; cmdq_copy is the host's copy, pinned. */
+	void			*cmdq_hw;
+	void			*cmdq_copy;
+	u32			cmdq_size;
+
+	/* The next slot to forward, at the offset last written to GITS_CWRITER. */
+	u32			cmd_offset;
+
+	/* The GITS_CREADR offset last read. */
+	u32			hw_creadr;
+};
+
+static int its_pin_host_pages(phys_addr_t pa, u64 nr_pages)
+{
+	void *va = hyp_phys_to_virt(pa);
+	u64 pfn = PHYS_PFN(pa), i;
+	int ret;
+
+	for (i = 0; i < nr_pages; i++) {
+		ret = __pkvm_host_share_hyp(pfn + i);
+		if (ret)
+			return ret;
+	}
+
+	return hyp_pin_shared_mem(va, va + (nr_pages << PAGE_SHIFT));
+}
+
+static bool its_process_cmd(struct its_priv *its, struct its_cmd_block *cmd)
+{
+	switch (its_cmd_get_command(cmd->raw_cmd)) {
+	case GITS_CMD_MAPD:
+	case GITS_CMD_MAPC:
+	case GITS_CMD_MAPTI:
+	case GITS_CMD_MAPI:
+	case GITS_CMD_MOVI:
+	case GITS_CMD_DISCARD:
+	case GITS_CMD_INV:
+	case GITS_CMD_MOVALL:
+	case GITS_CMD_INVALL:
+	case GITS_CMD_INT:
+	case GITS_CMD_CLEAR:
+	case GITS_CMD_SYNC:
+		return true;
+	default:
+		/* The vLPI commands and anything unknown. */
+		return false;
+	}
+}
+
+static u64 its_read_creadr(struct its_priv *its)
+{
+	u64 creadr = readq_relaxed(its->base + GITS_CREADR);
+	u32 offset = creadr & GITS_CREADR_OFFSET;
+
+	its->hw_creadr = offset;
+
+	return creadr;
+}
+
+/* Returns false on a rejected command, after forwarding the ones before it. */
+static bool its_cwriter_write(struct its_priv *its, u64 val)
+{
+	u32 new = val & GITS_CWRITER_OFFSET;
+	struct its_cmd_block cmd;
+	bool rejected = false;
+	unsigned int i;
+	u64 *slot;
+
+	if (new >= its->cmdq_size)
+		return false;
+
+	its_read_creadr(its);
+
+	/* The queue is full 32 bytes behind GITS_CREADR (IHI 0069H.b 5.2.8): reject an overrun. */
+	if ((its->cmd_offset + its->cmdq_size - its->hw_creadr) % its->cmdq_size +
+	    (new + its->cmdq_size - its->cmd_offset) % its->cmdq_size >
+	    its->cmdq_size - sizeof(cmd))
+		return false;
+
+	/*
+	 * A value below cmd_offset wraps around the queue, for the ITS too. The
+	 * driver writes one only as the 0 after a GITS_CBASER write, which has
+	 * already reset cmd_offset to 0.
+	 */
+	while (its->cmd_offset != new) {
+		/* The host can change its copy: read the command once. */
+		slot = its->cmdq_copy + its->cmd_offset;
+		for (i = 0; i < ARRAY_SIZE(cmd.raw_cmd); i++)
+			cmd.raw_cmd[i] = READ_ONCE(slot[i]);
+		if (!its_process_cmd(its, &cmd)) {
+			rejected = true;
+			break;
+		}
+
+		memcpy(its->cmdq_hw + its->cmd_offset, &cmd, sizeof(cmd));
+		/*
+		 * Whatever GITS_CBASER says: it can read shareable on an ITS the
+		 * driver treats as non-coherent (ITS_FLAGS_CMDQ_NEEDS_FLUSHING).
+		 */
+		gic_flush_dcache_to_poc(its->cmdq_hw + its->cmd_offset, sizeof(cmd));
+		its->cmd_offset = (its->cmd_offset + sizeof(cmd)) % its->cmdq_size;
+	}
+
+	/* Retry has no effect unless the hardware stalled on a command error. */
+	dsb(ishst);
+	writeq_relaxed(its->cmd_offset | (val & GITS_CWRITER_RETRY), its->base + GITS_CWRITER);
+	return !rejected;
+}
+
+/* A reset clears GITS_CBASER.Valid, and an enable with it clear is UNPREDICTABLE. */
+static bool its_bases_intact(struct its_priv *its)
+{
+	return readq_relaxed(its->base + GITS_CBASER) == its->cbaser;
+}
+
+static bool its_ctlr_write(struct its_priv *its, u64 val)
+{
+	u32 ctlr = readl_relaxed(its->base + GITS_CTLR);
+
+	if ((val & GITS_CTLR_ENABLE) && !(ctlr & GITS_CTLR_ENABLE)) {
+		/*
+		 * Enabling while not Quiescent is UNPREDICTABLE. Dropped, not
+		 * rejected: the driver writes it when a quiesce times out on suspend.
+		 */
+		if (!(ctlr & GITS_CTLR_QUIESCENT))
+			return true;
+		if (!its_bases_intact(its))
+			return false;
+	}
+
+	writel_relaxed(val, its->base + GITS_CTLR);
+	return true;
+}
+
+/* GITS_CBASER and GITS_BASER<n> are UNPREDICTABLE to write otherwise. */
+static bool its_disabled_quiescent(struct its_priv *its)
+{
+	u32 ctlr = readl_relaxed(its->base + GITS_CTLR);
+
+	return !(ctlr & GITS_CTLR_ENABLE) && (ctlr & GITS_CTLR_QUIESCENT);
+}
+
+static bool its_cbaser_write(struct its_priv *its)
+{
+	if (!its_disabled_quiescent(its))
+		return false;
+
+	writeq_relaxed(its->cbaser, its->base + GITS_CBASER);
+
+	/* The write reset GITS_CREADR to 0; nothing queued before it completes. */
+	its->cmd_offset = 0;
+	its->hw_creadr = 0;
+	writeq_relaxed(0, its->base + GITS_CWRITER);
+	return true;
+}
+
+static bool its_emulate(struct its_priv *its, u64 offset, bool write, u64 *val, u8 size)
+{
+	if (write) {
+		switch (offset) {
+		case GITS_CTLR:
+			return size == sizeof(u32) && its_ctlr_write(its, *val);
+		case GITS_CBASER:
+			return size == sizeof(u64) && its_cbaser_write(its);
+		case GITS_CWRITER:
+			/* The driver writes the low word; the high word is RES0. */
+			return its_cwriter_write(its, *val);
+		case GITS_BASER ... GITS_BASER + (GITS_BASER_NR_REGS - 1) * sizeof(u64):
+			if (size != sizeof(u64))
+				return false;
+			writeq_relaxed(*val, its->base + offset);
+			return true;
+		default:
+			return false;
+		}
+	}
+
+	*val = size == sizeof(u64) ? readq_relaxed(its->base + offset) :
+				     readl_relaxed(its->base + offset);
+	return true;
+}
+
+bool pkvm_its_emulate_handler(struct pkvm_protected_reg *region, u64 offset,
+			      bool write, u64 *val, u8 size)
+{
+	struct its_priv *its = region->priv;
+	bool ret;
+
+	if (size < sizeof(u32) || !IS_ALIGNED(offset, size))
+		return false;
+
+	hyp_spin_lock(&its->lock);
+	ret = its_emulate(its, offset, write, val, size);
+	hyp_spin_unlock(&its->lock);
+
+	return ret;
+}
+
+static struct pkvm_protected_reg *its_find_region(phys_addr_t its_base)
+{
+	int i;
+
+	for (i = 0; i < num_protected_reg; i++) {
+		if (PFN_PHYS(pkvm_protected_regs[i].pfn) == its_base)
+			return &pkvm_protected_regs[i];
+	}
+
+	return NULL;
+}
+
+static int its_take_cmdq(struct its_priv *its, phys_addr_t copy_pa)
+{
+	phys_addr_t hw_pa = GITS_CBASER_ADDRESS(its->cbaser);
+	size_t size = SZ_4K * (FIELD_GET(GITS_CBASER_SIZE, its->cbaser) + 1);
+	u32 cwriter = readq_relaxed(its->base + GITS_CWRITER) & GITS_CWRITER_OFFSET;
+	u32 creadr = readq_relaxed(its->base + GITS_CREADR) & GITS_CREADR_OFFSET;
+	u64 nr_pages = size >> PAGE_SHIFT;
+	int ret;
+
+	ret = __pkvm_host_donate_hyp(PHYS_PFN(hw_pa), nr_pages);
+	if (ret)
+		return ret;
+
+	ret = its_pin_host_pages(copy_pa, nr_pages);
+	if (ret)
+		return ret;
+
+	its->cmdq_hw = hyp_phys_to_virt(hw_pa);
+	its->cmdq_copy = hyp_phys_to_virt(copy_pa);
+	its->cmdq_size = size;
+	its->cmd_offset = cwriter;
+	its->hw_creadr = creadr;
+	return 0;
+}
+
+int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
+			   size_t desc_size)
+{
+	u64 desc_pages = desc_size >> PAGE_SHIFT;
+	struct pkvm_protected_reg *region;
+	struct its_pkvm_desc *desc;
+	struct its_priv *its;
+	u64 priv_pages;
+	int ret;
+
+	region = its_find_region(its_base);
+	if (!region)
+		return -ENODEV;
+	if (region->priv)
+		return -EBUSY;
+
+	/*
+	 * Called once per ITS by the still-privileged host, so its arguments and
+	 * the registers it programmed are trusted. No unwind on failure: the host
+	 * is then never deprivileged.
+	 */
+	ret = __pkvm_host_donate_hyp(PHYS_PFN(desc_pa), desc_pages);
+	if (ret)
+		return ret;
+	desc = hyp_phys_to_virt(desc_pa);
+
+	priv_pages = desc->priv_pages;
+	ret = __pkvm_host_donate_hyp(PHYS_PFN(desc->priv_pa), priv_pages);
+	if (ret)
+		return ret;
+	its = hyp_phys_to_virt(desc->priv_pa);
+	memset(its, 0, priv_pages * PAGE_SIZE);
+
+	hyp_spin_lock_init(&its->lock);
+	its->base = region->hyp_va;
+	its->cbaser = readq_relaxed(its->base + GITS_CBASER);
+
+	ret = its_take_cmdq(its, desc->cmdq_copy_pa);
+	if (ret)
+		return ret;
+
+	WARN_ON(__pkvm_hyp_donate_host(PHYS_PFN(desc_pa), desc_pages));
+	/* Published by stop_machine()'s exit: no host CPU traps until then. */
+	region->priv = its;
+	return 0;
+}
diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
index 8d45f45d30e6a..4730f628b9497 100644
--- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c
+++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
@@ -13,10 +13,12 @@
 #include <asm/kvm_pkvm.h>
 #include <asm/stage2_pgtable.h>
 
+#include <hyp/adjust_pc.h>
 #include <hyp/fault.h>
 
 #include <nvhe/arm-smccc.h>
 #include <nvhe/gfp.h>
+#include <nvhe/its_emulate.h>
 #include <nvhe/memory.h>
 #include <nvhe/mem_protect.h>
 #include <nvhe/mm.h>
@@ -779,6 +781,65 @@ static void host_inject_mem_abort(struct kvm_cpu_context *host_ctxt)
 	inject_host_exception(esr);
 }
 
+static struct pkvm_protected_reg *find_protected_reg(u64 addr, u8 size)
+{
+	struct pkvm_protected_reg *region;
+	u64 start, end;
+	int i;
+
+	for (i = 0; i < num_protected_reg; i++) {
+		region = &pkvm_protected_regs[i];
+		start = PFN_PHYS(region->pfn);
+		end = start + PFN_PHYS(region->nr_pages);
+
+		if (addr >= start && addr + size <= end)
+			return region;
+	}
+
+	return NULL;
+}
+
+/*
+ * Returns false when the access cannot be emulated or the emulation rejects it;
+ * the caller's host stage-2 path then injects the abort.
+ */
+static bool handle_host_mmio_trap(struct kvm_cpu_context *host_ctxt, u64 esr,
+				  u64 addr)
+{
+	struct pkvm_protected_reg *region;
+	u64 val = 0;
+	u8 size, rt;
+	bool write;
+
+	/* No syndrome, no emulation. */
+	if (ESR_ELx_EC(esr) != ESR_ELx_EC_DABT_LOW || !(esr & ESR_ELx_ISV))
+		return false;
+
+	size = BIT(FIELD_GET(ESR_ELx_SAS, esr));
+	region = find_protected_reg(addr, size);
+	if (!region)
+		return false;
+
+	rt = FIELD_GET(ESR_ELx_SRT_MASK, esr);
+	write = esr & ESR_ELx_WNR;
+	if (write && rt != 31)
+		val = host_ctxt->regs.regs[rt] & GENMASK_ULL(size * 8 - 1, 0);
+
+	if (!pkvm_its_emulate_handler(region, addr - PFN_PHYS(region->pfn), write, &val, size))
+		return false;
+
+	if (!write && rt != 31) {
+		if ((esr & ESR_ELx_SSE) && size < sizeof(u64))
+			val = sign_extend64(val, size * 8 - 1);
+		if (!(esr & ESR_ELx_SF))
+			val = lower_32_bits(val);
+		host_ctxt->regs.regs[rt] = val;
+	}
+
+	kvm_skip_host_instr();
+	return true;
+}
+
 void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt)
 {
 	struct kvm_vcpu_fault_info fault;
@@ -801,6 +862,10 @@ void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt)
 	BUG_ON(!(fault.hpfar_el2 & HPFAR_EL2_NS));
 	addr = FIELD_GET(HPFAR_EL2_FIPA, fault.hpfar_el2) << 12;
 
+	if (num_protected_reg &&
+	    handle_host_mmio_trap(host_ctxt, esr, addr | FAR_TO_FIPA_OFFSET(fault.far_el2)))
+		return;
+
 	switch (host_stage2_idmap(addr)) {
 	case -EPERM:
 		host_inject_mem_abort(host_ctxt);
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index 16dfb249b4961..3293e3c7353b8 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -6,6 +6,7 @@
 
 #include <linux/init.h>
 #include <linux/interval_tree_generic.h>
+#include <linux/irqchip/arm-gic-v3.h>
 #include <linux/kmemleak.h>
 #include <linux/kvm_host.h>
 #include <asm/kvm_mmu.h>
@@ -43,6 +44,53 @@ static int __init register_memblock_regions(void)
 	return 0;
 }
 
+static int __init pkvm_register_its_region(phys_addr_t its_base)
+{
+	unsigned int idx = kvm_nvhe_sym(num_protected_reg);
+	struct pkvm_protected_reg *region;
+
+	if (idx >= PKVM_PROTECTED_REGS_NUM) {
+		kvm_err("ITS@%pa: no protected region slot left\n", &its_base);
+		return -ENOSPC;
+	}
+
+	/*
+	 * The control frame only: GITS_TRANSLATER stays host-mapped in the next
+	 * frame, as devices raise MSIs through it.
+	 */
+	region = &kvm_nvhe_sym(pkvm_protected_regs)[idx];
+	region->pfn = PHYS_PFN(its_base);
+	region->nr_pages = PFN_DOWN(GITS_TRANSLATER);
+	kvm_nvhe_sym(num_protected_reg) = idx + 1;
+
+	return 0;
+}
+
+int __init pkvm_register_its_regions(void)
+{
+	int ret;
+
+	ret = its_pkvm_iter_its(pkvm_register_its_region);
+	if (ret)
+		kvm_err("Failed to register the ITS regions with hyp: %d\n", ret);
+
+	return ret;
+}
+
+/* Called by the driver with every CPU parked: no allocation here. */
+static int __init pkvm_init_its_emulation(phys_addr_t its_base,
+					  struct its_pkvm_desc *desc,
+					  size_t desc_size)
+{
+	int ret;
+
+	ret = kvm_call_hyp_nvhe(__pkvm_its_emulate_setup, its_base, virt_to_phys(desc), desc_size);
+	if (ret)
+		kvm_err("ITS@%pa: emulation setup failed: %d\n", &its_base, ret);
+
+	return ret;
+}
+
 void __init kvm_hyp_reserve(void)
 {
 	u64 hyp_mem_pages = 0;
@@ -299,6 +347,10 @@ int pkvm_init_host_vm(struct kvm *kvm)
 {
 	int ret;
 
+	/* The ITS takeover failed, so the hypervisor never finalised. */
+	if (!is_pkvm_initialized())
+		return -ENODEV;
+
 	/* Reserve the VM in hyp and obtain a hyp handle for the VM. */
 	ret = kvm_call_hyp_nvhe(__pkvm_reserve_vm);
 	if (ret < 0)
@@ -313,19 +365,61 @@ int pkvm_init_host_vm(struct kvm *kvm)
 	return 0;
 }
 
+struct pkvm_its_takeover {
+	int	ret;
+	bool	done;
+};
+
 static int __init __pkvm_drop_host_privileges(void *arg)
 {
+	struct pkvm_its_takeover *takeover = arg;
+
+	if (smp_processor_id() == cpumask_first(cpu_online_mask)) {
+		takeover->ret = its_pkvm_setup_commit(pkvm_init_its_emulation);
+		/* Publishes ret to the CPUs spinning below. */
+		smp_store_release(&takeover->done, true);
+	}
+
+	/* Pairs with the release above: ret is read after done. */
+	while (!smp_load_acquire(&takeover->done))
+		cpu_relax();
+
+	/* No CPU is deprivileged unless the hypervisor took every ITS. */
+	if (takeover->ret)
+		return 0;
+
 	return WARN_ON(kvm_call_hyp_nvhe(__pkvm_prot_finalize)) ? -EINVAL : 0;
 }
 
 static int __init pkvm_drop_host_privileges(void)
 {
+	struct pkvm_its_takeover takeover = {};
+	int ret;
+
+	/* Drivers take dev_alloc_lock, through the MSI layer, with it held. */
+	cpus_read_lock();
+	ret = its_pkvm_setup_prepare();
+	if (ret)
+		goto out;
+
 	/*
 	 * Flip the static key upfront as that may no longer be possible
 	 * once the host stage 2 is installed.
 	 */
-	static_branch_enable(&kvm_protected_mode_initialized);
-	return stop_machine(__pkvm_drop_host_privileges, NULL, cpu_online_mask);
+	static_branch_enable_cpuslocked(&kvm_protected_mode_initialized);
+	ret = stop_machine_cpuslocked(__pkvm_drop_host_privileges, &takeover, cpu_online_mask);
+	if (takeover.ret) {
+		/* No stage 2 was installed: back to before finalisation, no VMs. */
+		static_branch_disable_cpuslocked(&kvm_protected_mode_initialized);
+		ret = takeover.ret;
+	}
+out:
+	its_pkvm_setup_finish();
+	cpus_read_unlock();
+	if (ret && !is_pkvm_initialized())
+		pr_err("pKVM: ITS takeover failed (%d): host not deprivileged, VMs refused\n", ret);
+
+	return ret;
 }
 
 static void __init pkvm_selftests(void)
-- 
2.47.3

[PATCH v3 10/13] KVM: arm64: Take the ITS tables and restrict GITS_BASER<n>

[Fuad Tabba]

The ITS writes device and collection entries into the tables
GITS_BASER<n> points to. At setup the hypervisor takes every such table
and every level-2 page an indirect device table already points to. That
includes the vPE table, which the driver allocates whenever the ITS has
one. With GICv4 off for the host, only the device table gains level-2
pages after setup.

From the takeover on, the driver writes new level-1 entries into a copy
of the device table, which the hypervisor pins. On the first MAPD that
needs a new entry, the hypervisor publishes the copy's entry if it holds
only Valid and the address of an aligned level-2 page, and takes and
zeroes that page. A valid hardware entry never changes, and a MAPD whose
entry stays invalid is rejected, as the ITS would discard its write. So
is a MAPD past the end of a flat device table.

GITS_BASER<n> writes now go through only to a register that pointed to
a table at setup, with the ITS disabled and quiescent. The hypervisor
then writes back the value it read at setup, and an enable aborts unless
every such register still holds it.

Co-developed-by: Sebastian Ene <sebastianene@google.com>
Signed-off-by: Sebastian Ene <sebastianene@google.com>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---

Notes:
    Changes since v2 (v2 patches 07 and 12, and the level-1 half of 08):
    - The tables come from GITS_BASER<n> as the hypervisor reads them,
      not from a host-supplied address, value, order and page size
      (sashiko)
    - A level-1 entry is write-once: only Valid 0 -> 1 is published (was
      any Valid flip, the page given back on 1 -> 0)
    - A published entry is read once and must name an aligned level-2
      page, which is zeroed, and cleaned for a non-shareable table
      (sashiko)
    - Level-2 pages are taken as DIV_ROUND_UP(psz, PAGE_SIZE) kernel
      pages, where v2's psz >> PAGE_SHIFT was 0 below PAGE_SIZE
      (sashiko)
    - Level-1 entries are read and written as __le64 (sashiko)
    - Page_Size, Size, Entry_Size and the 64K address encoding are
      decoded per the architecture (v2 didn't fold PA[51:48])
    - Only the device table has a level-1 copy (was a copy of every
      table)
    - A vPE table that GICv4.1 ITSes share is taken once (was a second
      donation, which fails)
    - A MAPD whose level-1 entry stays invalid is rejected
    - A GITS_BASER<n> write stores the value read at setup, whole,
      rather than the host's value under a mask (sashiko)
    - A GITS_BASER<n> write to a register that named no table, or while
      the ITS is enabled or not quiescent, aborts (was a silent drop)
    - An enable also aborts unless every GITS_BASER<n> that named a
      table still holds its value

 arch/arm64/kvm/hyp/nvhe/its_emulate.c | 252 +++++++++++++++++++++++++-
 drivers/irqchip/irq-gic-v3-its.c      |  28 ++-
 include/linux/irqchip/arm-gic-v3.h    |   3 +-
 3 files changed, 275 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/kvm/hyp/nvhe/its_emulate.c b/arch/arm64/kvm/hyp/nvhe/its_emulate.c
index b0181cb3fe9ba..5dd959451f989 100644
--- a/arch/arm64/kvm/hyp/nvhe/its_emulate.c
+++ b/arch/arm64/kvm/hyp/nvhe/its_emulate.c
@@ -21,12 +21,27 @@
 #define GITS_CWRITER_RETRY	BIT_ULL(0)
 #define GITS_CREADR_OFFSET	GENMASK_ULL(19, 5)
 #define GITS_CBASER_SIZE	GENMASK_ULL(7, 0)
+#define GITS_LVL1_ADDRESS	GENMASK_ULL(51, 12)
+
+/* A GITS_BASER<n> table, as the register described it at setup. */
+struct its_table {
+	/* hw is hyp-owned; l1_copy is the host's level-1 copy, pinned. */
+	void			*hw;
+	void			*l1_copy;
+	u64			baser;
+	size_t			size;
+	u32			psz;
+	u8			esz;
+	bool			indirect;
+};
 
 /* Per-ITS state, in the pages the host donated at setup. */
 struct its_priv {
 	hyp_spinlock_t		lock;
 	void __iomem		*base;
 	u64			cbaser;
+	struct its_table	tables[GITS_BASER_NR_REGS];
+	struct its_table	*dev_table;
 
 	/* cmdq_hw is hyp-owned; cmdq_copy is the host's copy, pinned. */
 	void			*cmdq_hw;
@@ -55,10 +70,75 @@ static int its_pin_host_pages(phys_addr_t pa, u64 nr_pages)
 	return hyp_pin_shared_mem(va, va + (nr_pages << PAGE_SHIFT));
 }
 
+static size_t its_l2_size(struct its_table *t)
+{
+	return DIV_ROUND_UP(t->psz, PAGE_SIZE) << PAGE_SHIFT;
+}
+
+/*
+ * The ITS discards a device table write through an invalid level-1 entry, so
+ * make the entry for device_id valid from the host's copy. A hardware entry
+ * only ever goes from invalid to valid.
+ */
+static bool its_publish_l1(struct its_priv *its, u32 device_id)
+{
+	struct its_table *t = its->dev_table;
+	size_t l2_size;
+	u64 idx, entry;
+	phys_addr_t pa;
+	__le64 *hw;
+
+	/* No device table in memory, so no level-1 entry to publish. */
+	if (!t)
+		return true;
+
+	if (!t->indirect)
+		return device_id < t->size / t->esz;
+
+	idx = device_id / (t->psz / t->esz);
+	if (idx >= t->size / GITS_LVL1_ENTRY_SIZE)
+		return false;
+
+	hw = t->hw;
+	if (le64_to_cpu(hw[idx]) & GITS_BASER_VALID)
+		return true;
+
+	/* The host can change its copy: read the entry once. */
+	entry = le64_to_cpu(READ_ONCE(((__le64 *)t->l1_copy)[idx]));
+
+	l2_size = its_l2_size(t);
+	pa = entry & GITS_LVL1_ADDRESS;
+	if (entry != (pa | GITS_BASER_VALID) || !IS_ALIGNED(pa, l2_size))
+		return false;
+	if (__pkvm_host_donate_hyp(PHYS_PFN(pa), l2_size >> PAGE_SHIFT))
+		return false;
+
+	/* Using a new level-2 table that isn't all zeros is UNPREDICTABLE (IHI 0069H.b 5.2.1). */
+	memset(hyp_phys_to_virt(pa), 0, l2_size);
+	if (!(t->baser & GITS_BASER_SHAREABILITY_MASK))
+		gic_flush_dcache_to_poc(hyp_phys_to_virt(pa), l2_size);
+	/* Zero the page before the entry is valid: the ITS can walk it at any time. */
+	dma_wmb();
+	hw[idx] = cpu_to_le64(entry);
+	if (!(t->baser & GITS_BASER_SHAREABILITY_MASK))
+		gic_flush_dcache_to_poc(&hw[idx], sizeof(*hw));
+	/* Visible to the ITS before the command that relies on it is queued. */
+	dsb(sy);
+	return true;
+}
+
+static bool its_mapd(struct its_priv *its, struct its_cmd_block *cmd)
+{
+	u32 device_id = its_cmd_get_deviceid(cmd->raw_cmd);
+
+	return its_publish_l1(its, device_id);
+}
+
 static bool its_process_cmd(struct its_priv *its, struct its_cmd_block *cmd)
 {
 	switch (its_cmd_get_command(cmd->raw_cmd)) {
 	case GITS_CMD_MAPD:
+		return its_mapd(its, cmd);
 	case GITS_CMD_MAPC:
 	case GITS_CMD_MAPTI:
 	case GITS_CMD_MAPI:
@@ -137,10 +217,26 @@ static bool its_cwriter_write(struct its_priv *its, u64 val)
 	return !rejected;
 }
 
-/* A reset clears GITS_CBASER.Valid, and an enable with it clear is UNPREDICTABLE. */
+/*
+ * A reset clears the Valid bits of GITS_CBASER and GITS_BASER<n>, and an enable
+ * with GITS_CBASER's or the device table's clear is UNPREDICTABLE.
+ */
 static bool its_bases_intact(struct its_priv *its)
 {
-	return readq_relaxed(its->base + GITS_CBASER) == its->cbaser;
+	unsigned int n;
+
+	if (readq_relaxed(its->base + GITS_CBASER) != its->cbaser)
+		return false;
+
+	for (n = 0; n < GITS_BASER_NR_REGS; n++) {
+		u64 baser = its->tables[n].baser;
+
+		if ((baser & GITS_BASER_VALID) &&
+		    readq_relaxed(its->base + GITS_BASER + n * sizeof(u64)) != baser)
+			return false;
+	}
+
+	return true;
 }
 
 static bool its_ctlr_write(struct its_priv *its, u64 val)
@@ -184,6 +280,21 @@ static bool its_cbaser_write(struct its_priv *its)
 	return true;
 }
 
+static bool its_baser_write(struct its_priv *its, unsigned int index)
+{
+	u64 baser = its->tables[index].baser;
+
+	if (!its_disabled_quiescent(its))
+		return false;
+
+	/* Refuse a table the hypervisor didn't take: the driver never restores one. */
+	if (!(baser & GITS_BASER_VALID))
+		return false;
+
+	writeq_relaxed(baser, its->base + GITS_BASER + index * sizeof(u64));
+	return true;
+}
+
 static bool its_emulate(struct its_priv *its, u64 offset, bool write, u64 *val, u8 size)
 {
 	if (write) {
@@ -196,10 +307,8 @@ static bool its_emulate(struct its_priv *its, u64 offset, bool write, u64 *val,
 			/* The driver writes the low word; the high word is RES0. */
 			return its_cwriter_write(its, *val);
 		case GITS_BASER ... GITS_BASER + (GITS_BASER_NR_REGS - 1) * sizeof(u64):
-			if (size != sizeof(u64))
-				return false;
-			writeq_relaxed(*val, its->base + offset);
-			return true;
+			return size == sizeof(u64) &&
+			       its_baser_write(its, (offset - GITS_BASER) / sizeof(u64));
 		default:
 			return false;
 		}
@@ -263,6 +372,133 @@ static int its_take_cmdq(struct its_priv *its, phys_addr_t copy_pa)
 	return 0;
 }
 
+static u32 its_baser_psz(u64 baser)
+{
+	switch (FIELD_GET(GITS_BASER_PAGE_SIZE_MASK, baser)) {
+	case GIC_PAGE_SIZE_4K:
+		return SZ_4K;
+	case GIC_PAGE_SIZE_16K:
+		return SZ_16K;
+	default:
+		/* 0b11 is reserved and treated as 64K. */
+		return SZ_64K;
+	}
+}
+
+static phys_addr_t its_baser_pa(u64 baser, u32 psz)
+{
+	/* With 64K pages, bits [15:12] of the register hold PA[51:48]. */
+	if (psz == SZ_64K)
+		return GITS_BASER_ADDR_48_to_52(baser);
+
+	return baser & GENMASK_ULL(47, 12);
+}
+
+static int its_take_level2(struct its_table *t, u64 nr_entries)
+{
+	u64 l2_pages = its_l2_size(t) >> PAGE_SHIFT;
+	__le64 *table = t->hw;
+	u64 i, entry;
+	int ret;
+
+	for (i = 0; i < nr_entries; i++) {
+		entry = le64_to_cpu(table[i]);
+		if (!(entry & GITS_BASER_VALID))
+			continue;
+
+		ret = __pkvm_host_donate_hyp(PHYS_PFN(entry & GITS_LVL1_ADDRESS), l2_pages);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int its_take_table(struct its_table *t, phys_addr_t copy_pa)
+{
+	u64 baser = t->baser, nr_pages;
+	phys_addr_t pa;
+	int ret;
+
+	t->psz = its_baser_psz(baser);
+	t->esz = GITS_BASER_ENTRY_SIZE(baser);
+	t->size = GITS_BASER_NR_PAGES(baser) * t->psz;
+	t->indirect = baser & GITS_BASER_INDIRECT;
+	pa = its_baser_pa(baser, t->psz);
+	nr_pages = DIV_ROUND_UP(t->size, PAGE_SIZE);
+
+	ret = __pkvm_host_donate_hyp(PHYS_PFN(pa), nr_pages);
+	if (ret)
+		return ret;
+	t->hw = hyp_phys_to_virt(pa);
+
+	if (!t->indirect)
+		return 0;
+
+	if (copy_pa) {
+		ret = its_pin_host_pages(copy_pa, nr_pages);
+		if (ret)
+			return ret;
+		t->l1_copy = hyp_phys_to_virt(copy_pa);
+	}
+
+	return its_take_level2(t, t->size / GITS_LVL1_ENTRY_SIZE);
+}
+
+/* GICv4.1 ITSes with a common affinity share one vPE table (GITS_TYPER.SVPET). */
+static bool its_vpe_table_taken(u64 baser)
+{
+	unsigned int i, n;
+
+	for (i = 0; i < num_protected_reg; i++) {
+		struct its_priv *its = pkvm_protected_regs[i].priv;
+
+		if (!its)
+			continue;
+
+		for (n = 0; n < GITS_BASER_NR_REGS; n++) {
+			if (its->tables[n].hw && its->tables[n].baser == baser)
+				return true;
+		}
+	}
+
+	return false;
+}
+
+static int its_take_tables(struct its_priv *its, struct its_pkvm_desc *desc)
+{
+	unsigned int n;
+	int ret;
+
+	for (n = 0; n < GITS_BASER_NR_REGS; n++) {
+		struct its_table *t = &its->tables[n];
+		phys_addr_t copy_pa = 0;
+
+		t->baser = readq_relaxed(its->base + GITS_BASER + n * sizeof(u64));
+		if (!(t->baser & GITS_BASER_VALID))
+			continue;
+
+		switch (GITS_BASER_TYPE(t->baser)) {
+		case GITS_BASER_TYPE_DEVICE:
+			/* The driver publishes level-1 entries through its copy. */
+			copy_pa = desc->dev_l1_copy_pa;
+			its->dev_table = t;
+			break;
+		case GITS_BASER_TYPE_VCPU:
+			/* Taken: writing it while the ITS is enabled is UNPREDICTABLE (5.2.1). */
+			if (its_vpe_table_taken(t->baser))
+				continue;
+			break;
+		}
+
+		ret = its_take_table(t, copy_pa);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
 int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
 			   size_t desc_size)
 {
@@ -304,6 +540,10 @@ int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
 	if (ret)
 		return ret;
 
+	ret = its_take_tables(its, desc);
+	if (ret)
+		return ret;
+
 	WARN_ON(__pkvm_hyp_donate_host(PHYS_PFN(desc_pa), desc_pages));
 	/* Published by stop_machine()'s exit: no host CPU traps until then. */
 	region->priv = its;
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 298e4ca1b0759..cc89691f43cb4 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -5247,11 +5247,14 @@ static int __init its_compute_its_list_map(struct its_node *its)
 
 /*
  * Protected KVM: from the takeover on, the driver writes commands into a copy
- * of the command queue, and the hypervisor checks each before the ITS sees it.
+ * of the command queue and level-1 entries into a copy of the device table,
+ * and the hypervisor checks each before the ITS sees it.
  */
 struct its_pkvm_state {
 	void			*priv;
 	void			*cmdq;
+	struct its_baser	*dev_baser;
+	void			*dev_l1;
 	struct its_pkvm_desc	*desc;
 	size_t			desc_size;
 };
@@ -5265,6 +5268,7 @@ static DEFINE_MUTEX(its_pkvm_lock);
 static int __init its_pkvm_alloc_copies(struct its_node *its,
 					struct its_pkvm_state *st)
 {
+	struct its_baser *baser = its_get_baser(its, GITS_BASER_TYPE_DEVICE);
 	struct page *page;
 
 	page = its_alloc_pages_node(its->numa_node, GFP_KERNEL, ITS_PKVM_PRIV_ORDER);
@@ -5278,6 +5282,15 @@ static int __init its_pkvm_alloc_copies(struct its_node *its,
 		return -ENOMEM;
 	st->cmdq = page_address(page);
 
+	if (!baser || !(baser->val & GITS_BASER_INDIRECT))
+		return 0;
+
+	page = its_alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO, baser->order);
+	if (!page)
+		return -ENOMEM;
+	st->dev_baser = baser;
+	st->dev_l1 = page_address(page);
+
 	return 0;
 }
 
@@ -5315,6 +5328,8 @@ static int __init its_pkvm_prepare_one(struct its_node *its)
 	desc->priv_pa = virt_to_phys(st->priv);
 	desc->priv_pages = 1 << ITS_PKVM_PRIV_ORDER;
 	desc->cmdq_copy_pa = virt_to_phys(st->cmdq);
+	if (st->dev_l1)
+		desc->dev_l1_copy_pa = virt_to_phys(st->dev_l1);
 
 	i = 0;
 	list_for_each_entry(dev, &its->its_device_list, entry) {
@@ -5333,6 +5348,9 @@ static int __init its_pkvm_hand_over(struct its_node *its,
 {
 	struct its_pkvm_state *st = its->pkvm;
 
+	if (st->dev_l1)
+		memcpy(st->dev_l1, st->dev_baser->base, PAGE_ORDER_TO_SIZE(st->dev_baser->order));
+
 	return setup(its->phys_base, st->desc, st->desc_size);
 }
 
@@ -5344,6 +5362,12 @@ static void __init its_pkvm_switch(struct its_node *its)
 	its->cmd_base = st->cmdq;
 	st->cmdq = NULL;
 	st->priv = NULL;
+
+	/* val keeps naming the hardware table, which the hypervisor now owns. */
+	if (st->dev_l1) {
+		st->dev_baser->base = st->dev_l1;
+		st->dev_l1 = NULL;
+	}
 }
 
 static void __init its_pkvm_finish_one(struct its_node *its)
@@ -5354,6 +5378,8 @@ static void __init its_pkvm_finish_one(struct its_node *its)
 		its_free_pages(st->priv, ITS_PKVM_PRIV_ORDER);
 	if (st->cmdq)
 		its_free_pages(st->cmdq, get_order(ITS_CMD_QUEUE_SZ));
+	if (st->dev_l1)
+		its_free_pages(st->dev_l1, st->dev_baser->order);
 
 	if (st->desc)
 		its_free_pages(st->desc, get_order(st->desc_size));
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index a708e188898ef..64c594c23bd46 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -689,12 +689,13 @@ struct its_pkvm_device {
 /*
  * What the host hands the hypervisor at setup: pages for its per-ITS state,
  * and the memory it cannot discover from the ITS itself. Physical addresses,
- * page-aligned.
+ * page-aligned; dev_l1_copy_pa is set when the device table is indirect.
  */
 struct its_pkvm_desc {
 	u64			priv_pa;
 	u64			priv_pages;
 	u64			cmdq_copy_pa;
+	u64			dev_l1_copy_pa;
 	u32			nr_devices;
 	struct its_pkvm_device	devices[];
 };
-- 
2.47.3

[PATCH v3 11/13] KVM: arm64: Take the ITT for the lifetime of a device mapping

[Fuad Tabba]

Once a device is mapped, the ITS writes its ITT on every MAPTI and reads
it on every translation, indexing the collection and vPE tables with
whatever ICID and vPEID the ITE holds (IHI 0069H.b B.7). As "The format
of the ITT entries is IMPLEMENTATION DEFINED" (5.3.10), the hypervisor
can't check an ITT the host can write.

On a MAPD with V == 1, the hypervisor takes the ITT's pages and zeroes
them before forwarding the command, and rejects the MAPD if the DeviceID
is already mapped, the ITT isn't page-aligned or Size exceeds
GITS_TYPER.ID_bits. It sizes the ITT from the MAPD's Size and
GITS_TYPER.ITT_entry_size, except on QDF2400 rev 1, whose ITS writes
16-byte entries whatever ITT_entry_size holds (erratum 0065). The ITTs
of the devices mapped before the takeover, which the driver lists, are
taken as they stand.

A MAPD with V == 0 gives the pages back once the hypervisor has seen the
ITS consume it, on a GITS_CREADR read or a GITS_CWRITER or GITS_CTLR
write. The driver's unmap polls GITS_CREADR before freeing the ITT, so
the pages are back by then, unless the poll times out on a stalled ITS.
The host's next access to the pages it freed then aborts.

A GITS_CBASER write resets the queue, so it now aborts until the ITS has
consumed every forwarded command, as a dropped MAPD would leave a device
tracked that the ITS never mapped. The driver writes GITS_CBASER only on
resume, after its waits, so a resume aborts only if a wait timed out
before suspend.

The hypervisor tracks a fixed number of devices per ITS and returns that
number from the setup call. its_create_device() fails a device past it
rather than queue a MAPD the hypervisor would reject.

Co-developed-by: Sebastian Ene <sebastianene@google.com>
Signed-off-by: Sebastian Ene <sebastianene@google.com>
Co-developed-by: Bartłomiej Grzesik <bgrzesik@google.com>
Signed-off-by: Bartłomiej Grzesik <bgrzesik@google.com>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---

Notes:
    Changes since v2 (the ITT half of v2 patch 08):
    - The ITT is donated to the hypervisor on a MAPD, zeroed and
      cleaned, and given back after the unmap (was shared and pinned, so
      the host could still write it)
    - The ITT pages go back only once the ITS is seen past the unmap
    - A GITS_CBASER write aborts until the ITS has consumed every
      command forwarded
    - The ITT must be page-aligned, where v2 under-counted the pages of
      one starting mid-page (sashiko)
    - The page count uses 16-byte entries on QDF2400 rev 1 (erratum
      0065)
    - A MAPD V=1 for a DeviceID already mapped is rejected, where v2
      shared the ITT a second time (sashiko)
    - One tracker entry per device, in a sorted array, instead of one
      per page keyed on the pfn (sashiko)
    - The devices mapped before the takeover are taken at setup (v2
      never tracked them)
    - A MAPD V=0 for a DeviceID that isn't mapped is forwarded once its
      level-1 entry is valid (was -EINVAL)
    - its_create_device() fails a device past the tracker's capacity,
      which setup returns

 arch/arm64/kvm/hyp/nvhe/its_emulate.c | 193 +++++++++++++++++++++++++-
 arch/arm64/kvm/pkvm.c                 |   2 +-
 drivers/irqchip/irq-gic-v3-its.c      |  18 ++-
 include/linux/irqchip/arm-gic-v3.h    |   5 +-
 4 files changed, 209 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/kvm/hyp/nvhe/its_emulate.c b/arch/arm64/kvm/hyp/nvhe/its_emulate.c
index 5dd959451f989..8f06b738ad118 100644
--- a/arch/arm64/kvm/hyp/nvhe/its_emulate.c
+++ b/arch/arm64/kvm/hyp/nvhe/its_emulate.c
@@ -22,6 +22,8 @@
 #define GITS_CREADR_OFFSET	GENMASK_ULL(19, 5)
 #define GITS_CBASER_SIZE	GENMASK_ULL(7, 0)
 #define GITS_LVL1_ADDRESS	GENMASK_ULL(51, 12)
+/* Erratum 0065: the ITS writes 16-byte ITEs where GITS_TYPER says 8. */
+#define GITS_IIDR_QDF2400_R1	0x00001070
 
 /* A GITS_BASER<n> table, as the register described it at setup. */
 struct its_table {
@@ -35,14 +37,33 @@ struct its_table {
 	bool			indirect;
 };
 
+/* A mapped device; its ITT pages are hyp-owned until the ITS consumes the unmap. */
+struct its_device_map {
+	u64			itt_pa;
+	u64			release_seq;
+	u32			device_id;
+	u32			nr_pages;
+	u8			event_bits;
+	bool			release_pending;
+};
+
 /* Per-ITS state, in the pages the host donated at setup. */
 struct its_priv {
 	hyp_spinlock_t		lock;
 	void __iomem		*base;
+	u64			typer;
 	u64			cbaser;
+	u8			id_bits;
+	u8			itt_entry_size;
 	struct its_table	tables[GITS_BASER_NR_REGS];
 	struct its_table	*dev_table;
 
+	/* Sorted by device_id, in the rest of the donated pages. */
+	struct its_device_map	*devices;
+	u32			nr_devices;
+	u32			max_devices;
+	u32			nr_pending;
+
 	/* cmdq_hw is hyp-owned; cmdq_copy is the host's copy, pinned. */
 	void			*cmdq_hw;
 	void			*cmdq_copy;
@@ -53,6 +74,9 @@ struct its_priv {
 
 	/* The GITS_CREADR offset last read. */
 	u32			hw_creadr;
+	/* Commands forwarded to the ITS, and consumed by it. */
+	u64			seq_submitted;
+	u64			seq_completed;
 };
 
 static int its_pin_host_pages(phys_addr_t pa, u64 nr_pages)
@@ -127,11 +151,118 @@ static bool its_publish_l1(struct its_priv *its, u32 device_id)
 	return true;
 }
 
+/* The index of device_id, or where it would be inserted. */
+static u32 its_dev_search(struct its_priv *its, u32 device_id, bool *found)
+{
+	u32 lo = 0, hi = its->nr_devices;
+
+	while (lo < hi) {
+		u32 mid = lo + (hi - lo) / 2;
+
+		if (its->devices[mid].device_id < device_id)
+			lo = mid + 1;
+		else
+			hi = mid;
+	}
+
+	*found = lo < its->nr_devices && its->devices[lo].device_id == device_id;
+	return lo;
+}
+
+static void its_dev_remove(struct its_priv *its, u32 idx)
+{
+	its->nr_devices--;
+	memmove(&its->devices[idx], &its->devices[idx + 1],
+		(its->nr_devices - idx) * sizeof(*its->devices));
+}
+
+/*
+ * The driver page-aligns ITTs under protected mode. A MAPD on an ITT that isn't
+ * all zeros is UNPREDICTABLE, so @zero is set for a runtime MAPD.
+ */
+static int its_take_itt(struct its_priv *its, u32 device_id, u8 event_bits,
+			phys_addr_t itt_pa, bool zero)
+{
+	struct its_device_map dev = {
+		.itt_pa = itt_pa,
+		.device_id = device_id,
+		.event_bits = event_bits,
+	};
+	size_t size;
+	bool found;
+	u32 idx;
+	int ret;
+
+	if (event_bits > its->id_bits || !PAGE_ALIGNED(itt_pa))
+		return -EINVAL;
+
+	idx = its_dev_search(its, device_id, &found);
+	if (found)
+		return -EEXIST;
+	if (its->nr_devices == its->max_devices)
+		return -ENOSPC;
+
+	size = PAGE_ALIGN((1UL << event_bits) * its->itt_entry_size);
+	dev.nr_pages = size >> PAGE_SHIFT;
+	ret = __pkvm_host_donate_hyp(PHYS_PFN(itt_pa), dev.nr_pages);
+	if (ret)
+		return ret;
+
+	if (zero) {
+		memset(hyp_phys_to_virt(itt_pa), 0, size);
+		gic_flush_dcache_to_poc(hyp_phys_to_virt(itt_pa), size);
+	}
+
+	memmove(&its->devices[idx + 1], &its->devices[idx], (its->nr_devices - idx) * sizeof(dev));
+	its->devices[idx] = dev;
+	its->nr_devices++;
+	return 0;
+}
+
+/* The ITT goes back to the host once the ITS has consumed this unmap. */
+static void its_arm_release(struct its_priv *its, u32 device_id)
+{
+	struct its_device_map *dev;
+	bool found;
+
+	dev = &its->devices[its_dev_search(its, device_id, &found)];
+	if (!found || dev->release_pending)
+		return;
+
+	dev->release_pending = true;
+	dev->release_seq = its->seq_submitted + 1;
+	its->nr_pending++;
+}
+
+static void its_retire_releases(struct its_priv *its)
+{
+	u32 i = its->nr_devices;
+
+	while (its->nr_pending && i--) {
+		struct its_device_map *dev = &its->devices[i];
+
+		if (!dev->release_pending || dev->release_seq > its->seq_completed)
+			continue;
+
+		WARN_ON(__pkvm_hyp_donate_host(PHYS_PFN(dev->itt_pa), dev->nr_pages));
+		its->nr_pending--;
+		its_dev_remove(its, i);
+	}
+}
+
 static bool its_mapd(struct its_priv *its, struct its_cmd_block *cmd)
 {
 	u32 device_id = its_cmd_get_deviceid(cmd->raw_cmd);
 
-	return its_publish_l1(its, device_id);
+	if (!its_publish_l1(its, device_id))
+		return false;
+
+	if (its_cmd_get_validbit(cmd->raw_cmd))
+		return !its_take_itt(its, device_id, its_cmd_get_size(cmd->raw_cmd),
+				     its_cmd_get_ittaddr(cmd->raw_cmd), true);
+
+	its_arm_release(its, device_id);
+	return true;
 }
 
 static bool its_process_cmd(struct its_priv *its, struct its_cmd_block *cmd)
@@ -157,12 +288,21 @@ static bool its_process_cmd(struct its_priv *its, struct its_cmd_block *cmd)
 	}
 }
 
+/* Count the commands consumed since the last read, and release the ITTs they unmapped. */
 static u64 its_read_creadr(struct its_priv *its)
 {
 	u64 creadr = readq_relaxed(its->base + GITS_CREADR);
 	u32 offset = creadr & GITS_CREADR_OFFSET;
 
+	/*
+	 * A disabled ITS consumes no commands, so GITS_CREADR moving while it is
+	 * disabled, as after a reset behind the hypervisor, isn't progress.
+	 */
+	if (readl_relaxed(its->base + GITS_CTLR) & GITS_CTLR_ENABLE)
+		its->seq_completed += ((offset + its->cmdq_size - its->hw_creadr) %
+				       its->cmdq_size) / sizeof(struct its_cmd_block);
 	its->hw_creadr = offset;
+	its_retire_releases(its);
 
 	return creadr;
 }
@@ -209,6 +349,7 @@ static bool its_cwriter_write(struct its_priv *its, u64 val)
 		 */
 		gic_flush_dcache_to_poc(its->cmdq_hw + its->cmd_offset, sizeof(cmd));
 		its->cmd_offset = (its->cmd_offset + sizeof(cmd)) % its->cmdq_size;
+		its->seq_submitted++;
 	}
 
 	/* Retry has no effect unless the hardware stalled on a command error. */
@@ -241,7 +382,11 @@ static bool its_bases_intact(struct its_priv *its)
 
 static bool its_ctlr_write(struct its_priv *its, u64 val)
 {
-	u32 ctlr = readl_relaxed(its->base + GITS_CTLR);
+	u32 ctlr;
+
+	/* Read while still disabled, so a reset GITS_CREADR isn't counted as progress. */
+	its_read_creadr(its);
+	ctlr = readl_relaxed(its->base + GITS_CTLR);
 
 	if ((val & GITS_CTLR_ENABLE) && !(ctlr & GITS_CTLR_ENABLE)) {
 		/*
@@ -268,12 +413,13 @@ static bool its_disabled_quiescent(struct its_priv *its)
 
 static bool its_cbaser_write(struct its_priv *its)
 {
-	if (!its_disabled_quiescent(its))
+	/* The write resets the queue, dropping any command the ITS hasn't consumed. */
+	if (!its_disabled_quiescent(its) || its->seq_completed != its->seq_submitted)
 		return false;
 
 	writeq_relaxed(its->cbaser, its->base + GITS_CBASER);
 
-	/* The write reset GITS_CREADR to 0; nothing queued before it completes. */
+	/* The write reset GITS_CREADR to 0. */
 	its->cmd_offset = 0;
 	its->hw_creadr = 0;
 	writeq_relaxed(0, its->base + GITS_CWRITER);
@@ -314,6 +460,13 @@ static bool its_emulate(struct its_priv *its, u64 offset, bool write, u64 *val,
 		}
 	}
 
+	if (offset == GITS_CREADR) {
+		*val = its_read_creadr(its);
+		if (size == sizeof(u32))
+			*val = lower_32_bits(*val);
+		return true;
+	}
+
 	*val = size == sizeof(u64) ? readq_relaxed(its->base + offset) :
 				     readl_relaxed(its->base + offset);
 	return true;
@@ -369,6 +522,8 @@ static int its_take_cmdq(struct its_priv *its, phys_addr_t copy_pa)
 	its->cmdq_size = size;
 	its->cmd_offset = cwriter;
 	its->hw_creadr = creadr;
+	/* Commands the host queued before setup are submitted, not yet consumed. */
+	its->seq_submitted = ((cwriter + size - creadr) % size) / sizeof(struct its_cmd_block);
 	return 0;
 }
 
@@ -499,6 +654,22 @@ static int its_take_tables(struct its_priv *its, struct its_pkvm_desc *desc)
 	return 0;
 }
 
+static int its_take_devices(struct its_priv *its, struct its_pkvm_desc *desc)
+{
+	u32 i;
+	int ret;
+
+	for (i = 0; i < desc->nr_devices; i++) {
+		struct its_pkvm_device dev = desc->devices[i];
+
+		ret = its_take_itt(its, dev.device_id, dev.event_bits, dev.itt_pa, false);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
 int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
 			   size_t desc_size)
 {
@@ -534,7 +705,15 @@ int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
 
 	hyp_spin_lock_init(&its->lock);
 	its->base = region->hyp_va;
+	its->typer = readq_relaxed(its->base + GITS_TYPER);
 	its->cbaser = readq_relaxed(its->base + GITS_CBASER);
+	its->id_bits = FIELD_GET(GITS_TYPER_IDBITS, its->typer) + 1;
+	its->itt_entry_size = FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1;
+	its->devices = (void *)(its + 1);
+	its->max_devices = (priv_pages * PAGE_SIZE - sizeof(*its)) / sizeof(*its->devices);
+
+	if (readl_relaxed(its->base + GITS_IIDR) == GITS_IIDR_QDF2400_R1)
+		its->itt_entry_size = 16;
 
 	ret = its_take_cmdq(its, desc->cmdq_copy_pa);
 	if (ret)
@@ -544,8 +723,12 @@ int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
 	if (ret)
 		return ret;
 
+	ret = its_take_devices(its, desc);
+	if (ret)
+		return ret;
+
 	WARN_ON(__pkvm_hyp_donate_host(PHYS_PFN(desc_pa), desc_pages));
 	/* Published by stop_machine()'s exit: no host CPU traps until then. */
 	region->priv = its;
-	return 0;
+	return its->max_devices;
 }
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index 3293e3c7353b8..3e9c734c264bc 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -85,7 +85,7 @@ static int __init pkvm_init_its_emulation(phys_addr_t its_base,
 	int ret;
 
 	ret = kvm_call_hyp_nvhe(__pkvm_its_emulate_setup, its_base, virt_to_phys(desc), desc_size);
-	if (ret)
+	if (ret < 0)
 		kvm_err("ITS@%pa: emulation setup failed: %d\n", &its_base, ret);
 
 	return ret;
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index cc89691f43cb4..35ab7bebd36f0 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -128,6 +128,7 @@ struct its_node {
 	u32			pre_its_base; /* for Socionext Synquacer */
 	int			vlpi_redist_offset;
 	struct its_pkvm_state	*pkvm;
+	u32			pkvm_max_devices;
 };
 
 static DEFINE_PER_CPU(struct its_node *, local_4_1_its);
@@ -3468,6 +3469,11 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	int id_bits;
 	int sz;
 
+	/* The hypervisor rejects a MAPD past its device tracker's capacity. */
+	if (its->pkvm_max_devices &&
+	    list_count_nodes(&its->its_device_list) >= its->pkvm_max_devices)
+		return NULL;
+
 	if (!its_alloc_device_table(its, dev_id))
 		return NULL;
 
@@ -5251,6 +5257,7 @@ static int __init its_compute_its_list_map(struct its_node *its)
  * and the hypervisor checks each before the ITS sees it.
  */
 struct its_pkvm_state {
+	u32			max_devices;
 	void			*priv;
 	void			*cmdq;
 	struct its_baser	*dev_baser;
@@ -5262,7 +5269,7 @@ struct its_pkvm_state {
 /* Held, with every ITS's dev_alloc_lock, from prepare to finish. */
 static DEFINE_MUTEX(its_pkvm_lock);
 
-/* Pages for the hypervisor's per-ITS state. */
+/* The hypervisor's per-ITS state, its device tracker included. */
 #define ITS_PKVM_PRIV_ORDER	3
 
 static int __init its_pkvm_alloc_copies(struct its_node *its,
@@ -5347,11 +5354,17 @@ static int __init its_pkvm_hand_over(struct its_node *its,
 				     its_pkvm_setup_fn setup)
 {
 	struct its_pkvm_state *st = its->pkvm;
+	int ret;
 
 	if (st->dev_l1)
 		memcpy(st->dev_l1, st->dev_baser->base, PAGE_ORDER_TO_SIZE(st->dev_baser->order));
 
-	return setup(its->phys_base, st->desc, st->desc_size);
+	ret = setup(its->phys_base, st->desc, st->desc_size);
+	if (ret < 0)
+		return ret;
+
+	st->max_devices = ret;
+	return 0;
 }
 
 static void __init its_pkvm_switch(struct its_node *its)
@@ -5360,6 +5373,7 @@ static void __init its_pkvm_switch(struct its_node *its)
 
 	its->cmd_write = (struct its_cmd_block *)st->cmdq + (its->cmd_write - its->cmd_base);
 	its->cmd_base = st->cmdq;
+	its->pkvm_max_devices = st->max_devices;
 	st->cmdq = NULL;
 	st->priv = NULL;
 
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 64c594c23bd46..794d37c3bbf55 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -700,7 +700,10 @@ struct its_pkvm_desc {
 	struct its_pkvm_device	devices[];
 };
 
-/* Called with every CPU parked: no allocation. */
+/*
+ * Returns < 0 or the number of devices the hypervisor can track on this ITS.
+ * Called with every CPU parked: no allocation.
+ */
 typedef int (*its_pkvm_setup_fn)(phys_addr_t its_base,
 				 struct its_pkvm_desc *desc, size_t desc_size);
 
-- 
2.47.3

[PATCH v3 12/13] KVM: arm64: Reject ITS commands whose IDs are out of range

[Fuad Tabba]

The ITS can't be relied on to reject a command with an out-of-range ID:
IHI 0069H.b 5.3.2 makes the response to a command error CONSTRAINED
UNPREDICTABLE, and one permitted outcome is "The data that generated the
error or errors is treated as having a legal value, and the command is
processed accordingly". An out-of-range DeviceID, EventID or ICID then
has the ITS access memory past the device table, the ITT or the
collection table.

Bound each ID before forwarding the command: the DeviceID by
GITS_TYPER.Devbits and the device table, the EventID by the size its
device was mapped with, and the ICID by the ICID width and the number of
collections. Also reject a command for an event of a device whose unmap
is queued, as its ITE would land in an ITT the hypervisor may already
have released. The driver loses nothing. It never allocates an
out-of-range DeviceID or EventID, or sends a command for a device after
its unmap. Its ICIDs are CPU numbers, which exceed the collections only
on an ITS that already treats its MAPC as a command error (IHI 0069H.b
5.3.9).

MAPTI's pINTID, which the ITS only stores in the ITE, and the RDbase of
MAPC, MOVALL and SYNC are left to the redistributor series, as both
concern the redistributors.

Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 arch/arm64/kvm/hyp/nvhe/its_emulate.c | 49 +++++++++++++++++++++++++--
 include/linux/irqchip/arm-gic-v3.h    |  3 ++
 2 files changed, 49 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/kvm/hyp/nvhe/its_emulate.c b/arch/arm64/kvm/hyp/nvhe/its_emulate.c
index 8f06b738ad118..8bcdb06632b65 100644
--- a/arch/arm64/kvm/hyp/nvhe/its_emulate.c
+++ b/arch/arm64/kvm/hyp/nvhe/its_emulate.c
@@ -55,8 +55,12 @@ struct its_priv {
 	u64			cbaser;
 	u8			id_bits;
 	u8			itt_entry_size;
+	u8			dev_bits;
+	u8			icid_bits;
+	u32			max_collections;
 	struct its_table	tables[GITS_BASER_NR_REGS];
 	struct its_table	*dev_table;
+	struct its_table	*coll_table;
 
 	/* Sorted by device_id, in the rest of the donated pages. */
 	struct its_device_map	*devices;
@@ -254,7 +258,7 @@ static bool its_mapd(struct its_priv *its, struct its_cmd_block *cmd)
 {
 	u32 device_id = its_cmd_get_deviceid(cmd->raw_cmd);
 
-	if (!its_publish_l1(its, device_id))
+	if (device_id >= BIT_ULL(its->dev_bits) || !its_publish_l1(its, device_id))
 		return false;
 
 	if (its_cmd_get_validbit(cmd->raw_cmd))
@@ -265,21 +269,42 @@ static bool its_mapd(struct its_priv *its, struct its_cmd_block *cmd)
 	return true;
 }
 
+/* An event of a device whose unmap is queued is out of range too. */
+static bool its_event_in_range(struct its_priv *its, struct its_cmd_block *cmd)
+{
+	struct its_device_map *dev;
+	bool found;
+
+	dev = &its->devices[its_dev_search(its, its_cmd_get_deviceid(cmd->raw_cmd), &found)];
+	return found && !dev->release_pending &&
+	       its_cmd_get_id(cmd->raw_cmd) < BIT_ULL(dev->event_bits);
+}
+
+static bool its_collection_in_range(struct its_priv *its, struct its_cmd_block *cmd)
+{
+	u32 icid = its_cmd_get_collection(cmd->raw_cmd);
+
+	return icid < BIT_ULL(its->icid_bits) && icid < its->max_collections;
+}
+
 static bool its_process_cmd(struct its_priv *its, struct its_cmd_block *cmd)
 {
 	switch (its_cmd_get_command(cmd->raw_cmd)) {
 	case GITS_CMD_MAPD:
 		return its_mapd(its, cmd);
 	case GITS_CMD_MAPC:
+	case GITS_CMD_INVALL:
+		return its_collection_in_range(its, cmd);
 	case GITS_CMD_MAPTI:
 	case GITS_CMD_MAPI:
 	case GITS_CMD_MOVI:
+		return its_event_in_range(its, cmd) && its_collection_in_range(its, cmd);
 	case GITS_CMD_DISCARD:
 	case GITS_CMD_INV:
-	case GITS_CMD_MOVALL:
-	case GITS_CMD_INVALL:
 	case GITS_CMD_INT:
 	case GITS_CMD_CLEAR:
+		return its_event_in_range(its, cmd);
+	case GITS_CMD_MOVALL:
 	case GITS_CMD_SYNC:
 		return true;
 	default:
@@ -639,6 +664,9 @@ static int its_take_tables(struct its_priv *its, struct its_pkvm_desc *desc)
 			copy_pa = desc->dev_l1_copy_pa;
 			its->dev_table = t;
 			break;
+		case GITS_BASER_TYPE_COLLECTION:
+			its->coll_table = t;
+			break;
 		case GITS_BASER_TYPE_VCPU:
 			/* Taken: writing it while the ITS is enabled is UNPREDICTABLE (5.2.1). */
 			if (its_vpe_table_taken(t->baser))
@@ -654,6 +682,17 @@ static int its_take_tables(struct its_priv *its, struct its_pkvm_desc *desc)
 	return 0;
 }
 
+/* GITS_TYPER.CCT indicates whether the ITS's HCC collections add to the memory table's. */
+static u32 its_collection_count(struct its_priv *its)
+{
+	u32 hcc = GITS_TYPER_HCC(its->typer), in_mem = 0;
+
+	if (its->coll_table)
+		in_mem = its->coll_table->size / its->coll_table->esz;
+
+	return its->typer & GITS_TYPER_CCT ? hcc + in_mem : max(hcc, in_mem);
+}
+
 static int its_take_devices(struct its_priv *its, struct its_pkvm_desc *desc)
 {
 	u32 i;
@@ -709,6 +748,9 @@ int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
 	its->cbaser = readq_relaxed(its->base + GITS_CBASER);
 	its->id_bits = FIELD_GET(GITS_TYPER_IDBITS, its->typer) + 1;
 	its->itt_entry_size = FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1;
+	its->dev_bits = FIELD_GET(GITS_TYPER_DEVBITS, its->typer) + 1;
+	its->icid_bits = its->typer & GITS_TYPER_CIL ?
+			 FIELD_GET(GITS_TYPER_CIDBITS, its->typer) + 1 : 16;
 	its->devices = (void *)(its + 1);
 	its->max_devices = (priv_pages * PAGE_SIZE - sizeof(*its)) / sizeof(*its->devices);
 
@@ -722,6 +764,7 @@ int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
 	ret = its_take_tables(its, desc);
 	if (ret)
 		return ret;
+	its->max_collections = its_collection_count(its);
 
 	ret = its_take_devices(its, desc);
 	if (ret)
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 794d37c3bbf55..9e2a42a5ceb40 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -392,6 +392,7 @@
 
 #define GITS_TYPER_PLPIS		(1UL << 0)
 #define GITS_TYPER_VLPIS		(1UL << 1)
+#define GITS_TYPER_CCT			(1UL << 2)
 #define GITS_TYPER_ITT_ENTRY_SIZE_SHIFT	4
 #define GITS_TYPER_ITT_ENTRY_SIZE	GENMASK_ULL(7, 4)
 #define GITS_TYPER_IDBITS		GENMASK_ULL(12, 8)
@@ -401,6 +402,8 @@
 #define GITS_TYPER_PTA			(1UL << 19)
 #define GITS_TYPER_HCC_SHIFT		24
 #define GITS_TYPER_HCC(r)		(((r) >> GITS_TYPER_HCC_SHIFT) & 0xff)
+#define GITS_TYPER_CIDBITS		GENMASK_ULL(35, 32)
+#define GITS_TYPER_CIL			(1ULL << 36)
 #define GITS_TYPER_VMOVP		(1ULL << 37)
 #define GITS_TYPER_VMAPP		(1ULL << 40)
 #define GITS_TYPER_SVPET		GENMASK_ULL(42, 41)
-- 
2.47.3

[PATCH v3 13/13] Documentation: KVM: arm64: Describe the ITS hardening

[Fuad Tabba]

Add the ITS hardening to pkvm.rst: why the hypervisor emulates the
host's ITS, and how protected mode differs from non-protected mode as a
result.

Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 Documentation/virt/kvm/arm/pkvm.rst | 32 +++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/Documentation/virt/kvm/arm/pkvm.rst b/Documentation/virt/kvm/arm/pkvm.rst
index b75bb8ad6f26d..1ebae35f8373e 100644
--- a/Documentation/virt/kvm/arm/pkvm.rst
+++ b/Documentation/virt/kvm/arm/pkvm.rst
@@ -89,6 +89,38 @@ DMA isolation using an IOMMU
 
 Status: **Unimplemented.**
 
+GICv3 ITS hardening
+-------------------
+
+Status: Host accesses to the ITS control registers are trapped and
+emulated by the hypervisor. The LPI configuration and pending tables
+remain under host control.
+
+The ITS reads and writes memory at addresses programmed by the host, so
+the hypervisor emulates the ITS control registers and command queue to
+ensure that the host cannot use the ITS as a confused deputy to write
+to hypervisor or pVM memory. The command queue, the ITS tables and each
+device's interrupt translation table are owned by the hypervisor while
+the ITS can access them.
+
+This gives rise to some user-visible differences when compared to
+non-protected mode:
+
+* Protected mode requires the hypervisor to take control of every ITS
+  in the system: KVM does not initialise with more than 8 ITSes, and
+  if the takeover fails the host is not deprivileged and
+  ``KVM_CREATE_VM`` returns ``-ENODEV``.
+* An ITS command or register write that the hypervisor rejects is
+  delivered to the host as a data abort.
+* The number of devices that can be mapped through each ITS is limited
+  by the hypervisor, and MSI allocation fails with ``-ENOMEM`` beyond
+  that limit.
+* Interrupt translation tables are page-aligned, so each device
+  consumes at least one page.
+* GICv4 is disabled: direct injection of LPIs is not available to
+  non-protected guests, and a guest saved with ``GICD_TYPER2.nASSGIcap``
+  set cannot be restored (``-EINVAL``).
+
 Proxying of Trustzone services
 ------------------------------
 
-- 
2.47.3