From: Fuad Tabba <fuad.tabba@linux.dev>
To: Marc Zyngier <maz@kernel.org>, Oliver Upton <oupton@kernel.org>
Cc: "Will Deacon" <will@kernel.org>,
"Catalin Marinas" <catalin.marinas@arm.com>,
"Joey Gouly" <joey.gouly@arm.com>,
"Mark Rutland" <mark.rutland@arm.com>,
"Suzuki K Poulose" <suzuki.poulose@arm.com>,
"Raghavendra Rao Ananta" <rananta@google.com>,
"Sascha Bischoff" <Sascha.Bischoff@arm.com>,
"Thomas Gleixner" <tglx@kernel.org>,
"Sebastian Ene" <sebastianene@google.com>,
"Bartłomiej Grzesik" <bgrzesik@google.com>,
"Mostafa Saleh" <smostafa@google.com>,
"Vincent Donnefort" <vdonnefort@google.com>,
"Vladimir Murzin" <vladimir.murzin@arm.com>,
"Zenghui Yu" <yuzenghui@huawei.com>,
"Zenghui Yu" <zenghui.yu@linux.dev>,
"Steffen Eiden" <seiden@linux.ibm.com>,
"Radu Rendec" <radu@rendec.net>, "Fuad Tabba" <tabba@google.com>,
kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org,
linux-kernel@vger.kernel.org
Subject: [PATCH v4 09/13] KVM: arm64: Emulate the ITS command queue
Date: Wed, 18 Nov 2026 15:19:00 +0000 [thread overview]
Message-ID: <20261118151904.4099560-10-fuad.tabba@linux.dev> (raw)
In-Reply-To: <20261118151904.4099560-1-fuad.tabba@linux.dev>
From: Sebastian Ene <sebastianene@google.com>
The ITS writes to the memory its commands point to, so a host that
queues commands freely can aim those writes at the hypervisor or a
protected guest.
Register each ITS's control frame with the hypervisor, so that every
host access to it traps once the host is deprivileged. Each ITS takes
one of the hypervisor's eight MMIO region slots, so on a host with more
than eight ITSes KVM doesn't initialise in protected mode. Inside the
stop_machine() callback that deprivileges the host, the hypervisor reads
GITS_CBASER, GITS_CWRITER and GITS_CREADR itself, takes the queue and
pins the driver's copy. If that fails, no CPU is deprivileged and
KVM_CREATE_VM fails with -ENODEV.
On a GITS_CWRITER write, the hypervisor reads each new command once from
the driver's copy of the queue, checks it and copies it into the
hardware queue. For a rejected command, the host's GITS_CWRITER write
takes a data abort, after the commands before it have been forwarded.
For now that's a vLPI command or an unknown encoding, neither of which
the driver queues.
An access aborts if its syndrome isn't valid, or if it is narrower than
four bytes or misaligned. Reads go to the hardware for the registers the
driver reads, each at the size the driver uses, and other reads abort.
GITS_CREADR, which the driver reads at 32 bits, can be read at 64 bits
too.
Writes abort, except to GITS_CWRITER, GITS_BASER<n>, GITS_CTLR and
GITS_CBASER. The next patch restricts GITS_BASER<n>, and the driver
writes GITS_CTLR and GITS_CBASER on suspend and resume. A GITS_CBASER
write is accepted only with the ITS disabled and quiescent, and the
hypervisor writes the value it read at setup in place of the host's. A
write that enables the ITS aborts unless GITS_CBASER holds that value.
While the ITS isn't quiescent, the hypervisor drops such a write rather
than abort it, which leaves the ITS disabled. The driver re-enables the
ITS from a syscore callback when a quiesce times out, and an abort
there would take the host down.
Add to arm-gic-v3.h the GITS_CBASER, GITS_CWRITER and GITS_CREADR
fields that the hypervisor decodes.
Signed-off-by: Sebastian Ene <sebastianene@google.com>
Co-developed-by: Bartłomiej Grzesik <bgrzesik@google.com>
Signed-off-by: Bartłomiej Grzesik <bgrzesik@google.com>
Co-developed-by: Fuad Tabba <fuad.tabba@linux.dev>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
arch/arm64/include/asm/kvm_asm.h | 2 +
arch/arm64/include/asm/kvm_hcall.h | 2 +
arch/arm64/include/asm/kvm_pkvm.h | 4 +
arch/arm64/kvm/arm.c | 4 +
arch/arm64/kvm/hyp/include/nvhe/its_emulate.h | 16 +
arch/arm64/kvm/hyp/nvhe/Makefile | 3 +-
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 8 +
arch/arm64/kvm/hyp/nvhe/its_emulate.c | 323 ++++++++++++++++++
arch/arm64/kvm/hyp/nvhe/mem_protect.c | 65 ++++
arch/arm64/kvm/pkvm.c | 98 +++++-
include/linux/irqchip/arm-gic-v3.h | 5 +
11 files changed, 527 insertions(+), 3 deletions(-)
create mode 100644 arch/arm64/kvm/hyp/include/nvhe/its_emulate.h
create mode 100644 arch/arm64/kvm/hyp/nvhe/its_emulate.c
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index abda476e023de..339850e2702d0 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -122,6 +122,8 @@ enum __kvm_host_smccc_func {
__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_sync_state,
__KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
+ /* Called with kvm_protected_mode_initialized on, before __pkvm_prot_finalize. */
+ __KVM_HOST_SMCCC_FUNC___pkvm_its_emulate_setup,
MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
};
diff --git a/arch/arm64/include/asm/kvm_hcall.h b/arch/arm64/include/asm/kvm_hcall.h
index 5f77a8aa29635..0748232c387ec 100644
--- a/arch/arm64/include/asm/kvm_hcall.h
+++ b/arch/arm64/include/asm/kvm_hcall.h
@@ -254,5 +254,7 @@ DECLARE_KVM_HOST_HCALL0(void, __pkvm_vcpu_put)
DECLARE_KVM_HOST_HCALL0(void, __pkvm_vcpu_sync_state)
DECLARE_KVM_HOST_HCALL(void, __pkvm_tlb_flush_vmid,
pkvm_handle_t, handle)
+DECLARE_KVM_HOST_HCALL(int, __pkvm_its_emulate_setup,
+ phys_addr_t, its_base, phys_addr_t, desc_pa, size_t, desc_size)
#endif /* __ARM64_KVM_HCALL_H__ */
diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
index 4695c35b31e1a..e1203b06ec7b3 100644
--- a/arch/arm64/include/asm/kvm_pkvm.h
+++ b/arch/arm64/include/asm/kvm_pkvm.h
@@ -24,11 +24,15 @@ struct pkvm_protected_reg {
u64 pfn;
u64 nr_pages;
void __iomem *hyp_va;
+ /* The ITS's state once it is taken. */
+ void *priv;
};
extern struct pkvm_protected_reg kvm_nvhe_sym(pkvm_protected_regs)[];
extern unsigned int kvm_nvhe_sym(num_protected_reg);
+int pkvm_register_its_regions(void);
+
int pkvm_init_host_vm(struct kvm *kvm);
int pkvm_create_hyp_vm(struct kvm *kvm);
bool pkvm_hyp_vm_is_created(struct kvm *kvm);
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 157bf4b812020..789f6ca13ae12 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -3128,6 +3128,10 @@ static int __init init_hyp_mode(void)
if (err)
goto out_err;
+ err = pkvm_register_its_regions();
+ if (err)
+ goto out_err;
+
err = kvm_hyp_init_protection(hyp_va_bits);
if (err) {
kvm_err("Failed to init hyp memory protection\n");
diff --git a/arch/arm64/kvm/hyp/include/nvhe/its_emulate.h b/arch/arm64/kvm/hyp/include/nvhe/its_emulate.h
new file mode 100644
index 0000000000000..0b0d4aa25a5b7
--- /dev/null
+++ b/arch/arm64/kvm/hyp/include/nvhe/its_emulate.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 Google LLC
+ * Author: Sebastian Ene <sebastianene@google.com>
+ */
+#ifndef __ARM64_KVM_NVHE_ITS_EMULATE_H__
+#define __ARM64_KVM_NVHE_ITS_EMULATE_H__
+
+#include <asm/kvm_pkvm.h>
+
+int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
+ size_t desc_size);
+bool pkvm_its_emulate_handler(struct pkvm_protected_reg *region, u64 offset,
+ bool write, u64 *val, u8 size);
+
+#endif /* __ARM64_KVM_NVHE_ITS_EMULATE_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
index 56a7e2c05f73a..2c50940849782 100644
--- a/arch/arm64/kvm/hyp/nvhe/Makefile
+++ b/arch/arm64/kvm/hyp/nvhe/Makefile
@@ -24,7 +24,8 @@ CFLAGS_switch.nvhe.o += -Wno-override-init
hyp-obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \
hyp-main.o hyp-smp.o psci-relay.o alloc.o early_alloc.o page_alloc.o \
- cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o
+ cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o \
+ its_emulate.o
hyp-obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
../hyp-entry.o ../exception.o ../pgtable.o ../vgic-v5-sr.o
hyp-obj-y += ../../../kernel/smccc-call.o
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 179d7c8277b88..5efaa32037a5b 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -22,6 +22,7 @@
#include <nvhe/alloc.h>
#include <nvhe/ffa.h>
+#include <nvhe/its_emulate.h>
#include <nvhe/mem_protect.h>
#include <nvhe/mm.h>
#include <nvhe/pkvm.h>
@@ -1169,6 +1170,12 @@ DEFINE_KVM_HOST_HCALL(void, __pkvm_tlb_flush_vmid,
put_pkvm_hyp_vm(hyp_vm);
}
+DEFINE_KVM_HOST_HCALL(int, __pkvm_its_emulate_setup,
+ phys_addr_t, its_base, phys_addr_t, desc_pa, size_t, desc_size)
+{
+ return pkvm_its_emulate_setup(its_base, desc_pa, desc_size);
+}
+
DEFINE_KVM_HOST_HCALL(void, __kvm_flush_cpu_context,
struct kvm_s2_mmu __kern *, mmu)
{
@@ -1603,6 +1610,7 @@ static const hcall_t host_hcall[] = {
HANDLE_FUNC(__pkvm_vcpu_put),
HANDLE_FUNC(__pkvm_vcpu_sync_state),
HANDLE_FUNC(__pkvm_tlb_flush_vmid),
+ HANDLE_FUNC(__pkvm_its_emulate_setup),
};
static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
diff --git a/arch/arm64/kvm/hyp/nvhe/its_emulate.c b/arch/arm64/kvm/hyp/nvhe/its_emulate.c
new file mode 100644
index 0000000000000..dd077abb1b24b
--- /dev/null
+++ b/arch/arm64/kvm/hyp/nvhe/its_emulate.c
@@ -0,0 +1,323 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Emulation of the host's accesses to a GICv3 ITS in pKVM.
+ *
+ * Copyright (C) 2026 Google LLC
+ * Author: Sebastian Ene <sebastianene@google.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/io.h>
+#include <linux/irqchip/arm-gic-v3.h>
+
+#include <asm/kvm_pkvm.h>
+
+#include <nvhe/its_emulate.h>
+#include <nvhe/mem_protect.h>
+#include <nvhe/memory.h>
+#include <nvhe/spinlock.h>
+
+/* Per-ITS state, in the pages the host donated at setup. */
+struct its_priv {
+ hyp_spinlock_t lock;
+ void __iomem *base;
+ u64 cbaser;
+
+ /* The hardware queue, which is hyp-owned, and the host's pinned copy of it. */
+ void *cmdq_hw;
+ void *cmdq_copy;
+ u32 cmdq_size;
+
+ /* The next slot to forward, at the offset last written to GITS_CWRITER. */
+ u32 cmd_offset;
+
+ /* The GITS_CREADR offset last read. */
+ u32 hw_creadr;
+};
+
+static int its_pin_host_pages(phys_addr_t pa, u64 nr_pages)
+{
+ void *va = hyp_phys_to_virt(pa);
+ u64 pfn = PHYS_PFN(pa), i;
+ int ret;
+
+ for (i = 0; i < nr_pages; i++) {
+ ret = __pkvm_host_share_hyp(pfn + i);
+ if (ret)
+ return ret;
+ }
+
+ return hyp_pin_shared_mem(va, va + (nr_pages << PAGE_SHIFT));
+}
+
+static bool its_process_cmd(struct its_priv *its, struct its_cmd_block *cmd)
+{
+ switch (its_cmd_get_command(cmd->raw_cmd)) {
+ case GITS_CMD_MAPD:
+ case GITS_CMD_MAPC:
+ case GITS_CMD_MAPTI:
+ case GITS_CMD_MAPI:
+ case GITS_CMD_MOVI:
+ case GITS_CMD_DISCARD:
+ case GITS_CMD_INV:
+ case GITS_CMD_MOVALL:
+ case GITS_CMD_INVALL:
+ case GITS_CMD_INT:
+ case GITS_CMD_CLEAR:
+ case GITS_CMD_SYNC:
+ return true;
+ default:
+ /* The vLPI commands and anything unknown. */
+ return false;
+ }
+}
+
+static u64 its_read_creadr(struct its_priv *its)
+{
+ u64 creadr = readq_relaxed(its->base + GITS_CREADR);
+ u32 offset = creadr & GITS_CREADR_OFFSET;
+
+ its->hw_creadr = offset;
+
+ return creadr;
+}
+
+/* Returns false on a rejected command, after forwarding the ones before it. */
+static bool its_cwriter_write(struct its_priv *its, u64 val)
+{
+ u32 new = val & GITS_CWRITER_OFFSET;
+ struct its_cmd_block cmd;
+ bool rejected = false;
+ unsigned int i;
+ u64 *slot;
+
+ if (new >= its->cmdq_size)
+ return false;
+
+ its_read_creadr(its);
+
+ /* The queue is full with GITS_CWRITER 32 bytes behind GITS_CREADR, so reject an overrun. */
+ if ((its->cmd_offset + its->cmdq_size - its->hw_creadr) % its->cmdq_size +
+ (new + its->cmdq_size - its->cmd_offset) % its->cmdq_size >
+ its->cmdq_size - sizeof(cmd))
+ return false;
+
+ /* A value below cmd_offset wraps around the queue, as it does for the ITS. */
+ while (its->cmd_offset != new) {
+ /* Read the command once, as the host can change its copy. */
+ slot = its->cmdq_copy + its->cmd_offset;
+ for (i = 0; i < ARRAY_SIZE(cmd.raw_cmd); i++)
+ cmd.raw_cmd[i] = READ_ONCE(slot[i]);
+ if (!its_process_cmd(its, &cmd)) {
+ rejected = true;
+ break;
+ }
+
+ memcpy(its->cmdq_hw + its->cmd_offset, &cmd, sizeof(cmd));
+ /*
+ * Flush whatever GITS_CBASER says, as it can read as shareable on an ITS
+ * that the driver treats as non-coherent (ITS_FLAGS_CMDQ_NEEDS_FLUSHING).
+ */
+ gic_flush_dcache_to_poc(its->cmdq_hw + its->cmd_offset, sizeof(cmd));
+ its->cmd_offset = (its->cmd_offset + sizeof(cmd)) % its->cmdq_size;
+ }
+
+ dsb(ishst);
+ /* Retry has no effect unless the hardware stalled on a command error. */
+ writeq_relaxed(its->cmd_offset | (val & GITS_CWRITER_RETRY), its->base + GITS_CWRITER);
+ return !rejected;
+}
+
+/* A reset clears GITS_CBASER.Valid, and an enable with it clear is UNPREDICTABLE. */
+static bool its_bases_intact(struct its_priv *its)
+{
+ return readq_relaxed(its->base + GITS_CBASER) == its->cbaser;
+}
+
+static bool its_ctlr_write(struct its_priv *its, u64 val)
+{
+ u32 ctlr = readl_relaxed(its->base + GITS_CTLR);
+
+ if ((val & GITS_CTLR_ENABLE) && !(ctlr & GITS_CTLR_ENABLE)) {
+ /*
+ * Enabling while not Quiescent is UNPREDICTABLE. Drop the write rather than
+ * abort it, as the driver enables the ITS when a quiesce times out on suspend.
+ */
+ if (!(ctlr & GITS_CTLR_QUIESCENT))
+ return true;
+ if (!its_bases_intact(its))
+ return false;
+ }
+
+ writel_relaxed(val, its->base + GITS_CTLR);
+ return true;
+}
+
+/* GITS_CBASER and GITS_BASER<n> are UNPREDICTABLE to write otherwise. */
+static bool its_disabled_quiescent(struct its_priv *its)
+{
+ u32 ctlr = readl_relaxed(its->base + GITS_CTLR);
+
+ return !(ctlr & GITS_CTLR_ENABLE) && (ctlr & GITS_CTLR_QUIESCENT);
+}
+
+static bool its_cbaser_write(struct its_priv *its)
+{
+ if (!its_disabled_quiescent(its))
+ return false;
+
+ writeq_relaxed(its->cbaser, its->base + GITS_CBASER);
+
+ /* The write reset GITS_CREADR to 0, and the ITS drops what was queued before it. */
+ its->cmd_offset = 0;
+ its->hw_creadr = 0;
+ writeq_relaxed(0, its->base + GITS_CWRITER);
+ return true;
+}
+
+static bool its_emulate(struct its_priv *its, u64 offset, bool write, u64 *val, u8 size)
+{
+ if (write) {
+ switch (offset) {
+ case GITS_CTLR:
+ return size == sizeof(u32) && its_ctlr_write(its, *val);
+ case GITS_CBASER:
+ return size == sizeof(u64) && its_cbaser_write(its);
+ case GITS_CWRITER:
+ /* The driver writes the low word, and the high word is RES0. */
+ return its_cwriter_write(its, *val);
+ case GITS_BASER ... GITS_BASER + (GITS_BASER_NR_REGS - 1) * sizeof(u64):
+ if (size != sizeof(u64))
+ return false;
+ writeq_relaxed(*val, its->base + offset);
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ /* The driver doesn't read any other register. */
+ switch (offset) {
+ case GITS_CTLR:
+ case GITS_IIDR:
+ case GITS_MPIDR:
+ case GITS_PIDR2:
+ if (size != sizeof(u32))
+ return false;
+ break;
+ case GITS_TYPER:
+ case GITS_CBASER:
+ case GITS_BASER ... GITS_BASER + (GITS_BASER_NR_REGS - 1) * sizeof(u64):
+ if (size != sizeof(u64))
+ return false;
+ break;
+ case GITS_CREADR:
+ break;
+ default:
+ return false;
+ }
+
+ *val = size == sizeof(u64) ? readq_relaxed(its->base + offset) :
+ readl_relaxed(its->base + offset);
+ return true;
+}
+
+bool pkvm_its_emulate_handler(struct pkvm_protected_reg *region, u64 offset,
+ bool write, u64 *val, u8 size)
+{
+ struct its_priv *its = region->priv;
+ bool ret;
+
+ if (size < sizeof(u32) || !IS_ALIGNED(offset, size))
+ return false;
+
+ hyp_spin_lock(&its->lock);
+ ret = its_emulate(its, offset, write, val, size);
+ hyp_spin_unlock(&its->lock);
+
+ return ret;
+}
+
+static struct pkvm_protected_reg *its_find_region(phys_addr_t its_base)
+{
+ int i;
+
+ for (i = 0; i < num_protected_reg; i++) {
+ if (PFN_PHYS(pkvm_protected_regs[i].pfn) == its_base)
+ return &pkvm_protected_regs[i];
+ }
+
+ return NULL;
+}
+
+static int its_take_cmdq(struct its_priv *its, phys_addr_t copy_pa)
+{
+ u32 cwriter = readq_relaxed(its->base + GITS_CWRITER) & GITS_CWRITER_OFFSET;
+ u32 creadr = readq_relaxed(its->base + GITS_CREADR) & GITS_CREADR_OFFSET;
+ u32 size = SZ_4K * (FIELD_GET(GITS_CBASER_SIZE, its->cbaser) + 1);
+ phys_addr_t hw_pa = GITS_CBASER_ADDRESS(its->cbaser);
+ u64 nr_pages = size >> PAGE_SHIFT;
+ int ret;
+
+ ret = __pkvm_host_donate_hyp(PHYS_PFN(hw_pa), nr_pages);
+ if (ret)
+ return ret;
+
+ ret = its_pin_host_pages(copy_pa, nr_pages);
+ if (ret)
+ return ret;
+
+ its->cmdq_hw = hyp_phys_to_virt(hw_pa);
+ its->cmdq_copy = hyp_phys_to_virt(copy_pa);
+ its->cmdq_size = size;
+ its->cmd_offset = cwriter;
+ its->hw_creadr = creadr;
+ return 0;
+}
+
+int pkvm_its_emulate_setup(phys_addr_t its_base, phys_addr_t desc_pa,
+ size_t desc_size)
+{
+ u64 desc_pages = desc_size >> PAGE_SHIFT;
+ struct pkvm_protected_reg *region;
+ struct its_pkvm_desc *desc;
+ struct its_priv *its;
+ u64 priv_pages;
+ int ret;
+
+ region = its_find_region(its_base);
+ if (!region)
+ return -ENODEV;
+ if (region->priv)
+ return -EBUSY;
+
+ /*
+ * Called once per ITS by the still-privileged host, so its arguments and
+ * the registers it programmed are trusted. A failure isn't unwound, as the
+ * host is then never deprivileged.
+ */
+ ret = __pkvm_host_donate_hyp(PHYS_PFN(desc_pa), desc_pages);
+ if (ret)
+ return ret;
+ desc = hyp_phys_to_virt(desc_pa);
+
+ priv_pages = desc->priv_pages;
+ ret = __pkvm_host_donate_hyp(PHYS_PFN(desc->priv_pa), priv_pages);
+ if (ret)
+ return ret;
+ its = hyp_phys_to_virt(desc->priv_pa);
+ memset(its, 0, priv_pages * PAGE_SIZE);
+
+ hyp_spin_lock_init(&its->lock);
+ its->base = region->hyp_va;
+ its->cbaser = readq_relaxed(its->base + GITS_CBASER);
+
+ ret = its_take_cmdq(its, desc->cmdq_copy_pa);
+ if (ret)
+ return ret;
+
+ WARN_ON(__pkvm_hyp_donate_host(PHYS_PFN(desc_pa), desc_pages));
+ /* Published by stop_machine()'s exit, and no host CPU traps until then. */
+ region->priv = its;
+ return 0;
+}
diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
index 8d45f45d30e6a..58cc1691b67a6 100644
--- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c
+++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
@@ -13,10 +13,12 @@
#include <asm/kvm_pkvm.h>
#include <asm/stage2_pgtable.h>
+#include <hyp/adjust_pc.h>
#include <hyp/fault.h>
#include <nvhe/arm-smccc.h>
#include <nvhe/gfp.h>
+#include <nvhe/its_emulate.h>
#include <nvhe/memory.h>
#include <nvhe/mem_protect.h>
#include <nvhe/mm.h>
@@ -779,6 +781,65 @@ static void host_inject_mem_abort(struct kvm_cpu_context *host_ctxt)
inject_host_exception(esr);
}
+static struct pkvm_protected_reg *find_protected_reg(u64 addr, u8 size)
+{
+ struct pkvm_protected_reg *region;
+ u64 start, end;
+ int i;
+
+ for (i = 0; i < num_protected_reg; i++) {
+ region = &pkvm_protected_regs[i];
+ start = PFN_PHYS(region->pfn);
+ end = start + PFN_PHYS(region->nr_pages);
+
+ if (addr >= start && addr + size <= end)
+ return region;
+ }
+
+ return NULL;
+}
+
+/*
+ * Returns false when the access cannot be emulated or the emulation rejects it.
+ * The caller's host stage-2 path then injects the abort.
+ */
+static bool handle_host_mmio_trap(struct kvm_cpu_context *host_ctxt, u64 esr,
+ u64 addr)
+{
+ struct pkvm_protected_reg *region;
+ u64 val = 0;
+ u8 size, rt;
+ bool write;
+
+ /* An access without a valid syndrome can't be emulated. */
+ if (ESR_ELx_EC(esr) != ESR_ELx_EC_DABT_LOW || !(esr & ESR_ELx_ISV))
+ return false;
+
+ size = BIT(FIELD_GET(ESR_ELx_SAS, esr));
+ region = find_protected_reg(addr, size);
+ if (!region)
+ return false;
+
+ rt = FIELD_GET(ESR_ELx_SRT_MASK, esr);
+ write = esr & ESR_ELx_WNR;
+ if (write && rt != 31)
+ val = cpu_reg(host_ctxt, rt) & GENMASK_ULL(size * 8 - 1, 0);
+
+ if (!pkvm_its_emulate_handler(region, addr - PFN_PHYS(region->pfn), write, &val, size))
+ return false;
+
+ if (!write && rt != 31) {
+ if ((esr & ESR_ELx_SSE) && size < sizeof(u64))
+ val = sign_extend64(val, size * 8 - 1);
+ if (!(esr & ESR_ELx_SF))
+ val = lower_32_bits(val);
+ cpu_reg(host_ctxt, rt) = val;
+ }
+
+ kvm_skip_host_instr();
+ return true;
+}
+
void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt)
{
struct kvm_vcpu_fault_info fault;
@@ -801,6 +862,10 @@ void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt)
BUG_ON(!(fault.hpfar_el2 & HPFAR_EL2_NS));
addr = FIELD_GET(HPFAR_EL2_FIPA, fault.hpfar_el2) << 12;
+ if (num_protected_reg &&
+ handle_host_mmio_trap(host_ctxt, esr, addr | FAR_TO_FIPA_OFFSET(fault.far_el2)))
+ return;
+
switch (host_stage2_idmap(addr)) {
case -EPERM:
host_inject_mem_abort(host_ctxt);
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index 16dfb249b4961..ddd797d21d42d 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -6,6 +6,7 @@
#include <linux/init.h>
#include <linux/interval_tree_generic.h>
+#include <linux/irqchip/arm-gic-v3.h>
#include <linux/kmemleak.h>
#include <linux/kvm_host.h>
#include <asm/kvm_mmu.h>
@@ -43,6 +44,53 @@ static int __init register_memblock_regions(void)
return 0;
}
+static int __init pkvm_register_its_region(phys_addr_t its_base)
+{
+ unsigned int idx = kvm_nvhe_sym(num_protected_reg);
+ struct pkvm_protected_reg *region;
+
+ if (idx >= PKVM_PROTECTED_REGS_NUM) {
+ kvm_err("ITS@%pa: no protected region slot left\n", &its_base);
+ return -ENOSPC;
+ }
+
+ /*
+ * Only the control frame is taken. GITS_TRANSLATER is in the next frame and
+ * stays host-mapped, as devices raise MSIs through it.
+ */
+ region = &kvm_nvhe_sym(pkvm_protected_regs)[idx];
+ region->pfn = PHYS_PFN(its_base);
+ region->nr_pages = PFN_DOWN(GITS_TRANSLATER);
+ kvm_nvhe_sym(num_protected_reg) = idx + 1;
+
+ return 0;
+}
+
+int __init pkvm_register_its_regions(void)
+{
+ int ret;
+
+ ret = its_pkvm_iter_its(pkvm_register_its_region);
+ if (ret)
+ kvm_err("Failed to register the ITS regions with hyp: %d\n", ret);
+
+ return ret;
+}
+
+/* Called by the driver with every CPU stopped. */
+static int __init pkvm_init_its_emulation(phys_addr_t its_base,
+ struct its_pkvm_desc *desc,
+ size_t desc_size)
+{
+ int ret;
+
+ ret = kvm_call_hyp_nvhe(__pkvm_its_emulate_setup, its_base, virt_to_phys(desc), desc_size);
+ if (ret)
+ kvm_err("ITS@%pa: emulation setup failed: %d\n", &its_base, ret);
+
+ return ret;
+}
+
void __init kvm_hyp_reserve(void)
{
u64 hyp_mem_pages = 0;
@@ -299,6 +347,10 @@ int pkvm_init_host_vm(struct kvm *kvm)
{
int ret;
+ /* The ITS takeover failed, so the hypervisor never finalised. */
+ if (!is_pkvm_initialized())
+ return -ENODEV;
+
/* Reserve the VM in hyp and obtain a hyp handle for the VM. */
ret = kvm_call_hyp_nvhe(__pkvm_reserve_vm);
if (ret < 0)
@@ -313,19 +365,61 @@ int pkvm_init_host_vm(struct kvm *kvm)
return 0;
}
+struct pkvm_its_takeover {
+ int ret;
+ bool done;
+};
+
static int __init __pkvm_drop_host_privileges(void *arg)
{
+ struct pkvm_its_takeover *takeover = arg;
+
+ if (smp_processor_id() == cpumask_first(cpu_online_mask)) {
+ takeover->ret = its_pkvm_setup_commit(pkvm_init_its_emulation);
+ /* Publishes ret to the CPUs spinning below. */
+ smp_store_release(&takeover->done, true);
+ }
+
+ /* Pairs with the release above, so that ret is read after done. */
+ while (!smp_load_acquire(&takeover->done))
+ cpu_relax();
+
+ /* No CPU is deprivileged unless the hypervisor took every ITS. */
+ if (takeover->ret)
+ return 0;
+
return WARN_ON(kvm_call_hyp_nvhe(__pkvm_prot_finalize)) ? -EINVAL : 0;
}
static int __init pkvm_drop_host_privileges(void)
{
+ struct pkvm_its_takeover takeover = {};
+ int ret;
+
+ /* Device drivers take dev_alloc_lock, through the MSI layer, with the cpus lock held. */
+ cpus_read_lock();
+ ret = its_pkvm_setup_prepare();
+ if (ret)
+ goto out;
+
/*
* Flip the static key upfront as that may no longer be possible
* once the host stage 2 is installed.
*/
- static_branch_enable(&kvm_protected_mode_initialized);
- return stop_machine(__pkvm_drop_host_privileges, NULL, cpu_online_mask);
+ static_branch_enable_cpuslocked(&kvm_protected_mode_initialized);
+ ret = stop_machine_cpuslocked(__pkvm_drop_host_privileges, &takeover, cpu_online_mask);
+ if (takeover.ret) {
+ /* No stage 2 was installed. Clear the key again, which also refuses every VM. */
+ static_branch_disable_cpuslocked(&kvm_protected_mode_initialized);
+ ret = takeover.ret;
+ }
+out:
+ its_pkvm_setup_finish();
+ cpus_read_unlock();
+ if (ret && !is_pkvm_initialized())
+ pr_err("pKVM: ITS takeover failed (%d): host not deprivileged, VMs refused\n", ret);
+
+ return ret;
}
static void __init pkvm_selftests(void)
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 1d31d5a72e009..01110162914f7 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -435,6 +435,11 @@
#define GITS_CBASER_RaWaWb GIC_BASER_CACHEABILITY(GITS_CBASER, INNER, RaWaWb)
#define GITS_CBASER_ADDRESS(cbaser) ((cbaser) & GENMASK_ULL(51, 12))
+#define GITS_CBASER_SIZE GENMASK_ULL(7, 0)
+
+#define GITS_CWRITER_RETRY (1ULL << 0)
+#define GITS_CWRITER_OFFSET GENMASK_ULL(19, 5)
+#define GITS_CREADR_OFFSET GENMASK_ULL(19, 5)
#define GITS_BASER_NR_REGS 8
--
2.47.3
next prev parent reply other threads:[~2026-11-18 15:19 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 15:18 [PATCH v4 00/13] KVM: ITS hardening for pKVM Fuad Tabba
2026-11-18 15:18 ` [PATCH v4 01/13] KVM: arm64: Donate MMIO to the hypervisor Fuad Tabba
2026-11-18 15:18 ` [PATCH v4 02/13] KVM: arm64: Take host-registered MMIO regions at pKVM finalisation Fuad Tabba
2026-11-18 15:18 ` [PATCH v4 03/13] irqchip/gic-v3: Disable GICv4 features in pKVM Fuad Tabba
2026-11-18 15:18 ` [PATCH v4 04/13] irqchip/gic-v3-its: Page-align ITT allocations " Fuad Tabba
2026-11-18 15:18 ` [PATCH v4 05/13] irqchip/gic-v3-its: Compute the wait target under the ITS lock Fuad Tabba
2026-11-18 15:18 ` [PATCH v4 06/13] KVM: arm64: Share the ITS command field accessors with the hypervisor Fuad Tabba
2026-11-18 15:18 ` [PATCH v4 07/13] KVM: arm64: Deprivilege the host under stop_machine() Fuad Tabba
2026-11-18 15:18 ` [PATCH v4 08/13] irqchip/gic-v3-its: Add the pKVM takeover entry points Fuad Tabba
2026-11-18 15:19 ` Fuad Tabba [this message]
2026-11-18 15:19 ` [PATCH v4 10/13] KVM: arm64: Take the ITS tables and restrict GITS_BASER<n> Fuad Tabba
2026-11-18 15:19 ` [PATCH v4 11/13] KVM: arm64: Take the ITT of every device the host maps Fuad Tabba
2026-11-18 15:19 ` [PATCH v4 12/13] KVM: arm64: Reject ITS commands whose IDs are out of range Fuad Tabba
2026-11-18 15:19 ` [PATCH v4 13/13] Documentation: KVM: arm64: Describe the ITS hardening Fuad Tabba
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118151904.4099560-10-fuad.tabba@linux.dev \
--to=fuad.tabba@linux.dev \
--cc=Sascha.Bischoff@arm.com \
--cc=bgrzesik@google.com \
--cc=catalin.marinas@arm.com \
--cc=joey.gouly@arm.com \
--cc=kvmarm@lists.linux.dev \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=maz@kernel.org \
--cc=oupton@kernel.org \
--cc=radu@rendec.net \
--cc=rananta@google.com \
--cc=sebastianene@google.com \
--cc=seiden@linux.ibm.com \
--cc=smostafa@google.com \
--cc=suzuki.poulose@arm.com \
--cc=tabba@google.com \
--cc=tglx@kernel.org \
--cc=vdonnefort@google.com \
--cc=vladimir.murzin@arm.com \
--cc=will@kernel.org \
--cc=yuzenghui@huawei.com \
--cc=zenghui.yu@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®