From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=217.140.110.172 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794847491; b=36VY6xWz23Oyeff8gvbxxGGXSC+mVHHlYntDdejydO5FZGWbiAYU1SBCPkaHrRj0v3rr ZOhbY22kQUt9MNhp8gLbAsqF5T7wAR/hw+rniQdvBZM+aw+uzfxg6Okl5dy14QXs9O7Dw HL0nFICBkFiIrv3hiOt0Rn2alD8iqaIvhlFIZ7t7WyNRnsbQbPLK23qr+ggbt0CmSoxJf glDpuO3cXfs7HKrZOUegEkFRCq/UxFTHf5N72jzcjnHzjCNVc4otC8QVXW5hObKoLY7Lh /LJl3kFAbDxM/VADAuE3AR6Z8w3pP+k44gy4Y67N8v65IrZNSWqqWvPLxEj8iWwFQxg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794847491; h=DMARC-Filter:Received:Received:DKIM-Signature:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=P8fw+2VBy+uX2tPICMYbTuK8vMTxdV9LiIiG9pw/syY=; b=NdFATl5cwmVOzfstL4gXzsazpQyDmDmLmbEtPPEdQvMG2cwfPIxsj2B0Ht1LRv9FItPa YN2EhAqPkPPW8ZWW1fHHkqeisTh0aXZPxFOwhFPTJHl4eXyTp8umYk2wxp91a1ZEOptuV bxzTEfe0nDJqQ0XQrRiJ+uOPIGivt1RXcQVZZRjA66Igg991uIdBd8PVmIsLRH/kden9E BKp3r3dEWG/V1Y3pax7l8t8XLuOpU2H1QprLUkva+9gcfN9gsWoTM6VO6pL2gR19BO/EM q2TWXzx3ZF5INdXQxtRtWEOuV4rK0ldwgS/u7wFq/DkSVcXDoA/O5+Lj+pshWxzpl/A== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=foss header.b=OigS+mbp; arc=none smtp.remote-ip=217.140.110.172 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 95191606A32F; Mon, 16 Nov 2026 16:44:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 95191606A32F Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=foss header.b=OigS+mbp Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 12BC01A25; Mon, 16 Nov 2026 08:44:46 -0800 (PST) Received: from e122027.arm.com (unknown [10.57.63.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D88353F882; Mon, 16 Nov 2026 08:44:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1794847489; bh=L7k3oDluQ75plIAHMometskgbywci2Xw4DnhGOUlVGQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OigS+mbpRYgFW6w369DBA1mWX4OHDIw4M7CHLgH095/Al9SZ8kmXACdwDiQBETI7W f4UgKcj1ZSNTHQxib4CZQMgJsiyeKeXHMOmdal6UAhMdjp3Ci0G2Tgmk7p/UtrYf9O 2ejtm7HC/l5I/cmvBU6fOoXcGO+V6AEAjuvrAEvM= From: Steven Price To: kvm@vger.kernel.org, kvmarm@lists.linux.dev Cc: Steven Price , Catalin Marinas , Marc Zyngier , Will Deacon , James Morse , Oliver Upton , Suzuki K Poulose , Zenghui Yu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Joey Gouly , Alexandru Elisei , Christoffer Dall , Fuad Tabba , linux-coco@lists.linux.dev, Ganapatrao Kulkarni , Gavin Shan , Shanker Donthineni , Alper Gun , "Aneesh Kumar K . V" , Emi Kisanuki , Vishal Annapurve , WeiLin.Chang@arm.com, Lorenzo Pieralisi , Kohei Enju Subject: [PATCH v3 06/11] KVM: arm64: CCA: Add Realm RTT mapping and invalidation helpers Date: Mon, 16 Nov 2026 16:43:52 +0000 Message-ID: <20261116164358.142785-7-steven.price@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261116164358.142785-1-steven.price@arm.com> References: <20261116164358.142785-1-steven.price@arm.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Add the RMI RTT creation and unmap helpers needed by CCA Realm stage-2 memory operations. REC exit handling and guest requested RIPAS changes follow in the vCPU series. Signed-off-by: Steven Price --- Changes since v2: - Use a local MMU cache during notifier invalidation, handle failed RTT splits, and validate RMI levels and cache state. Changes since v1: - Rework RTT creation and folding around missing or existing RTTs. Use a shared RMI block-size helper and handle alignment, partial progress and errors in range operations. --- arch/arm64/include/asm/kvm_rmi.h | 8 + arch/arm64/kvm/rmi.c | 421 +++++++++++++++++++++++++++++++ drivers/firmware/arm_rmm/rmi.c | 10 - include/linux/arm-rmi-cmds.h | 8 + 4 files changed, 437 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/kvm_rmi.h b/arch/arm64/include/asm/kvm_rmi.h index dd16c004b831..11e227172faf 100644 --- a/arch/arm64/include/asm/kvm_rmi.h +++ b/arch/arm64/include/asm/kvm_rmi.h @@ -80,6 +80,14 @@ int kvm_init_realm(struct kvm *kvm); void kvm_destroy_realm(struct kvm *kvm); int kvm_realm_teardown_stage2(struct kvm *kvm); +enum kvm_gfn_range_filter; + +void kvm_realm_unmap_range_filter(struct kvm *kvm, + unsigned long ipa, + unsigned long size, + bool may_block, + enum kvm_gfn_range_filter attr_filter); + static inline bool kvm_realm_is_private_address(struct realm *realm, unsigned long addr) { diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c index 74b89da4e801..838f95c201ad 100644 --- a/arch/arm64/kvm/rmi.c +++ b/arch/arm64/kvm/rmi.c @@ -25,6 +25,76 @@ static int get_start_level(struct realm *realm) return 4 - stage2_pgtable_levels(realm->ia_bits); } +static int find_map_level(struct realm *realm, + unsigned long start, + unsigned long end) +{ + int start_level = get_start_level(realm); + int level = KVM_PGTABLE_LAST_LEVEL; + + while (level > start_level) { + unsigned long map_size = kvm_granule_size(level - 1); + + if (!IS_ALIGNED(start, map_size) || + (start + map_size) > end) + break; + + level--; + } + + return level; +} + +static int undelegate_range_desc(unsigned long desc) +{ + unsigned long size = rmi_addr_block_size_to_bytes(RMI_ADDR_RANGE_BLOCK_SIZE(desc)); + unsigned long count = RMI_ADDR_RANGE_COUNT(desc); + unsigned long addr = RMI_ADDR_RANGE_ADDR(desc); + unsigned long state = RMI_ADDR_RANGE_STATE(desc); + + if (state == RMI_OP_MEM_UNDELEGATED) + return 0; + + if (size * count == 0) + return 0; + + return rmi_undelegate_range(addr, size * count); +} + +static phys_addr_t alloc_delegated_granule(struct kvm_mmu_memory_cache *mc) +{ + phys_addr_t phys; + void *virt; + + if (mc && kvm_mmu_memory_cache_nr_free_objects(mc)) { + virt = kvm_mmu_memory_cache_alloc(mc); + } else { + virt = (void *)__get_free_page(GFP_ATOMIC | __GFP_ZERO | + __GFP_ACCOUNT); + } + + if (!virt) + return PHYS_ADDR_MAX; + + phys = virt_to_phys(virt); + if (rmi_delegate_page(phys)) { + free_page((unsigned long)virt); + return PHYS_ADDR_MAX; + } + + return phys; +} + +static phys_addr_t alloc_rtt(struct kvm_mmu_memory_cache *mc) +{ + phys_addr_t phys = alloc_delegated_granule(mc); + + if (phys != PHYS_ADDR_MAX) + kvm_account_pgtable_pages(phys_to_virt(phys), 1); + + return phys; +} + static void free_rtt(phys_addr_t phys) { kvm_account_pgtable_pages(phys_to_virt(phys), -1); @@ -32,6 +102,32 @@ static void free_rtt(phys_addr_t phys) kvm_account_pgtable_pages(phys_to_virt(phys), 1); } +static long realm_rtt_create(struct realm *realm, + unsigned long addr, + int level, + phys_addr_t phys) +{ + addr = ALIGN_DOWN(addr, kvm_granule_size(level - 1)); + return rmi_rtt_create(realm->rd_phys, phys, addr, level); +} + +static long realm_rtt_fold(struct realm *realm, + unsigned long addr, + int level, + phys_addr_t *rtt_granule) +{ + unsigned long out_rtt; + long ret; + + addr = ALIGN_DOWN(addr, kvm_granule_size(level - 1)); + ret = rmi_rtt_fold(realm->rd_phys, addr, level, &out_rtt); + + if (ret == RMI_SUCCESS && rtt_granule) + *rtt_granule = out_rtt; + + return ret; +} + /* * realm_rtt_destroy - Destroy an RTT at @level for @addr. * @@ -56,6 +152,47 @@ static long realm_rtt_destroy(struct realm *realm, unsigned long addr, return ret; } +static int realm_create_rtt_levels(struct realm *realm, + unsigned long ipa, + int level, + int max_level, + struct kvm_mmu_memory_cache *mc) +{ + long ret = 0; + phys_addr_t rtt = PHYS_ADDR_MAX; + + while (level++ < max_level) { + if (rtt == PHYS_ADDR_MAX) + rtt = alloc_rtt(mc); + if (rtt == PHYS_ADDR_MAX) + return -ENOMEM; + + ret = realm_rtt_create(realm, ipa, level, rtt); + if (ret < 0) + break; + + if (RMI_RESULT_STATUS(ret) == RMI_ERROR_RTT && + RMI_RESULT_DATA_LEVEL(ret) == level - 1) { + /* The RTT already exists, reuse the granule at the next level. */ + ret = 0; + continue; + } + + if (ret) { + WARN(1, "Failed to create RTT at level %d: %ld\n", + level, ret); + ret = -ENXIO; + break; + } + rtt = PHYS_ADDR_MAX; + } + + if (rtt != PHYS_ADDR_MAX) + free_rtt(rtt); + + return ret; +} + static int realm_tear_down_rtt_level(struct realm *realm, int level, unsigned long start, unsigned long end) { @@ -154,12 +291,94 @@ static int realm_tear_down_rtt_range(struct realm *realm, start, end); } +/* + * Returns 0 on successful fold, a negative value on error, a positive value if + * we were not able to fold all tables at this level. + */ +static int realm_fold_rtt_level(struct realm *realm, int level, + unsigned long start, unsigned long end) +{ + int not_folded = 0; + ssize_t map_size; + unsigned long addr, next_addr; + + if (WARN_ON(level > KVM_PGTABLE_LAST_LEVEL)) + return -EINVAL; + + map_size = kvm_granule_size(level - 1); + + for (addr = start; addr < end; addr = next_addr) { + phys_addr_t rtt_granule; + long ret; + + next_addr = ALIGN(addr + 1, map_size); + + ret = realm_rtt_fold(realm, addr, level, &rtt_granule); + if (ret < 0) + return ret; + + switch (RMI_RESULT_STATUS(ret)) { + case RMI_SUCCESS: + free_rtt(rtt_granule); + break; + case RMI_ERROR_RTT: { + int err_level = RMI_RESULT_DATA_LEVEL(ret); + + if (err_level < level) { + /* No RTT at this level; skip the missing ancestor. */ + if (WARN_ON(err_level < get_start_level(realm))) + return -EINVAL; + next_addr = ALIGN(addr + 1, + kvm_granule_size(err_level)); + break; + } + if (level == KVM_PGTABLE_LAST_LEVEL) { + not_folded++; + break; + } + /* Recurse a level deeper */ + ret = realm_fold_rtt_level(realm, + level + 1, + addr, + min(next_addr, end)); + if (ret < 0) + return ret; + if (ret) { + not_folded++; + break; + } + /* Try once more after folding the child RTTs. */ + ret = realm_rtt_fold(realm, addr, level, &rtt_granule); + if (ret < 0) + return ret; + if (RMI_RESULT_STATUS(ret) == RMI_SUCCESS) { + free_rtt(rtt_granule); + break; + } + if (RMI_RESULT_STATUS(ret) != RMI_ERROR_RTT) { + WARN_ON(1); + return -ENXIO; + } + not_folded++; + break; + } + default: + WARN_ON(1); + return -ENXIO; + } + } + + return not_folded; +} + static int realm_destroy_rtts(struct kvm *kvm) { struct realm *realm = &kvm->arch.realm; unsigned int ia_bits = realm->ia_bits; lockdep_assert_held(&kvm->arch.config_lock); + if (WARN_ON_ONCE(!realm->stage2_unmapped)) + return -EINVAL; return realm_tear_down_rtt_range(realm, 0, (1UL << ia_bits)); } @@ -189,6 +408,208 @@ int kvm_realm_teardown_stage2(struct kvm *kvm) return 0; } +static bool realm_unmap_cache_topup(struct kvm *kvm, + struct kvm_mmu_memory_cache *mc, int needed) +{ + int ret; + + if (!mc || kvm_mmu_memory_cache_nr_free_objects(mc) >= needed) + return false; + + lockdep_assert_held_write(&kvm->mmu_lock); + write_unlock(&kvm->mmu_lock); + ret = __kvm_mmu_topup_memory_cache(mc, KVM_PGTABLE_LAST_LEVEL, needed); + write_lock(&kvm->mmu_lock); + + return !ret; +} + +static void realm_unmap_shared_range(struct kvm *kvm, + unsigned long start, + unsigned long end, + bool may_block, + struct kvm_mmu_memory_cache *mc) +{ + struct realm *realm = &kvm->arch.realm; + unsigned long rd = realm->rd_phys; + unsigned long next_addr, addr; + unsigned long shared_bit = BIT(realm->ia_bits - 1); + + if (start >= end) + return; + + /* + * Callers may pass either an IPA range or its shared alias. Add the + * shared bit only for the former. Use addition so an exclusive end at + * shared_bit advances to the end of the IPA space. + */ + if (start < shared_bit) { + start += shared_bit; + end += shared_bit; + } + + for (addr = start; addr < end; addr = next_addr) { + long ret; + + ret = rmi_rtt_unprot_unmap(rd, addr, end, RMI_ADDR_TYPE_NONE, + 0, &next_addr, NULL, NULL); + if (WARN_ON(ret < 0)) + return; + + switch (RMI_RESULT_STATUS(ret)) { + case RMI_SUCCESS: + break; + case RMI_ERROR_RTT: { + int err_level = RMI_RESULT_DATA_LEVEL(ret); + int level = find_map_level(realm, addr, end); + + if (WARN_ON(err_level < get_start_level(realm) || + err_level > KVM_PGTABLE_LAST_LEVEL)) + return; + + if (err_level >= level) { + /* Nothing present, so skip */ + next_addr = ALIGN(addr + 1, + kvm_granule_size(err_level)); + break; + } + + if (may_block && realm_unmap_cache_topup(kvm, mc, + level - err_level)) { + next_addr = addr; + break; + } + + ret = realm_create_rtt_levels(realm, addr, err_level, + level, mc); + if (!ret) { + /* Retry with the RTT levels in place. */ + next_addr = addr; + break; + } + WARN_ON(ret != -ENOMEM); + + /* Splitting failed; unmap the entire existing block. */ + next_addr = ALIGN_DOWN(addr, kvm_granule_size(err_level)); + end = max(end, next_addr + kvm_granule_size(err_level)); + break; + } + default: + WARN_ON(1); + return; + } + + if (may_block) + cond_resched_rwlock_write(&kvm->mmu_lock); + } + + realm_fold_rtt_level(realm, get_start_level(realm) + 1, + start, end); +} + +static void realm_unmap_private_range(struct kvm *kvm, + unsigned long start, + unsigned long end, + bool may_block, + struct kvm_mmu_memory_cache *mc) +{ + struct realm *realm = &kvm->arch.realm; + unsigned long rd = realm->rd_phys; + unsigned long next_addr, addr; + long ret; + + for (addr = start; addr < end; addr = next_addr) { + unsigned long out_range; + unsigned long flags = RMI_ADDR_TYPE_SINGLE; + /* TODO: Optimise using RMI_ADDR_TYPE_LIST */ + + ret = rmi_rtt_data_unmap(rd, addr, end, flags, 0, + &next_addr, &out_range, NULL); + if (WARN_ON(ret < 0)) + return; + + switch (RMI_RESULT_STATUS(ret)) { + case RMI_SUCCESS: + ret = undelegate_range_desc(out_range); + if (WARN_ON(ret)) + return; + break; + case RMI_ERROR_RTT: { + int err_level = RMI_RESULT_DATA_LEVEL(ret); + int level = find_map_level(realm, addr, end); + + if (WARN_ON(err_level < get_start_level(realm) || + err_level > KVM_PGTABLE_LAST_LEVEL)) + return; + + if (err_level >= level) { + /* Nothing present, so skip this RTT entry. */ + next_addr = ALIGN(addr + 1, + kvm_granule_size(err_level)); + break; + } + + if (may_block && realm_unmap_cache_topup(kvm, mc, + level - err_level)) { + next_addr = addr; + break; + } + + ret = realm_create_rtt_levels(realm, addr, err_level, + level, mc); + if (!ret) { + next_addr = addr; + break; + } + + /* A failed split cannot leave the backing pages mapped. */ + KVM_BUG_ON(1, kvm); + next_addr = ALIGN_DOWN(addr, kvm_granule_size(err_level)); + end = max(end, next_addr + kvm_granule_size(err_level)); + break; + } + default: + WARN_ON(1); + return; + } + + if (may_block) + cond_resched_rwlock_write(&kvm->mmu_lock); + } + + realm_fold_rtt_level(realm, get_start_level(realm) + 1, + start, end); +} + +void kvm_realm_unmap_range_filter(struct kvm *kvm, unsigned long start, + unsigned long size, bool may_block, + enum kvm_gfn_range_filter attr_filter) +{ + /* + * MMU notifier callbacks cannot take slots_lock, which protects the + * eager split cache. Use a cache owned by this invalidation instead. + */ + struct kvm_mmu_memory_cache cache = { .gfp_zero = __GFP_ZERO }; + unsigned long end = start + size; + struct realm *realm = &kvm->arch.realm; + + if (!kvm_realm_is_created(kvm)) + return; + + end = min(BIT(realm->ia_bits - 1), end); + + if (attr_filter & KVM_FILTER_SHARED) + realm_unmap_shared_range(kvm, start, end, may_block, &cache); + if (attr_filter & KVM_FILTER_PRIVATE) + realm_unmap_private_range(kvm, start, end, may_block, &cache); + + if (cache.objects) { + write_unlock(&kvm->mmu_lock); + kvm_mmu_free_memory_cache(&cache); + write_lock(&kvm->mmu_lock); + } +} + void kvm_destroy_realm(struct kvm *kvm) { struct realm *realm = &kvm->arch.realm; diff --git a/drivers/firmware/arm_rmm/rmi.c b/drivers/firmware/arm_rmm/rmi.c index 77255a90e0a6..a9f42c9a4f79 100644 --- a/drivers/firmware/arm_rmm/rmi.c +++ b/drivers/firmware/arm_rmm/rmi.c @@ -11,7 +11,6 @@ #include #include -#include /* RMM v2.0 defines RmiFeatureRegister0 to RmiFeatureRegister4. */ static unsigned long rmi_feat_reg_cache[5] __ro_after_init; @@ -183,15 +182,6 @@ int rmi_free_delegated_page(phys_addr_t phys) } EXPORT_SYMBOL_GPL(rmi_free_delegated_page); -/* - * Convert the RmiAddrBlockSize to actual size. This is used in RmiDonateReq - * and RmiAddrRangeDesc*. - */ -static unsigned long rmi_addr_block_size_to_bytes(unsigned long block_size_fld) -{ - return BIT(ARM64_HW_PGTABLE_LEVEL_SHIFT(3 - block_size_fld)); -} - /* * free_addr_range: Free memory described by the address range entry, which may * be partially consumed by RMM. diff --git a/include/linux/arm-rmi-cmds.h b/include/linux/arm-rmi-cmds.h index a84e50ac851c..1a885f1017df 100644 --- a/include/linux/arm-rmi-cmds.h +++ b/include/linux/arm-rmi-cmds.h @@ -13,8 +13,16 @@ #include #include +#include + #define RMI_MAX_ADDR_LIST 256 +/* Convert the block size used by RmiDonateReq and RmiAddrRangeDesc. */ +static inline unsigned long rmi_addr_block_size_to_bytes(unsigned long block_size) +{ + return BIT(ARM64_HW_PGTABLE_LEVEL_SHIFT(3 - block_size)); +} + struct rmi_sro_state { struct arm_smccc_1_2_regs regs; int addr_count; -- 2.43.0