From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7832455623; Mon, 14 Sep 2026 12:27:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789388851; cv=none; b=iax3ekfw6NRikpbCN2Uljg6rmlOaHI1uUdX46jfR/p7U88ljZfXhj0gV8KrfWKK6O4bIIFWwhZMYEyMQrqIo/7f4wHQEap7tiL1Pebl46ucuBMA58I1K7cxeqwWF5EdXaSLWisq8l1zuxImPmcNdyR0akursWfiur9quvoVBFSQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789388851; c=relaxed/simple; bh=zGFS584HnUtx3DB3mxkwTl+niPKdC9PKA7LK2k9UJ98=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=e0zpfmRU3gUFOBKELvlBp7Jabav1UEHZqy2tTaoCu4TRZIFneUvHOFn2UrAF4/b9ICJU2nY5GJH6JIooZv77bzZxUpoR7eAjCkpwaNCEqOGIisfxli/20uRR3wRWwTFyG/srtxQxtvBlAx0ZV/lF69jhkQZtA24w0RIBTSphaCQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IN67ho5V; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IN67ho5V" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DA2761F00893; Mon, 14 Sep 2026 12:27:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789388849; bh=uc+x321Bebrgu/kzizKcirXOM0C2dCttTdV9zeKiSgc=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=IN67ho5VaBAydk8qvmgod7tjfM46rHy+asK+Bv9GyqD7YCTT0S8RuaziBM1idGRSn JPzTZ/p4icHXsKqG88Va3EyQUp1eDwzfgR7NuzyrSPbYr2SHz7xcWLhGm48kwpObQz kVWLxoPcr3rGMLyGCfm/8ChtoCV7VnkWv/CKgLyy8rvy9oD4i6GWS3fyv6nN73qqda lDsoPh/62QRTl7rm1yD5ralj860Eew667WNyF1lsvzfLou3NalEUOC+259znGaUqVB kUAQ+aY7O4IIfozegfxVDrraoJx6HAUsRyK/+xp8+Z+w9/rdw6LYtlSOAllsXtcpDq IC2v88j2YomTA== From: "Lorenzo Stoakes (ARM)" Date: Mon, 14 Sep 2026 13:26:20 +0100 Subject: [PATCH v2 09/13] KVM: arm64: Implement KVM_PRE_FAULT_MEMORY Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260914-kvm-arm-prefault-v2-9-26fb47f74b73@kernel.org> References: <20260914-kvm-arm-prefault-v2-0-26fb47f74b73@kernel.org> In-Reply-To: <20260914-kvm-arm-prefault-v2-0-26fb47f74b73@kernel.org> To: Catalin Marinas , Will Deacon , Marc Zyngier , Oliver Upton , Fuad Tabba , Joey Gouly , Steffen Eiden , Suzuki K Poulose , Zenghui Yu , Paolo Bonzini , Jonathan Corbet Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Jack Thomson , Jack Thomson , Alexandru Elisei , Vincent Donnefort , "Aneesh Kumar K.V" , Sean Christopherson , Claudio Imbrenda , Leo Soares Passos , "Lorenzo Stoakes (ARM)" X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=7848; i=ljs@kernel.org; h=from:subject:message-id; bh=zGFS584HnUtx3DB3mxkwTl+niPKdC9PKA7LK2k9UJ98=; b=owGbwMvMwCV2fu7ZrsZH9SKMp9WSGLKWP//64Lfs1rOmp8vrKixKb/i1TTGu/9S7XiNCrj3Fy TBM77hCRykLgxgXg6yYIsvzL+L7g0TC5nVe8HeDmcPKBDKEgYtTACayt57hr0B4ZIMcT71Y7apH AVe72bkcHrM/rX4c/WeVl/VVd81LxowMzfo2Dl8l62r9q1eXHvYtevbF+OL0+E96jPmCioznVaZ yAgA= X-Developer-Key: i=ljs@kernel.org; a=openpgp; fpr=E7F417BF5214569E89D04F46CF9DCD8A81E27F14 Implement KVM stage 2 page table pre-faulting for the arm64 architecture, with the core of the implementation in kvm_arch_vcpu_pre_fault_memory(). Pre-fault by first trying a page table walk under the MMU read lock then, if it fails, injecting a synthetic data abort at the page table level at which the page table walk failed. This is necessarily racey as reclaim might happen at any time. Successfully pre-faulting can therefore only guarantee that each GPA was observed to be mapped at least once. Protected KVM (pKVM) is not supported at all because pKVM creates VMs and vCPUs when first run, meaning any attempt to pre-fault prior to this cannot succeed. A retry mechanic is implemented when user_mem_abort() or gmem_abort() fail to map memory due to a benign failure where a hardware abort would not result in an error. An invalid memslot (i.e. a memslot with the KVM_MEMSLOT_INVALID flag set) results in the operation returning -EAGAIN without a retry mechanic, because an invalid memslot means the pre-fault operation raced with memslot reclaim, and since the SRCU lock is held, progress cannot be made, and the user must retry. This work is based with gratitude on Jack Thomson's original series, its previous revisions and the feedback they received. Link: https://patch.msgid.link/20260612162354.73378-1-jackabt.amazon@gmail.com/ Signed-off-by: Lorenzo Stoakes (ARM) --- arch/arm64/include/asm/kvm_pkvm.h | 2 +- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 141 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 144 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h index beea00e693a0..4d6e5765e9e5 100644 --- a/arch/arm64/include/asm/kvm_pkvm.h +++ b/arch/arm64/include/asm/kvm_pkvm.h @@ -44,9 +44,9 @@ static inline bool kvm_pkvm_ext_allowed(struct kvm *kvm, long ext) case KVM_CAP_ARM_PTRAUTH_GENERIC: return true; case KVM_CAP_ARM_MTE: - return false; case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE: case KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES: + case KVM_CAP_PRE_FAULT_MEMORY: return false; default: return !kvm || !kvm_vm_is_protected(kvm); diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 449154f9a485..71233068b7cb 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -37,6 +37,7 @@ menuconfig KVM select SCHED_INFO select GUEST_PERF_EVENTS if PERF_EVENTS select KVM_GUEST_MEMFD + select KVM_GENERIC_PRE_FAULT_MEMORY help Support hosting virtualized guest machines. diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index ccae82c1242b..58ac70f31d84 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -393,6 +393,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_COUNTER_OFFSET: case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS: case KVM_CAP_ARM_SEA_TO_USER: + case KVM_CAP_PRE_FAULT_MEMORY: r = 1; break; case KVM_CAP_SET_GUEST_DEBUG2: diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 9c922b498adb..17f1f6d36f2b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -5,6 +5,7 @@ */ #include +#include #include #include #include @@ -2835,3 +2836,143 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled); } + +/* + * Try to walk to the specified GPA in canonical mmu - if unmapped returns 0, if + * mapped returns the granule size, otherwise returns an error. + */ +static long kvm_walk_s2(struct kvm_pgtable *pgt, + gpa_t gpa, s8 *level) +{ + struct kvm *kvm = kvm_s2_mmu_to_kvm(pgt->mmu); + kvm_pte_t pte; + long ret; + + guard(read_lock)(&kvm->mmu_lock); + + ret = kvm_pgtable_get_leaf(pgt, gpa, &pte, level, + KVM_PGTABLE_WALK_SHARED); + if (ret) + return ret; + /* Unpopulated, must fault. */ + if (!kvm_pte_valid(pte)) + return 0; + return kvm_granule_size(*level); +} + +/* Synthesised data abort at specified page table level. */ +#define PRE_FAULT_ESR(level) \ + ((ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) | \ + ESR_ELx_IL | ESR_ELx_FSC_FAULT_L(level)) + +/* Retrieve either a read-only or a read/write hva. */ +static hva_t gfn_to_hva_memslot_read(struct kvm_memory_slot *slot, gfn_t gfn) +{ + return gfn_to_hva_memslot_prot(slot, gfn, /*writable=*/NULL); +} + +static long __pre_fault_s2(struct kvm_s2_mmu *mmu, struct kvm_vcpu *vcpu, + gpa_t gpa, struct kvm_memory_slot *memslot, s8 level) +{ + const bool is_gmem = kvm_slot_has_gmem(memslot); + const gfn_t gfn = gpa_to_gfn(gpa); + const hva_t hva = is_gmem ? 0 : gfn_to_hva_memslot_read(memslot, gfn); + const struct kvm_s2_fault_desc s2fd = { + .vcpu = vcpu, + .fault_ipa = gpa, + .nested = NULL, + .memslot = memslot, + .hva = hva, + .esr = PRE_FAULT_ESR(level), + .mmu = mmu, + }; + struct kvm_s2_fault_result result = {}; + long ret; + + if (kvm_is_error_hva(hva)) + return -EFAULT; + + if (is_gmem) + ret = gmem_abort(&s2fd, &result); + else + ret = user_mem_abort(&s2fd, &result); + if (IS_ERR_VALUE(ret)) + return ret; + return result.mapping_size; +} + +static long pre_fault_s2(struct kvm_s2_mmu *mmu, struct kvm_vcpu *vcpu, + gpa_t gpa, struct kvm_memory_slot *memslot) +{ + s8 level; + long ret; + + /* Try a walk first. */ + ret = kvm_walk_s2(mmu->pgt, gpa, &level); + if (ret) + return ret; + /* OK, have to fault page in. */ + return __pre_fault_s2(mmu, vcpu, gpa, memslot, level); +} + +static unsigned long +pre_fault_bytes_consumed(gpa_t gpa, unsigned long granule_size, + unsigned long bytes_remaining) +{ + /* Granules are always a power-of-2. */ + const unsigned long granule_bytes_remaining = + granule_size - (gpa % granule_size); + + return min(granule_bytes_remaining, bytes_remaining); +} + +/* If you lose the race this many times, time to give up. */ +#define MAX_PRE_FAULT_RETRIES 3 + +/** + * kvm_arch_vcpu_pre_fault_memory - pre-fault stage-2 page tables for the + * specified GPA. + * @vcpu: The VCPU pointer + * @range: {gpa, size, flags} tuple + * + * The mapping performed is always best-effort - faulting in is necessarily + * racey. The ranges faulted in are canonical, nested page tables are ignored. + * + * @range->gpa specifies the GPA to pre-fault, @range->size specifies how many + * bytes remain to be pre-faulted and @range->flags is reserved and must be 0. + * + * Returns: the number of bytes the pre-fault consumed, or an error. + */ +long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, + struct kvm_pre_fault_memory *range) +{ + struct kvm *kvm = vcpu->kvm; + const u64 bytes_remaining = range->size; + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; /* Canonical. */ + struct kvm_memory_slot *memslot; + const gpa_t gpa = range->gpa; + int num_retries = 0; + long ret; + + /* + * pKVM is unsupported as their vCPUs are instantiated on first run and + * pre-faulting only running vCPUs would be inconsistent and confusing. + */ + if (is_protected_kvm_enabled()) + return -EOPNOTSUPP; + + memslot = gfn_to_memslot(kvm, gpa_to_gfn(gpa)); + if (!memslot) + return -ENOENT; + /* SRCU must be released for progress and only userland can do that. */ + if (memslot->flags & KVM_MEMSLOT_INVALID) + return -EAGAIN; + + do { + ret = pre_fault_s2(mmu, vcpu, gpa, memslot); + } while (ret == -EAGAIN && num_retries++ < MAX_PRE_FAULT_RETRIES); + + if (IS_ERR_VALUE(ret)) + return ret; + return pre_fault_bytes_consumed(gpa, ret, bytes_remaining); +} -- 2.55.0