* [PATCH v2 01/14] KVM: x86: disallow pre-fault for SNP VMs before initialization
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 02/14] KVM: guest_memfd: return folio from __kvm_gmem_get_pfn() Paolo Bonzini
` (13 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
KVM_PRE_FAULT_MEMORY for an SNP guest can race with
sev_gmem_post_populate() in bad ways. The following sequence for
instance can potentially trigger an RMP fault:
thread A, sev_gmem_post_populate: called
thread B, sev_gmem_prepare: places below 'pfn' in a private state in RMP
thread A, sev_gmem_post_populate: *vaddr = kmap_local_pfn(pfn + i);
thread A, sev_gmem_post_populate: copy_from_user(vaddr, src + i * PAGE_SIZE, PAGE_SIZE);
RMP #PF
Fix this by only allowing KVM_PRE_FAULT_MEMORY to run after a guest's
initial private memory contents have been finalized via
KVM_SEV_SNP_LAUNCH_FINISH.
Beyond fixing this issue, it just sort of makes sense to enforce this,
since the KVM_PRE_FAULT_MEMORY documentation states:
"KVM maps memory as if the vCPU generated a stage-2 read page fault"
which sort of implies we should be acting on the same guest state that a
vCPU would see post-launch after the initial guest memory is all set up.
Co-developed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
Documentation/virt/kvm/api.rst | 6 ++++++
arch/x86/include/asm/kvm_host.h | 1 +
arch/x86/kvm/mmu/mmu.c | 3 +++
arch/x86/kvm/svm/sev.c | 8 ++++++++
arch/x86/kvm/svm/svm.c | 1 +
arch/x86/kvm/x86.c | 3 +++
6 files changed, 22 insertions(+)
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index ec1cd8aa1d56..7b512286f8d2 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6402,6 +6402,12 @@ for the current vCPU state. KVM maps memory as if the vCPU generated a
stage-2 read page fault, e.g. faults in memory as needed, but doesn't break
CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed.
+In the case of confidential VM types where there is an initial set up of
+private guest memory before the guest is 'finalized'/measured, this ioctl
+should only be issued after completing all the necessary setup to put the
+guest into a 'finalized' state so that the above semantics can be reliably
+ensured.
+
In some cases, multiple vCPUs might share the page tables. In this
case, the ioctl can be called in parallel.
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 950a03e0181e..94e7b5a4fafe 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1305,6 +1305,7 @@ struct kvm_arch {
u8 vm_type;
bool has_private_mem;
bool has_protected_state;
+ bool pre_fault_allowed;
struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES];
struct list_head active_mmu_pages;
struct list_head zapped_obsolete_pages;
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 901be9e420a4..26ef5b6ac3c1 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4743,6 +4743,9 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
u64 end;
int r;
+ if (!vcpu->kvm->arch.pre_fault_allowed)
+ return -EOPNOTSUPP;
+
/*
* reload is efficient when called repeatedly, so we can do it on
* every iteration.
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index a16c873b3232..6589091e8ce0 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -2549,6 +2549,14 @@ static int snp_launch_finish(struct kvm *kvm, struct kvm_sev_cmd *argp)
data->gctx_paddr = __psp_pa(sev->snp_context);
ret = sev_issue_cmd(kvm, SEV_CMD_SNP_LAUNCH_FINISH, data, &argp->error);
+ /*
+ * Now that there will be no more SNP_LAUNCH_UPDATE ioctls, private pages
+ * can be given to the guest simply by marking the RMP entry as private.
+ * This can happen on first access and also with KVM_PRE_FAULT_MEMORY.
+ */
+ if (!ret)
+ kvm->arch.pre_fault_allowed = true;
+
kfree(id_auth);
e_free_id_block:
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index c115d26844f7..d6f252555ab3 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4949,6 +4949,7 @@ static int svm_vm_init(struct kvm *kvm)
to_kvm_sev_info(kvm)->need_init = true;
kvm->arch.has_private_mem = (type == KVM_X86_SNP_VM);
+ kvm->arch.pre_fault_allowed = !kvm->arch.has_private_mem;
}
if (!pause_filter_count || !pause_filter_thresh)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index af6c8cf6a37a..52778689cef4 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12646,6 +12646,9 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
kvm->arch.vm_type = type;
kvm->arch.has_private_mem =
(type == KVM_X86_SW_PROTECTED_VM);
+ /* Decided by the vendor code for other VM types. */
+ kvm->arch.pre_fault_allowed =
+ type == KVM_X86_DEFAULT_VM || type == KVM_X86_SW_PROTECTED_VM;
ret = kvm_page_track_init(kvm);
if (ret)
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 02/14] KVM: guest_memfd: return folio from __kvm_gmem_get_pfn()
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 01/14] KVM: x86: disallow pre-fault for SNP VMs before initialization Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 03/14] KVM: guest_memfd: delay folio_mark_uptodate() until after successful preparation Paolo Bonzini
` (12 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
Right now this is simply more consistent and avoids use of pfn_to_page()
and put_page(). It will be put to more use in upcoming patches, to
ensure that the up-to-date flag is set at the very end of both the
kvm_gmem_get_pfn() and kvm_gmem_populate() flows.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
virt/kvm/guest_memfd.c | 37 ++++++++++++++++++++-----------------
1 file changed, 20 insertions(+), 17 deletions(-)
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 1c509c351261..522e1b28e7ae 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -541,34 +541,34 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
fput(file);
}
-static int __kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
- gfn_t gfn, kvm_pfn_t *pfn, int *max_order, bool prepare)
+static struct folio *
+__kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
+ gfn_t gfn, kvm_pfn_t *pfn, int *max_order, bool prepare)
{
pgoff_t index = gfn - slot->base_gfn + slot->gmem.pgoff;
struct kvm_gmem *gmem = file->private_data;
struct folio *folio;
struct page *page;
- int r;
if (file != slot->gmem.file) {
WARN_ON_ONCE(slot->gmem.file);
- return -EFAULT;
+ return ERR_PTR(-EFAULT);
}
gmem = file->private_data;
if (xa_load(&gmem->bindings, index) != slot) {
WARN_ON_ONCE(xa_load(&gmem->bindings, index));
- return -EIO;
+ return ERR_PTR(-EIO);
}
folio = kvm_gmem_get_folio(file_inode(file), index, prepare);
if (IS_ERR(folio))
- return PTR_ERR(folio);
+ return folio;
if (folio_test_hwpoison(folio)) {
folio_unlock(folio);
folio_put(folio);
- return -EHWPOISON;
+ return ERR_PTR(-EHWPOISON);
}
page = folio_file_page(folio, index);
@@ -577,25 +577,25 @@ static int __kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
if (max_order)
*max_order = 0;
- r = 0;
-
folio_unlock(folio);
-
- return r;
+ return folio;
}
int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
gfn_t gfn, kvm_pfn_t *pfn, int *max_order)
{
struct file *file = kvm_gmem_get_file(slot);
- int r;
+ struct folio *folio;
if (!file)
return -EFAULT;
- r = __kvm_gmem_get_pfn(file, slot, gfn, pfn, max_order, true);
+ folio = __kvm_gmem_get_pfn(file, slot, gfn, pfn, max_order, true);
fput(file);
- return r;
+ if (IS_ERR(folio))
+ return PTR_ERR(folio);
+
+ return 0;
}
EXPORT_SYMBOL_GPL(kvm_gmem_get_pfn);
@@ -625,6 +625,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
npages = min_t(ulong, slot->npages - (start_gfn - slot->base_gfn), npages);
for (i = 0; i < npages; i += (1 << max_order)) {
+ struct folio *folio;
gfn_t gfn = start_gfn + i;
kvm_pfn_t pfn;
@@ -633,9 +634,11 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
break;
}
- ret = __kvm_gmem_get_pfn(file, slot, gfn, &pfn, &max_order, false);
- if (ret)
+ folio = __kvm_gmem_get_pfn(file, slot, gfn, &pfn, &max_order, false);
+ if (IS_ERR(folio)) {
+ ret = PTR_ERR(folio);
break;
+ }
if (!IS_ALIGNED(gfn, (1 << max_order)) ||
(npages - i) < (1 << max_order))
@@ -644,7 +647,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
p = src ? src + i * PAGE_SIZE : NULL;
ret = post_populate(kvm, gfn, pfn, p, max_order, opaque);
- put_page(pfn_to_page(pfn));
+ folio_put(folio);
if (ret)
break;
}
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 03/14] KVM: guest_memfd: delay folio_mark_uptodate() until after successful preparation
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 01/14] KVM: x86: disallow pre-fault for SNP VMs before initialization Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 02/14] KVM: guest_memfd: return folio from __kvm_gmem_get_pfn() Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 04/14] KVM: guest_memfd: do not go through struct page Paolo Bonzini
` (11 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
The up-to-date flag as is now is not too useful; it tells guest_memfd not
to overwrite the contents of a folio, but it doesn't say that the page
is ready to be mapped into the guest. For encrypted guests, mapping
a private page requires that the "preparation" phase has succeeded,
and at the same time the same page cannot be prepared twice.
So, ensure that folio_mark_uptodate() is only called on a prepared page. If
kvm_gmem_prepare_folio() or the post_populate callback fail, the folio
will not be marked up-to-date; it's not a problem to call clear_highpage()
again on such a page prior to the next preparation attempt.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
virt/kvm/guest_memfd.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 522e1b28e7ae..1ea632dbae57 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -73,8 +73,6 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index, bool
for (i = 0; i < nr_pages; i++)
clear_highpage(folio_page(folio, i));
-
- folio_mark_uptodate(folio);
}
if (prepare) {
@@ -84,6 +82,8 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index, bool
folio_put(folio);
return ERR_PTR(r);
}
+
+ folio_mark_uptodate(folio);
}
/*
@@ -646,6 +646,8 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
p = src ? src + i * PAGE_SIZE : NULL;
ret = post_populate(kvm, gfn, pfn, p, max_order, opaque);
+ if (!ret)
+ folio_mark_uptodate(folio);
folio_put(folio);
if (ret)
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 04/14] KVM: guest_memfd: do not go through struct page
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (2 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 03/14] KVM: guest_memfd: delay folio_mark_uptodate() until after successful preparation Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 05/14] KVM: rename CONFIG_HAVE_KVM_GMEM_* to CONFIG_HAVE_KVM_ARCH_GMEM_* Paolo Bonzini
` (10 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
We have a perfectly usable folio, use it to retrieve the pfn and order.
All that's needed is a version of folio_file_page that returns a pfn.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
virt/kvm/guest_memfd.c | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 1ea632dbae57..5221b584288f 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -13,6 +13,18 @@ struct kvm_gmem {
struct list_head entry;
};
+/**
+ * folio_file_pfn - like folio_file_page, but return a pfn.
+ * @folio: The folio which contains this index.
+ * @index: The index we want to look up.
+ *
+ * Return: The pfn for this index.
+ */
+static inline kvm_pfn_t folio_file_pfn(struct folio *folio, pgoff_t index)
+{
+ return folio_pfn(folio) + (index & (folio_nr_pages(folio) - 1));
+}
+
static int kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct folio *folio)
{
#ifdef CONFIG_HAVE_KVM_GMEM_PREPARE
@@ -22,7 +34,6 @@ static int kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct fol
list_for_each_entry(gmem, gmem_list, entry) {
struct kvm_memory_slot *slot;
struct kvm *kvm = gmem->kvm;
- struct page *page;
kvm_pfn_t pfn;
gfn_t gfn;
int rc;
@@ -34,13 +45,12 @@ static int kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct fol
if (!slot)
continue;
- page = folio_file_page(folio, index);
- pfn = page_to_pfn(page);
+ pfn = folio_file_pfn(folio, index);
gfn = slot->base_gfn + index - slot->gmem.pgoff;
- rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, compound_order(compound_head(page)));
+ rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
if (rc) {
- pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
- index, gfn, pfn, rc);
+ pr_warn_ratelimited("gmem: Failed to prepare folio for GFN %llx PFN %llx error %d.\n",
+ gfn, pfn, rc);
return rc;
}
}
@@ -548,7 +558,6 @@ __kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
pgoff_t index = gfn - slot->base_gfn + slot->gmem.pgoff;
struct kvm_gmem *gmem = file->private_data;
struct folio *folio;
- struct page *page;
if (file != slot->gmem.file) {
WARN_ON_ONCE(slot->gmem.file);
@@ -571,9 +580,7 @@ __kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
return ERR_PTR(-EHWPOISON);
}
- page = folio_file_page(folio, index);
-
- *pfn = page_to_pfn(page);
+ *pfn = folio_file_pfn(folio, index);
if (max_order)
*max_order = 0;
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 05/14] KVM: rename CONFIG_HAVE_KVM_GMEM_* to CONFIG_HAVE_KVM_ARCH_GMEM_*
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (3 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 04/14] KVM: guest_memfd: do not go through struct page Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 06/14] KVM: guest_memfd: return locked folio from __kvm_gmem_get_pfn Paolo Bonzini
` (9 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
Add "ARCH" to the symbols; shortly, the "prepare" phase will include both
the arch-independent step to clear out contents left in the page by the
host, and the arch-dependent step enabled by CONFIG_HAVE_KVM_GMEM_PREPARE.
For consistency do the same for CONFIG_HAVE_KVM_GMEM_INVALIDATE as well.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
arch/x86/kvm/Kconfig | 4 ++--
arch/x86/kvm/x86.c | 4 ++--
include/linux/kvm_host.h | 4 ++--
virt/kvm/Kconfig | 4 ++--
virt/kvm/guest_memfd.c | 6 +++---
5 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
index 4287a8071a3a..472a1537b7a9 100644
--- a/arch/x86/kvm/Kconfig
+++ b/arch/x86/kvm/Kconfig
@@ -141,8 +141,8 @@ config KVM_AMD_SEV
depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
select ARCH_HAS_CC_PLATFORM
select KVM_GENERIC_PRIVATE_MEM
- select HAVE_KVM_GMEM_PREPARE
- select HAVE_KVM_GMEM_INVALIDATE
+ select HAVE_KVM_ARCH_GMEM_PREPARE
+ select HAVE_KVM_ARCH_GMEM_INVALIDATE
help
Provides support for launching Encrypted VMs (SEV) and Encrypted VMs
with Encrypted State (SEV-ES) on AMD processors.
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 52778689cef4..4db18924b2f2 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -13644,7 +13644,7 @@ bool kvm_arch_no_poll(struct kvm_vcpu *vcpu)
}
EXPORT_SYMBOL_GPL(kvm_arch_no_poll);
-#ifdef CONFIG_HAVE_KVM_GMEM_PREPARE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
bool kvm_arch_gmem_prepare_needed(struct kvm *kvm)
{
return kvm->arch.vm_type == KVM_X86_SNP_VM;
@@ -13656,7 +13656,7 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
}
#endif
-#ifdef CONFIG_HAVE_KVM_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
{
kvm_x86_call(gmem_invalidate)(start, end);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 689e8be873a7..344d90771844 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2445,7 +2445,7 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
}
#endif /* CONFIG_KVM_PRIVATE_MEM */
-#ifdef CONFIG_HAVE_KVM_GMEM_PREPARE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order);
bool kvm_arch_gmem_prepare_needed(struct kvm *kvm);
#endif
@@ -2477,7 +2477,7 @@ typedef int (*kvm_gmem_populate_cb)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
long kvm_gmem_populate(struct kvm *kvm, gfn_t gfn, void __user *src, long npages,
kvm_gmem_populate_cb post_populate, void *opaque);
-#ifdef CONFIG_HAVE_KVM_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
#endif
diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
index b14e14cdbfb9..fd6a3010afa8 100644
--- a/virt/kvm/Kconfig
+++ b/virt/kvm/Kconfig
@@ -113,10 +113,10 @@ config KVM_GENERIC_PRIVATE_MEM
select KVM_PRIVATE_MEM
bool
-config HAVE_KVM_GMEM_PREPARE
+config HAVE_KVM_ARCH_GMEM_PREPARE
bool
depends on KVM_PRIVATE_MEM
-config HAVE_KVM_GMEM_INVALIDATE
+config HAVE_KVM_ARCH_GMEM_INVALIDATE
bool
depends on KVM_PRIVATE_MEM
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 5221b584288f..76139332f2f3 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -27,7 +27,7 @@ static inline kvm_pfn_t folio_file_pfn(struct folio *folio, pgoff_t index)
static int kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct folio *folio)
{
-#ifdef CONFIG_HAVE_KVM_GMEM_PREPARE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
struct list_head *gmem_list = &inode->i_mapping->i_private_list;
struct kvm_gmem *gmem;
@@ -353,7 +353,7 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
return MF_DELAYED;
}
-#ifdef CONFIG_HAVE_KVM_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
static void kvm_gmem_free_folio(struct folio *folio)
{
struct page *page = folio_page(folio, 0);
@@ -368,7 +368,7 @@ static const struct address_space_operations kvm_gmem_aops = {
.dirty_folio = noop_dirty_folio,
.migrate_folio = kvm_gmem_migrate_folio,
.error_remove_folio = kvm_gmem_error_folio,
-#ifdef CONFIG_HAVE_KVM_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
.free_folio = kvm_gmem_free_folio,
#endif
};
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 06/14] KVM: guest_memfd: return locked folio from __kvm_gmem_get_pfn
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (4 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 05/14] KVM: rename CONFIG_HAVE_KVM_GMEM_* to CONFIG_HAVE_KVM_ARCH_GMEM_* Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 07/14] KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest Paolo Bonzini
` (8 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
Allow testing the up-to-date flag in the caller without taking the
lock again.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
virt/kvm/guest_memfd.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 76139332f2f3..9271aba9b7b3 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -59,6 +59,7 @@ static int kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct fol
return 0;
}
+/* Returns a locked folio on success. */
static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index, bool prepare)
{
struct folio *folio;
@@ -551,6 +552,7 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
fput(file);
}
+/* Returns a locked folio on success. */
static struct folio *
__kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
gfn_t gfn, kvm_pfn_t *pfn, int *max_order, bool prepare)
@@ -584,7 +586,6 @@ __kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
if (max_order)
*max_order = 0;
- folio_unlock(folio);
return folio;
}
@@ -602,6 +603,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
if (IS_ERR(folio))
return PTR_ERR(folio);
+ folio_unlock(folio);
return 0;
}
EXPORT_SYMBOL_GPL(kvm_gmem_get_pfn);
@@ -647,6 +649,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
break;
}
+ folio_unlock(folio);
if (!IS_ALIGNED(gfn, (1 << max_order)) ||
(npages - i) < (1 << max_order))
max_order = 0;
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 07/14] KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (5 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 06/14] KVM: guest_memfd: return locked folio from __kvm_gmem_get_pfn Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-29 21:40 ` Michael Roth
2024-07-26 18:51 ` [PATCH v2 08/14] KVM: guest_memfd: make kvm_gmem_prepare_folio() operate on a single struct kvm Paolo Bonzini
` (7 subsequent siblings)
14 siblings, 1 reply; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
Initializing the contents of the folio on fallocate() is unnecessarily
restrictive. It means that the page is registered with the firmware and
then it cannot be touched anymore. In particular, this loses the
possibility of using fallocate() to pre-allocate the page for SEV-SNP
guests, because kvm_arch_gmem_prepare() then fails.
It's only when the guest actually accesses the page (and therefore
kvm_gmem_get_pfn() is called) that the page must be cleared from any
stale host data and registered with the firmware. The up-to-date flag
is clear if this has to be done (i.e. it is the first access and
kvm_gmem_populate() has not been called).
All in all, there are enough differences between kvm_gmem_get_pfn() and
kvm_gmem_populate(), that it's better to separate the two flows completely.
Extract the bulk of kvm_gmem_get_folio(), which take a folio and end up
setting its up-to-date flag, to a new function kvm_gmem_prepare_folio();
these are now done only by the non-__-prefixed kvm_gmem_get_pfn().
As a bonus, __kvm_gmem_get_pfn() loses its ugly "bool prepare" argument.
One difference is that fallocate(PUNCH_HOLE) can now race with a
page fault. Potentially this causes a page to be prepared and into the
filemap even after fallocate(PUNCH_HOLE). This is harmless, as it can be
fixed by another hole punching operation, and can be avoided by clearing
the private-page attribute prior to invoking fallocate(PUNCH_HOLE).
This way, the page fault will cause an exit to user space.
The previous semantics, where fallocate() could be used to prepare
the pages in advance of running the guest, can be accessed with
KVM_PRE_FAULT_MEMORY.
For now, accessing a page in one VM will attempt to call
kvm_arch_gmem_prepare() in all of those that have bound the guest_memfd.
Cleaning this up is left to a separate patch.
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
virt/kvm/guest_memfd.c | 110 ++++++++++++++++++++++++-----------------
1 file changed, 66 insertions(+), 44 deletions(-)
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 9271aba9b7b3..5af278c7adba 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -25,7 +25,7 @@ static inline kvm_pfn_t folio_file_pfn(struct folio *folio, pgoff_t index)
return folio_pfn(folio) + (index & (folio_nr_pages(folio) - 1));
}
-static int kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct folio *folio)
+static int __kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct folio *folio)
{
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
struct list_head *gmem_list = &inode->i_mapping->i_private_list;
@@ -59,49 +59,63 @@ static int kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct fol
return 0;
}
-/* Returns a locked folio on success. */
-static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index, bool prepare)
+/*
+ * Process @folio, which contains @gfn, so that the guest can use it.
+ * The folio must be locked and the gfn must be contained in @slot.
+ * On successful return the guest sees a zero page so as to avoid
+ * leaking host data and the up-to-date flag is set.
+ */
+static int kvm_gmem_prepare_folio(struct file *file, struct kvm_memory_slot *slot,
+ gfn_t gfn, struct folio *folio)
{
- struct folio *folio;
+ unsigned long nr_pages, i;
+ pgoff_t index;
+ int r;
- /* TODO: Support huge pages. */
- folio = filemap_grab_folio(inode->i_mapping, index);
- if (IS_ERR(folio))
- return folio;
+ if (folio_test_uptodate(folio))
+ return 0;
+
+ nr_pages = folio_nr_pages(folio);
+ for (i = 0; i < nr_pages; i++)
+ clear_highpage(folio_page(folio, i));
/*
- * Use the up-to-date flag to track whether or not the memory has been
- * zeroed before being handed off to the guest. There is no backing
- * storage for the memory, so the folio will remain up-to-date until
- * it's removed.
+ * Preparing huge folios should always be safe, since it should
+ * be possible to split them later if needed.
*
- * TODO: Skip clearing pages when trusted firmware will do it when
- * assigning memory to the guest.
+ * Right now the folio order is always going to be zero, but the
+ * code is ready for huge folios. The only assumption is that
+ * the base pgoff of memslots is naturally aligned with the
+ * requested page order, ensuring that huge folios can also use
+ * huge page table entries for GPA->HPA mapping.
+ *
+ * The order will be passed when creating the guest_memfd, and
+ * checked when creating memslots.
*/
- if (!folio_test_uptodate(folio)) {
- unsigned long nr_pages = folio_nr_pages(folio);
- unsigned long i;
-
- for (i = 0; i < nr_pages; i++)
- clear_highpage(folio_page(folio, i));
- }
-
- if (prepare) {
- int r = kvm_gmem_prepare_folio(inode, index, folio);
- if (r < 0) {
- folio_unlock(folio);
- folio_put(folio);
- return ERR_PTR(r);
- }
+ WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, 1 << folio_order(folio)));
+ index = gfn - slot->base_gfn + slot->gmem.pgoff;
+ index = ALIGN_DOWN(index, 1 << folio_order(folio));
+ r = __kvm_gmem_prepare_folio(file_inode(file), index, folio);
+ if (!r)
folio_mark_uptodate(folio);
- }
- /*
- * Ignore accessed, referenced, and dirty flags. The memory is
- * unevictable and there is no storage to write back to.
- */
- return folio;
+ return r;
+}
+
+/*
+ * Returns a locked folio on success. The caller is responsible for
+ * setting the up-to-date flag before the memory is mapped into the guest.
+ * There is no backing storage for the memory, so the folio will remain
+ * up-to-date until it's removed.
+ *
+ * Ignore accessed, referenced, and dirty flags. The memory is
+ * unevictable and there is no storage to write back to.
+ */
+static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
+{
+ /* TODO: Support huge pages. */
+ return filemap_grab_folio(inode->i_mapping, index);
}
static void kvm_gmem_invalidate_begin(struct kvm_gmem *gmem, pgoff_t start,
@@ -201,7 +215,7 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
break;
}
- folio = kvm_gmem_get_folio(inode, index, true);
+ folio = kvm_gmem_get_folio(inode, index);
if (IS_ERR(folio)) {
r = PTR_ERR(folio);
break;
@@ -555,7 +569,7 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
/* Returns a locked folio on success. */
static struct folio *
__kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
- gfn_t gfn, kvm_pfn_t *pfn, int *max_order, bool prepare)
+ gfn_t gfn, kvm_pfn_t *pfn, int *max_order)
{
pgoff_t index = gfn - slot->base_gfn + slot->gmem.pgoff;
struct kvm_gmem *gmem = file->private_data;
@@ -572,7 +586,7 @@ __kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
return ERR_PTR(-EIO);
}
- folio = kvm_gmem_get_folio(file_inode(file), index, prepare);
+ folio = kvm_gmem_get_folio(file_inode(file), index);
if (IS_ERR(folio))
return folio;
@@ -594,17 +608,25 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
{
struct file *file = kvm_gmem_get_file(slot);
struct folio *folio;
+ int r = 0;
if (!file)
return -EFAULT;
- folio = __kvm_gmem_get_pfn(file, slot, gfn, pfn, max_order, true);
- fput(file);
- if (IS_ERR(folio))
- return PTR_ERR(folio);
+ folio = __kvm_gmem_get_pfn(file, slot, gfn, pfn, max_order);
+ if (IS_ERR(folio)) {
+ r = PTR_ERR(folio);
+ goto out;
+ }
+ r = kvm_gmem_prepare_folio(file, slot, gfn, folio);
folio_unlock(folio);
- return 0;
+ if (r < 0)
+ folio_put(folio);
+
+out:
+ fput(file);
+ return r;
}
EXPORT_SYMBOL_GPL(kvm_gmem_get_pfn);
@@ -643,7 +665,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
break;
}
- folio = __kvm_gmem_get_pfn(file, slot, gfn, &pfn, &max_order, false);
+ folio = __kvm_gmem_get_pfn(file, slot, gfn, &pfn, &max_order);
if (IS_ERR(folio)) {
ret = PTR_ERR(folio);
break;
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH v2 07/14] KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest
2024-07-26 18:51 ` [PATCH v2 07/14] KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest Paolo Bonzini
@ 2024-07-29 21:40 ` Michael Roth
0 siblings, 0 replies; 18+ messages in thread
From: Michael Roth @ 2024-07-29 21:40 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: linux-kernel, kvm, seanjc
On Fri, Jul 26, 2024 at 02:51:50PM -0400, Paolo Bonzini wrote:
> Initializing the contents of the folio on fallocate() is unnecessarily
> restrictive. It means that the page is registered with the firmware and
> then it cannot be touched anymore. In particular, this loses the
> possibility of using fallocate() to pre-allocate the page for SEV-SNP
> guests, because kvm_arch_gmem_prepare() then fails.
>
> It's only when the guest actually accesses the page (and therefore
> kvm_gmem_get_pfn() is called) that the page must be cleared from any
> stale host data and registered with the firmware. The up-to-date flag
> is clear if this has to be done (i.e. it is the first access and
> kvm_gmem_populate() has not been called).
>
> All in all, there are enough differences between kvm_gmem_get_pfn() and
> kvm_gmem_populate(), that it's better to separate the two flows completely.
> Extract the bulk of kvm_gmem_get_folio(), which take a folio and end up
> setting its up-to-date flag, to a new function kvm_gmem_prepare_folio();
> these are now done only by the non-__-prefixed kvm_gmem_get_pfn().
> As a bonus, __kvm_gmem_get_pfn() loses its ugly "bool prepare" argument.
>
> One difference is that fallocate(PUNCH_HOLE) can now race with a
> page fault. Potentially this causes a page to be prepared and into the
> filemap even after fallocate(PUNCH_HOLE). This is harmless, as it can be
> fixed by another hole punching operation, and can be avoided by clearing
> the private-page attribute prior to invoking fallocate(PUNCH_HOLE).
> This way, the page fault will cause an exit to user space.
>
> The previous semantics, where fallocate() could be used to prepare
> the pages in advance of running the guest, can be accessed with
> KVM_PRE_FAULT_MEMORY.
>
> For now, accessing a page in one VM will attempt to call
> kvm_arch_gmem_prepare() in all of those that have bound the guest_memfd.
> Cleaning this up is left to a separate patch.
>
> Suggested-by: Sean Christopherson <seanjc@google.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> virt/kvm/guest_memfd.c | 110 ++++++++++++++++++++++++-----------------
> 1 file changed, 66 insertions(+), 44 deletions(-)
>
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index 9271aba9b7b3..5af278c7adba 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
> @@ -25,7 +25,7 @@ static inline kvm_pfn_t folio_file_pfn(struct folio *folio, pgoff_t index)
> return folio_pfn(folio) + (index & (folio_nr_pages(folio) - 1));
> }
>
> -static int kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct folio *folio)
> +static int __kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct folio *folio)
> {
> #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
> struct list_head *gmem_list = &inode->i_mapping->i_private_list;
> @@ -59,49 +59,63 @@ static int kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct fol
> return 0;
> }
>
> -/* Returns a locked folio on success. */
> -static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index, bool prepare)
> +/*
> + * Process @folio, which contains @gfn, so that the guest can use it.
> + * The folio must be locked and the gfn must be contained in @slot.
> + * On successful return the guest sees a zero page so as to avoid
> + * leaking host data and the up-to-date flag is set.
> + */
> +static int kvm_gmem_prepare_folio(struct file *file, struct kvm_memory_slot *slot,
> + gfn_t gfn, struct folio *folio)
> {
> - struct folio *folio;
> + unsigned long nr_pages, i;
> + pgoff_t index;
> + int r;
>
> - /* TODO: Support huge pages. */
> - folio = filemap_grab_folio(inode->i_mapping, index);
> - if (IS_ERR(folio))
> - return folio;
> + if (folio_test_uptodate(folio))
> + return 0;
> +
> + nr_pages = folio_nr_pages(folio);
> + for (i = 0; i < nr_pages; i++)
> + clear_highpage(folio_page(folio, i));
>
> /*
> - * Use the up-to-date flag to track whether or not the memory has been
> - * zeroed before being handed off to the guest. There is no backing
> - * storage for the memory, so the folio will remain up-to-date until
> - * it's removed.
> + * Preparing huge folios should always be safe, since it should
> + * be possible to split them later if needed.
> *
> - * TODO: Skip clearing pages when trusted firmware will do it when
> - * assigning memory to the guest.
> + * Right now the folio order is always going to be zero, but the
> + * code is ready for huge folios. The only assumption is that
> + * the base pgoff of memslots is naturally aligned with the
> + * requested page order, ensuring that huge folios can also use
> + * huge page table entries for GPA->HPA mapping.
> + *
> + * The order will be passed when creating the guest_memfd, and
> + * checked when creating memslots.
> */
> - if (!folio_test_uptodate(folio)) {
> - unsigned long nr_pages = folio_nr_pages(folio);
> - unsigned long i;
> -
> - for (i = 0; i < nr_pages; i++)
> - clear_highpage(folio_page(folio, i));
> - }
> -
> - if (prepare) {
> - int r = kvm_gmem_prepare_folio(inode, index, folio);
> - if (r < 0) {
> - folio_unlock(folio);
> - folio_put(folio);
> - return ERR_PTR(r);
> - }
> + WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, 1 << folio_order(folio)));
For this patchset, I think the WARN_ON() is appropriate since without
mapping_set_large_folios() we'd only ever expect 4K pages, in which
case gmem.pgoff would necessarilly be naturally-aligned to 4K and
it'll be good to notice if that unexpectedly changes.
However, it's worth noting now that that likely will change once THP
is enabled, since QEMU will generally set gmem.pgoff to be the same
as slot.base_gfn rather than doing any sort of padding out.
For instance, filemap considers gmem.pgoff=0 to be the start of the
virtual range it backs, and will allocate large folio's on
2MB-aligned gmem.pgoff values. However, QEMU might align some slots
on non-2MB-aligned gmem.pgoff boundaries to handle things like legacy
regions. E.g., the following example:
kvm_set_user_memory AddrSpace#0 Slot#0 flags=0x0 gpa=0x0 size=0x0 ua=0x7fc257e00000 guest_memfd=19 guest_memfd_offset=0x0 ret=0
kvm_set_user_memory AddrSpace#0 Slot#0 flags=0x4 gpa=0x0 size=0xc0000 ua=0x7fc257e00000 guest_memfd=19 guest_memfd_offset=0x0 ret=0
kvm_set_user_memory AddrSpace#0 Slot#3 flags=0x4 gpa=0xc0000 size=0x20000 ua=0x7fc06de00000 guest_memfd=534 guest_memfd_offset=0x0 ret=0
kvm_set_user_memory AddrSpace#0 Slot#4 flags=0x4 gpa=0xe0000 size=0x7ff20000 ua=0x7fc257ee0000 guest_memfd=19 guest_memfd_offset=0xe0000 ret=0
will result in everything at GFN 0xe0 and up (slot 4) being associated
with a gmem.pgoff value of 0xe0, so it will always fail the WARN_ON().
But:
a) GFN 0xe0 might still be backed by a large folio that's rooted at
gmem.pgoff==0. It's still technically possible to prepare the
entire 2MB folio in the RMP table/etc., since KVM MMU will split
the NPT mapping and trigger an RMP #PF to psmash the 2MB RMP
entry if necessary.
b) GFN 0x200 and up could still be backed by large folios even
though they'd trigger this WARN_ON().
Once we move to more granular per-4K tracking rather than per-folio
tracking I think a lot of this nuance goes away however (or maybe
requires some re-thinking), so probably makes sense to revisit in
the context of that work, but I thought it was worth mentioning.
Reviewed-by: Michael Roth <michael.roth@amd.com>
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v2 08/14] KVM: guest_memfd: make kvm_gmem_prepare_folio() operate on a single struct kvm
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (6 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 07/14] KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 09/14] KVM: remove kvm_arch_gmem_prepare_needed() Paolo Bonzini
` (6 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
This is now possible because preparation is done by kvm_gmem_get_pfn()
instead of fallocate(). In practice this is not a limitation, because
even though guest_memfd can be bound to multiple struct kvm, for
hardware implementations of confidential computing only one guest
(identified by an ASID on SEV-SNP, or an HKID on TDX) will be able
to access it.
In the case of intra-host migration (not implemented yet for SEV-SNP,
but we can use SEV-ES as an idea of how it will work), the new struct
kvm inherits the same ASID and preparation need not be repeated.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
virt/kvm/guest_memfd.c | 47 ++++++++++++++++--------------------------
1 file changed, 18 insertions(+), 29 deletions(-)
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 5af278c7adba..444ded162154 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -25,37 +25,27 @@ static inline kvm_pfn_t folio_file_pfn(struct folio *folio, pgoff_t index)
return folio_pfn(folio) + (index & (folio_nr_pages(folio) - 1));
}
-static int __kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct folio *folio)
+static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
+ pgoff_t index, struct folio *folio)
{
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
- struct list_head *gmem_list = &inode->i_mapping->i_private_list;
- struct kvm_gmem *gmem;
+ kvm_pfn_t pfn;
+ gfn_t gfn;
+ int rc;
- list_for_each_entry(gmem, gmem_list, entry) {
- struct kvm_memory_slot *slot;
- struct kvm *kvm = gmem->kvm;
- kvm_pfn_t pfn;
- gfn_t gfn;
- int rc;
+ if (!kvm_arch_gmem_prepare_needed(kvm))
+ return 0;
- if (!kvm_arch_gmem_prepare_needed(kvm))
- continue;
-
- slot = xa_load(&gmem->bindings, index);
- if (!slot)
- continue;
-
- pfn = folio_file_pfn(folio, index);
- gfn = slot->base_gfn + index - slot->gmem.pgoff;
- rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
- if (rc) {
- pr_warn_ratelimited("gmem: Failed to prepare folio for GFN %llx PFN %llx error %d.\n",
- gfn, pfn, rc);
- return rc;
- }
+ pfn = folio_file_pfn(folio, index);
+ gfn = slot->base_gfn + index - slot->gmem.pgoff;
+ rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
+ if (rc) {
+ pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
+ index, gfn, pfn, rc);
+ return rc;
}
-
#endif
+
return 0;
}
@@ -65,7 +55,7 @@ static int __kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct f
* On successful return the guest sees a zero page so as to avoid
* leaking host data and the up-to-date flag is set.
*/
-static int kvm_gmem_prepare_folio(struct file *file, struct kvm_memory_slot *slot,
+static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
gfn_t gfn, struct folio *folio)
{
unsigned long nr_pages, i;
@@ -95,8 +85,7 @@ static int kvm_gmem_prepare_folio(struct file *file, struct kvm_memory_slot *slo
WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, 1 << folio_order(folio)));
index = gfn - slot->base_gfn + slot->gmem.pgoff;
index = ALIGN_DOWN(index, 1 << folio_order(folio));
-
- r = __kvm_gmem_prepare_folio(file_inode(file), index, folio);
+ r = __kvm_gmem_prepare_folio(kvm, slot, index, folio);
if (!r)
folio_mark_uptodate(folio);
@@ -619,7 +608,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
goto out;
}
- r = kvm_gmem_prepare_folio(file, slot, gfn, folio);
+ r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
folio_unlock(folio);
if (r < 0)
folio_put(folio);
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 09/14] KVM: remove kvm_arch_gmem_prepare_needed()
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (7 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 08/14] KVM: guest_memfd: make kvm_gmem_prepare_folio() operate on a single struct kvm Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 10/14] KVM: guest_memfd: move check for already-populated page to common code Paolo Bonzini
` (5 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
It is enough to return 0 if a guest need not do any preparation.
This is in fact how sev_gmem_prepare() works for non-SNP guests,
and it extends naturally to Intel hosts: the x86 callback for
gmem_prepare is optional and returns 0 if not defined.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
arch/x86/kvm/x86.c | 5 -----
include/linux/kvm_host.h | 1 -
virt/kvm/guest_memfd.c | 13 +++----------
3 files changed, 3 insertions(+), 16 deletions(-)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 4db18924b2f2..ef3d3511e4af 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -13645,11 +13645,6 @@ bool kvm_arch_no_poll(struct kvm_vcpu *vcpu)
EXPORT_SYMBOL_GPL(kvm_arch_no_poll);
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-bool kvm_arch_gmem_prepare_needed(struct kvm *kvm)
-{
- return kvm->arch.vm_type == KVM_X86_SNP_VM;
-}
-
int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order)
{
return kvm_x86_call(gmem_prepare)(kvm, pfn, gfn, max_order);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 344d90771844..45373d42f314 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2447,7 +2447,6 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order);
-bool kvm_arch_gmem_prepare_needed(struct kvm *kvm);
#endif
/**
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 444ded162154..191fd42067c0 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -29,16 +29,9 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
pgoff_t index, struct folio *folio)
{
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
- kvm_pfn_t pfn;
- gfn_t gfn;
- int rc;
-
- if (!kvm_arch_gmem_prepare_needed(kvm))
- return 0;
-
- pfn = folio_file_pfn(folio, index);
- gfn = slot->base_gfn + index - slot->gmem.pgoff;
- rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
+ kvm_pfn_t pfn = folio_file_pfn(folio, index);
+ gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
+ int rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
if (rc) {
pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
index, gfn, pfn, rc);
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 10/14] KVM: guest_memfd: move check for already-populated page to common code
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (8 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 09/14] KVM: remove kvm_arch_gmem_prepare_needed() Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 11/14] KVM: cleanup and add shortcuts to kvm_range_has_memory_attributes() Paolo Bonzini
` (4 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
Do not allow populating the same page twice with startup data. In the
case of SEV-SNP, for example, the firmware does not allow it anyway,
since the launch-update operation is only possible on pages that are
still shared in the RMP.
Even if it worked, kvm_gmem_populate()'s callback is meant to have side
effects such as updating launch measurements, and updating the same
page twice is unlikely to have the desired results.
Races between calls to the ioctl are not possible because
kvm_gmem_populate() holds slots_lock and the VM should not be running.
But again, even if this worked on other confidential computing technology,
it doesn't matter to guest_memfd.c whether this is something fishy
such as missing synchronization in userspace, or rather something
intentional. One of the racers wins, and the page is initialized by
either kvm_gmem_prepare_folio() or kvm_gmem_populate().
Anyway, out of paranoia, adjust sev_gmem_post_populate() anyway to use
the same errno that kvm_gmem_populate() is using.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
arch/x86/kvm/svm/sev.c | 2 +-
virt/kvm/guest_memfd.c | 7 +++++++
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 6589091e8ce0..752d2fff0f10 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -2290,7 +2290,7 @@ static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn_start, kvm_pfn_t pf
if (ret || assigned) {
pr_debug("%s: Failed to ensure GFN 0x%llx RMP entry is initial shared state, ret: %d assigned: %d\n",
__func__, gfn, ret, assigned);
- ret = -EINVAL;
+ ret = ret ? -EINVAL : -EEXIST;
goto err;
}
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 191fd42067c0..319ec491ca47 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -653,6 +653,13 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
break;
}
+ if (folio_test_uptodate(folio)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ ret = -EEXIST;
+ break;
+ }
+
folio_unlock(folio);
if (!IS_ALIGNED(gfn, (1 << max_order)) ||
(npages - i) < (1 << max_order))
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 11/14] KVM: cleanup and add shortcuts to kvm_range_has_memory_attributes()
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (9 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 10/14] KVM: guest_memfd: move check for already-populated page to common code Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 12/14] KVM: extend kvm_range_has_memory_attributes() to check subset of attributes Paolo Bonzini
` (3 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
Use a guard to simplify early returns, and add two more easy
shortcuts. If the requested attributes are invalid, the attributes
xarray will never show them as set. And if testing a single page,
kvm_get_memory_attributes() is more efficient.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
virt/kvm/kvm_main.c | 42 ++++++++++++++++++++----------------------
1 file changed, 20 insertions(+), 22 deletions(-)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index d0788d0a72cc..30328ff2d840 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2398,6 +2398,14 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
#endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */
#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+static u64 kvm_supported_mem_attributes(struct kvm *kvm)
+{
+ if (!kvm || kvm_arch_has_private_mem(kvm))
+ return KVM_MEMORY_ATTRIBUTE_PRIVATE;
+
+ return 0;
+}
+
/*
* Returns true if _all_ gfns in the range [@start, @end) have attributes
* matching @attrs.
@@ -2406,40 +2414,30 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
unsigned long attrs)
{
XA_STATE(xas, &kvm->mem_attr_array, start);
+ unsigned long mask = kvm_supported_mem_attributes(kvm);
unsigned long index;
- bool has_attrs;
void *entry;
- rcu_read_lock();
+ if (attrs & ~mask)
+ return false;
- if (!attrs) {
- has_attrs = !xas_find(&xas, end - 1);
- goto out;
- }
+ if (end == start + 1)
+ return kvm_get_memory_attributes(kvm, start) == attrs;
+
+ guard(rcu)();
+ if (!attrs)
+ return !xas_find(&xas, end - 1);
- has_attrs = true;
for (index = start; index < end; index++) {
do {
entry = xas_next(&xas);
} while (xas_retry(&xas, entry));
- if (xas.xa_index != index || xa_to_value(entry) != attrs) {
- has_attrs = false;
- break;
- }
+ if (xas.xa_index != index || xa_to_value(entry) != attrs)
+ return false;
}
-out:
- rcu_read_unlock();
- return has_attrs;
-}
-
-static u64 kvm_supported_mem_attributes(struct kvm *kvm)
-{
- if (!kvm || kvm_arch_has_private_mem(kvm))
- return KVM_MEMORY_ATTRIBUTE_PRIVATE;
-
- return 0;
+ return true;
}
static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 12/14] KVM: extend kvm_range_has_memory_attributes() to check subset of attributes
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (10 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 11/14] KVM: cleanup and add shortcuts to kvm_range_has_memory_attributes() Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 13/14] KVM: guest_memfd: let kvm_gmem_populate() operate only on private gfns Paolo Bonzini
` (2 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
While currently there is no other attribute than KVM_MEMORY_ATTRIBUTE_PRIVATE,
KVM code such as kvm_mem_is_private() is written to expect their existence.
Allow using kvm_range_has_memory_attributes() as a multi-page version of
kvm_mem_is_private(), without it breaking later when more attributes are
introduced.
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
arch/x86/kvm/mmu/mmu.c | 2 +-
include/linux/kvm_host.h | 2 +-
virt/kvm/kvm_main.c | 13 +++++++------
3 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 26ef5b6ac3c1..43a02891babb 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -7513,7 +7513,7 @@ static bool hugepage_has_attrs(struct kvm *kvm, struct kvm_memory_slot *slot,
const unsigned long end = start + KVM_PAGES_PER_HPAGE(level);
if (level == PG_LEVEL_2M)
- return kvm_range_has_memory_attributes(kvm, start, end, attrs);
+ return kvm_range_has_memory_attributes(kvm, start, end, ~0, attrs);
for (gfn = start; gfn < end; gfn += KVM_PAGES_PER_HPAGE(level - 1)) {
if (hugepage_test_mixed(slot, gfn, level - 1) ||
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 45373d42f314..c223b97df03e 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2414,7 +2414,7 @@ static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn
}
bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
- unsigned long attrs);
+ unsigned long mask, unsigned long attrs);
bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
struct kvm_gfn_range *range);
bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 30328ff2d840..92901656a0d4 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2408,21 +2408,21 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
/*
* Returns true if _all_ gfns in the range [@start, @end) have attributes
- * matching @attrs.
+ * such that the bits in @mask match @attrs.
*/
bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
- unsigned long attrs)
+ unsigned long mask, unsigned long attrs)
{
XA_STATE(xas, &kvm->mem_attr_array, start);
- unsigned long mask = kvm_supported_mem_attributes(kvm);
unsigned long index;
void *entry;
+ mask &= kvm_supported_mem_attributes(kvm);
if (attrs & ~mask)
return false;
if (end == start + 1)
- return kvm_get_memory_attributes(kvm, start) == attrs;
+ return (kvm_get_memory_attributes(kvm, start) & mask) == attrs;
guard(rcu)();
if (!attrs)
@@ -2433,7 +2433,8 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
entry = xas_next(&xas);
} while (xas_retry(&xas, entry));
- if (xas.xa_index != index || xa_to_value(entry) != attrs)
+ if (xas.xa_index != index ||
+ (xa_to_value(entry) & mask) != attrs)
return false;
}
@@ -2532,7 +2533,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
mutex_lock(&kvm->slots_lock);
/* Nothing to do if the entire range as the desired attributes. */
- if (kvm_range_has_memory_attributes(kvm, start, end, attributes))
+ if (kvm_range_has_memory_attributes(kvm, start, end, ~0, attributes))
goto out_unlock;
/*
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 13/14] KVM: guest_memfd: let kvm_gmem_populate() operate only on private gfns
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (11 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 12/14] KVM: extend kvm_range_has_memory_attributes() to check subset of attributes Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-26 18:51 ` [PATCH v2 14/14] KVM: guest_memfd: abstract how prepared folios are recorded Paolo Bonzini
2024-07-29 22:06 ` [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Michael Roth
14 siblings, 0 replies; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
This check is currently performed by sev_gmem_post_populate(), but it
applies to all callers of kvm_gmem_populate(): the point of the function
is that the memory is being encrypted and some work has to be done
on all the gfns in order to encrypt them.
Therefore, check the KVM_MEMORY_ATTRIBUTE_PRIVATE attribute prior
to invoking the callback, and stop the operation if a shared page
is encountered. Because CONFIG_KVM_PRIVATE_MEM in principle does
not require attributes, this makes kvm_gmem_populate() depend on
CONFIG_KVM_GENERIC_PRIVATE_MEM (which does require them).
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
arch/x86/kvm/svm/sev.c | 7 -------
include/linux/kvm_host.h | 2 ++
virt/kvm/guest_memfd.c | 12 ++++++++++++
3 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 752d2fff0f10..532df12b43c5 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -2279,13 +2279,6 @@ static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn_start, kvm_pfn_t pf
bool assigned;
int level;
- if (!kvm_mem_is_private(kvm, gfn)) {
- pr_debug("%s: Failed to ensure GFN 0x%llx has private memory attribute set\n",
- __func__, gfn);
- ret = -EINVAL;
- goto err;
- }
-
ret = snp_lookup_rmpentry((u64)pfn + i, &assigned, &level);
if (ret || assigned) {
pr_debug("%s: Failed to ensure GFN 0x%llx RMP entry is initial shared state, ret: %d assigned: %d\n",
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index c223b97df03e..79a6b1a63027 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2449,6 +2449,7 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order);
#endif
+#ifdef CONFIG_KVM_GENERIC_PRIVATE_MEM
/**
* kvm_gmem_populate() - Populate/prepare a GPA range with guest data
*
@@ -2475,6 +2476,7 @@ typedef int (*kvm_gmem_populate_cb)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
long kvm_gmem_populate(struct kvm *kvm, gfn_t gfn, void __user *src, long npages,
kvm_gmem_populate_cb post_populate, void *opaque);
+#endif
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 319ec491ca47..95e338a29910 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -612,6 +612,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
}
EXPORT_SYMBOL_GPL(kvm_gmem_get_pfn);
+#ifdef CONFIG_KVM_GENERIC_PRIVATE_MEM
long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long npages,
kvm_gmem_populate_cb post_populate, void *opaque)
{
@@ -665,11 +666,21 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
(npages - i) < (1 << max_order))
max_order = 0;
+ ret = -EINVAL;
+ while (!kvm_range_has_memory_attributes(kvm, gfn, gfn + (1 << max_order),
+ KVM_MEMORY_ATTRIBUTE_PRIVATE,
+ KVM_MEMORY_ATTRIBUTE_PRIVATE)) {
+ if (!max_order)
+ goto put_folio_and_exit;
+ max_order--;
+ }
+
p = src ? src + i * PAGE_SIZE : NULL;
ret = post_populate(kvm, gfn, pfn, p, max_order, opaque);
if (!ret)
folio_mark_uptodate(folio);
+put_folio_and_exit:
folio_put(folio);
if (ret)
break;
@@ -681,3 +692,4 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
return ret && !i ? ret : i;
}
EXPORT_SYMBOL_GPL(kvm_gmem_populate);
+#endif
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v2 14/14] KVM: guest_memfd: abstract how prepared folios are recorded
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (12 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 13/14] KVM: guest_memfd: let kvm_gmem_populate() operate only on private gfns Paolo Bonzini
@ 2024-07-26 18:51 ` Paolo Bonzini
2024-07-29 21:59 ` Michael Roth
2024-07-29 22:06 ` [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Michael Roth
14 siblings, 1 reply; 18+ messages in thread
From: Paolo Bonzini @ 2024-07-26 18:51 UTC (permalink / raw)
To: linux-kernel, kvm; +Cc: seanjc, michael.roth
Right now, large folios are not supported in guest_memfd, and therefore the order
used by kvm_gmem_populate() is always 0. In this scenario, using the up-to-date
bit to track prepared-ness is nice and easy because we have one bit available
per page.
In the future, however, we might have large pages that are partially populated;
for example, in the case of SEV-SNP, if a large page has both shared and private
areas inside, it is necessary to populate it at a granularity that is smaller
than that of the guest_memfd's backing store. In that case we will have
to track preparedness at a 4K level, probably as a bitmap.
In preparation for that, do not use explicitly folio_test_uptodate() and
folio_mark_uptodate(). Return the state of the page directly from
__kvm_gmem_get_pfn(), so that it is expected to apply to 2^N pages
with N=*max_order. The function to mark a range as prepared for now
takes just a folio, but is expected to take also an index and order
(or something like that) when large pages are introduced.
Thanks to Michael Roth for pointing out the issue with large pages.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
virt/kvm/guest_memfd.c | 32 +++++++++++++++++++-------------
1 file changed, 19 insertions(+), 13 deletions(-)
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 95e338a29910..f4fa5e040fd1 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -42,6 +42,11 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
return 0;
}
+static inline void kvm_gmem_mark_prepared(struct folio *folio)
+{
+ folio_mark_uptodate(folio);
+}
+
/*
* Process @folio, which contains @gfn, so that the guest can use it.
* The folio must be locked and the gfn must be contained in @slot.
@@ -55,9 +60,6 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
pgoff_t index;
int r;
- if (folio_test_uptodate(folio))
- return 0;
-
nr_pages = folio_nr_pages(folio);
for (i = 0; i < nr_pages; i++)
clear_highpage(folio_page(folio, i));
@@ -80,7 +82,7 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
index = ALIGN_DOWN(index, 1 << folio_order(folio));
r = __kvm_gmem_prepare_folio(kvm, slot, index, folio);
if (!r)
- folio_mark_uptodate(folio);
+ kvm_gmem_mark_prepared(folio);
return r;
}
@@ -551,7 +553,8 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
/* Returns a locked folio on success. */
static struct folio *
__kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
- gfn_t gfn, kvm_pfn_t *pfn, int *max_order)
+ gfn_t gfn, kvm_pfn_t *pfn, bool *is_prepared,
+ int *max_order)
{
pgoff_t index = gfn - slot->base_gfn + slot->gmem.pgoff;
struct kvm_gmem *gmem = file->private_data;
@@ -582,6 +585,7 @@ __kvm_gmem_get_pfn(struct file *file, struct kvm_memory_slot *slot,
if (max_order)
*max_order = 0;
+ *is_prepared = folio_test_uptodate(folio);
return folio;
}
@@ -590,18 +594,21 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
{
struct file *file = kvm_gmem_get_file(slot);
struct folio *folio;
+ bool is_prepared = false;
int r = 0;
if (!file)
return -EFAULT;
- folio = __kvm_gmem_get_pfn(file, slot, gfn, pfn, max_order);
+ folio = __kvm_gmem_get_pfn(file, slot, gfn, pfn, &is_prepared, max_order);
if (IS_ERR(folio)) {
r = PTR_ERR(folio);
goto out;
}
- r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
+ if (!is_prepared)
+ r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
+
folio_unlock(folio);
if (r < 0)
folio_put(folio);
@@ -641,6 +648,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
for (i = 0; i < npages; i += (1 << max_order)) {
struct folio *folio;
gfn_t gfn = start_gfn + i;
+ bool is_prepared = false;
kvm_pfn_t pfn;
if (signal_pending(current)) {
@@ -648,13 +656,13 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
break;
}
- folio = __kvm_gmem_get_pfn(file, slot, gfn, &pfn, &max_order);
+ folio = __kvm_gmem_get_pfn(file, slot, gfn, &pfn, &is_prepared, &max_order);
if (IS_ERR(folio)) {
ret = PTR_ERR(folio);
break;
}
- if (folio_test_uptodate(folio)) {
+ if (is_prepared) {
folio_unlock(folio);
folio_put(folio);
ret = -EEXIST;
@@ -662,9 +670,8 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
}
folio_unlock(folio);
- if (!IS_ALIGNED(gfn, (1 << max_order)) ||
- (npages - i) < (1 << max_order))
- max_order = 0;
+ WARN_ON(!IS_ALIGNED(gfn, 1 << max_order) ||
+ (npages - i) < (1 << max_order));
ret = -EINVAL;
while (!kvm_range_has_memory_attributes(kvm, gfn, gfn + (1 << max_order),
@@ -678,7 +684,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
p = src ? src + i * PAGE_SIZE : NULL;
ret = post_populate(kvm, gfn, pfn, p, max_order, opaque);
if (!ret)
- folio_mark_uptodate(folio);
+ kvm_gmem_mark_prepared(folio);
put_folio_and_exit:
folio_put(folio);
--
2.43.0
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH v2 14/14] KVM: guest_memfd: abstract how prepared folios are recorded
2024-07-26 18:51 ` [PATCH v2 14/14] KVM: guest_memfd: abstract how prepared folios are recorded Paolo Bonzini
@ 2024-07-29 21:59 ` Michael Roth
0 siblings, 0 replies; 18+ messages in thread
From: Michael Roth @ 2024-07-29 21:59 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: linux-kernel, kvm, seanjc
On Fri, Jul 26, 2024 at 02:51:57PM -0400, Paolo Bonzini wrote:
> Right now, large folios are not supported in guest_memfd, and therefore the order
> used by kvm_gmem_populate() is always 0. In this scenario, using the up-to-date
> bit to track prepared-ness is nice and easy because we have one bit available
> per page.
>
> In the future, however, we might have large pages that are partially populated;
> for example, in the case of SEV-SNP, if a large page has both shared and private
> areas inside, it is necessary to populate it at a granularity that is smaller
> than that of the guest_memfd's backing store. In that case we will have
> to track preparedness at a 4K level, probably as a bitmap.
>
> In preparation for that, do not use explicitly folio_test_uptodate() and
> folio_mark_uptodate(). Return the state of the page directly from
> __kvm_gmem_get_pfn(), so that it is expected to apply to 2^N pages
> with N=*max_order. The function to mark a range as prepared for now
> takes just a folio, but is expected to take also an index and order
> (or something like that) when large pages are introduced.
>
> Thanks to Michael Roth for pointing out the issue with large pages.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP
2024-07-26 18:51 [PATCH v2 00/14] KVM: guest_memfd: lazy preparation of pages + prefault support for SEV-SNP Paolo Bonzini
` (13 preceding siblings ...)
2024-07-26 18:51 ` [PATCH v2 14/14] KVM: guest_memfd: abstract how prepared folios are recorded Paolo Bonzini
@ 2024-07-29 22:06 ` Michael Roth
14 siblings, 0 replies; 18+ messages in thread
From: Michael Roth @ 2024-07-29 22:06 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: linux-kernel, kvm, seanjc, pratikrajesh.sampat
On Fri, Jul 26, 2024 at 02:51:43PM -0400, Paolo Bonzini wrote:
> Paolo Bonzini (14):
> KVM: x86: disallow pre-fault for SNP VMs before initialization
> KVM: guest_memfd: return folio from __kvm_gmem_get_pfn()
> KVM: guest_memfd: delay folio_mark_uptodate() until after successful
> preparation
> KVM: guest_memfd: do not go through struct page
> KVM: rename CONFIG_HAVE_KVM_GMEM_* to CONFIG_HAVE_KVM_ARCH_GMEM_*
> KVM: guest_memfd: return locked folio from __kvm_gmem_get_pfn
> KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is
> passed to the guest
> KVM: guest_memfd: make kvm_gmem_prepare_folio() operate on a single
> struct kvm
> KVM: remove kvm_arch_gmem_prepare_needed()
> KVM: guest_memfd: move check for already-populated page to common code
> KVM: cleanup and add shortcuts to kvm_range_has_memory_attributes()
> KVM: extend kvm_range_has_memory_attributes() to check subset of
> attributes
> KVM: guest_memfd: let kvm_gmem_populate() operate only on private gfns
> KVM: guest_memfd: abstract how prepared folios are recorded
Re-tested series with multiple SNP guests with/without experimental THP
patches on top.
Also re-tested using the following SNP self tests (I believe Pratik has
a newer version he'll be submitting soon):
https://github.com/mdroth/linux/commits/snp-uptodate2-kst4/
x86_64/coco_pre_fault_memory_test
Series:
Tested-by: Michael Roth <michael.roth@amd.com>
>
> v1->v2:
> - patch 1: new
> - patch 7: point out benign race between page fault and hole-punch
> - patch 7: clean up comments
> - patch 10: clean up commit message
> - patch 11: remove ";;"
> - patch 13: fix length argument to kvm_range_has_memory_attributes()
> - patch 14: new
>
> Documentation/virt/kvm/api.rst | 6 +
> arch/x86/include/asm/kvm_host.h | 1 +
> arch/x86/kvm/Kconfig | 4 +-
> arch/x86/kvm/mmu/mmu.c | 5 +-
> arch/x86/kvm/svm/sev.c | 17 +--
> arch/x86/kvm/svm/svm.c | 1 +
> arch/x86/kvm/x86.c | 12 +-
> include/linux/kvm_host.h | 9 +-
> virt/kvm/Kconfig | 4 +-
> virt/kvm/guest_memfd.c | 226 +++++++++++++++++++-------------
> virt/kvm/kvm_main.c | 73 +++++------
> 11 files changed, 206 insertions(+), 152 deletions(-)
>
> --
> 2.43.0
>
>
^ permalink raw reply [flat|nested] 18+ messages in thread