* [PATCH v1 0/5] KVM: riscv: Age G-stage PTEs locklessly
@ 2026-09-21 11:13 SeungJu Cheon
2026-09-21 11:13 ` [PATCH v1 1/5] KVM: riscv: Age all G-stage PTEs in a GFN range SeungJu Cheon
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: SeungJu Cheon @ 2026-09-21 11:13 UTC (permalink / raw)
To: Anup Patel
Cc: Atish Patra, Paul Walmsley, Palmer Dabbelt, Albert Ou,
Alexandre Ghiti, Paolo Bonzini, Andrew Jones, Jinyu Tang,
Wang Yechao, kvm-riscv, kvm, linux-riscv, linux-kernel,
SeungJu Cheon
Aging G-stage PTEs currently runs with mmu_lock held for write, taken
by the common MMU notifier code. When MGLRU or kswapd ages a large
range, every vCPU taking a G-stage fault blocks on the lock, and the
dirty-logging read-side fast path blocks behind aging as well.
This series lets RISC-V select KVM_MMU_LOCKLESS_AGING, so G-stage
aging can run without mmu_lock while protecting page-table walks with
RCU.
1. Age all G-stage PTEs in a GFN range
Bug fix, posted separately [1] and included unchanged so the
series applies on riscv_kvm_queue. Drop it if already applied.
2. Read G-stage PTEs once when walking page tables
A lockless walker must derive presence, leaf-ness and the child
address from a single snapshot of each entry.
3. Write-protect G-stage PTEs atomically
Aging will clear the Accessed bit without mmu_lock; the plain
read-modify-write in GSTAGE_OP_WP could clobber that update.
4. Free G-stage page tables after an RCU grace period
Table pages and the root outlive any RCU reader. Root teardown
keeps pgd_levels intact so a walker holding the old root sees
consistent metadata.
5. Age G-stage PTEs locklessly
Select KVM_MMU_LOCKLESS_AGING and walk under rcu_read_lock().
This builds on Jinyu Tang's rwlock and cmpxchg helpers already in
riscv_kvm_queue. Wang Yechao's huge page recovery series [2] adds a
second page-table free site in kvm_riscv_gstage_recover_huge();
whichever lands second will need to switch that put_page() to
gstage_free_page_table(). I can rebase the series as needed.
A possible follow-up is to run clear-dirty-log and the general fault
path under the read side of mmu_lock, as arm64 and x86 do. Patch 3
provides the atomic write-protection needed for the former.
Testing
-------
QEMU virt, -cpu rv64,h=true (Svadu), 4 vCPUs, TCG. The host kernel
was built with KASAN, PROVE_LOCKING, PROVE_RCU, LOCK_STAT and LRU_GEN.
After each patch:
- dirty_log_perf_test -v 2 -b 256M -i 3
- access_tracking_perf_test -v 2 -b 256M -s anonymous_thp
(2M THP disabled and 64K mTHP enabled)
- 20 VM create/destroy iterations
With MGLRU aging forced on the VM's cgroup every 50-100ms:
- dirty_log_perf_test -v 3 -b 256M -i 3
- 30 VM create/destroy iterations
With CONFIG_KVM=m, 5 module load/unload cycles, each with a VM created
and destroyed immediately before module unload, to exercise module
teardown after queuing RCU frees.
No WARN, KASAN, lockdep or RCU reports were observed. The kvm_age_hva
tracepoint recorded approximately 65k events per aging pass.
Performance results from dirty_log_perf_test -v 3 -b 256M -i 3, with
lock_stat enabled for mmu_lock and 86 aging passes:
no aging aging
before after before after
guest dirty time 12.5s 12.4s 18.6s 16.7s
mmu_lock W wait total 0.15s 0.17s 44.4s 0.15s
mmu_lock W contentions 35k 35k 1.0M 30k
mmu_lock R wait total 0 0 7.5s 0
Before the series, 93% of write-lock waiters were from
kvm_mmu_notifier_clear_young(), and the dirty-log read-side fast path
waited for mmu_lock 352k times. After the series, aging does not
acquire mmu_lock.
Absolute numbers are inflated by TCG; only the relative change is
meaningful. Under TCG, the atomic write-protect change in patch 3
increased the clear-dirty-log time from approximately 16ms to 30ms,
as TCG serialises AMO instructions; hardware numbers would be welcome.
[1] https://lore.kernel.org/all/20260918072622.284188-1-suunj1331@gmail.com/
[2] https://lore.kernel.org/kvm-riscv/20260914055936.3672758-1-wang.yechao255@zte.com.cn/
SeungJu Cheon (5):
KVM: riscv: Age all G-stage PTEs in a GFN range
KVM: riscv: Read G-stage PTEs once when walking page tables
KVM: riscv: Write-protect G-stage PTEs atomically
KVM: riscv: Free G-stage page tables after an RCU grace period
KVM: riscv: Age G-stage PTEs locklessly
arch/riscv/include/asm/kvm_gstage.h | 4 +-
arch/riscv/kvm/Kconfig | 1 +
arch/riscv/kvm/gstage.c | 111 +++++++++++++++++++++-------
arch/riscv/kvm/main.c | 3 +
arch/riscv/kvm/mmu.c | 56 +++++++-------
5 files changed, 117 insertions(+), 58 deletions(-)
base-commit: 41e81f7e3ef96594fb840445343c0ee7723aa550
--
2.52.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v1 1/5] KVM: riscv: Age all G-stage PTEs in a GFN range
2026-09-21 11:13 [PATCH v1 0/5] KVM: riscv: Age G-stage PTEs locklessly SeungJu Cheon
@ 2026-09-21 11:13 ` SeungJu Cheon
2026-09-21 11:13 ` [PATCH v1 2/5] KVM: riscv: Read G-stage PTEs once when walking page tables SeungJu Cheon
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: SeungJu Cheon @ 2026-09-21 11:13 UTC (permalink / raw)
To: Anup Patel
Cc: Atish Patra, Paul Walmsley, Palmer Dabbelt, Albert Ou,
Alexandre Ghiti, Paolo Bonzini, Andrew Jones, Jinyu Tang,
Wang Yechao, kvm-riscv, kvm, linux-riscv, linux-kernel,
SeungJu Cheon
kvm_age_gfn() and kvm_test_age_gfn() only operate on the G-stage
leaf covering range->start. They also warn unless the range size is
exactly 4K, 2M, or 1G.
The MM can pass larger ranges when batching the PTEs of a large folio.
For example, a 64K mTHP range triggers the warning and only the first
4K leaf is aged, leaving the Accessed bits of the remaining leaves
unchanged.
This can be reproduced on an rv64 host with Svadu by disabling 2M THP,
enabling 64K mTHP, and running:
access_tracking_perf_test -s anonymous_thp
This results in:
WARNING: arch/riscv/kvm/mmu.c:362 at kvm_age_gfn+0x210/0x370
...
[<ffffffff800c8fc8>] kvm_age_gfn+0x210/0x370
[<ffffffff80095c90>] kvm_mmu_notifier_clear_young+0x250/0x478
[<ffffffff80ae532e>] __mmu_notifier_clear_young+0xde/0x1a0
[<ffffffff80932f76>] walk_pud_range+0x86e/0x1688
[<ffffffff80a18720>] walk_pgd_range+0x518/0x17b0
[<ffffffff80a19aa2>] __walk_page_range+0xea/0x580
[<ffffffff80a1a422>] walk_page_range+0x3a/0x80
[<ffffffff80931dc0>] try_to_inc_max_seq+0x520/0xe68
[<ffffffff80939dc0>] lru_gen_seq_write+0xd08/0x1310
Walk all G-stage leaves in the requested range. Advance by the leaf
size when a mapping is found, or by the size of the non-present region
at the level where the walk stopped. When testing the age, return as
soon as a young leaf is found.
Remove the range-size warning since arbitrary GFN ranges are valid.
Fixes: 9955371cc014 ("RISC-V: KVM: Implement MMU notifiers")
Signed-off-by: SeungJu Cheon <suunj1331@gmail.com>
---
arch/riscv/include/asm/kvm_gstage.h | 2 ++
arch/riscv/kvm/gstage.c | 34 +++++++++++++++++++++++++++++
arch/riscv/kvm/mmu.c | 22 ++++---------------
3 files changed, 40 insertions(+), 18 deletions(-)
diff --git a/arch/riscv/include/asm/kvm_gstage.h b/arch/riscv/include/asm/kvm_gstage.h
index aaf080ba1b77..a03db1a10095 100644
--- a/arch/riscv/include/asm/kvm_gstage.h
+++ b/arch/riscv/include/asm/kvm_gstage.h
@@ -81,6 +81,8 @@ bool kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage,
gpa_t start, gpa_t size, bool may_block);
bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end);
+bool kvm_riscv_gstage_age_range(struct kvm_gstage *gstage, gpa_t start,
+ gpa_t end, bool test_only);
bool kvm_riscv_gstage_wp_pt_masked(struct kvm_gstage *gstage, gfn_t base_gfn,
unsigned long mask);
diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c
index e5002cb9cbef..f61ba434093c 100644
--- a/arch/riscv/kvm/gstage.c
+++ b/arch/riscv/kvm/gstage.c
@@ -485,6 +485,40 @@ bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end
return flush;
}
+bool kvm_riscv_gstage_age_range(struct kvm_gstage *gstage, gpa_t start,
+ gpa_t end, bool test_only)
+{
+ unsigned long page_size;
+ bool young = false;
+ gpa_t addr = start;
+ pte_t *ptep;
+ u32 level;
+ bool found;
+
+ while (addr < end) {
+ found = kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &level);
+ if (gstage_level_to_page_size(gstage, level, &page_size))
+ break;
+
+ if (found) {
+ if (test_only) {
+ if (pte_young(ptep_get(ptep)))
+ return true;
+ } else {
+ young |= ptep_test_and_clear_young(NULL, 0, ptep);
+ }
+ }
+
+ /*
+ * Advance past this leaf, or past the non-present region at
+ * the level where the walk stopped.
+ */
+ addr = ALIGN_DOWN(addr, page_size) + page_size;
+ }
+
+ return young;
+}
+
static inline void clear_huge_mask(unsigned long *mask, unsigned long page_size,
gfn_t base_gfn, gpa_t addr)
{
diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
index 3e955d808743..342f606399e5 100644
--- a/arch/riscv/kvm/mmu.c
+++ b/arch/riscv/kvm/mmu.c
@@ -351,42 +351,28 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
- pte_t *ptep;
- u32 ptep_level = 0;
- u64 size = (range->end - range->start) << PAGE_SHIFT;
struct kvm_gstage gstage;
if (!kvm->arch.pgd)
return false;
- WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
-
kvm_riscv_gstage_init(&gstage, kvm);
- if (!kvm_riscv_gstage_get_leaf(&gstage, range->start << PAGE_SHIFT,
- &ptep, &ptep_level))
- return false;
- return ptep_test_and_clear_young(NULL, 0, ptep);
+ return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT,
+ range->end << PAGE_SHIFT, false);
}
bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
- pte_t *ptep;
- u32 ptep_level = 0;
- u64 size = (range->end - range->start) << PAGE_SHIFT;
struct kvm_gstage gstage;
if (!kvm->arch.pgd)
return false;
- WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
-
kvm_riscv_gstage_init(&gstage, kvm);
- if (!kvm_riscv_gstage_get_leaf(&gstage, range->start << PAGE_SHIFT,
- &ptep, &ptep_level))
- return false;
- return pte_young(ptep_get(ptep));
+ return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT,
+ range->end << PAGE_SHIFT, true);
}
static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
--
2.52.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v1 2/5] KVM: riscv: Read G-stage PTEs once when walking page tables
2026-09-21 11:13 [PATCH v1 0/5] KVM: riscv: Age G-stage PTEs locklessly SeungJu Cheon
2026-09-21 11:13 ` [PATCH v1 1/5] KVM: riscv: Age all G-stage PTEs in a GFN range SeungJu Cheon
@ 2026-09-21 11:13 ` SeungJu Cheon
2026-09-21 11:14 ` [PATCH v1 3/5] KVM: riscv: Write-protect G-stage PTEs atomically SeungJu Cheon
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: SeungJu Cheon @ 2026-09-21 11:13 UTC (permalink / raw)
To: Anup Patel
Cc: Atish Patra, Paul Walmsley, Palmer Dabbelt, Albert Ou,
Alexandre Ghiti, Paolo Bonzini, Andrew Jones, Jinyu Tang,
Wang Yechao, kvm-riscv, kvm, linux-riscv, linux-kernel,
SeungJu Cheon
gstage_pte_leaf() dereferences the PTE pointer, causing walkers to read
the same entry multiple times. This is safe under mmu_lock, but a
lockless walker could observe different values when deciding whether an
entry is a leaf and which child table to follow.
Make gstage_pte_leaf() take a PTE value and use a single snapshot for
presence, leaf and child-table checks. Also remove a redundant leaf
check in kvm_riscv_gstage_set_pte(), where mmu_lock prevents the entry
from changing.
Signed-off-by: SeungJu Cheon <suunj1331@gmail.com>
---
arch/riscv/kvm/gstage.c | 48 +++++++++++++++++++++++------------------
1 file changed, 27 insertions(+), 21 deletions(-)
diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c
index f61ba434093c..944fa4c95aea 100644
--- a/arch/riscv/kvm/gstage.c
+++ b/arch/riscv/kvm/gstage.c
@@ -19,8 +19,8 @@ unsigned long kvm_riscv_gstage_max_pgd_levels __ro_after_init = 3;
unsigned long kvm_riscv_gstage_max_pgd_levels __ro_after_init = 2;
#endif
-#define gstage_pte_leaf(__ptep) \
- (pte_val(*(__ptep)) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC))
+#define gstage_pte_leaf(__pte) \
+ (pte_val(__pte) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC))
static inline unsigned long gstage_pte_index(struct kvm_gstage *gstage,
gpa_t addr, u32 level)
@@ -84,14 +84,18 @@ static int gstage_level_to_page_size(struct kvm_gstage *gstage, u32 level,
bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr,
pte_t **ptepp, u32 *ptep_level)
{
- pte_t *ptep;
+ pte_t *ptep, pte;
u32 current_level = gstage->pgd_levels - 1;
*ptep_level = current_level;
ptep = (pte_t *)gstage->pgd;
ptep = &ptep[gstage_pte_index(gstage, addr, current_level)];
- while (ptep && pte_val(ptep_get(ptep))) {
- if (gstage_pte_leaf(ptep)) {
+ while (ptep) {
+ pte = ptep_get(ptep);
+ if (!pte_val(pte))
+ break;
+
+ if (gstage_pte_leaf(pte)) {
*ptep_level = current_level;
*ptepp = ptep;
return true;
@@ -100,7 +104,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr,
if (current_level) {
current_level--;
*ptep_level = current_level;
- ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ ptep = (pte_t *)gstage_pte_page_vaddr(pte);
ptep = &ptep[gstage_pte_index(gstage, addr, current_level)];
} else {
ptep = NULL;
@@ -151,10 +155,12 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
return -EINVAL;
while (current_level != map->level) {
- if (gstage_pte_leaf(ptep))
+ pte_t pte = ptep_get(ptep);
+
+ if (gstage_pte_leaf(pte))
return -EEXIST;
- if (!pte_val(ptep_get(ptep))) {
+ if (!pte_val(pte)) {
if (!pcache)
return -ENOMEM;
next_ptep = kvm_mmu_memory_cache_alloc(pcache);
@@ -163,9 +169,7 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)),
__pgprot(_PAGE_TABLE)));
} else {
- if (gstage_pte_leaf(ptep))
- return -EEXIST;
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(pte);
}
current_level--;
@@ -175,7 +179,7 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
if (pte_val(*ptep) != pte_val(map->pte)) {
bool was_invalid = !pte_val(*ptep);
set_pte(ptep, map->pte);
- if (gstage_pte_leaf(ptep) &&
+ if (gstage_pte_leaf(map->pte) &&
!(was_invalid && riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)))
gstage_tlb_flush(gstage, current_level, map->addr);
}
@@ -316,7 +320,7 @@ bool kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage,
unsigned long huge_pte, child_pte;
unsigned long child_page_size;
bool need_flush = false;
- pte_t *ptep;
+ pte_t *ptep, pte;
int i, ret;
if (!pcache)
@@ -325,16 +329,17 @@ bool kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage,
while(current_level > target_level) {
ptep = (pte_t *)&next_ptep[gstage_pte_index(gstage, addr, current_level)];
- if (!pte_val(ptep_get(ptep)))
+ pte = ptep_get(ptep);
+ if (!pte_val(pte))
break;
- if (!gstage_pte_leaf(ptep)) {
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ if (!gstage_pte_leaf(pte)) {
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(pte);
current_level--;
continue;
}
- huge_pte = pte_val(ptep_get(ptep));
+ huge_pte = pte_val(pte);
ret = gstage_level_to_page_size(gstage, current_level - 1, &child_page_size);
if (ret)
@@ -373,7 +378,7 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op)
{
int i, ret;
- pte_t old_pte, *next_ptep;
+ pte_t old_pte, pte, *next_ptep;
u32 next_ptep_level;
unsigned long next_page_size, page_size;
bool flush = false;
@@ -384,11 +389,12 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
WARN_ON(addr & (page_size - 1));
- if (!pte_val(ptep_get(ptep)))
+ pte = ptep_get(ptep);
+ if (!pte_val(pte))
return false;
- if (ptep_level && !gstage_pte_leaf(ptep)) {
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ if (ptep_level && !gstage_pte_leaf(pte)) {
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(pte);
next_ptep_level = ptep_level - 1;
ret = gstage_level_to_page_size(gstage, next_ptep_level, &next_page_size);
if (ret)
--
2.52.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v1 3/5] KVM: riscv: Write-protect G-stage PTEs atomically
2026-09-21 11:13 [PATCH v1 0/5] KVM: riscv: Age G-stage PTEs locklessly SeungJu Cheon
2026-09-21 11:13 ` [PATCH v1 1/5] KVM: riscv: Age all G-stage PTEs in a GFN range SeungJu Cheon
2026-09-21 11:13 ` [PATCH v1 2/5] KVM: riscv: Read G-stage PTEs once when walking page tables SeungJu Cheon
@ 2026-09-21 11:14 ` SeungJu Cheon
2026-09-21 11:14 ` [PATCH v1 4/5] KVM: riscv: Free G-stage page tables after an RCU grace period SeungJu Cheon
2026-09-21 11:14 ` [PATCH v1 5/5] KVM: riscv: Age G-stage PTEs locklessly SeungJu Cheon
4 siblings, 0 replies; 6+ messages in thread
From: SeungJu Cheon @ 2026-09-21 11:14 UTC (permalink / raw)
To: Anup Patel
Cc: Atish Patra, Paul Walmsley, Palmer Dabbelt, Albert Ou,
Alexandre Ghiti, Paolo Bonzini, Andrew Jones, Jinyu Tang,
Wang Yechao, kvm-riscv, kvm, linux-riscv, linux-kernel,
SeungJu Cheon
GSTAGE_OP_WP clears the W bit with a read-modify-write of the entire
PTE. Lockless aging can concurrently clear the Accessed bit, allowing
write-protection to overwrite the update with a stale PTE value.
Clear W atomically with test_and_clear_bit() so that concurrent
Accessed-bit updates are preserved. Its return value also preserves
the existing behavior of requesting a TLB flush only when W was
actually cleared.
Signed-off-by: SeungJu Cheon <suunj1331@gmail.com>
---
arch/riscv/kvm/gstage.c | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c
index 944fa4c95aea..f7e4756ce15b 100644
--- a/arch/riscv/kvm/gstage.c
+++ b/arch/riscv/kvm/gstage.c
@@ -378,7 +378,7 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op)
{
int i, ret;
- pte_t old_pte, pte, *next_ptep;
+ pte_t pte, *next_ptep;
u32 next_ptep_level;
unsigned long next_page_size, page_size;
bool flush = false;
@@ -408,13 +408,16 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
if (op == GSTAGE_OP_CLEAR)
put_page(virt_to_page(next_ptep));
} else {
- old_pte = *ptep;
- if (op == GSTAGE_OP_CLEAR)
+ if (op == GSTAGE_OP_CLEAR) {
set_pte(ptep, __pte(0));
- else if (op == GSTAGE_OP_WP)
- set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE));
- if (pte_val(*ptep) != pte_val(old_pte))
flush = true;
+ } else if (op == GSTAGE_OP_WP) {
+ /*
+ * Clear W atomically to avoid clobbering a concurrent
+ * Accessed-bit update by lockless aging.
+ */
+ flush = test_and_clear_bit(__ffs(_PAGE_WRITE), &ptep->pte);
+ }
}
return flush;
--
2.52.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v1 4/5] KVM: riscv: Free G-stage page tables after an RCU grace period
2026-09-21 11:13 [PATCH v1 0/5] KVM: riscv: Age G-stage PTEs locklessly SeungJu Cheon
` (2 preceding siblings ...)
2026-09-21 11:14 ` [PATCH v1 3/5] KVM: riscv: Write-protect G-stage PTEs atomically SeungJu Cheon
@ 2026-09-21 11:14 ` SeungJu Cheon
2026-09-21 11:14 ` [PATCH v1 5/5] KVM: riscv: Age G-stage PTEs locklessly SeungJu Cheon
4 siblings, 0 replies; 6+ messages in thread
From: SeungJu Cheon @ 2026-09-21 11:14 UTC (permalink / raw)
To: Anup Patel
Cc: Atish Patra, Paul Walmsley, Palmer Dabbelt, Albert Ou,
Alexandre Ghiti, Paolo Bonzini, Andrew Jones, Jinyu Tang,
Wang Yechao, kvm-riscv, kvm, linux-riscv, linux-kernel,
SeungJu Cheon
G-stage page tables are currently freed immediately after being
unlinked under mmu_lock. This is safe while all walkers hold mmu_lock,
as a table cannot be freed while a walker is using it. A subsequent
change will allow aging walks without mmu_lock, where a walker may
obtain a child table pointer just before a concurrent unmap unlinks
and frees the table.
Defer freeing unlinked page-table pages with call_rcu(). The parent
entry is cleared before the child table is retired, so new walkers
cannot acquire it while existing RCU-protected walkers can safely
finish using it.
Apply the same lifetime rule to the root PGD. Unpublish the root with
WRITE_ONCE() and defer its free with call_rcu(). Keep pgd_levels
unchanged so a walker that observed the old root continues to use
matching page-table metadata.
Wait for pending G-stage page-table callbacks with rcu_barrier() when
the RISC-V KVM module exits so that they complete before the module is
unloaded.
Signed-off-by: SeungJu Cheon <suunj1331@gmail.com>
---
arch/riscv/include/asm/kvm_gstage.h | 2 +-
arch/riscv/kvm/gstage.c | 16 +++++++++++++++-
arch/riscv/kvm/main.c | 3 +++
arch/riscv/kvm/mmu.c | 16 +++++++++++++---
4 files changed, 32 insertions(+), 5 deletions(-)
diff --git a/arch/riscv/include/asm/kvm_gstage.h b/arch/riscv/include/asm/kvm_gstage.h
index a03db1a10095..caeed6de6dbe 100644
--- a/arch/riscv/include/asm/kvm_gstage.h
+++ b/arch/riscv/include/asm/kvm_gstage.h
@@ -111,7 +111,7 @@ static inline void kvm_riscv_gstage_init(struct kvm_gstage *gstage, struct kvm *
gstage->kvm = kvm;
gstage->flags = 0;
gstage->vmid = READ_ONCE(kvm->arch.vmid.vmid);
- gstage->pgd = kvm->arch.pgd;
+ gstage->pgd = READ_ONCE(kvm->arch.pgd);
gstage->pgd_levels = kvm->arch.pgd_levels;
}
diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c
index f7e4756ce15b..fc39d188b20a 100644
--- a/arch/riscv/kvm/gstage.c
+++ b/arch/riscv/kvm/gstage.c
@@ -22,6 +22,20 @@ unsigned long kvm_riscv_gstage_max_pgd_levels __ro_after_init = 2;
#define gstage_pte_leaf(__pte) \
(pte_val(__pte) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC))
+static void gstage_free_page_table_rcu(struct rcu_head *head)
+{
+ put_page(container_of(head, struct page, rcu_head));
+}
+
+/*
+ * Defer freeing an unlinked page table until lockless walkers
+ * that may have observed it have exited.
+ */
+static void gstage_free_page_table(pte_t *table)
+{
+ call_rcu(&virt_to_page(table)->rcu_head, gstage_free_page_table_rcu);
+}
+
static inline unsigned long gstage_pte_index(struct kvm_gstage *gstage,
gpa_t addr, u32 level)
{
@@ -406,7 +420,7 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
flush |= kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size,
&next_ptep[i], next_ptep_level, op);
if (op == GSTAGE_OP_CLEAR)
- put_page(virt_to_page(next_ptep));
+ gstage_free_page_table(next_ptep);
} else {
if (op == GSTAGE_OP_CLEAR) {
set_pte(ptep, __pte(0));
diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c
index 89568ccce01d..5fcf425f7150 100644
--- a/arch/riscv/kvm/main.c
+++ b/arch/riscv/kvm/main.c
@@ -262,6 +262,9 @@ static void __exit riscv_kvm_exit(void)
{
kvm_exit();
+ /* Wait for pending G-stage page-table RCU callbacks. */
+ rcu_barrier();
+
/* Unregister CPU PM notifier */
if (IS_ENABLED(CONFIG_CPU_PM))
cpu_pm_unregister_notifier(&kvm_riscv_cpu_pm_nb);
diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
index 342f606399e5..8aed69abf814 100644
--- a/arch/riscv/kvm/mmu.c
+++ b/arch/riscv/kvm/mmu.c
@@ -761,6 +761,13 @@ int kvm_riscv_mmu_alloc_pgd(struct kvm *kvm)
return 0;
}
+static void kvm_riscv_mmu_free_pgd_rcu(struct rcu_head *head)
+{
+ struct page *page = container_of(head, struct page, rcu_head);
+
+ __free_pages(page, get_order(kvm_riscv_gstage_pgd_size));
+}
+
void kvm_riscv_mmu_free_pgd(struct kvm *kvm)
{
struct kvm_gstage gstage;
@@ -773,9 +780,12 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm)
flush = kvm_riscv_gstage_unmap_range(&gstage, 0UL,
kvm_riscv_gstage_gpa_size(kvm->arch.pgd_levels), false);
pgd = READ_ONCE(kvm->arch.pgd);
- kvm->arch.pgd = NULL;
+ /*
+ * Keep pgd_levels unchanged for lockless walkers that already
+ * observed the old root.
+ */
+ WRITE_ONCE(kvm->arch.pgd, NULL);
kvm->arch.pgd_phys = 0;
- kvm->arch.pgd_levels = 0;
}
write_unlock(&kvm->mmu_lock);
@@ -783,7 +793,7 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm)
kvm_flush_remote_tlbs(kvm);
if (pgd)
- free_pages((unsigned long)pgd, get_order(kvm_riscv_gstage_pgd_size));
+ call_rcu(&virt_to_page(pgd)->rcu_head, kvm_riscv_mmu_free_pgd_rcu);
kvm_mmu_free_memory_cache(&kvm->arch.pgd_split_page_cache);
}
--
2.52.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v1 5/5] KVM: riscv: Age G-stage PTEs locklessly
2026-09-21 11:13 [PATCH v1 0/5] KVM: riscv: Age G-stage PTEs locklessly SeungJu Cheon
` (3 preceding siblings ...)
2026-09-21 11:14 ` [PATCH v1 4/5] KVM: riscv: Free G-stage page tables after an RCU grace period SeungJu Cheon
@ 2026-09-21 11:14 ` SeungJu Cheon
4 siblings, 0 replies; 6+ messages in thread
From: SeungJu Cheon @ 2026-09-21 11:14 UTC (permalink / raw)
To: Anup Patel
Cc: Atish Patra, Paul Walmsley, Palmer Dabbelt, Albert Ou,
Alexandre Ghiti, Paolo Bonzini, Andrew Jones, Jinyu Tang,
Wang Yechao, kvm-riscv, kvm, linux-riscv, linux-kernel,
SeungJu Cheon
Aging G-stage PTEs currently runs with mmu_lock held for write, taken
by the common MMU notifier code. When MGLRU or kswapd ages a large
range, every vCPU taking a G-stage fault blocks on the lock, and the
dirty-logging read-side fast path blocks behind aging as well.
The preceding patches prepare G-stage page-table walks for lockless
aging by using consistent PTE snapshots, preserving concurrent
Accessed-bit updates, and deferring page-table frees with RCU.
Select KVM_MMU_LOCKLESS_AGING so the common code no longer takes
mmu_lock for aging, and protect the G-stage walk with an RCU read-side
critical section.
Read the root inside the RCU read-side critical section and check the
resulting snapshot instead of checking kvm->arch.pgd separately before
initializing the G-stage context. This ensures that the root used by
the walk remains protected until the walk completes.
On QEMU TCG with 4 vCPUs, running dirty_log_perf_test -v 3 -b 256M -i 3
with MGLRU aging of the VM's cgroup forced every 100ms (86 passes):
before after
mmu_lock write wait, total 44.4 s 0.15 s
mmu_lock write contentions 1,002,092 30,092
mmu_lock read contentions 351,937 0
guest dirty-memory time 18.6 s 16.7 s
Before, 93% of write-lock waiters were kvm_mmu_notifier_clear_young().
With no aging, no meaningful difference was observed between the two
kernels.
Signed-off-by: SeungJu Cheon <suunj1331@gmail.com>
---
arch/riscv/kvm/Kconfig | 1 +
arch/riscv/kvm/mmu.c | 26 +++++++++++++-------------
2 files changed, 14 insertions(+), 13 deletions(-)
diff --git a/arch/riscv/kvm/Kconfig b/arch/riscv/kvm/Kconfig
index ec2cee0a39e0..77898d58ff9a 100644
--- a/arch/riscv/kvm/Kconfig
+++ b/arch/riscv/kvm/Kconfig
@@ -32,6 +32,7 @@ config KVM
select VIRT_XFER_TO_GUEST_WORK
select SCHED_INFO
select GUEST_PERF_EVENTS if PERF_EVENTS
+ select KVM_MMU_LOCKLESS_AGING
help
Support hosting virtualized guest machines.
diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
index 8aed69abf814..ff282bdbe492 100644
--- a/arch/riscv/kvm/mmu.c
+++ b/arch/riscv/kvm/mmu.c
@@ -349,30 +349,30 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
return false;
}
-bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+static bool kvm_riscv_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range,
+ bool test_only)
{
struct kvm_gstage gstage;
- if (!kvm->arch.pgd)
- return false;
+ guard(rcu)();
+ lockdep_assert_not_held(&kvm->mmu_lock);
kvm_riscv_gstage_init(&gstage, kvm);
+ if (!gstage.pgd)
+ return false;
return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT,
- range->end << PAGE_SHIFT, false);
+ range->end << PAGE_SHIFT, test_only);
}
-bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
- struct kvm_gstage gstage;
-
- if (!kvm->arch.pgd)
- return false;
-
- kvm_riscv_gstage_init(&gstage, kvm);
+ return kvm_riscv_age_gfn(kvm, range, false);
+}
- return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT,
- range->end << PAGE_SHIFT, true);
+bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+ return kvm_riscv_age_gfn(kvm, range, true);
}
static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
--
2.52.0
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-09-21 11:14 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-09-21 11:13 [PATCH v1 0/5] KVM: riscv: Age G-stage PTEs locklessly SeungJu Cheon
2026-09-21 11:13 ` [PATCH v1 1/5] KVM: riscv: Age all G-stage PTEs in a GFN range SeungJu Cheon
2026-09-21 11:13 ` [PATCH v1 2/5] KVM: riscv: Read G-stage PTEs once when walking page tables SeungJu Cheon
2026-09-21 11:14 ` [PATCH v1 3/5] KVM: riscv: Write-protect G-stage PTEs atomically SeungJu Cheon
2026-09-21 11:14 ` [PATCH v1 4/5] KVM: riscv: Free G-stage page tables after an RCU grace period SeungJu Cheon
2026-09-21 11:14 ` [PATCH v1 5/5] KVM: riscv: Age G-stage PTEs locklessly SeungJu Cheon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®