[PATCH] KVM: x86/mmu: Batch masked dirty clearing within leaf page tables

[PATCH] KVM: x86/mmu: Batch masked dirty clearing within leaf page tables

[Hao Zhang]

From: Hao Zhang <zhanghao1@kylinos.cn>

Walk the TDP page tables down to L2 when clearing a dirty-log bitmap word,
and update the selected 4KiB SPTEs directly in each child page table.

clear_dirty_pt_masked() currently uses the generic TDP iterator to visit
individual leaf mappings. For a densely populated bitmap word, this repeats
the iterator's traversal and entry checks for adjacent SPTEs in the same
page table. Sparse masks can also cause unselected entries to be visited.

Use the existing minimum-level iterator to locate each L1 page table
through its parent. The GFN range covered by a bitmap word can cross a
2MiB boundary. Clip a private copy of the mask to each table's coverage,
keeping bit indices relative to the original GFN.

Locate the first covered SPTE once per batch, then use relative offsets for
the selected bits. This avoids recomputing the SPTE index in the inner loop
and avoids reading and checking unselected 4KiB entries.

The write side of mmu_lock keeps the parent-child link stable, and the RCU
read-side critical section protects the page-table memory. Retain the
existing SPTE update helper and tracing semantics. Continue to skip huge
mappings, as before. Preserve the generic iterator and the caller's TLB
flush behavior.

For an 8-vCPU dirty-log benchmark with a 768MiB tracked region, reduce
KVM_CLEAR_DIRTY_LOG time by approximately 42-51% on the tested Intel host
with PML enabled.

Signed-off-by: Hao Zhang <zhanghao1@kylinos.cn>
---
Benchmark and testing details:

Base commit: 3857c2fe5449541e24afc5efdb0f81a8a8f9a3a0

Test setup:
  - Intel Xeon E5-2680 v4 @ 2.40GHz, two sockets, 14 cores per socket,
    two threads per core, 56 logical CPUs.
  - Approximately 31GiB host memory, all on NUMA node1; node0 has no memory.
  - GCC (Debian 12.2.0-14) 12.2.0, x86_64, CONFIG_PREEMPT=y.
  - kvm_intel.pml=Y, kvm.tdp_mmu=Y, kvm.eager_page_split=Y.
  - CPU governor=performance, THP=madvise, NUMA balancing=1.

Measured with a modified dirty_log_perf_test that allows selecting
MANUAL_DIRTY_LOG_PROTECT2 flags independently. The modified test is named
dirty_log_clear_probe in the commands below.

Representative commands, with the executable path relocated:

  CLEAR_MANUAL_CAPS=1 numactl --membind=1 \
    dirty_log_clear_probe \
    -v 8 -b 96M -c 3,5,7,9,11,13,15,17,27 -i 50 -w 100 -r 1

  CLEAR_MANUAL_CAPS=3 numactl --membind=1 \
    dirty_log_clear_probe \
    -v 8 -b 96M -c 3,5,7,9,11,13,15,17,27 -i 50 -w 100 -r 1

-b specifies per-vCPU tracked memory: 8 * 96MiB = 768MiB total. vCPU
threads are pinned to CPUs 3,5,...,17; the main thread is pinned to CPU27.
Memory is bound to node1. -w 100 makes all workload accesses writes,
and -r 1 fixes the random seed. Each vCPU accesses a disjoint range in
one anonymous-backed memslot, using 4KiB guest pages and sequential
page-stride accesses. This is not a full-region byte-write bandwidth test.
Adapt CPU/NUMA IDs on other machines. caps=1 enables MANUAL_PROTECT_ENABLE;
caps=3 also enables INITIALLY_SET.

Run each configuration seven times in fresh processes on each kernel.
The caps order alternates between repetitions (3,1 then 1,3). The recorded
runs interleaved these cases with other vCPU counts and region sizes.

The guest first populates the region. Each iteration waits for the vCPUs
to finish dirtying it, gets the dirty bitmap, and then clears it. CLEAR
timing uses CLOCK_MONOTONIC around memstress_clear_dirty_log() for the
single workload memslot. It includes helper/ioctl overhead, not only the
in-kernel SPTE loop. Populate, dirty-log enable, guest dirtying, GET, and
disable time are excluded. vCPUs wait for the next iteration during CLEAR;
this does not measure continuous-writer concurrency or guest pause latency.

Discard iteration 1 from each 50-iteration process. Take the median of the
remaining 49 CLEAR timings, then the median of the seven process medians.
Do not use the probe's printed overall average.

Results for 8 vCPUs and a 768MiB tracked region:

  INITIALLY_SET    Before (ms)    After (ms)    Reduction
  disabled          2.280414      1.128757        50.5%
  enabled           2.752557      1.607395        41.6%

Ranges of the seven process medians, in ms (not confidence intervals):
  disabled: before 2.277870..2.290569; after 1.125377..1.141876
  enabled:  before 2.750839..2.758421; after 1.601551..1.618283

CPU affinity, NUMA placement, kernel configuration and benchmark parameters
were held fixed. Timing runs did not enable perf or tracing; separate
profiles were excluded from these numbers. No new dmesg messages, kernel
taint, swap, or reclaim scanning were observed in the final measurement
window.

The original and final kernels were measured on separate boots, without
alternating between kernels. These results are CLEAR latency measurements,
not end-to-end migration throughput. Runtime coverage is limited to the
tested Intel/PML-enabled host; AMD and PML-disabled paths were not tested.

dirty_log_test -i 32 -I 10, nested_dirty_log_test, and 80 selective-clearing
cases covering same-table/cross-2MiB boundaries, sparse masks, bit63,
nonzero first_page and slot tails passed on the patched kernel.

 arch/x86/kvm/mmu/tdp_mmu.c | 71 +++++++++++++++++++++++++++++++++-------------
 1 file changed, 51 insertions(+), 20 deletions(-)

diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 44dad106fad1..e11b41c1890d 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1668,6 +1668,48 @@ void kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm,
 				      slot->base_gfn + slot->npages);
 }
 
+static void clear_dirty_leaf_pt_masked(const struct tdp_iter *parent,
+				       gfn_t gfn, unsigned long mask, u64 dbit)
+{
+	tdp_ptep_t pt, sptep;
+	unsigned int first, last;
+	unsigned long offset;
+	u64 old_spte;
+
+	/* Keep the mask in the original bitmap word coordinates. */
+	first = max(gfn, parent->gfn) - gfn;
+	last = min(gfn + BITS_PER_LONG,
+		   parent->gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M)) - gfn - 1;
+	mask &= GENMASK(last, first);
+	if (!mask)
+		return;
+
+	pt = spte_to_child_pt(parent->old_spte, parent->level);
+	/* Anchor at the first covered bit, including for cross-table masks. */
+	pt += SPTE_INDEX(((gfn + first) | parent->gfn_bits) << PAGE_SHIFT,
+			 PG_LEVEL_4K);
+	while (mask) {
+		offset = __ffs(mask);
+		mask &= mask - 1;
+		sptep = pt + (offset - first);
+		old_spte = kvm_tdp_mmu_read_spte(sptep);
+		if (!is_shadow_present_pte(old_spte))
+			continue;
+
+		KVM_MMU_WARN_ON(dbit == shadow_dirty_mask &&
+				spte_ad_need_write_protect(old_spte));
+
+		if (!(old_spte & dbit))
+			continue;
+
+		old_spte = tdp_mmu_clear_spte_bits(sptep, old_spte, dbit,
+						   PG_LEVEL_4K);
+		trace_kvm_tdp_mmu_spte_changed(parent->as_id, gfn + offset,
+					       PG_LEVEL_4K, old_spte,
+					       old_spte & ~dbit);
+	}
+}
+
 static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *root,
 				  gfn_t gfn, unsigned long mask, bool wrprot)
 {
@@ -1679,30 +1721,19 @@ static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *root,
 
 	rcu_read_lock();
 
-	tdp_root_for_each_leaf_pte(iter, kvm, root, gfn + __ffs(mask),
-				    gfn + BITS_PER_LONG) {
-		if (!mask)
-			break;
-
-		KVM_MMU_WARN_ON(dbit == shadow_dirty_mask &&
-				spte_ad_need_write_protect(iter.old_spte));
-
-		if (iter.level > PG_LEVEL_4K ||
-		    !(mask & (1UL << (iter.gfn - gfn))))
+	for_each_tdp_pte_min_level(iter, kvm, root, PG_LEVEL_2M,
+				   gfn + __ffs(mask), gfn + __fls(mask) + 1) {
+		if (!is_shadow_present_pte(iter.old_spte))
 			continue;
 
-		mask &= ~(1UL << (iter.gfn - gfn));
-
-		if (!(iter.old_spte & dbit))
+		if (is_last_spte(iter.old_spte, iter.level)) {
+			KVM_MMU_WARN_ON(dbit == shadow_dirty_mask &&
+					spte_ad_need_write_protect(iter.old_spte));
 			continue;
+		}
 
-		iter.old_spte = tdp_mmu_clear_spte_bits(iter.sptep,
-							iter.old_spte, dbit,
-							iter.level);
-
-		trace_kvm_tdp_mmu_spte_changed(iter.as_id, iter.gfn, iter.level,
-					       iter.old_spte,
-					       iter.old_spte & ~dbit);
+		if (iter.level == PG_LEVEL_2M)
+			clear_dirty_leaf_pt_masked(&iter, gfn, mask, dbit);
 	}
 
 	rcu_read_unlock();

base-commit: 3857c2fe5449541e24afc5efdb0f81a8a8f9a3a0
-- 
2.15.0