Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Sean Christopherson <seanjc@google.com>]

Please send mails as plain text, otherwise they get ignored by the mailing lists
and don't get archived by lore.

On Thu, Oct 29, 2026, Jackie Yu wrote:
> Subject: Re: [PATCH v2 00/29] KVM: nSVM: Optimize nSVM TLB flushes
> 
> Hi Yosry,
> 
> Thank you for the quick response. Two clarifications from our side:
> 
> 1. About the boot failures: no emulation is involved in our
> experiment. Our out-of-tree changes enable direct guest
> INVLPGB/TLBSYNC execution by clearing the KVM intercepts, without any
> software emulation of the instruction. The failures occur with this
> unintercepted passthrough combined with static per-vCPU ASIDs, so they
> are not missing cases in an emulation path. Under the per-vCPU design
> we did not attempt to intercept and emulate INVLPGB at all.
> 
> 2. We do not use nested virtualization in our deployments. L1 guests
> in our production scenarios run without nested guests. Therefore, the
> ASID-replacement behavior you quoted from the APM does not affect our
> current use case: with one shared ASID per non-nested VM, an
> unintercepted guest INVLPGB would be scoped to that VM's ASID, which
> matches what our per-VM configuration relies on today.
> 
> Regarding the direction: we understand that with per-vCPU ASIDs,
> correct handling would require intercepting INVLPGB and re-issuing a
> flush for every ASID in use by the VM, which you note is unlikely to
> perform well on large VMs. We also note your point that as long as any
> vCPU uses nested, KVM must intercept INVLPGB anyway because the
> hardware replaces the requested ASID with the current ASID.
> 
> If we read the situation correctly, the practical options seem to be
> either (a) a per-VM shared ASID mode restricted to non-nested VMs,
> with unintercepted guest INVLPGB, plus mandatory interception whenever
> nested is in use, or (b) always intercepting and virtualizing INVLPGB
> under per-vCPU ASIDs. Either way, direct passthrough would at most
> cover the non-nested case. We would appreciate Sean's and Paolo's
> input on which direction upstream prefers; we are glad to provide more
> details of our configuration or run tests that would help decide
> between these options.
> 
> Thanks,
> 
> Jackie
> 
> 
> Yosry Ahmed <yosry@kernel.org> 于2026年10月29日周四 14:08写道：
> 
> > On Wed, Oct 28, 2026 at 1:15 PM Yosry Ahmed <yosry@kernel.org> wrote:
> > >
> > > On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@gmail.com> wrote:
> > > >
> > > > Hi, guys!
> > > >
> > > > Jackie Yu <ufo19890607@gmail.com> 于2026年10月28日周三 21:36写道：
> > > >>
> > > >> Hi Yosry and KVM folks,
> > > >>
> > > >> We would appreciate your guidance on supporting INVLPGB in ordinary
> > (non-SEV)
> > > >> L1 guests alongside the proposed static per-vCPU ASIDs.
> > > >>
> > > >> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
> > > >> "enabling INVLPGB virtualization" among its motivations [1]. The June
> > > >> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
> > > >> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
> > > >> while the remaining patches primarily address nested TLB flushes [3].
> > > >> Our question is how guest INVLPGB should be supported under the
> > > >> per-vCPU design.
> > > >>
> > > >> Some of our workloads frequently discard and repopulate memory using
> > > >> patterns involving madvise() and memset(). Broadcast IPI-based TLB
> > > >> shootdowns are expensive for these workloads, so we are interested in
> > > >> making AMD INVLPGB available to guests. We use a small benchmark to
> > > >> approximate the resulting shared-address-space pressure:
> > > >>
> > > >>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
> > > >>     memset(region, 0xab, 256 MiB);       // Initial population
> > > >>     start N workers, pinned across 64 allowed CPUs;
> > > >>     each worker repeatedly reads pages in its own slice of region;
> > > >>
> > > >>     repeat 500 warmup + 2,000 measured iterations:
> > > >>         start = clock_gettime(CLOCK_MONOTONIC);
> > > >>         madvise(region, 256 MiB, MADV_DONTNEED);
> > > >>         record elapsed madvise() time;
> > > >>         write one page in every 64 pages to repopulate part of region;
> > > >>
> > > >>     stop and join the workers;
> > > >>
> > > >> The benchmark uses memset() for initial population; subsequent
> > > >> iterations re-touch pages rather than repeatedly calling memset().
> > > >> The numbers below measure the advisor thread's madvise() syscall
> > > >> latency, not worker throughput.
> > > >>
> > > >> In our experimental KVM4 configuration with a VM-shared ASID and our
> > > >> separate guest INVLPGB passthrough changes, we observed:
> > > >>
> > > >>                          INVLPGB off     INVLPGB on    Difference
> > > >>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
> > > >>                 P99        32.760 ms       21.518 ms     -34.32%
> > > >>     128 workers mean       29.553 ms       20.459 ms     -30.77%
> > > >>                 P99        39.628 ms       28.160 ms     -28.94%
> > > >>
> > > >> These are single, sequential A/B runs for each worker count, with
> > > >> 500 warmup and 2,000 measured calls per run. The process had 64
> > > >> allowed CPUs, so the 128-worker case was oversubscribed. The
> > > >> measurements show a useful latency difference, but do not by
> > > >> themselves establish INVLPGB instruction counts or IPI reductions.
> > > >>
> > > >> We then experimentally combined the October per-vCPU ASID changes for
> > > >> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
> > > >> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
> > > >> otherwise comparable two-vCPU and 64-vCPU guests fail early during
> > > >> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
> > > >> to the same guest kernel allows the multi-vCPU guest to boot; the
> > > >> guest logs "Clearing CPUID bits: invlpgb".
> > >
> > > I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
> > > to intercept INVLPGB and basically issue an INVLPGB for each ASID used
> > > by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
> > > this would perform well on large VMs, not sure if virtualizing INVLPGB
> > > would still make sense.
> > >
> > > Regarding the boot failure, I would guess there are some missing cases
> > > due to the complexity of emulating INVLPGB without a shared per-VM
> > > ASID.
> > >
> > > >> These controlled boot comparisons identify a problem with combining
> > > >> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
> > > >> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
> > > >> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
> > > >> with INVLPGB enabled.
> > > >>
> > > >> In this configuration, an unintercepted guest INVLPGB targets the
> > > >> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
> > > >> ASIDs.
> > >
> > > It's neat that INVLPGB will automatically use the current ASID and not
> > > ASID=0 for guest usage.
> >
> > Hmm, actually, how is this supposed to work with nested
> > virtualization? The APM says:
> >
> >     A guest that executes a legal INVLPGB that is not intercepted will have
> >     the requested ASID field replaced by the current ASID and the valid
> >     ASID bit set before doing the broadcast invalidation.
> >
> > If any ASID passed by an L1 guest is replaced by its own ASID, then if
> > L1 is flushing L2's ASID (i.e. a non-zero ASID), passing through
> > INVLPGB would do the wrong thing and flush L1's ASID instead, right?
> >
> > Seems like if the VM is using nested (on any vCPU), KVM must intercept
> > INVLPGB anyway to correctly handle the ASID?
> >
> > The APM also says:
> >
> >     Two forms of INVLPGB intercepts, conditional and unconditional, are
> >     available to the hypervisor. The unconditional intercept traps all
> > guest
> >     usage of INVLPGB. The conditional intercept traps only
> >     illegally-specified INVLPGB instructions.
> >
> > When I read "conditional", I was really hoping it was conditional on
> > the ASID being zero or non-zero, as KVM really only needs to intercept
> > INVLPGB with non-zero ASIDs (assuming the ASID is shared to begin
> > with).
> >
> > Am I missing something?
> >