Re: [RFC PATCH 5/5] KVM: arm64: Enable HAFDBS for guests not on migration

[Leonardo Bras <leo.bras@arm.com>]

On Tue, Oct 27, 2026 at 10:02:55AM +0800, Tian Zheng wrote:
> 
> 
> On 10/23/2026 8:01 PM, Leonardo Bras wrote:
> > On Fri, Oct 23, 2026 at 03:23:04PM +0800, Tian Zheng wrote:
> > > 
> > > 
> > > On 10/14/2026 11:56 PM, Leonardo Bras wrote:
> > > > On Thu, Sep 17, 2026 at 02:40:42PM +0100, Leonardo Bras wrote:
> > > > > On Wed, Sep 16, 2026 at 04:27:19PM -0700, Oliver Upton wrote:
> > > > > > On Wed, Sep 16, 2026 at 03:00:46PM +0100, Leonardo Bras wrote:
> > > > > > > On Tue, Sep 15, 2026 at 05:10:02PM -0700, Oliver Upton wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > On Tue, Sep 01, 2026 at 06:15:56PM +0100, Leonardo Bras wrote:
> > > > > > > > > When dirty-logging is disabled, even non-write faults make a page dirty,
> > > > > > > > > which avoids a second fault when the page is actually written to.
> > > > > > > > > 
> > > > > > > > > On dirty-logging enable, this approach causes all (writable) pages on the
> > > > > > > > > memslot to be marked clean, even if they were not written to, which can
> > > > > > > > > take a lot of time, while holding the MMU lock, doing atomic writes to
> > > > > > > > > PTEs.
> > > > > > > > 
> > > > > > > > Do you have any performance numbers for this? Enabling HAFDBS seems a
> > > > > > > > bit involved to avoid some stores on the first pass.
> > > > > > > 
> > > > > > > Not yet, but if the idea does not look too crazy I can find hardware and
> > > > > > > collect some data :)
> > > > > > 
> > > > > > TBH this looks like a micro-optimization so I'm not expecting the
> > > > > > performance gains to justify the behavior change.
> > > > > > 
> > > > > 
> > > > > Okay then, I will get a machine to figure the performance gains :)
> > > > 
> > > > Got the performance numbers here.
> > > > I tested both the case where all PTEs are dirty, and the case that 50% of
> > > > the PTEs are dirty.
> > > > 
> > > > For that test, actually patch #5 could make kvm_pgtable_stage2_wrprotect()
> > > > slower, which is counter-intuitive, in all platforms tested:
> > > > 
> > > > When we walk the S2 pagetable to wrprotect, using the new walker, the
> > > > write to the PTE itself is so cheap* that the cost of the branch
> > > > misprediction (write PTE or not) for the 50/50 case is significantly higher
> > > > than the cost of the 50% extra writes, which makes the 50/50 up to 60%
> > > > slower than the 100% dirty case. It is still slower in the '25% dirty' as
> > > > well, and starts to get even at '10% dirty'.
> > > > 
> > > > * (non-atomic, everything is in order, walker is small, cpu will probably
> > > > bundle it all in a single cacheline write)
> > > > 
> > > > Since the idea of patch #5 was to reduce PTE writes in the first wrprotect
> > > > walk, it becomes faster only if only a small portion (<10%) of the VM was
> > > > write-faulted while the remaining was read-faulted, which seems to be a
> > > > really reduced audience.
> > > > 
> > > > Since this comes with increased complexity, and also carry costs of
> > > > having HAFDBS itself enabled outside dirty-tracking, I don't think it's a
> > > > good idea to further pursue patch #5. It's only value is that it served as
> > > > a base for HDBSS implementation.
> > > > 
> > > > Thanks for reading!
> > > > Leo
> > > > 
> > > Hi Leo,
> > > 
> > > Thanks for running this, the numbers are really useful.
> > > 
> > > It means the hardware dirty mode (HD=1) should only be enabled while
> > > dirty logging is active with HDBSS support, and stays off otherwise.
> > 
> > Hi Tian,
> > 
> > Correct, it's actually counter-productive for it to stay on without HDBSS
> > before dirty-logging.
> > 
> > > 
> > > With HD only enabled during dirty logging, we also keep the DBM-based
> > > encoding for the HD=0 case. On systems that support FEAT_HAFDBS, DBM
> > > remains the write bit and S2AP[1] the dirty bit even with HD=0. This
> > > works because kvm_s2_fault_compute_prot() maps any writable page of a
> > > non-tracked memslot writable-dirty (WD), including on read faults:
> > > 
> > > ```c
> > > if (s2vi->map_writable) {
> > >          *prot |= KVM_PGTABLE_PROT_W;
> > > 
> > >          if (s2vi->device ||
> > >              !memslot_is_logging(s2fd->memslot) ||
> > >              kvm_is_write_fault(s2fd->vcpu))
> > >                  *prot |= KVM_PGTABLE_PROT_DIRTY;
> > > }
> > > ```
> > > 
> > > So there is no WC entry for hardware to promote, and HD is simply not
> > > needed outside dirty logging.
> > > 
> > > In the next HDBSS version, we'll only turn on HD=1 when dirty logging
> > > is active.
> > > 
> > 
> > Right!
> > I will also send a new version of the pre-patch (4 patches) for us to
> > discuss. The first 2 patches will change meaning a bit, but should still
> > work as they do today.
> > 
> > Thanks!
> > Leo
> > 
> Hi Leo,
> 
> OK, I've seen the new DBM RFC patchset. HDBSS v6 will be based on it.


Hi Tian, thanks for reading it, and for accepting it in your series!

Before rebasing, please wait for the ongoing discussion on it's 
correctness. Maybe I will have to send new versions before it can be used.

Thanks!
Leo