Re: [RFC PATCH 5/5] KVM: arm64: Enable HAFDBS for guests not on migration

[Tian Zheng <zhengtian10@huawei.com>]

On 10/14/2026 11:56 PM, Leonardo Bras wrote:
> On Thu, Sep 17, 2026 at 02:40:42PM +0100, Leonardo Bras wrote:
>> On Wed, Sep 16, 2026 at 04:27:19PM -0700, Oliver Upton wrote:
>>> On Wed, Sep 16, 2026 at 03:00:46PM +0100, Leonardo Bras wrote:
>>>> On Tue, Sep 15, 2026 at 05:10:02PM -0700, Oliver Upton wrote:
>>>>> Hi,
>>>>>
>>>>> On Tue, Sep 01, 2026 at 06:15:56PM +0100, Leonardo Bras wrote:
>>>>>> When dirty-logging is disabled, even non-write faults make a page dirty,
>>>>>> which avoids a second fault when the page is actually written to.
>>>>>>
>>>>>> On dirty-logging enable, this approach causes all (writable) pages on the
>>>>>> memslot to be marked clean, even if they were not written to, which can
>>>>>> take a lot of time, while holding the MMU lock, doing atomic writes to
>>>>>> PTEs.
>>>>>
>>>>> Do you have any performance numbers for this? Enabling HAFDBS seems a
>>>>> bit involved to avoid some stores on the first pass.
>>>>
>>>> Not yet, but if the idea does not look too crazy I can find hardware and
>>>> collect some data :)
>>>
>>> TBH this looks like a micro-optimization so I'm not expecting the
>>> performance gains to justify the behavior change.
>>>
>>
>> Okay then, I will get a machine to figure the performance gains :)
> 
> Got the performance numbers here.
> I tested both the case where all PTEs are dirty, and the case that 50% of
> the PTEs are dirty.
> 
> For that test, actually patch #5 could make kvm_pgtable_stage2_wrprotect()
> slower, which is counter-intuitive, in all platforms tested:
> 
> When we walk the S2 pagetable to wrprotect, using the new walker, the
> write to the PTE itself is so cheap* that the cost of the branch
> misprediction (write PTE or not) for the 50/50 case is significantly higher
> than the cost of the 50% extra writes, which makes the 50/50 up to 60%
> slower than the 100% dirty case. It is still slower in the '25% dirty' as
> well, and starts to get even at '10% dirty'.
> 
> * (non-atomic, everything is in order, walker is small, cpu will probably
> bundle it all in a single cacheline write)
> 
> Since the idea of patch #5 was to reduce PTE writes in the first wrprotect
> walk, it becomes faster only if only a small portion (<10%) of the VM was
> write-faulted while the remaining was read-faulted, which seems to be a
> really reduced audience.
> 
> Since this comes with increased complexity, and also carry costs of
> having HAFDBS itself enabled outside dirty-tracking, I don't think it's a
> good idea to further pursue patch #5. It's only value is that it served as
> a base for HDBSS implementation.
> 
> Thanks for reading!
> Leo
> 
Hi Leo,

Thanks for running this, the numbers are really useful.

It means the hardware dirty mode (HD=1) should only be enabled while
dirty logging is active with HDBSS support, and stays off otherwise.

With HD only enabled during dirty logging, we also keep the DBM-based
encoding for the HD=0 case. On systems that support FEAT_HAFDBS, DBM
remains the write bit and S2AP[1] the dirty bit even with HD=0. This
works because kvm_s2_fault_compute_prot() maps any writable page of a
non-tracked memslot writable-dirty (WD), including on read faults:

```c
if (s2vi->map_writable) {
         *prot |= KVM_PGTABLE_PROT_W;

         if (s2vi->device ||
             !memslot_is_logging(s2fd->memslot) ||
             kvm_is_write_fault(s2fd->vcpu))
                 *prot |= KVM_PGTABLE_PROT_DIRTY;
}
```

So there is no WC entry for hardware to promote, and HD is simply not
needed outside dirty logging.

In the next HDBSS version, we'll only turn on HD=1 when dirty logging
is active.

Thanks,
Tian