Re: DAMON reporting more hot memory on huge pages (was: "Re: [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports")

[Lian Wang <lianux.mm@gmail.com>]

Hi SJ,

Thank you for thinking further about this and for suggesting the damon_prep
direction.  Let me first clarify the real-world use case and correct one
expectation that I created in my previous mail.

SXF develops a hyperconverged infrastructure platform.  The workloads running
in the guests are customer workloads, and their types, memory sizes and access
distributions are not controlled by us.  Oracle is the workload in the case
that exposed the problem, but we cannot define one Oracle access distribution
as representative of the platform or provide one generally applicable
workload-specific THP performance number.

The platform has two requirements:

  1. Host THP is enabled by default.  Large mappings generally provide useful
     translation and virtualization benefits, so disabling THP merely to make
     monitoring easier is not an acceptable general solution.

  2. We also need an estimate of each VM's active memory as an input to later
     memory-tiering configuration.  If a small number of distributed 4 KiB
     accesses makes many 2 MiB mappings appear fully active, the hot-tier
     requirement can be substantially overestimated.  VM sizes vary, so the
     absolute error can become large on a large VM when accesses are spread
     over many huge mappings.

The reported setup is a KVM/QEMU VM whose large guest-RAM allocation is backed
by a shared tmpfs file.  Oracle runs in the guest, host THP is enabled, and
host DAMON vaddr monitoring targets the QEMU process.  The field observation
is that, for the same business memory-use case, the measured hot proportion is
much higher with THP.  The guest-side tmpfs 4 KiB/2 MiB writer is a controlled
diagnostic for isolating this mechanism, not a model of the complete Oracle
access distribution.

The DAMON setup is:

  operations=vaddr
  monitoring_attrs/nr_regions/min=500
  monitoring_attrs/nr_regions/max=2000
  monitoring_attrs/intervals/sample_us=500000
  monitoring_attrs/intervals/aggr_us=20000000
  monitoring_attrs/intervals/update_us=60000000
  schemes/nr_schemes=1
  schemes/0/action=stat
  schemes/0/access_pattern/age/min=0
  schemes/0/access_pattern/age/max=18446744073709551615
  schemes/0/access_pattern/nr_accesses/min=1
  schemes/0/access_pattern/nr_accesses/max=18446744073709551615
  schemes/0/access_pattern/sz/min=0
  schemes/0/access_pattern/sz/max=18446744073709551615

The hot proportion is derived from the bytes selected by this stat-only scheme
(at least one observed access in a 20-second aggregation window) relative to
the monitored QEMU scope.  It is used for later configuration, not as a
request for DAMOS to split or migrate memory.

As shared in my previous mail, our controlled PC and server runs reproduced a
large THP-off/on observation gap.  The exact-address capture also showed a
512-times difference in unique 4 KiB spatial coverage while the two aggregate
DAMON hot proportions remained nearly identical.  The conclusion is that the
current coarse observation can substantially inflate active-memory estimates,
and configuration-only tuning cannot reconstruct the missing within-PMD
spatial information.

> I still want to better understand the real world use case before digging into
> the specific solution.  But, I was thinking perf event based monitoring might
> not feasible for the case.

We agree that perf-event feasibility should be established rather than
assumed.  We will test the address space reported while a KVM vCPU is running,
per-VM attribution, coverage and loss before relying on that path.  If you had
a specific feasibility concern in mind, please let us know so that we can
include it in the test.

> That is, we can request DAMON to do some preparation action for the data
> attribute sampling, using damon_prep.  At the moment, set_pgidle prep action is
> supported.  Maybe we could add a new prep action, say, split_pmd?  Then, the
> following sampling memory access check probe (would be a kind of "allow
> pidle_unset" probe filter) will be able to show the access on exactly the pte
> accessed bit, not the pmd accessed bit.
>
> Breaking pmd would cause the overhead, but it would be capped by the
> max_nr_regions.  So the user could at least consider about the tradeoff.  I
> think this is simpler and more intuitive than DAMOS-based pmd breaking, at
> least.

This placement makes sense to us.  A split_pmd preparation for access sampling
is clearer than exposing PMD breaking as a DAMOS action.  We understand it as
refining the mapping used for observation, without splitting the underlying
huge folio, and then letting the following probe observe PTE accessed bits.

We will follow this idea and validate it.  In particular, we will measure the
active-memory result and mapping cost, check whether max_nr_regions also bounds
the cumulative number of mappings that remain PTE-mapped over time, and verify
whether splitting the host process PMD is sufficient in the QEMU case or KVM's
large secondary mapping remains the limiting layer.  We will compare it with
the perf-event path, stat-only first, without proposing a user-visible DAMOS
split action or physical folio split.

Please rest assured that we see this as a long-term technical discussion and
collaboration, not an attempt to rush one mechanism into the tree.  We will
continue sharing both positive and negative results.  I also plan to present
the scenario, evidence and tradeoffs in more detail at LPC.  Please feel free
to share any concerns or ideas at any time; we are happy to keep iterating on
them together.

This topic has now become broader than Ravi's hardware-sampled report series.
If you think it would keep the discussions clearer, I can start a separate
thread focused on huge-page observation and the damon_prep approach, while
keeping perf-event implementation and testing comments on Ravi's thread.

Please let me know whether this clarifies the intended use and whether our
reading of the split_pmd damon_prep proposal matches what you had in mind.

Thanks,
Lian