* [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports
@ 2026-09-10 17:16 Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 1/9] mm/damon/vaddr: support page fault access check primitive Ravi Jonnalagadda
` (10 more replies)
0 siblings, 11 replies; 12+ messages in thread
From: Ravi Jonnalagadda @ 2026-09-10 17:16 UTC (permalink / raw)
To: sj, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc
Cc: akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
ravis.opensrc, rientjes, weixugc, jic23, gourry
This series lets DAMON take its access information from a hardware sampler
instead of from a page-table scan, and lets a scheme's score be weighted by what
that sampler reported.
The change from v1 [1] is that it is now built on the data attribute probe
infrastructure that has since landed in mm-new: a PMU is expressed as one more
probe on a context, with its own weight and probe hits, rather than as a
mechanism of its own beside the ops set. The rest of this posting is the same
substrate reworked onto that, plus two consumers.
This is not a merge request. The series is based on damon/next at the
base-commit below, which moves, so the tree it was built and tested from is also
on
https://github.com/ravis-opensrc/linux/tree/damon/perf-rfc-v2-09-06-26
It is posted for design feedback, on the substrate and on where it belongs in
the roadmap for extending DAMON beyond the pte-accessed bit [2]. That roadmap's
second milestone, now open, is a first data attribute monitored through
`damon_report_access()`, and that is what a sampling PMU is here. So this
series keeps that function and its callers and replaces its body: the reporting
path a hardware sampler needs cannot take a mutex, and the drain has to reach a
virtual-address context as well as a physical one. The shape of the ring, the
drain and the sysfs surface are what is most useful to review.
## Why a unified perf-event substrate
DAMON derives its access information from the PTE Accessed bit. A sampling PMU
carries what that bit cannot: which addresses the hardware went to, and how
often it went there. Many machines already have such a unit, and more than one
kind of it, so what this series is after is letting DAMON's regions be tuned
from whichever perf-based hardware source a machine offers rather than from the
Accessed bit alone. A sampler does not arrive on a kdamond's terms, though: it
delivers an address when the hardware decides to, in NMI context, with no
relation to the sampling interval and no mm to walk.
The alternative is a backend per PMU vendor, each owning its own configuration,
sysfs knobs and lifecycle. The perf-event direction [3] avoids that: let DAMON
register kernel-counter perf events and consume samples from any sampling PMU the
perf core already knows about. This series follows it, adding one substrate below
the ops sets rather than an ops set per PMU -- a report ring that any in-kernel
access source can push into, and a drain that folds those reports into region
probe hits on the aggregation boundary the kdamond already has. What running it
across vendors needed on top of that direction is:
- per-CPU lockless rings between the NMI sample handler and the kdamond drain,
- a ring partitioned by probe class, so the page-fault primitive and a PMU can
populate one context without either seeing the other's reports,
- per-CPU events that follow CPU hotplug, armed when the kdamond starts and
disarmed and drained when it stops,
- a per-PMU owner, so two contexts cannot claim the same PMU type,
- whichever address a PMU does report carried on the report and matched
against the context's own address space, so one source serves a paddr or a
vaddr context without a backend per address space.
This is tested with PEBS on Intel and IBS on AMD, both configured as `perf_event`
attributes on a probe and using the perf core's event plumbing rather than
per-vendor MSR code. A third source has already been written against the same
ring: Kunwu Chan's ARM SPE backend [5], which reaches it through an AUX buffer
drained in process context instead of an overflow callback, and which the
roadmap [2] places in its third milestone.
The partitioning is what lets promotion and demotion run in one context. A
sampler says which regions are hot; it says nothing about which are cold, because
a sampler that reports nothing about a page cannot distinguish untouched from
unsampled. Region age is what a demotion scheme matches on, and age comes from
the page-fault primitive. With the ring partitioned by class both are live at
once: the probe supplies hotness, the primitive supplies age, and two schemes
over the same regions can move memory in both directions under one kdamond.
The two classes are not symmetric, and the asymmetry is in where a report comes
from rather than in what it carries. A page fault has no monitoring context at
report time, so a per-context ring is not expressible for it: those reports go to
one global set of per-CPU rings, drained by the single context whose page-fault
primitive is enabled. A PMU overflow handler does carry the context that armed
the event, so those reports go to that context's own per-CPU rings and each
context drains only what it owns. Both classes share the ring layout and the
producer side, so the barrier pairing cannot drift between them.
```
page-fault report PMU sample, NMI context
no ctx at report time handler carries the owning ctx
| |
v v
+----------------------------+ +----------------------------+
| one global per-CPU ring | | per-CPU rings per context |
| set, shared by every ctx | | that armed a perf event |
+----------------------------+ +----------------------------+
| drained by the one ctx | | each ctx drains only the |
| with the pf primitive on | | rings it owns |
+----------------------------+ +----------------------------+
| |
+------------------+------------------+
|
drain on the aggregation boundary,
each report matched to a region
|
+---------------+---------------+
| |
region age region probe hits
demotion scheme promotion scheme
```
That is the property this series exists to enable, and three things follow from
it:
- One measurement drives both directions, at whichever scope the mode gives.
Bandwidth comes from resctrl MBM, which is where the discussion on a
platform-independent bandwidth signal for tiering has pointed [6]: a
virtual-address context reads a monitoring group created for the processes
it monitors, and a physical-address context reads the root monitoring
group, so the same controller is scoped to a set of processes or to the
whole machine without changing what it does. Reading it there also means a
bandwidth allocation reserved for an application bounds what the controller
optimises within. It spreads the hot set across nodes in the ratio that
maximises achieved bandwidth when bandwidth is what the workload is short of,
and it demotes cold pages off the near node in the same context at the same
time -- either proactively under a fixed quota or only once DRAM is under
memory pressure. Neither direction is configured as a target ratio; the
proportion or weight the controller writes bounds how much memory moves while
the score decides which pages move, and a hardware sample makes that score
proportional to the traffic a region generates rather than one bit per page
per scan. Earlier work on bandwidth-driven interleaving [4] measured the same
effect from userspace.
- Bandwidth and capacity are expanded at once rather than one being chosen.
Static weighted interleaving expands bandwidth, but it is a placement rule
rather than a decision about any particular page, so cold pages keep their
share of DRAM. NUMA balancing expands usable DRAM by promoting whatever is
hot, but it does so without reference to what the far node can deliver, so it
keeps promoting after the near node has stopped being the better place to
read from. Here the hot set's distribution comes from a bandwidth
measurement and the cold set is demoted on region age, in one context, so the
far node's bandwidth is used and DRAM is not held by memory nothing is
touching. The decision itself is a quota goal the kernel already carries:
`node_eligible_mem_bp` [8] bounds how much of a node a scheme may fill, so
what the controller supplies is the bandwidth measurement, not the
placement.
- When bandwidth is not the constraint it reverts to latency-first tiering on
its own. The search starts with everything on the near node and moves memory
outward only when the measurement improves, so on a workload that does not
saturate that node no step outward improves anything: the hot set stays in
DRAM for the lower access latency and cold pages keep being demoted. The
same configuration covers both cases, with nothing to switch on and no ratio
to revise.
## What the series adds
1. `mm/damon/vaddr: support page fault access check primitive` -- implements
the page-fault primitive for a virtual address space context; it was
available for physical addresses only. The marker whose fault reports the
access is installed through the target mm and the vma of the sampling
address directly, and prepare_access_checks() dispatches on the enabled
primitive as the physical address space one does.
2. `mm/damon/core: read the CPU number with preemption disabled` -- the report
path runs in fault context, which is preemptible. Read the CPU number with
preemption disabled and keep it disabled across the report, so the recorded
CPU stays consistent with the per-CPU ring the report is queued to.
3. `mm/damon/paddr: lock the folio for the page fault primitive rmap walk` --
take the folio lock unconditionally around the walk, as the other reverse
mapping walks in DAMON do, so every mapping type reaches it locked, and
release the reference the lookup took on each exit path.
4. `mm/damon: add damos_node_eligible_mem_bp tracepoint` -- a per-tick
tracepoint over the node-eligible-memory quota goal evaluation, exposing
the goal's target and current values, so the loop a bandwidth-driven
controller steers is visible to a tracer.
5. `mm/damon/core: add per-probe-class report rings and unified drain` -- the
substrate. Per-CPU SPSC rings an NMI-context source can publish into,
partitioned by probe index into a global page-fault ring and a per-context
perf ring, plus the kdamond-side drain that matches each report to a region
by binary search over a per-target snapshot and credits it to that region's
probe hits on the aggregation boundary. The address space of the target
selects which address of a report is matched, and pid targets are filtered
by thread group id.
6. `mm/damon: add perf-event overflow handler feeding the report ring` -- an
ops-agnostic perf-event source whose overflow handler turns a PMU sample
into a report, setting each address field the PMU reported as valid. A PMU
that reports a virtual address only can therefore drive a virtual-address
context, one that reports a physical address a physical-address context, and
the same source serves either without a backend per address space. Per-CPU
events are armed and released through cpuhp callbacks, and a per-PMU owner
keeps two contexts from claiming the same PMU type.
7. `mm/damon/ops-common: use probe-weighted score when probe weights are set`
-- lets a scheme's frequency subscore come from the probe hits, weighted per
probe class, so what the sampler reported reaches the tiering decision.
With no weights set the subscore comes from the access rate as before.
8. `mm/damon: add perf_event prep for PMU-driven hotness probes` -- the sysfs
surface and the event lifecycle: a perf_event prep action carrying the PMU
type, the event config and the sample attributes per probe, with per-CPU or
single-instance arming depending on how many counters the PMU needs.
Arming is deferred on a context built for a commit, so a weight-only commit
leaves the running event untouched.
9. `mm/damon/tests/drain-kunit: kunit for report rings and unified drain` --
unit tests for the rings and the drain: inject and drain, overflow on wrap,
producer routing by probe index, ring partitioning, pf-ring ownership,
thread-group filtering, the address space match, and both primitives live
on one context.
Patches 1-4 apply standalone on damon/next and are useful without the rest;
5-8 are the substrate and its first two consumers, and 9 covers them. If the
first four read right, they need not wait on the rest of this series -- happy
to repost them on their own for damon/next if that is easier to take. Two of
them, 2 and 3, correct code that is already in mm-new rather than adding
anything new, so they could instead go as a small fixes series carrying
Fixes: tags; guidance on which of the two routes is preferred is welcome.
## Userspace setup model
The runs were driven by an `auto_tier` subcommand added to damo on the branch
below, which reads achieved bandwidth from resctrl MBM and searches for the hot
set's distribution that maximises it, while the demotion scheme runs alongside
it in the same context. That tooling is not part of this posting; it is on
https://github.com/ravis-opensrc/damo/tree/damo/auto-tier-bw-2026-09-08
- `--bw_source resctrl` is the option this adds: the controller reads achieved
bandwidth from an MBM monitoring group and searches, rather than being given
a distribution to install.
- Everything else -- intervals, schemes, filters, the probe's `perf_event`
attributes -- is what the generator writes from the command line shown: 5 ms
sampling, 100 ms aggregation, a 1 s ops update.
- Both configurations select proactive demotion, so they demote continuously
rather than waiting for pressure, and both were left closed loop throughout.
- The search is the algorithm described in [4].
### Configuration A: AMD IBS Op, paddr ops, system-wide
```
$ sudo mount -t resctrl resctrl /sys/fs/resctrl
$ sudo python3 tools/damon_tier_gen.py --hotness ibs \
--near_node 0 --far_node 4 \
--cold_demote --cold_demote_mode proactive \
-o tier.yaml
$ sudo damo auto_tier tier.yaml --bw_source resctrl --verbose
```
- Scope is the machine, not a process set. The distribution is steered
through `node_eligible_mem_bp` quota goals over each node's own physical
ranges, which the generator reads from `/proc/iomem`.
- AMD Turin, DRAM on node 0 and a CXL node. IBS Op at a `sample_period` of
262144 with `sample_phys_addr` set, which is what a physical-address context
needs from the sample.
- Workload is a hot-and-cold mix so both schemes have something to act on:
`multiload` drives the bandwidth-hungry hot set, and a second process
allocates on the near node, touches it once and goes idle, leaving pages
that age out for the demotion scheme to find.
### Configuration B: Intel PEBS L3-miss, vaddr ops, per-PID
```
$ sudo mount -t resctrl resctrl /sys/fs/resctrl
$ sudo python3 tools/damon_tier_gen.py --hotness pebs \
--pid $HOT_PID --pid $COLD_PID \
--near_node 0 --far_node 1 \
--cold_demote --cold_demote_mode proactive \
--min_nr_regions 1000 --max_nr_regions 20000 \
-o tier.yaml
$ sudo damo auto_tier tier.yaml --bw_source resctrl --verbose
```
- Scope is the processes the monitoring group names and nothing else. The
distribution is steered through the hot scheme's `DamosDest` weights [9].
- Intel Granite Rapids, DRAM on node 0 and CXL on node 1. PEBS L3-miss at
`sample_freq` 5003 with `precise_ip` 2.
- Same workload shape.
## What the runs show
Per-node reference, each figure measured by binding the same workload to one
node:
```
Granite Rapids, 64 threads x 8 GiB
node 0 1.0 TB DRAM DRAM only 269,885 MB/s
node 1 2.0 TB CXL CXL only 249,903 MB/s
Turin, 32 threads x 4 GiB
node 0 386 GB DRAM DRAM only 113,596 MB/s
node 4 1.0 TB CXL CXL only 33,253 MB/s
```
- The two machines differ in the way that matters: on the Turin the far node
is 3.4x slower than the near one, on the Granite Rapids the two are within
8%.
- Each configuration gets one graph, the climb: the run itself, one mark per
decision taken from the bandwidth just measured, with the share it moved to
under each mark. The settled line below it is the median over the samples
the run went on to hold at that share.
### Configuration B, virtual-address mode
```
MB/s climb
460k | *
440k | *
420k | *
400k |
380k | *
360k | *
340k | * *
320k | *
300k | * *
280k | *
260k | * *
--------+------------+--------------+---------------+------------+--
1 4 7 10 13
92 80 68 56 44
decision index, near-node share (%)
settled share 44, 430,547 MB/s held
```
- Thirteen decisions to settle at 44%, no target given: about 60% more than
the better of the two nodes on its own, and the share does not change again
over the 81 samples that follow.
- Cold demotion runs over the same interval, in the same context, actuated by
the same kdamond: 34 regions totalling 15.4 GiB applied, and the idle
process ends with 4,099,980 pages -- 15.6 GiB -- on the CXL node having
started on DRAM.
- The promotion scheme's applied byte count is flat across the demotion ramp,
291.1 GiB before against 291.3 GiB after, so the recovered capacity is the
demotion scheme's.
### Configuration A, physical-address mode
```
MB/s climb
136k |
134k | * *
132k | * *
130k | *
128k |
126k | *
124k |
122k | *
120k |
118k | *
+----+-----+-----+-----+-----+-----+-----+-----+--
1 2 3 4 5 6 7 8
92 88 84 80 76 78 80 78
decision index, near-node share (%)
settled share 78, 134,107 MB/s held
```
- Eight decisions to settle at 78, from a start with almost everything on the
near node: the same algorithm reaches the optimum share on its own from the
bandwidth reading alone, and cold pages are demoted to the far node while it
does so.
- A distribution beats either end because both nodes supply at once: at the
settled share the near node is supplying 104,618 MB/s against the 113,596 it
reaches with the whole working set bound to it, so the further 37,669 MB/s
arriving over CXL is bandwidth it could not have supplied at any share.
- The idle process ends with 2,863,156 pages -- 10.9 GiB -- on the CXL node,
out of the 32 GiB it allocated on DRAM and stopped touching. Residency is
the attributable figure in this mode rather than a per-scheme applied count,
because the schemes share physical ranges.
### What the runs are meant to establish
Not the settled share and not the absolute bandwidth; those are properties of
these machines and these workloads. What reproduces is:
- A controller reading achieved bandwidth converges unattended, from a
configuration naming no target, to a distribution that beats either node on
its own, and holds it once found.
- A demotion scheme recovers near-node capacity in the same context while that
happens.
- One code path does both, steering `node_eligible_mem_bp` over physical
ranges system-wide on one machine and `DamosDest` weights over a named
process group on the other.
## Beyond a CPU PMU
Nothing above is specific to PEBS or IBS. A source qualifies if it can report
an accessed address to the ring, which is the only thing the substrate asks of
it. A CXL device's Hotness Monitoring Unit, whose r3.2 perf driver has been
posted [7] exposing its hotlist through an AUX buffer, or a custom monitoring
unit on an accelerator or a memory expander, reports exactly that, and a
backend delivering those reports through a perf event reaches the same drain,
the same probe hits and the same schemes already in the tree. Such a unit sees
the traffic that reaches its own device rather than a system-wide view, so what
it contributes is the hot set resident on that device: the tiering logic can
promote those pages to DRAM without the host having to detect that hotness
itself, and without depending on what produced the reports. That posting names
driving tiering from such a unit as the intent and the in-kernel step as future
work, and consuming it as a DAMON access-check primitive was raised there as
one way to get it; a backend on this substrate is that path.
[1] v1 of this series
https://lore.kernel.org/damon/20260529165640.820-1-ravis.opensrc@gmail.com/
[2] Roadmap for extending DAMON beyond pte-accessed bit
https://lore.kernel.org/damon/20260525225208.1179-1-sj@kernel.org/
[3] mm/damon: introduce perf event based access check
https://lore.kernel.org/damon/20260423004211.7037-1-akinobu.mita@gmail.com/
[4] B. Tabatabai, R. Jonnalagadda et al., "Bandwidth Speaks, We Listen:
Dynamic Interleaving for Tiered Memory", ISMM 2026.
https://dl.acm.org/doi/10.1145/3814942.3816137
[5] mm/damon/perf: add ARM SPE AUX backend
https://lore.kernel.org/damon/20260816142222.689624-1-kunwu.chan@linux.dev/
[6] A platform-independent subsystem for bandwidth information, and resctrl
as that source
https://lore.kernel.org/linux-mm/d952a84f-332e-8f7a-4816-2c1cbd8f5b00@google.com/
[7] CXL Hotness Monitoring Unit perf driver
https://lore.kernel.org/linux-mm/20241121101845.1815660-1-Jonathan.Cameron@huawei.com/
[8] mm/damon: add node_eligible_mem_bp goal metric, merged for v7.2
https://lore.kernel.org/linux-mm/20260428030520.701-1-ravis.opensrc@gmail.com/
[9] mm/damon/vaddr: allow interleaving in migrate_{hot,cold} actions, merged
for v6.17
https://lore.kernel.org/linux-mm/20250709005952.17776-1-bijan311@gmail.com/
Ravi Jonnalagadda (9):
mm/damon/vaddr: support page fault access check primitive
mm/damon/core: read the CPU number with preemption disabled
mm/damon/paddr: lock the folio for the page fault primitive rmap walk
mm/damon: add damos_node_eligible_mem_bp tracepoint
mm/damon/core: add per-probe-class report rings and unified drain
mm/damon: add perf-event overflow handler feeding the report ring
mm/damon/ops-common: use probe-weighted score when probe weights are
set
mm/damon: add perf_event prep for PMU-driven hotness probes
mm/damon/tests/drain-kunit: kunit for report rings and unified drain
include/linux/damon.h | 136 ++++-
include/trace/events/damon.h | 32 +
mm/damon/Kconfig | 18 +
mm/damon/Makefile | 1 +
mm/damon/core.c | 995 +++++++++++++++++++++++++++----
mm/damon/ops-common.c | 21 +-
mm/damon/paddr.c | 37 +-
mm/damon/perf_source.c | 489 +++++++++++++++
mm/damon/perf_source.h | 54 ++
mm/damon/sysfs.c | 266 ++++++++-
mm/damon/tests/.kunitconfig | 4 +
mm/damon/tests/core-kunit.h | 2 +-
mm/damon/tests/drain-kunit.h | 1091 ++++++++++++++++++++++++++++++++++
mm/damon/tests/perf-kunit.h | 133 +++++
mm/damon/vaddr.c | 83 ++-
15 files changed, 3220 insertions(+), 142 deletions(-)
create mode 100644 mm/damon/perf_source.c
create mode 100644 mm/damon/perf_source.h
create mode 100644 mm/damon/tests/drain-kunit.h
create mode 100644 mm/damon/tests/perf-kunit.h
base-commit: e1f34dce183a96fc93bf9a42dcdc0ec0bf82a3d2
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 1/9] mm/damon/vaddr: support page fault access check primitive
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
@ 2026-09-10 17:16 ` Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 2/9] mm/damon/core: read the CPU number with preemption disabled Ravi Jonnalagadda
` (9 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Ravi Jonnalagadda @ 2026-09-10 17:16 UTC (permalink / raw)
To: sj, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc
Cc: akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
ravis.opensrc, rientjes, weixugc, jic23, gourry
The page-fault sampling primitive was implemented for the physical
address space only, so a virtual address space context had no way to
enable it.
Implement it for the virtual address space. The prep installs the marker
whose fault reports the access on the sampling address of each region,
mirroring the physical address space primitive, and prepare_access_checks()
now dispatches on the enabled primitive as the physical address space one
does.
The target mm and the vma of the sampling address are both available
here, so the marker is installed on the mapping directly. The physical
address space primitive has only a physical address and reaches its
mappings through a reverse mapping walk.
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@gmail.com>
---
mm/damon/vaddr.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 64 insertions(+), 1 deletion(-)
diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c
index d5dde97b3cd0d..1f2d923e569d6 100644
--- a/mm/damon/vaddr.c
+++ b/mm/damon/vaddr.c
@@ -13,6 +13,7 @@
#include <linux/pagemap.h>
#include <linux/pagewalk.h>
#include <linux/sched/mm.h>
+#include <asm/tlb.h>
#include "../internal.h"
#include "ops-common.h"
@@ -329,7 +330,8 @@ static void __damon_va_prepare_access_check(struct mm_struct *mm,
damon_va_mkold(mm, r->sampling_addr);
}
-static void damon_va_prepare_access_checks(struct damon_ctx *ctx)
+/* Use page table accessed bits */
+static void damon_va_prepare_access_checks_abit(struct damon_ctx *ctx)
{
struct damon_target *t;
struct mm_struct *mm;
@@ -345,6 +347,67 @@ static void damon_va_prepare_access_checks(struct damon_ctx *ctx)
}
}
+/*
+ * Install the page table marker whose fault reports the access. The target
+ * mm and the vma of the address are available here, so the marker is set on
+ * the mapping directly, without a reverse mapping walk.
+ */
+static void damon_va_change_protection(struct mm_struct *mm,
+ unsigned long addr)
+{
+ struct vm_area_struct *vma;
+ struct mmu_gather tlb;
+
+ /*
+ * The sampling address is a random offset within the region, so round
+ * it down to the page it falls in. change_protection() walks whole
+ * page table entries and expects a page aligned range.
+ */
+ addr = ALIGN_DOWN(addr, PAGE_SIZE);
+
+ mmap_read_lock(mm);
+
+ vma = vma_lookup(mm, addr);
+ if (!vma || !vma_is_accessible(vma) || (vma->vm_flags & VM_PFNMAP))
+ goto unlock;
+
+ tlb_gather_mmu(&tlb, mm);
+ /* todo: batch or remove tlb flushing */
+ change_protection(&tlb, vma, addr, addr + PAGE_SIZE, MM_CP_DAMON);
+ tlb_finish_mmu(&tlb);
+
+unlock:
+ mmap_read_unlock(mm);
+}
+
+/* Use page faults */
+static void damon_va_prepare_access_checks_faults(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+ struct mm_struct *mm;
+ struct damon_region *r;
+
+ damon_for_each_target(t, ctx) {
+ mm = damon_get_mm(t);
+ if (!mm)
+ continue;
+ damon_for_each_region(r, t) {
+ r->sampling_addr = damon_rand(ctx, r->ar.start,
+ r->ar.end);
+ damon_va_change_protection(mm, r->sampling_addr);
+ }
+ mmput(mm);
+ }
+}
+
+static void damon_va_prepare_access_checks(struct damon_ctx *ctx)
+{
+ if (ctx->sample_control.primitives_enabled.page_table)
+ damon_va_prepare_access_checks_abit(ctx);
+ if (ctx->sample_control.primitives_enabled.page_fault)
+ damon_va_prepare_access_checks_faults(ctx);
+}
+
struct damon_young_walk_private {
bool young;
};
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 2/9] mm/damon/core: read the CPU number with preemption disabled
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 1/9] mm/damon/vaddr: support page fault access check primitive Ravi Jonnalagadda
@ 2026-09-10 17:16 ` Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 3/9] mm/damon/paddr: lock the folio for the page fault primitive rmap walk Ravi Jonnalagadda
` (8 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Ravi Jonnalagadda @ 2026-09-10 17:16 UTC (permalink / raw)
To: sj, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc
Cc: akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
ravis.opensrc, rientjes, weixugc, jic23, gourry
damon_report_page_fault() runs in fault context, which is preemptible, so
reading the CPU number there triggers a "using smp_processor_id() in
preemptible" report.
Read it with preemption disabled, and keep it disabled across the report
so the recorded CPU stays consistent with the per-CPU ring the report is
queued to, which is selected the same way.
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@gmail.com>
---
mm/damon/core.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 5fdb5922edb78..991ddb9a09414 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -2514,7 +2514,6 @@ void damon_report_page_fault(struct vm_fault *vmf, bool huge_pmd)
struct damon_access_report access_report = {
.vaddr = vmf->address,
.size = 1, /* todo: set appripriately */
- .cpu = smp_processor_id(),
.tid = task_pid_vnr(current),
.is_write = vmf->flags & FAULT_FLAG_WRITE,
};
@@ -2524,7 +2523,15 @@ void damon_report_page_fault(struct vm_fault *vmf, bool huge_pmd)
else
access_report.paddr = PFN_PHYS(pte_pfn(vmf->orig_pte));
+ /*
+ * This runs in fault context, which is preemptible, so the CPU number
+ * is read with preemption disabled. Holding it across the report also
+ * keeps the recorded CPU consistent with the per-CPU ring the report is
+ * queued to, which damon_report_access() selects the same way.
+ */
+ access_report.cpu = get_cpu();
damon_report_access(&access_report);
+ put_cpu();
}
#endif
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 3/9] mm/damon/paddr: lock the folio for the page fault primitive rmap walk
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 1/9] mm/damon/vaddr: support page fault access check primitive Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 2/9] mm/damon/core: read the CPU number with preemption disabled Ravi Jonnalagadda
@ 2026-09-10 17:16 ` Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 4/9] mm/damon: add damos_node_eligible_mem_bp tracepoint Ravi Jonnalagadda
` (7 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Ravi Jonnalagadda @ 2026-09-10 17:16 UTC (permalink / raw)
To: sj, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc
Cc: akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
ravis.opensrc, rientjes, weixugc, jic23, gourry
rmap_walk() requires the folio to be locked for every mapping type it
walks, and the page fault access check primitive takes the lock only for
file-backed and KSM folios, so a plain anonymous folio reaches the walk
unlocked and the walk does not install the marker.
Take the lock unconditionally, as the other reverse mapping walks in DAMON
do, and release the folio reference the lookup took on every exit path.
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@gmail.com>
---
mm/damon/paddr.c | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index f4fa7c231e551..5fcef3005c536 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -91,21 +91,25 @@ static void damon_pa_change_protection(unsigned long paddr)
.rmap_one = damon_pa_change_protection_one,
.anon_lock = folio_lock_anon_vma_read,
};
- bool need_lock;
if (!folio)
return;
if (!folio_mapped(folio) || !folio_raw_mapping(folio))
- return;
+ goto put;
- need_lock = !folio_test_anon(folio) || folio_test_ksm(folio);
- if (need_lock && !folio_trylock(folio))
- return;
+ /*
+ * rmap_walk() requires a locked folio for every mapping type, so the
+ * lock is taken unconditionally, as the other reverse mapping walks in
+ * DAMON do.
+ */
+ if (!folio_trylock(folio))
+ goto put;
rmap_walk(folio, &rwc);
- if (need_lock)
- folio_unlock(folio);
+ folio_unlock(folio);
+put:
+ folio_put(folio);
}
static void damon_pa_prepare_access_checks_faults(struct damon_ctx *ctx)
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 4/9] mm/damon: add damos_node_eligible_mem_bp tracepoint
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
` (2 preceding siblings ...)
2026-09-10 17:16 ` [RFC PATCH v2 3/9] mm/damon/paddr: lock the folio for the page fault primitive rmap walk Ravi Jonnalagadda
@ 2026-09-10 17:16 ` Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 5/9] mm/damon/core: add per-probe-class report rings and unified drain Ravi Jonnalagadda
` (6 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Ravi Jonnalagadda @ 2026-09-10 17:16 UTC (permalink / raw)
To: sj, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc
Cc: akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
ravis.opensrc, rientjes, weixugc, jic23, gourry
Add a per-tick tracepoint for DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP goal
evaluation, exposing the goal target/current values for observability of
the node-eligible-memory bandwidth quota loop. Bandwidth-feedback tiering
loops read this goal metric to detect convergence.
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@gmail.com>
---
include/trace/events/damon.h | 32 ++++++++++++++++++++++++++++++++
mm/damon/core.c | 28 ++++++++++++++++++++++++++++
2 files changed, 60 insertions(+)
diff --git a/include/trace/events/damon.h b/include/trace/events/damon.h
index 4dfc9467ea4b5..b2a472eb45d95 100644
--- a/include/trace/events/damon.h
+++ b/include/trace/events/damon.h
@@ -74,6 +74,38 @@ TRACE_EVENT(damos_esz,
__entry->esz)
);
+/* Per-tick DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP goal evaluation. */
+TRACE_EVENT(damos_node_eligible_mem_bp,
+
+ TP_PROTO(unsigned int context_idx, unsigned int scheme_idx,
+ int nid,
+ unsigned long target_value, unsigned long current_value),
+
+ TP_ARGS(context_idx, scheme_idx, nid, target_value, current_value),
+
+ TP_STRUCT__entry(
+ __field(unsigned int, context_idx)
+ __field(unsigned int, scheme_idx)
+ __field(int, nid)
+ __field(unsigned long, target_value)
+ __field(unsigned long, current_value)
+ ),
+
+ TP_fast_assign(
+ __entry->context_idx = context_idx;
+ __entry->scheme_idx = scheme_idx;
+ __entry->nid = nid;
+ __entry->target_value = target_value;
+ __entry->current_value = current_value;
+ ),
+
+ TP_printk("ctx_idx=%u scheme_idx=%u nid=%d "
+ "target_value=%lu current_value=%lu",
+ __entry->context_idx, __entry->scheme_idx,
+ __entry->nid,
+ __entry->target_value, __entry->current_value)
+);
+
TRACE_EVENT_CONDITION(damos_before_apply,
TP_PROTO(unsigned int context_idx, unsigned int scheme_idx,
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 991ddb9a09414..6b56a3e961c4e 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -3461,6 +3461,34 @@ static unsigned long damos_quota_score(struct damon_ctx *c, struct damos *s)
highest_score = max(highest_score,
mult_frac(goal->current_value, 10000,
goal->target_value));
+
+ /*
+ * Per-tick visibility of NODE_ELIGIBLE_MEM_BP goal evaluation
+ * for userspace convergence-detection.
+ */
+ if (goal->metric == DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP &&
+ trace_damos_node_eligible_mem_bp_enabled()) {
+ /*
+ * cidx is hardcoded 0 because DAMON sysfs currently
+ * supports a single context per kdamond. Other DAMOS
+ * tracepoints (damos_before_apply, damos_esz,
+ * damos_stat_after_apply_interval) follow the same
+ * convention; once multi-ctx-per-kdamond lands the
+ * field becomes meaningful.
+ */
+ unsigned int cidx = 0, sidx = 0;
+ struct damos *siter;
+
+ damon_for_each_scheme(siter, c) {
+ if (siter == s)
+ break;
+ sidx++;
+ }
+ trace_damos_node_eligible_mem_bp(cidx, sidx,
+ goal->nid,
+ goal->target_value,
+ goal->current_value);
+ }
}
return highest_score;
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 5/9] mm/damon/core: add per-probe-class report rings and unified drain
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
` (3 preceding siblings ...)
2026-09-10 17:16 ` [RFC PATCH v2 4/9] mm/damon: add damos_node_eligible_mem_bp tracepoint Ravi Jonnalagadda
@ 2026-09-10 17:16 ` Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 6/9] mm/damon: add perf-event overflow handler feeding the report ring Ravi Jonnalagadda
` (5 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Ravi Jonnalagadda @ 2026-09-10 17:16 UTC (permalink / raw)
To: sj, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc
Cc: akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
ravis.opensrc, rientjes, weixugc, jic23, gourry
damon_report_access() takes a mutex, so it cannot be called from NMI
context, which is where a perf-event overflow handler runs. Provide
per-CPU SPSC rings instead, partitioned by the globally-meaningful
probe_idx boundary into two classes:
- Page-fault reports (probe_idx == DAMON_PROBE_IDX_NONE) use a global
per-CPU ring. The page_fault primitive has no damon_ctx at report
time, so a per-context ring is not expressible; the global ring is
drained by the single context whose page_fault primitive is enabled.
- Perf-event reports (probe_idx >= 1) use a per-context per-CPU ring.
The overflow handler carries a pointer to its owning ctx, so each
perf-driven context enqueues into and drains only its own ring, and no
cross-context ring owner is needed.
The per-ctx perf ring is allocated when the first perf probe is armed and
freed in damon_destroy_ctx() after all perf events are released, so no
in-flight NMI can reach freed storage.
damon_report_access() routes each report to the global pf ring or to the
report's ctx perf ring by probe_idx, sharing one copy of the NMI-safe
publish sequence so the barrier pairing cannot drift between the two.
Each ring has a per-CPU busy counter to detect and drop re-entrant NMI
nesting on the same CPU. The producer writes the entry, issues smp_wmb(),
then advances head with WRITE_ONCE(); the consumer reads head and pairs
with smp_rmb() before reading the entry.
Add the kdamond consumer, kdamond_check_reported_accesses(): a dispatcher
that drains the ring(s) the context's enabled primitives feed, with the
per-ring drain body parameterised only by ring and pending mask. Each
entry carries its own probe_idx, so the drain resolves the probe_hits[]
slot without a list walk. damon_drains_ring_pf() and
damon_drains_ring_perf() name which ring a context drives, so which ring
is drained and which access-check path runs follow from the enabled
primitive rather than from whether probe weights are set.
Build a per-target sorted region snapshot once per drain so each entry is
matched to its region by binary search rather than a linear region walk.
Reports straddling a region boundary are rejected. The address space of
the monitoring target selects which address of a report is matched, so a
context whose targets carry a pid matches the virtual address and the
others match the physical address; a report carrying no address for that
space is dropped. For pid-target contexts, filter by thread group id so
entries from unrelated processes are not credited to the wrong target. The
report carries that id in a field of its own, beside the thread id the
threads type sample filter matches against.
Ownership of the global pf ring is enforced under damon_lock at
damon_start(), on the live commit path, and cleared on kdamond exit. The
perf ring needs no such guard because it is per-ctx. Staleness uses
sample_interval rather than aggr_interval, so stale data does not inflate
hot scores across aggregation boundaries.
Co-developed-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@gmail.com>
---
include/linux/damon.h | 106 +++++-
mm/damon/core.c | 820 ++++++++++++++++++++++++++++++++++++------
2 files changed, 817 insertions(+), 109 deletions(-)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 33929a400aa7a..88a459a60b296 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -6,6 +6,7 @@
#ifndef _DAMON_H_
#define _DAMON_H_
+#include <asm/local.h>
#include <linux/math64.h>
#include <linux/memcontrol.h>
#include <linux/mutex.h>
@@ -17,9 +18,19 @@
#define DAMON_MIN_REGION_SZ PAGE_SIZE
/* Maximum number of monitoring probes. */
#define DAMON_MAX_PROBES (4)
+/*
+ * Sentinel value for damon_access_report.probe_idx: 0 means no probe
+ * attribution (matches zero-init of struct damon_access_report on the stack).
+ * Perf-event probe indices start at 1.
+ */
+#define DAMON_PROBE_IDX_NONE 0
/* Max priority score for DAMON-based operation schemes */
#define DAMOS_MAX_SCORE (99)
+/* Per-CPU SPSC ring: size must be a power of two. */
+#define DAMON_REPORT_RING_SIZE 256
+#define DAMON_REPORT_RING_MASK (DAMON_REPORT_RING_SIZE - 1)
+
/**
* struct damon_addr_range - Represents an address region of [@start, @end).
* @start: Start address of the region (inclusive).
@@ -110,7 +121,15 @@ struct damon_target {
* @size: The size of the accessed address range.
* @cpu: The id of the CPU that made the access.
* @tid: The task id of the task that made the access.
+ * @tgid: The thread group id of the task that made the access. A
+ * monitoring target created for a process carries this id,
+ * so it is the id a report is matched against.
* @is_write: Whether the access is write.
+ * @probe_idx: Index into probe_hits[] for the reporting probe; set by
+ * the perf-event overflow handler so the drain can credit
+ * the correct slot without a list walk.
+ * 0 is reserved (no probe attribution; matches zero-init);
+ * perf-event probe indices start at 1.
*
* Any DAMON API callers that notified access events can report the information
* to DAMON using damon_report_access(). This struct contains the reporting
@@ -122,11 +141,53 @@ struct damon_access_report {
unsigned long size;
unsigned int cpu;
pid_t tid;
+ pid_t tgid;
bool is_write;
+ int probe_idx;
+ /*
+ * Owning context for perf-event reports (probe_idx >= 1): the overflow
+ * handler sets this so the producer enqueues into that ctx's own perf
+ * ring. NULL for page_fault reports (probe_idx == DAMON_PROBE_IDX_NONE),
+ * which route to the global pf ring by address at drain time.
+ */
+ struct damon_ctx *ctx;
/* private: */
unsigned long report_jiffies; /* when this report is made */
};
+/**
+ * struct damon_report_ring - Per-CPU SPSC ring for NMI-safe access reports.
+ *
+ * @head: Write index; updated by the NMI producer.
+ * @tail: Read index; updated by the kdamond consumer.
+ * @entries: Ring buffer entries.
+ *
+ * One ring per CPU: page-fault reports use a global set, and each context
+ * with a perf-event probe has its own set. The producer (NMI overflow
+ * handler) writes to @head; the consumer (kdamond) reads from @tail. Both
+ * indices are unsigned and wrap modulo DAMON_REPORT_RING_SIZE.
+ */
+struct damon_report_ring {
+ unsigned int head; /* written by producer (NMI) */
+ unsigned int tail /* written by consumer (kdamond) */
+ ____cacheline_aligned_in_smp;
+ struct damon_access_report entries[DAMON_REPORT_RING_SIZE]
+ ____cacheline_aligned_in_smp;
+};
+
+/*
+ * struct damon_target_lookup - Cached, sorted region snapshot for one target.
+ * @regions: Array of region pointers, sorted by ar.start (address order).
+ * @nr_regions: Number of entries in @regions.
+ *
+ * Built once per aggregation tick by damon_build_target_lookup() so the ring
+ * drain can binary-search a target's regions instead of walking the list.
+ */
+struct damon_target_lookup {
+ struct damon_region **regions;
+ unsigned int nr_regions;
+};
+
/**
* enum damos_action - Represents an action of a Data Access Monitoring-based
* Operation Scheme.
@@ -874,6 +935,7 @@ struct damon_filter {
*/
struct damon_probe {
unsigned int weight;
+ bool event_driven; /* hits arrive via ring drain, not apply_probes */
/* private: */
/* Preparation actions to apply to each probing memory. */
struct list_head preps;
@@ -1090,6 +1152,34 @@ struct damon_ctx {
/* @rnd_state: Per-ctx PRNG state for damon_rand(). */
struct rnd_state rnd_state;
+
+ /* Reusable drain-loop snapshot buffer (avoids per-tick kmalloc). */
+ struct {
+ struct damon_target_lookup *lookups;
+ unsigned int nr_lookups;
+ struct damon_region **region_buf;
+ unsigned int region_buf_cap;
+ } drain_snapshot;
+
+ /*
+ * Per-context perf-event report ring. Unlike the page_fault primitive
+ * (which has no ctx at report time and so uses a global ring), a perf
+ * overflow handler is armed by -- and carries a pointer to -- its owning
+ * ctx (damon_access_report.ctx), so its reports route to this per-ctx
+ * ring. This gives full per-context isolation (two perf-driven ctxs
+ * never share a ring) and removes any need for a cross-ctx perf ring
+ * owner guard.
+ *
+ * Allocated lazily when the first perf probe is armed
+ * (damon_ctx_alloc_perf_ring, from damon_perf_probe_setup) and freed in
+ * damon_destroy_ctx() AFTER all perf events are released, so no in-flight
+ * NMI can reach freed storage. perf_rings == NULL means "no perf ring
+ * yet" and any perf report is dropped, so a build/config without a perf
+ * source simply never allocates it (lazy alloc = zero cost when unused).
+ */
+ struct damon_report_ring __percpu *perf_rings;
+ int __percpu *perf_ring_busy;
+ cpumask_t perf_pending;
};
/* Get a random number in [@l, @r) using @ctx's lockless PRNG. */
@@ -1208,6 +1298,7 @@ void damon_destroy_filter(struct damon_filter *f);
struct damon_probe *damon_new_probe(void);
void damon_add_probe(struct damon_ctx *ctx, struct damon_probe *probe);
+bool damon_has_event_driven_probes(struct damon_ctx *ctx);
struct damon_region *damon_new_region(unsigned long start, unsigned long end);
unsigned int damon_nr_accesses_mvsum(struct damon_region *r,
@@ -1297,7 +1388,8 @@ int damon_kdamond_pid(struct damon_ctx *ctx);
int damon_call(struct damon_ctx *ctx, struct damon_call_control *control);
int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control);
-void damon_report_access(struct damon_access_report *report);
+bool damon_report_access(struct damon_access_report *report);
+int damon_ctx_alloc_perf_ring(struct damon_ctx *ctx);
#ifdef CONFIG_MMU
void damon_report_page_fault(struct vm_fault *vmf, bool huge_pmd);
#else
@@ -1311,16 +1403,22 @@ int damon_set_region_system_rams_default(struct damon_target *t,
unsigned long addr_unit,
unsigned long min_region_sz);
-#ifdef CONFIG_ACMA
+unsigned long damon_get_report_overflow(void);
+unsigned long damon_get_report_ring_full(void);
+unsigned long damon_get_report_busy_drop(void);
+unsigned long damon_get_samples_drained(void);
+unsigned long damon_get_samples_stale_drained(void);
+unsigned long damon_get_samples_no_region(void);
+#ifdef CONFIG_ACMA
unsigned long damon_alloced_bytes(void);
-
#endif
#else /* CONFIG_DAMON */
-static inline void damon_report_access(struct damon_access_report *report)
+static inline bool damon_report_access(struct damon_access_report *report)
{
+ return false;
}
static inline void damon_report_page_fault(struct vm_fault *vmf, bool huge_pmd)
{
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 6b56a3e961c4e..850880f791c9e 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -22,22 +22,136 @@
#define CREATE_TRACE_POINTS
#include <trace/events/damon.h>
-#define DAMON_ACCESS_REPORTS_CAP 1000
+/*
+ * Report rings are partitioned by the globally-meaningful probe_idx boundary,
+ * and the two classes live in different places:
+ *
+ * page-fault reports (probe_idx == DAMON_PROBE_IDX_NONE, 0): a GLOBAL per-CPU
+ * ring (damon_report_rings_pf). The page_fault primitive has no damon_ctx
+ * at report time, so a per-context ring is not expressible; the global ring
+ * is drained by the single context whose page_fault primitive is enabled
+ * (damon_report_ring_owner_pf).
+ * perf-event reports (probe_idx >= 1): a PER-CONTEXT per-CPU ring
+ * (ctx->perf_rings). The overflow handler carries the owning ctx, so each
+ * perf-driven context drains only its own ring -- no global ring and no
+ * cross-context owner needed. See struct damon_ctx.
+ *
+ * Each ring (the global pf ring, and every context's perf ring) has its OWN
+ * per-CPU storage, per-CPU busy flag, pending cpumask, and overflow counter;
+ * the producer/consumer barrier sequence in damon_report_access() is shared so
+ * the pairing cannot drift between the two.
+ */
+/*
+ * Report drops are counted per reason. A ring-full drop means the consumer
+ * did not keep up with the producer; a busy-guard drop means an NMI nested on
+ * top of a same-CPU producer.
+ */
+static DEFINE_PER_CPU(unsigned long, damon_report_ring_full_pf);
+static DEFINE_PER_CPU(unsigned long, damon_report_ring_full_perf);
+static DEFINE_PER_CPU(unsigned long, damon_report_busy_drop_pf);
+static DEFINE_PER_CPU(unsigned long, damon_report_busy_drop_perf);
+static DEFINE_PER_CPU(unsigned long, damon_samples_drained);
+static DEFINE_PER_CPU(unsigned long, damon_samples_stale_drained);
+static DEFINE_PER_CPU(unsigned long, damon_samples_no_region);
+
+static DEFINE_PER_CPU(struct damon_report_ring, damon_report_rings_pf);
+static DEFINE_PER_CPU(int, damon_report_ring_busy_pf);
+/*
+ * Producer (NMI) sets after publishing a report; consumer (kdamond) clears
+ * before draining the corresponding ring. Hot-write under sampling load -
+ * do NOT mark __read_mostly. One pending mask per ring.
+ */
+static cpumask_t damon_rings_pending_pf;
+
+unsigned long damon_get_report_ring_full(void)
+{
+ unsigned long sum = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ sum += per_cpu(damon_report_ring_full_pf, cpu);
+ sum += per_cpu(damon_report_ring_full_perf, cpu);
+ }
+ return sum;
+}
+EXPORT_SYMBOL_GPL(damon_get_report_ring_full);
+
+unsigned long damon_get_report_busy_drop(void)
+{
+ unsigned long sum = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ sum += per_cpu(damon_report_busy_drop_pf, cpu);
+ sum += per_cpu(damon_report_busy_drop_perf, cpu);
+ }
+ return sum;
+}
+EXPORT_SYMBOL_GPL(damon_get_report_busy_drop);
+
+/* Reports dropped for either reason. Kept so existing users need no change. */
+unsigned long damon_get_report_overflow(void)
+{
+ return damon_get_report_ring_full() + damon_get_report_busy_drop();
+}
+EXPORT_SYMBOL_GPL(damon_get_report_overflow);
+
+unsigned long damon_get_samples_drained(void)
+{
+ unsigned long sum = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu)
+ sum += per_cpu(damon_samples_drained, cpu);
+ return sum;
+}
+EXPORT_SYMBOL_GPL(damon_get_samples_drained);
+
+unsigned long damon_get_samples_stale_drained(void)
+{
+ unsigned long sum = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu)
+ sum += per_cpu(damon_samples_stale_drained, cpu);
+ return sum;
+}
+EXPORT_SYMBOL_GPL(damon_get_samples_stale_drained);
+
+unsigned long damon_get_samples_no_region(void)
+{
+ unsigned long sum = 0;
+ int cpu;
+ for_each_possible_cpu(cpu)
+ sum += per_cpu(damon_samples_no_region, cpu);
+ return sum;
+}
+EXPORT_SYMBOL_GPL(damon_get_samples_no_region);
static DEFINE_MUTEX(damon_lock);
static int nr_running_ctxs;
static bool running_exclusive_ctxs;
+/*
+ * Single-consumer owner of each global report ring. A ring is a destructive
+ * SPSC channel that must be drained by exactly one kdamond. The pf ring is
+ * claimed by the (single) ctx whose page_fault primitive is enabled; the perf
+ * ring by the (single) ctx that has event-driven probes. Because the two
+ * rings are independent, they have independent owners: distinct ctxs may
+ * concurrently own the pf ring and the perf ring, but no ring may be shared
+ * by two draining ctxs. Accessed only under damon_lock.
+ *
+ * Only the pf ring has a global owner: it is a global ring with no ctx at
+ * report time. The perf ring is per-ctx (ctx->perf_rings), so it needs no
+ * cross-ctx owner -- each ctx drains exclusively its own perf ring.
+ */
+static struct damon_ctx *damon_report_ring_owner_pf;
+
static DEFINE_MUTEX(damon_ops_lock);
static struct damon_operations damon_registered_ops[NR_DAMON_OPS];
static struct kmem_cache *damon_region_cache __ro_after_init;
-static DEFINE_MUTEX(damon_access_reports_lock);
-static struct damon_access_report damon_access_reports[
- DAMON_ACCESS_REPORTS_CAP];
-static int damon_access_reports_len;
-
/* Should be called under damon_ops_lock with id smaller than NR_DAMON_OPS */
static bool __damon_is_registered_ops(enum damon_ops_id id)
{
@@ -288,6 +402,38 @@ static bool damon_has_probe_weights(struct damon_ctx *c)
return false;
}
+/**
+ * damon_has_event_driven_probes() - return true if @ctx has any event-driven
+ * probes registered.
+ *
+ * Event-driven probes (e.g. perf-event IBS/PEBS) populate probe_hits[] via
+ * the SPSC ring drain rather than the apply_probes vtable. Callers use this
+ * to decide whether to arm hardware sampling.
+ */
+bool damon_has_event_driven_probes(struct damon_ctx *ctx)
+{
+ struct damon_probe *p;
+
+ damon_for_each_probe(p, ctx) {
+ if (p->event_driven)
+ return true;
+ }
+ return false;
+}
+EXPORT_SYMBOL_GPL(damon_has_event_driven_probes);
+
+/* Does @ctx drive (and thus need exclusive drain of) the pf report ring? */
+static bool damon_drains_ring_pf(struct damon_ctx *ctx)
+{
+ return ctx->sample_control.primitives_enabled.page_fault;
+}
+
+/* Does @ctx drive (and thus need exclusive drain of) the perf report ring? */
+static bool damon_drains_ring_perf(struct damon_ctx *ctx)
+{
+ return damon_has_event_driven_probes(ctx);
+}
+
/*
* damon_mvsum() - Returns pseudo moving sum value for a time window.
* @current_nr: The value of the current aggregation window.
@@ -973,13 +1119,6 @@ static struct damon_sample_filter *damon_nth_sample_filter(int n,
return NULL;
}
-static struct damon_sample_filter *damon_last_sample_filter_or_null(
- struct damon_sample_control *ctrl)
-{
- return list_last_entry_or_null(&ctrl->sample_filters,
- struct damon_sample_filter, list);
-}
-
struct damon_ctx *damon_new_ctx(void)
{
struct damon_ctx *ctx;
@@ -1024,6 +1163,40 @@ struct damon_ctx *damon_new_ctx(void)
return ctx;
}
+/*
+ * Lazily allocate the per-ctx perf report ring. Called from the perf probe
+ * setup path BEFORE any perf event is armed, so an overflow can never observe
+ * a half-built ring. Idempotent: a ctx with several perf probes allocates
+ * once. The ring is freed in damon_destroy_ctx() after all perf events are
+ * released (damon_perf_probe_teardown), so no in-flight NMI can reach it.
+ */
+int damon_ctx_alloc_perf_ring(struct damon_ctx *ctx)
+{
+ if (ctx->perf_rings)
+ return 0; /* already allocated for an earlier probe */
+ ctx->perf_rings = alloc_percpu(struct damon_report_ring);
+ if (!ctx->perf_rings)
+ return -ENOMEM;
+ ctx->perf_ring_busy = alloc_percpu(int);
+ if (!ctx->perf_ring_busy) {
+ free_percpu(ctx->perf_rings);
+ ctx->perf_rings = NULL;
+ return -ENOMEM;
+ }
+ cpumask_clear(&ctx->perf_pending);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(damon_ctx_alloc_perf_ring);
+
+/* Free the per-ctx perf ring. Caller must ensure no perf event is armed. */
+static void damon_ctx_free_perf_ring(struct damon_ctx *ctx)
+{
+ free_percpu(ctx->perf_rings);
+ ctx->perf_rings = NULL;
+ free_percpu(ctx->perf_ring_busy);
+ ctx->perf_ring_busy = NULL;
+}
+
static void damon_destroy_targets(struct damon_ctx *ctx)
{
struct damon_target *t, *next_t;
@@ -1049,6 +1222,16 @@ void damon_destroy_ctx(struct damon_ctx *ctx)
damon_for_each_sample_filter_safe(f, next_f, &ctx->sample_control)
damon_destroy_sample_filter(f, &ctx->sample_control);
+ /*
+ * All perf events were released by damon_perf_probe_teardown() in the
+ * probe loop above, so no overflow handler can still reach the ring.
+ * Safe to free now, before kfree(ctx). No-op if never allocated.
+ */
+ damon_ctx_free_perf_ring(ctx);
+
+ /* Free the reusable ring-drain region snapshot buffers. */
+ kfree(ctx->drain_snapshot.lookups);
+ kfree(ctx->drain_snapshot.region_buf);
kfree(ctx);
}
@@ -2061,7 +2244,8 @@ static int damon_commit_sample_control(
return damon_commit_sample_filters(dst, src);
}
-static int __damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src)
+static int __damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src,
+ bool commit_live)
{
int err;
struct damos *scheme;
@@ -2116,6 +2300,37 @@ static int __damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src)
&src->sample_control);
if (err)
return err;
+
+ /*
+ * A running ctx that (still) drives a global report ring must hold sole
+ * ownership of it; enforce it here since damon_start()'s claim only
+ * covers the not-running -> running transition. This runs only for the
+ * live commit (commit_live) into the running dst; dst's probes and
+ * sample_control were already committed above, so damon_drains_ring_*()
+ * reflect the post-commit config. A not-running dst claims ownership
+ * later, in damon_start(). Each ring is claimed/released independently.
+ */
+ if (commit_live && damon_is_running(dst)) {
+ mutex_lock(&damon_lock);
+ if (damon_drains_ring_pf(dst)) {
+ if (damon_report_ring_owner_pf &&
+ damon_report_ring_owner_pf != dst) {
+ mutex_unlock(&damon_lock);
+ return -EBUSY;
+ }
+ damon_report_ring_owner_pf = dst;
+ } else if (damon_report_ring_owner_pf == dst) {
+ damon_report_ring_owner_pf = NULL;
+ }
+ /*
+ * No perf ring owner check: the perf ring is per-ctx, so a live
+ * commit that (re)configures perf probes never contends a shared
+ * ring. This is what lets a running perf-driven ctx accept a
+ * commit without the cross-ctx -EBUSY the global ring imposed.
+ */
+ mutex_unlock(&damon_lock);
+ }
+
dst->addr_unit = src->addr_unit;
dst->min_region_sz = src->min_region_sz;
@@ -2131,7 +2346,7 @@ static struct damon_ctx *damon_new_test_ctx(struct damon_ctx *dst)
test_ctx = damon_new_ctx();
if (!test_ctx)
return NULL;
- err = __damon_commit_ctx(test_ctx, dst);
+ err = __damon_commit_ctx(test_ctx, dst, false);
if (err) {
damon_destroy_ctx(test_ctx);
return NULL;
@@ -2160,10 +2375,10 @@ int damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src)
test_ctx = damon_new_test_ctx(dst);
if (!test_ctx)
return -ENOMEM;
- err = __damon_commit_ctx(test_ctx, src);
+ err = __damon_commit_ctx(test_ctx, src, false);
if (err)
goto out;
- err = __damon_commit_ctx(dst, src);
+ err = __damon_commit_ctx(dst, src, true);
out:
damon_destroy_ctx(test_ctx);
return err;
@@ -2257,6 +2472,37 @@ static int __damon_start(struct damon_ctx *ctx)
return err;
}
+/*
+ * Claim the owner of ONE global report ring for a damon_start() batch.
+ * @drains: per-ring predicate (damon_drains_ring_pf / _perf).
+ * @owner: per-ring owner slot.
+ *
+ * A ring is a destructive drain owned by exactly one kdamond. Reject the
+ * start (-EBUSY) if another already-running ctx owns this ring, or if more
+ * than one ctx in this batch drains it. On success @owner points at the sole
+ * draining ctx (or is left NULL if no batch ctx drains this ring).
+ * Must be called under damon_lock. Returns 0 or -EBUSY.
+ */
+static int damon_claim_ring_owner_start(struct damon_ctx **ctxs, int nr_ctxs,
+ bool (*drains)(struct damon_ctx *), struct damon_ctx **owner)
+{
+ int i, j;
+
+ for (i = 0; i < nr_ctxs; i++) {
+ if (!drains(ctxs[i]))
+ continue;
+ if (*owner)
+ return -EBUSY;
+ for (j = i + 1; j < nr_ctxs; j++) {
+ if (drains(ctxs[j]))
+ return -EBUSY;
+ }
+ *owner = ctxs[i];
+ break;
+ }
+ return 0;
+}
+
/**
* damon_start() - Starts the monitorings for a given group of contexts.
* @ctxs: an array of the pointers for contexts to start monitoring
@@ -2284,7 +2530,7 @@ int damon_start(struct damon_ctx **ctxs, int nr_ctxs, bool exclusive)
if (!test_ctx)
return -ENOMEM;
- err = __damon_commit_ctx(test_ctx, ctxs[i]);
+ err = __damon_commit_ctx(test_ctx, ctxs[i], false);
damon_destroy_ctx(test_ctx);
if (err)
return err;
@@ -2297,11 +2543,43 @@ int damon_start(struct damon_ctx **ctxs, int nr_ctxs, bool exclusive)
return -EBUSY;
}
- for (i = 0; i < nr_ctxs; i++) {
- err = __damon_start(ctxs[i]);
- if (err)
- break;
- nr_running_ctxs++;
+ /*
+ * Each global report ring is drained by exactly one kdamond. Claim the
+ * pf and perf ring owners independently; distinct ctxs may own the two
+ * rings, but neither ring may be shared by two draining ctxs. Remember
+ * whether this batch newly set each owner so the failure paths release
+ * only those, never another batch's still-running owner.
+ */
+ {
+ struct damon_ctx *prev_owner_pf = damon_report_ring_owner_pf;
+
+ err = damon_claim_ring_owner_start(ctxs, nr_ctxs,
+ damon_drains_ring_pf,
+ &damon_report_ring_owner_pf);
+ if (err) {
+ /* Release only owners this batch just claimed. */
+ if (damon_report_ring_owner_pf != prev_owner_pf)
+ damon_report_ring_owner_pf = prev_owner_pf;
+ mutex_unlock(&damon_lock);
+ return err;
+ }
+
+ for (i = 0; i < nr_ctxs; i++) {
+ err = __damon_start(ctxs[i]);
+ if (err)
+ break;
+ nr_running_ctxs++;
+ }
+ /*
+ * A ctx that failed to start has no running kdamond, so its
+ * exit path will not release the ring owner assigned above.
+ * Release each owner set above whose kdamond is not running.
+ */
+ if (err) {
+ if (damon_report_ring_owner_pf != prev_owner_pf &&
+ !damon_report_ring_owner_pf->kdamond)
+ damon_report_ring_owner_pf = prev_owner_pf;
+ }
}
if (exclusive && nr_running_ctxs)
running_exclusive_ctxs = true;
@@ -2483,30 +2761,121 @@ int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control)
* damon_report_access() - Report identified access events to DAMON.
* @report: The reporting access information.
*
- * Report access events to DAMON.
+ * Report access events to DAMON via a per-CPU SPSC lockless ring. Producer
+ * is the local CPU (typically NMI from a hardware-sampling backend);
+ * consumer is the kdamond drain in kdamond_check_reported_accesses().
+ *
+ * The destination ring is selected by this_cpu_ptr(), i.e. by the CPU calling
+ * this function, not by @report->cpu, which is sample metadata used by the
+ * drain-side filter. The two coincide for a sample delivered by an interrupt
+ * on the CPU that produced it.
*
- * Context: May sleep.
+ * A backend whose PMU writes a record stream into a memory buffer instead of
+ * raising a per-sample interrupt, or one reading a device counter table, must
+ * therefore decode CPU N's buffer on CPU N -- for example by queueing per-CPU
+ * work with queue_work_on() -- rather than calling this function in a loop
+ * from one thread. A single-thread loop puts every report in that thread's
+ * ring, which caps machine-wide capacity at DAMON_REPORT_RING_SIZE - 1
+ * reports per drain regardless of the number of producing CPUs, and does not
+ * satisfy the single-producer invariant if the thread can migrate.
*
- * NOTE: we may be able to implement this as a lockless queue, and allow any
- * context. As the overhead is unknown, and region-based DAMON logics would
- * guarantee the reports would be not made that frequently, let's start with
- * this simple implementation.
+ * Context: any (NMI-safe). An NMI nesting on top of a process-context
+ * producer on the same CPU would otherwise stomp the same entries[head]
+ * slot; the busy guard detects and drops in that case.
+ *
+ * If the ring is full, the sample is dropped and the per-CPU ring-full
+ * counter incremented; a busy-guard drop increments the busy-drop counter.
+ *
+ * Return: true if the report was queued, false if it was dropped. A producer
+ * holding a single report may ignore this. A producer decoding a batch out
+ * of a hardware buffer should stop on false and leave the remainder in that
+ * buffer for the next round, since a report released from the buffer but not
+ * queued here is not delivered.
*/
-void damon_report_access(struct damon_access_report *report)
+bool damon_report_access(struct damon_access_report *report)
{
- struct damon_access_report *dst;
+ /*
+ * Route by the globally-meaningful probe_idx boundary:
+ * probe_idx == DAMON_PROBE_IDX_NONE (0) -> pf ring (page_fault)
+ * probe_idx >= 1 -> perf ring (event probes)
+ * The two rings are fully independent (own storage, busy flag, pending
+ * mask, overflow counter); only the selected per-CPU symbols differ.
+ * The NMI ordering sequence below is a single copy shared by both rings
+ * so the subtle producer/consumer barrier pairing cannot drift.
+ */
+ bool is_perf = report->probe_idx != DAMON_PROBE_IDX_NONE;
+ struct damon_report_ring *ring;
+ cpumask_t *pending;
+ int __percpu *busy_pcpu;
+ unsigned int head, next;
+ int busy;
+ bool queued = false;
+ struct damon_ctx *pctx = is_perf ? report->ctx : NULL;
- /* silently fail for races */
- if (!mutex_trylock(&damon_access_reports_lock))
- return;
- dst = &damon_access_reports[damon_access_reports_len++];
- /* just drop all existing reports in favor of simplicity. */
- if (damon_access_reports_len == DAMON_ACCESS_REPORTS_CAP)
- damon_access_reports_len = 0;
- *dst = *report;
- dst->report_jiffies = jiffies;
- mutex_unlock(&damon_access_reports_lock);
+ /*
+ * A perf report must carry its owning ctx (set by the overflow handler)
+ * and that ctx must have an allocated per-ctx perf ring. If either is
+ * missing (e.g. an overflow racing teardown after the ring was freed, or
+ * a report raised before the ring was allocated), drop the sample rather
+ * than touch NULL/freed storage.
+ */
+ if (is_perf && (!pctx || !pctx->perf_rings || !pctx->perf_ring_busy))
+ return false;
+
+ /* Pin to a CPU so the SPSC invariant holds for preemptible callers. */
+ preempt_disable();
+ if (is_perf)
+ busy_pcpu = pctx->perf_ring_busy;
+ else
+ busy_pcpu = &damon_report_ring_busy_pf;
+ busy = this_cpu_inc_return(*busy_pcpu);
+ if (busy != 1) {
+ /* NMI nested on a process-context producer; drop. */
+ if (is_perf)
+ this_cpu_inc(damon_report_busy_drop_perf);
+ else
+ this_cpu_inc(damon_report_busy_drop_pf);
+ goto out;
+ }
+
+ if (is_perf) {
+ ring = this_cpu_ptr(pctx->perf_rings);
+ pending = &pctx->perf_pending;
+ } else {
+ ring = this_cpu_ptr(&damon_report_rings_pf);
+ pending = &damon_rings_pending_pf;
+ }
+ head = ring->head;
+ next = (head + 1) & DAMON_REPORT_RING_MASK;
+
+ if (next == READ_ONCE(ring->tail)) {
+ if (is_perf)
+ this_cpu_inc(damon_report_ring_full_perf);
+ else
+ this_cpu_inc(damon_report_ring_full_pf);
+ goto out;
+ }
+
+ ring->entries[head] = *report;
+ ring->entries[head].report_jiffies = jiffies;
+ smp_wmb(); /* publish entry before head advance */
+ WRITE_ONCE(ring->head, next);
+ /*
+ * Order the head advance before publishing the pending bit so
+ * that the consumer, on observing the bit, is also guaranteed
+ * to observe the new head. cpumask_set_cpu / set_bit are
+ * documented as unordered RMW (atomic_bitops.txt), hence the
+ * explicit barrier.
+ */
+ smp_mb__before_atomic();
+ cpumask_set_cpu(smp_processor_id(), pending);
+ queued = true;
+out:
+ this_cpu_dec(*busy_pcpu);
+ preempt_enable();
+ return queued;
}
+EXPORT_SYMBOL_GPL(damon_report_access);
#ifdef CONFIG_MMU
void damon_report_page_fault(struct vm_fault *vmf, bool huge_pmd)
@@ -2515,7 +2884,9 @@ void damon_report_page_fault(struct vm_fault *vmf, bool huge_pmd)
.vaddr = vmf->address,
.size = 1, /* todo: set appripriately */
.tid = task_pid_vnr(current),
+ .tgid = task_tgid_vnr(current),
.is_write = vmf->flags & FAULT_FLAG_WRITE,
+ /* probe_idx intentionally zero (DAMON_PROBE_IDX_NONE): no probe */
};
if (huge_pmd)
@@ -4187,6 +4558,24 @@ static void kdamond_init_ctx(struct damon_ctx *ctx)
}
}
+static unsigned int kdamond_apply_zero_access_report(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+ struct damon_region *r;
+ unsigned int max_nr_accesses = 0;
+
+ damon_for_each_target(t, ctx) {
+ damon_for_each_region(r, t) {
+ if (r->access_reported)
+ r->access_reported = false;
+ else
+ damon_update_region_access_rate(r, false);
+ max_nr_accesses = max(max_nr_accesses, r->nr_accesses);
+ }
+ }
+ return max_nr_accesses;
+}
+
static bool damon_sample_filter_matching(struct damon_access_report *report,
struct damon_sample_filter *filter)
{
@@ -4214,89 +4603,301 @@ static bool damon_sample_filter_matching(struct damon_access_report *report,
return matched == filter->matching;
}
+/*
+ * Decide whether a drained report should be dropped per the ctx sample
+ * filters. A matching "!allow" (filter-out) filter drops the report; if no
+ * filter matched, the last filter's @allow acts as the default policy.
+ */
static bool damon_sample_filter_out(struct damon_access_report *report,
struct damon_sample_control *ctrl)
{
- struct damon_sample_filter *filter;
+ struct damon_sample_filter *filter, *last = NULL;
damon_for_each_sample_filter(filter, ctrl) {
+ last = filter;
if (damon_sample_filter_matching(report, filter) &&
!filter->allow)
return true;
}
- filter = damon_last_sample_filter_or_null(ctrl);
- if (!filter)
+ if (!last)
return false;
- return !filter->allow;
+ return !last->allow;
}
-static void kdamond_apply_access_report(struct damon_access_report *report,
- struct damon_target *t, struct damon_ctx *ctx)
+/*
+ * Build a snapshot of the ctx's targets and their region arrays for use by
+ * the ring drain loop. The snapshot buffer is reused across ticks, grown via
+ * krealloc only when a new high water mark is reached.
+ *
+ * The two-pass walk over adaptive_targets is safe even though krealloc_array()
+ * may sleep: target list mutation is funneled through damon_call onto the
+ * kdamond itself, so no other thread can mutate the list while kdamond runs
+ * this function. Regions within a target are kept address-sorted by DAMON, so
+ * the snapshot arrays are directly binary-searchable.
+ */
+static struct damon_target_lookup *damon_build_target_lookup(
+ struct damon_ctx *ctx, unsigned int *nr_targets_out)
{
- struct damon_region *r;
- unsigned long addr;
+ struct damon_target *t;
+ struct damon_target_lookup *tbl;
+ unsigned int nr_targets = 0, total_regions = 0, ti = 0, ri = 0;
- if (damon_sample_filter_out(report, &ctx->sample_control))
- return;
- if (damon_target_has_pid(ctx))
- addr = report->vaddr;
- else
- addr = report->paddr;
+ damon_for_each_target(t, ctx) {
+ nr_targets++;
+ total_regions += damon_nr_regions(t);
+ }
- /* todo: make search faster, e.g., binary search? */
- damon_for_each_region(r, t) {
- if (addr < r->ar.start)
- continue;
- if (r->ar.end < addr + report->size)
- continue;
- if (!r->access_reported)
- damon_update_region_access_rate(r, true);
- r->access_reported = true;
+ if (nr_targets > ctx->drain_snapshot.nr_lookups) {
+ tbl = krealloc_array(ctx->drain_snapshot.lookups,
+ nr_targets, sizeof(*tbl), GFP_KERNEL);
+ if (!tbl)
+ return NULL;
+ ctx->drain_snapshot.lookups = tbl;
+ ctx->drain_snapshot.nr_lookups = nr_targets;
+ }
+ tbl = ctx->drain_snapshot.lookups;
+
+ if (total_regions > ctx->drain_snapshot.region_buf_cap) {
+ struct damon_region **buf;
+
+ buf = krealloc_array(ctx->drain_snapshot.region_buf,
+ total_regions, sizeof(*buf), GFP_KERNEL);
+ if (!buf)
+ return NULL;
+ ctx->drain_snapshot.region_buf = buf;
+ ctx->drain_snapshot.region_buf_cap = total_regions;
+ }
+
+ damon_for_each_target(t, ctx) {
+ struct damon_region *r;
+
+ tbl[ti].regions = &ctx->drain_snapshot.region_buf[ri];
+ tbl[ti].nr_regions = damon_nr_regions(t);
+ damon_for_each_region(r, t)
+ ctx->drain_snapshot.region_buf[ri++] = r;
+ ti++;
}
+
+ *nr_targets_out = nr_targets;
+ return tbl;
}
-static unsigned int kdamond_apply_zero_access_report(struct damon_ctx *ctx)
-{
+/*
+ * Binary-search a sorted region snapshot for the region containing @addr and,
+ * on a hit, credit the report to @pidx. Returns true if a region was credited
+ * (straddling reports that spill past the region end are rejected).
+ */
+static bool damon_credit_report_bsearch(struct damon_region **regions,
+ unsigned int nr_regions, unsigned long addr,
+ unsigned long size, int pidx)
+{
+ struct damon_region *r = NULL;
+ int left = 0, right = (int)nr_regions - 1, mid;
+
+ while (left <= right) {
+ /* Avoid (left + right) overflow at large nr_regions. */
+ mid = left + (right - left) / 2;
+ if (addr < regions[mid]->ar.start)
+ right = mid - 1;
+ else if (addr >= regions[mid]->ar.end)
+ left = mid + 1;
+ else {
+ r = regions[mid];
+ break;
+ }
+ }
+ if (!r)
+ return false;
+ /* Reject reports straddling a region boundary. */
+ if (addr + size > r->ar.end)
+ return false;
+
+ /*
+ * pidx == 0 (page_fault / non-probe): credit access rate only.
+ * Ring probe_idx is 1-based (0 == DAMON_PROBE_IDX_NONE sentinel), but
+ * probe_hits[] storage is 0-based to match all readers (wsum, mvsum,
+ * update, aggregate reset, merge). Convert here: probe_hits[pidx - 1].
+ */
+ if (pidx > 0)
+ r->probe_hits[pidx - 1]++;
+ damon_update_region_access_rate(r, true);
+ r->access_reported = true;
+ return true;
+}
+
+/*
+ * __kdamond_drain_ring - drain ONE global per-CPU SPSC ring into region probe_hits.
+ * @ctx: draining context (owns @ring for this run).
+ * @tbl: pre-built sorted per-target region snapshot (shared).
+ * @ring_pcpu: the per-CPU ring base (pf or perf ring).
+ * @pending: the matching per-ring pending cpumask.
+ *
+ * The global page-fault ring and each context's per-CPU perf ring
+ * (ctx->perf_rings) are partitioned by probe_idx but drained by
+ * an identical loop; this helper is that loop, parameterised only by which
+ * ring + pending mask to consume. The bsearch / straddle-reject / tid-filter /
+ * stale-window / sample-filter / probe_hits-crediting logic is unchanged from
+ * the prior single-ring drain.
+ *
+ * Each ring entry carries its own probe_idx (set by the overflow handler), so
+ * no list walk is needed to resolve the probe_hits[] slot.
+ *
+ * A per-target sorted region snapshot is built once per drain (by the caller)
+ * so each entry is matched to its region via O(log R) binary search rather
+ * than a linear damon_for_each_region() walk. Iterates the ring's pending
+ * cpumask to drain only CPUs with published reports.
+ */
+static void __kdamond_drain_ring(struct damon_ctx *ctx,
+ struct damon_target_lookup *tbl,
+ struct damon_report_ring __percpu *ring_pcpu,
+ cpumask_t *pending)
+{
+ int cpu;
+ struct damon_report_ring *ring;
+ unsigned int tail, head;
+ struct damon_access_report *entry;
struct damon_target *t;
- struct damon_region *r;
- unsigned int max_nr_accesses = 0;
+ unsigned long match_addr;
+ bool found;
+ unsigned int ti;
- damon_for_each_target(t, ctx) {
- damon_for_each_region(r, t) {
- if (r->access_reported)
- r->access_reported = false;
+ /*
+ * Unified paddr/vaddr drain. The address space of the monitoring
+ * target selects which address of the report is matched: contexts
+ * whose targets carry a pid are monitoring a virtual address space and
+ * match report->vaddr, the others match report->paddr.
+ *
+ * For pid-target contexts, filter by thread group id so entries from
+ * unrelated processes are not credited to the wrong target.
+ * damon_target_has_pid(ctx) gates this filter: it is false for paddr
+ * ops, whose targets have no pid, so paddr crediting is unfiltered.
+ */
+ for_each_cpu(cpu, pending) {
+ ring = per_cpu_ptr(ring_pcpu, cpu);
+ cpumask_clear_cpu(cpu, pending);
+ /*
+ * Pair with the producer's smp_mb__before_atomic() between
+ * the head publish and cpumask_set_cpu(): order the bit clear
+ * before the head read so a producer publishing between the
+ * clear and the READ_ONCE(head) is observed via the bit it
+ * re-sets, not lost as a stale-head drain.
+ */
+ smp_mb__after_atomic();
+ head = READ_ONCE(ring->head);
+ smp_rmb(); /* pair with smp_wmb in producer */
+ tail = ring->tail;
+
+ while (tail != head) {
+ unsigned long stale_before;
+ int pidx;
+
+ entry = &ring->entries[tail];
+ /*
+ * Use sample_interval (not aggr_interval) as the
+ * staleness window: entries older than one sample
+ * interval are from a previous monitoring tick and
+ * should not inflate the current aggregation window.
+ */
+ stale_before = jiffies -
+ usecs_to_jiffies(ctx->attrs.sample_interval);
+ if (time_before(entry->report_jiffies, stale_before)) {
+ this_cpu_inc(damon_samples_stale_drained);
+ goto next;
+ }
+ pidx = entry->probe_idx;
+ /*
+ * probe_idx == 0 (DAMON_PROBE_IDX_NONE) is the
+ * page_fault / non-probe credit path: no probe_hits[]
+ * slot, but it still credits the region access rate.
+ * Reject only out-of-range indices (>= DAMON_MAX_PROBES)
+ * and, defensively, any negative value.
+ */
+ if (pidx < 0 || pidx >= DAMON_MAX_PROBES)
+ goto next;
+
+ /* Drop reports rejected by the ctx sample filters. */
+ if (damon_sample_filter_out(entry, &ctx->sample_control))
+ goto next;
+
+ /*
+ * Select the address that matches the address space
+ * the targets of this context are monitoring. A
+ * report that carries no address for that space
+ * cannot be credited.
+ */
+ if (damon_target_has_pid(ctx))
+ match_addr = entry->vaddr;
else
- damon_update_region_access_rate(r, false);
- max_nr_accesses = max(max_nr_accesses, r->nr_accesses);
+ match_addr = entry->paddr;
+ if (!match_addr)
+ goto next;
+
+ found = false;
+ ti = 0;
+ damon_for_each_target(t, ctx) {
+ /* pid targets: match the reporting process */
+ if (damon_target_has_pid(ctx) &&
+ pid_vnr(t->pid) != entry->tgid) {
+ ti++;
+ continue;
+ }
+ if (damon_credit_report_bsearch(tbl[ti].regions,
+ tbl[ti].nr_regions, match_addr,
+ entry->size, pidx)) {
+ this_cpu_inc(damon_samples_drained);
+ found = true;
+ break;
+ }
+ ti++;
+ }
+ if (!found)
+ this_cpu_inc(damon_samples_no_region);
+next:
+ tail = (tail + 1) & DAMON_REPORT_RING_MASK;
}
+ WRITE_ONCE(ring->tail, tail);
}
- return max_nr_accesses;
}
-static unsigned int kdamond_check_reported_accesses(struct damon_ctx *ctx)
+/*
+ * kdamond_check_reported_accesses - drain the global report ring(s) this ctx feeds.
+ * Called from kdamond main loop after each sampling interval.
+ *
+ * The global page-fault ring and each context's per-CPU perf ring
+ * (ctx->perf_rings) are partitioned by probe_idx: the pf ring holds
+ * page_fault (probe_idx == DAMON_PROBE_IDX_NONE) reports, the perf ring holds
+ * event-driven probe (probe_idx >= 1) reports. A ctx drains the ring(s) its
+ * enabled primitives feed:
+ * - page_fault primitive enabled -> drain pf ring.
+ * - event-driven probes registered -> drain perf ring.
+ * - both enabled -> drain both.
+ *
+ * The per-target sorted region snapshot is built once and shared across both
+ * ring drains (it is ring-agnostic).
+ */
+static void kdamond_check_reported_accesses(struct damon_ctx *ctx)
{
- int i;
- struct damon_access_report *report;
- struct damon_target *t;
+ struct damon_target_lookup *tbl;
+ unsigned int nr_targets = 0;
- /* currently damon_access_report supports only physical address */
- if (damon_target_has_pid(ctx))
- return 0;
-
- mutex_lock(&damon_access_reports_lock);
- for (i = 0; i < damon_access_reports_len; i++) {
- report = &damon_access_reports[i];
- if (time_before(report->report_jiffies,
- jiffies -
- usecs_to_jiffies(
- ctx->attrs.sample_interval)))
- continue;
- damon_for_each_target(t, ctx)
- kdamond_apply_access_report(report, t, ctx);
+ /*
+ * Build the sorted region snapshot once for this drain. If the alloc
+ * fails, skip the drain this tick rather than falling back to a linear
+ * scan (a missed tick self-heals; a linear scan does not).
+ */
+ tbl = damon_build_target_lookup(ctx, &nr_targets);
+ if (!tbl) {
+ pr_warn_ratelimited(
+ "damon: target-lookup alloc failed; ring drain skipped this tick\n");
+ return;
}
- mutex_unlock(&damon_access_reports_lock);
- /* For nr_accesses_bp, absence of access should also be reported. */
- return kdamond_apply_zero_access_report(ctx);
+
+ if (damon_drains_ring_pf(ctx))
+ __kdamond_drain_ring(ctx, tbl, &damon_report_rings_pf,
+ &damon_rings_pending_pf);
+ if (damon_drains_ring_perf(ctx))
+ __kdamond_drain_ring(ctx, tbl, ctx->perf_rings,
+ &ctx->perf_pending);
}
/*
@@ -4345,7 +4946,9 @@ static int kdamond_fn(void *data)
do_prep = ctx->ops.prep_probes && damon_has_prep(ctx);
- if (!access_check_disabled && ctx->ops.prepare_access_checks)
+ /* Page-fault sampling installs its markers from this callback. */
+ if ((!access_check_disabled || damon_drains_ring_pf(ctx)) &&
+ ctx->ops.prepare_access_checks)
ctx->ops.prepare_access_checks(ctx);
if (do_prep)
ctx->ops.prep_probes(ctx, access_check_disabled);
@@ -4353,14 +4956,18 @@ static int kdamond_fn(void *data)
kdamond_usleep(sample_interval);
ctx->passed_sample_intervals++;
- if (!access_check_disabled) {
- /* todo: make these non-exclusive */
- if (ctx->sample_control.primitives_enabled.page_fault)
- max_merge_score =
- kdamond_check_reported_accesses(ctx);
- else if (ctx->ops.check_accesses)
- max_merge_score = ctx->ops.check_accesses(ctx);
- }
+ /*
+ * Both perf-event and page-fault primitives feed damon_report_access()
+ * into the global per-CPU ring; the same drain consumes both.
+ */
+ if (damon_drains_ring_perf(ctx) || damon_drains_ring_pf(ctx))
+ kdamond_check_reported_accesses(ctx);
+
+ /* Page-fault sampling reports only the accessed regions. */
+ if (damon_drains_ring_pf(ctx))
+ max_merge_score = kdamond_apply_zero_access_report(ctx);
+ else if (!access_check_disabled && ctx->ops.check_accesses)
+ max_merge_score = ctx->ops.check_accesses(ctx);
if (ctx->ops.apply_probes) {
if (time_after_eq(ctx->passed_sample_intervals,
@@ -4475,6 +5082,9 @@ static int kdamond_fn(void *data)
nr_running_ctxs--;
if (!nr_running_ctxs && running_exclusive_ctxs)
running_exclusive_ctxs = false;
+ /* Release the global pf report ring if this ctx owned it. */
+ if (damon_report_ring_owner_pf == ctx)
+ damon_report_ring_owner_pf = NULL;
mutex_unlock(&damon_lock);
return 0;
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 6/9] mm/damon: add perf-event overflow handler feeding the report ring
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
` (4 preceding siblings ...)
2026-09-10 17:16 ` [RFC PATCH v2 5/9] mm/damon/core: add per-probe-class report rings and unified drain Ravi Jonnalagadda
@ 2026-09-10 17:16 ` Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 7/9] mm/damon/ops-common: use probe-weighted score when probe weights are set Ravi Jonnalagadda
` (4 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Ravi Jonnalagadda @ 2026-09-10 17:16 UTC (permalink / raw)
To: sj, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc
Cc: akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
ravis.opensrc, rientjes, weixugc, jic23, gourry
Add mm/damon/perf_source.c, an ops-agnostic perf-event source that turns
PMU overflow samples into DAMON access reports. The overflow handler runs
in NMI context and reports through damon_report_access(), so it works with
either paddr or vaddr ops.
The handler sets whichever address fields the PMU provides: paddr when
PERF_SAMPLE_PHYS_ADDR is set, vaddr when PERF_SAMPLE_ADDR is set, plus the
CPU it runs on. The unified drain matches on whichever is present. It
reports both the thread id and the thread group id, because a VA-only PMU
samples whichever thread of a process happened to access memory while the
drain matches a target created for that process.
Per-CPU perf events are armed and released through cpuhp callbacks, so a
CPU coming online while a probe is armed gets a counter and a CPU going
offline releases its own.
damon_perf_probe_alloc() allocates a perf-event-backed damon_probe with
event_driven set, so its hits are credited through the report-ring drain
rather than the apply_probes vtable; callers set the perf attributes after
allocation. Both ops sets honour that flag in their probe vtables: an
event-driven probe has no software prep action, and is skipped in the
apply_probes loop, so the drain is the only thing that credits its
probe_hits[].
damon_perf_probe_setup() computes probe_idx by walking ctx->probes,
assigning 1-based indices because 0 is the zero-init sentinel
DAMON_PROBE_IDX_NONE, and the overflow handler warns once if it sees 0.
A refcounted per-PMU owner tracks which context owns each PMU type: a
second context claiming the same type gets -EBUSY, and ownership is
released when the last probe for that type is torn down. Module exit
refuses to unload while any owner remains.
Co-developed-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@gmail.com>
---
mm/damon/Kconfig | 18 ++
mm/damon/Makefile | 1 +
mm/damon/paddr.c | 19 +-
mm/damon/perf_source.c | 402 +++++++++++++++++++++++++++++++++++++++++
mm/damon/perf_source.h | 26 +++
mm/damon/vaddr.c | 18 +-
6 files changed, 482 insertions(+), 2 deletions(-)
create mode 100644 mm/damon/perf_source.c
create mode 100644 mm/damon/perf_source.h
diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
index c7b6f3125e79e..4a4f4381dc3b7 100644
--- a/mm/damon/Kconfig
+++ b/mm/damon/Kconfig
@@ -64,6 +64,24 @@ config DAMON_PADDR
This builds the default data access monitoring operations for DAMON
that works for the physical address space.
+config DAMON_PERF_SOURCE
+ bool "perf-event source for DAMON probes"
+ depends on DAMON && PERF_EVENTS
+ help
+ Provides a PMU-agnostic perf-event overflow handler that feeds
+ physical-address access reports into DAMON via damon_report_access().
+ Required for DAMON probe-weighted tiering with any address-sampling PMU.
+
+ The overflow handler is NMI-safe: it writes to a per-CPU SPSC
+ ring rather than taking any lock. The PMU is selected at probe
+ creation time via perf_event_attr (e.g. AMD IBS Op, Intel PEBS).
+
+ A given PMU type may be driven by at most one DAMON context at a
+ time (hardware constraint); damon_perf_probe_setup() returns -EBUSY
+ if another context already owns that PMU type. Distinct contexts may
+ each drive different PMUs concurrently, each draining its own
+ per-context report ring.
+
config DAMON_VADDR_KUNIT_TEST
bool "Test for DAMON operations" if !KUNIT_ALL_TESTS
depends on DAMON_VADDR && KUNIT=y
diff --git a/mm/damon/Makefile b/mm/damon/Makefile
index 22494754f41e8..1abf6f2a5133f 100644
--- a/mm/damon/Makefile
+++ b/mm/damon/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_DAMON_RECLAIM) += modules-common.o reclaim.o
obj-$(CONFIG_DAMON_LRU_SORT) += modules-common.o lru_sort.o
obj-$(CONFIG_DAMON_STAT) += modules-common.o stat.o
obj-$(CONFIG_DAMON_ACMA) += modules-common.o acma.o
+obj-$(CONFIG_DAMON_PERF_SOURCE) += perf_source.o
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index 5fcef3005c536..c6ac44561ee46 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -179,6 +179,16 @@ static void damon_pa_prep_probes_region(struct damon_region *r,
{
struct damon_prep *p;
+ /*
+ * Event-driven probes have no software prep action: their hits arrive
+ * asynchronously through the per-CPU report rings and are applied by
+ * the kdamond ring drain, so skip the software prep path here.
+ * Reaching this for an event-driven probe is normal (prep runs for
+ * every probe); it is silently skipped, not a warning.
+ */
+ if (probe->event_driven)
+ return;
+
damon_for_each_prep(p, probe) {
switch (p->action) {
case DAMON_PREP_SET_PGIDLE:
@@ -268,7 +278,14 @@ static unsigned int damon_pa_apply_probes(struct damon_ctx *ctx,
ctx->addr_unit);
folio = damon_get_folio(PHYS_PFN(pa));
damon_for_each_probe(p, ctx) {
- if (damon_pa_filter_pass(folio, p))
+ /*
+ * Event-driven probes skip this software apply
+ * path; their hits are counted asynchronously by
+ * the kdamond ring drain. Only sampling-based
+ * probes credit probe_hits[] here.
+ */
+ if (!p->event_driven &&
+ damon_pa_filter_pass(folio, p))
r->probe_hits[i]++;
i++;
}
diff --git a/mm/damon/perf_source.c b/mm/damon/perf_source.c
new file mode 100644
index 0000000000000..1102426152eca
--- /dev/null
+++ b/mm/damon/perf_source.c
@@ -0,0 +1,402 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * DAMON perf-event source
+ *
+ * Provides a PMU-agnostic NMI-safe overflow handler that feeds
+ * physical-address access reports into DAMON via damon_report_access().
+ * The PMU is selected at probe creation time via perf_event_attr.
+ */
+
+#include <linux/cpuhotplug.h>
+#include <linux/damon.h>
+#include <linux/module.h>
+#include <linux/perf_event.h>
+#include <linux/slab.h>
+#include "perf_source.h"
+
+/* PMU event attribute for perf-event probe configuration */
+struct damon_perf_event_attr {
+ u32 type;
+ u64 config;
+ u64 config1;
+ u64 config2;
+ bool sample_phys_addr;
+ bool sample_weight_struct;
+ bool exclude_kernel;
+ bool exclude_hv;
+ bool freq;
+ u64 sample_freq;
+ u64 sample_period;
+ u32 wakeup_events;
+ u32 precise_ip;
+};
+
+struct damon_perf_probe_event {
+ struct damon_perf_event_attr attr;
+ void *priv; /* struct damon_perf_probe_state * */
+ struct hlist_node hlist_node;
+ int probe_idx; /* index into probe_hits[]; set at registration */
+};
+
+struct damon_perf_probe_state {
+ struct perf_event * __percpu *event;
+};
+
+static DEFINE_PER_CPU(unsigned long, damon_perf_samples_total);
+static DEFINE_PER_CPU(unsigned long, damon_perf_samples_filtered);
+static DEFINE_PER_CPU(unsigned long, damon_perf_samples_no_addr);
+
+static void damon_perf_overflow(struct perf_event *perf_event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs)
+{
+ int probe_idx = (int)(unsigned long)perf_event->overflow_handler_context;
+ struct damon_access_report report = {
+ .probe_idx = probe_idx,
+ .size = PAGE_SIZE,
+ .cpu = smp_processor_id(),
+ };
+
+ /* probe_idx 0 is the zero-init sentinel; a valid index must be >= 1 */
+ if (WARN_ONCE(probe_idx == 0,
+ "damon-perf: overflow handler called with probe_idx=0\n"))
+ return;
+
+ if (!data) {
+ this_cpu_inc(damon_perf_samples_no_addr);
+ return;
+ }
+
+ /*
+ * Populate whichever address fields the PMU provides.
+ * IBS provides PA (and optionally VA); PEBS provides VA only.
+ * Gate on sample_flags rather than testing for zero: the flag is
+ * the authoritative indicator of field validity.
+ */
+ if (data->sample_flags & PERF_SAMPLE_PHYS_ADDR)
+ report.paddr = data->phys_addr & PAGE_MASK;
+ if (data->sample_flags & PERF_SAMPLE_ADDR)
+ report.vaddr = data->addr;
+
+ if (!report.paddr && !report.vaddr) {
+ this_cpu_inc(damon_perf_samples_filtered);
+ return;
+ }
+
+ report.is_write = !!(data->data_src.mem_op & PERF_MEM_OP_STORE);
+ report.tid = task_pid_vnr(current);
+ report.tgid = task_tgid_vnr(current);
+ damon_report_access(&report);
+ this_cpu_inc(damon_perf_samples_total);
+}
+
+static enum cpuhp_state damon_perf_cpuhp_state;
+
+/*
+ * Per-PMU exclusivity: each PMU type may be owned by at most one damon_ctx.
+ * Multiple probes from the same ctx sharing a PMU type are allowed; a second
+ * ctx attempting to grab a PMU type already owned returns -EBUSY.
+ */
+struct damon_pmu_owner {
+ struct list_head node;
+ u32 pmu_type;
+ atomic_long_t owner_ctx;
+ atomic_t refcount;
+};
+
+static LIST_HEAD(damon_pmu_owner_list);
+static DEFINE_SPINLOCK(damon_pmu_owner_lock);
+
+static void damon_perf_event_init_attr(struct damon_perf_probe_event *event,
+ struct perf_event_attr *attr)
+{
+ u64 stype = PERF_SAMPLE_TIME | PERF_SAMPLE_PERIOD | PERF_SAMPLE_ADDR;
+
+ /*
+ * Gate PERF_SAMPLE_PHYS_ADDR on the probe attribute: Intel PEBS
+ * has sample_phys_addr=0 by default; forcing it triggers an
+ * NMI-context pagetable walk per sample and sets entry->paddr
+ * unconditionally, which prevents the vaddr drain branch from
+ * ever being exercised.
+ */
+ if (event->attr.sample_phys_addr)
+ stype |= PERF_SAMPLE_PHYS_ADDR;
+ if (event->attr.sample_weight_struct)
+ stype |= PERF_SAMPLE_WEIGHT_STRUCT;
+ stype |= PERF_SAMPLE_DATA_SRC;
+
+ *attr = (struct perf_event_attr) {
+ .size = sizeof(*attr),
+ .type = event->attr.type,
+ .config = event->attr.config,
+ .config1 = event->attr.config1,
+ .config2 = event->attr.config2,
+ .freq = event->attr.freq,
+ .sample_type = stype,
+ .precise_ip = event->attr.precise_ip,
+ .pinned = 1,
+ /*
+ * Created disabled, and enabled by the caller once the counter
+ * is fully set up.
+ */
+ .disabled = 1,
+ .wakeup_events = event->attr.wakeup_events,
+ .exclude_kernel = event->attr.exclude_kernel,
+ .exclude_hv = event->attr.exclude_hv,
+ };
+ if (event->attr.freq)
+ attr->sample_freq = event->attr.sample_freq;
+ else
+ attr->sample_period = event->attr.sample_period;
+}
+
+static int damon_perf_cpu_online(unsigned int cpu, struct hlist_node *node)
+{
+ struct damon_perf_probe_event *event = hlist_entry(node,
+ struct damon_perf_probe_event, hlist_node);
+ struct damon_perf_probe_state *perf = event->priv;
+ struct perf_event_attr attr;
+ struct perf_event *perf_event;
+
+ if (!perf)
+ return 0;
+
+ damon_perf_event_init_attr(event, &attr);
+
+ perf_event = perf_event_create_kernel_counter(&attr, cpu, NULL,
+ damon_perf_overflow,
+ (void *)(unsigned long)event->probe_idx);
+ if (IS_ERR(perf_event)) {
+ pr_warn_ratelimited("damon-perf: cpu %u event create failed: %ld\n",
+ cpu, PTR_ERR(perf_event));
+ return 0;
+ }
+ per_cpu(*perf->event, cpu) = perf_event;
+
+ perf_event_enable(perf_event);
+ return 0;
+}
+
+static int damon_perf_cpu_offline(unsigned int cpu, struct hlist_node *node)
+{
+ struct damon_perf_probe_event *event = hlist_entry(node,
+ struct damon_perf_probe_event, hlist_node);
+ struct damon_perf_probe_state *perf = event->priv;
+ struct perf_event *perf_event;
+
+ if (!perf)
+ return 0;
+
+ perf_event = per_cpu(*perf->event, cpu);
+ if (perf_event) {
+ perf_event_disable(perf_event);
+ perf_event_release_kernel(perf_event);
+ per_cpu(*perf->event, cpu) = NULL;
+ }
+ return 0;
+}
+
+/**
+ * damon_perf_probe_setup - arm perf_events for a DAMON probe.
+ * @ctx: DAMON context that owns the probe.
+ * @probe: the damon_probe being armed; its list position in ctx->probes
+ * determines the probe_idx stored in ring entries.
+ * @event: perf event descriptor (caller fills .attr fields)
+ *
+ * Computes probe_idx by walking ctx->probes so the caller does not need
+ * to track it externally. Returns 0 on success, negative errno on failure.
+ */
+int damon_perf_probe_setup(struct damon_ctx *ctx,
+ struct damon_probe *probe,
+ struct damon_perf_probe_event *event)
+{
+ struct damon_perf_probe_state *perf;
+ struct damon_pmu_owner *owner, *found = NULL;
+ struct damon_probe *p;
+ int idx = 0;
+ int err = -ENOMEM;
+
+ /*
+ * Per-PMU exclusivity: find or create an owner slot for this PMU type.
+ * Multiple probes from the same ctx sharing a PMU type are allowed;
+ * a second ctx attempting the same PMU type returns -EBUSY.
+ */
+ spin_lock(&damon_pmu_owner_lock);
+ list_for_each_entry(owner, &damon_pmu_owner_list, node) {
+ if (owner->pmu_type == event->attr.type) {
+ long cur = atomic_long_read(&owner->owner_ctx);
+
+ if (cur != 0L && cur != (long)ctx) {
+ spin_unlock(&damon_pmu_owner_lock);
+ return -EBUSY;
+ }
+ atomic_long_set(&owner->owner_ctx, (long)ctx);
+ atomic_inc(&owner->refcount);
+ found = owner;
+ break;
+ }
+ }
+ if (!found) {
+ /*
+ * GFP_ATOMIC: this allocation runs while holding
+ * damon_pmu_owner_lock (a spinlock), so it must not sleep.
+ */
+ owner = kzalloc_obj(*owner, GFP_ATOMIC);
+ if (!owner) {
+ spin_unlock(&damon_pmu_owner_lock);
+ return -ENOMEM;
+ }
+ owner->pmu_type = event->attr.type;
+ atomic_long_set(&owner->owner_ctx, (long)ctx);
+ atomic_set(&owner->refcount, 1);
+ list_add(&owner->node, &damon_pmu_owner_list);
+ found = owner;
+ }
+ spin_unlock(&damon_pmu_owner_lock);
+
+ /* Compute probe_idx by walking ctx->probes list */
+ damon_for_each_probe(p, ctx) {
+ if (p == probe)
+ break;
+ idx++;
+ }
+ /*
+ * Probe indices are 1-based (0 is the zero-init sentinel).
+ * With DAMON_MAX_PROBES slots (0..DAMON_MAX_PROBES-1), valid probe
+ * indices are 1..DAMON_MAX_PROBES-1, so the 0-based list position
+ * must be < DAMON_MAX_PROBES-1.
+ */
+ if (idx >= DAMON_MAX_PROBES - 1) {
+ err = -ENOSPC;
+ goto release_owner;
+ }
+ event->probe_idx = idx + 1; /* 1-based; 0 is reserved sentinel */
+
+ perf = kzalloc_obj(*perf, GFP_KERNEL);
+ if (!perf)
+ goto release_owner;
+
+ perf->event = alloc_percpu(typeof(*perf->event));
+ if (!perf->event)
+ goto free_perf;
+
+ event->priv = perf;
+ INIT_HLIST_NODE(&event->hlist_node);
+
+ err = cpuhp_state_add_instance(damon_perf_cpuhp_state,
+ &event->hlist_node);
+ if (err)
+ goto free_event;
+
+ return 0;
+
+free_event:
+ free_percpu(perf->event);
+free_perf:
+ kfree(perf);
+ event->priv = NULL;
+release_owner:
+ spin_lock(&damon_pmu_owner_lock);
+ if (atomic_dec_and_test(&found->refcount)) {
+ list_del(&found->node);
+ spin_unlock(&damon_pmu_owner_lock);
+ kfree(found);
+ } else {
+ spin_unlock(&damon_pmu_owner_lock);
+ }
+ return err;
+}
+EXPORT_SYMBOL_GPL(damon_perf_probe_setup);
+
+/**
+ * damon_perf_probe_teardown - disarm perf_events.
+ * @event: perf event descriptor previously passed to damon_perf_probe_setup()
+ */
+void damon_perf_probe_teardown(struct damon_ctx *ctx,
+ struct damon_perf_probe_event *event)
+{
+ struct damon_perf_probe_state *perf = event->priv;
+ struct damon_pmu_owner *owner, *tmp;
+
+ if (!perf)
+ return;
+
+ cpuhp_state_remove_instance(damon_perf_cpuhp_state,
+ &event->hlist_node);
+ free_percpu(perf->event);
+ kfree(perf);
+ event->priv = NULL;
+
+ /*
+ * Release per-PMU ownership when the last probe for this
+ * ctx/PMU-type pair is torn down.
+ */
+ spin_lock(&damon_pmu_owner_lock);
+ list_for_each_entry_safe(owner, tmp, &damon_pmu_owner_list, node) {
+ if (owner->pmu_type == event->attr.type &&
+ atomic_long_read(&owner->owner_ctx) == (long)ctx) {
+ /*
+ * Free under the lock so a concurrent same-PMU teardown
+ * cannot observe and free the same owner. kfree() under
+ * a non-irq spinlock in process context is safe.
+ */
+ if (atomic_dec_and_test(&owner->refcount)) {
+ list_del(&owner->node);
+ kfree(owner);
+ }
+ break;
+ }
+ }
+ spin_unlock(&damon_pmu_owner_lock);
+}
+EXPORT_SYMBOL_GPL(damon_perf_probe_teardown);
+
+/**
+ * damon_perf_probe_alloc - allocate and initialise a perf-event-backed damon_probe.
+ * @weight: probe weight for probe-weighted tiering scoring.
+ *
+ * The caller sets perf->attr directly after allocation to select the PMU
+ * and sampling parameters (e.g. AMD IBS Op, Intel PEBS).
+ *
+ * Returns a new probe on success, NULL on allocation failure.
+ */
+struct damon_probe *damon_perf_probe_alloc(unsigned int weight)
+{
+ struct damon_probe *probe = damon_new_probe();
+
+ if (!probe)
+ return NULL;
+ probe->weight = weight;
+ probe->event_driven = true;
+ return probe;
+}
+EXPORT_SYMBOL_GPL(damon_perf_probe_alloc);
+
+static int __init damon_perf_source_init(void)
+{
+ int ret;
+
+ ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN,
+ "mm/damon/perf_source:online",
+ damon_perf_cpu_online,
+ damon_perf_cpu_offline);
+ if (ret < 0)
+ return ret;
+ damon_perf_cpuhp_state = ret;
+ return 0;
+}
+
+static void __exit damon_perf_source_exit(void)
+{
+ spin_lock(&damon_pmu_owner_lock);
+ if (!list_empty(&damon_pmu_owner_list)) {
+ spin_unlock(&damon_pmu_owner_lock);
+ WARN(1, "damon_perf_source: unloading with active probes\n");
+ return;
+ }
+ spin_unlock(&damon_pmu_owner_lock);
+ cpuhp_remove_multi_state(damon_perf_cpuhp_state);
+}
+
+module_init(damon_perf_source_init);
+module_exit(damon_perf_source_exit);
diff --git a/mm/damon/perf_source.h b/mm/damon/perf_source.h
new file mode 100644
index 0000000000000..5de55a29785f4
--- /dev/null
+++ b/mm/damon/perf_source.h
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * DAMON perf-event source - public interface
+ *
+ * Callers that register perf-event probes include this header.
+ */
+
+#ifndef _DAMON_PERF_SOURCE_H
+#define _DAMON_PERF_SOURCE_H
+
+#ifdef CONFIG_DAMON_PERF_SOURCE
+
+#include <linux/damon.h>
+#include <linux/perf_event.h>
+
+struct damon_perf_probe_event;
+
+int damon_perf_probe_setup(struct damon_ctx *ctx,
+ struct damon_probe *probe,
+ struct damon_perf_probe_event *event);
+void damon_perf_probe_teardown(struct damon_ctx *ctx,
+ struct damon_perf_probe_event *event);
+struct damon_probe *damon_perf_probe_alloc(unsigned int weight);
+
+#endif /* CONFIG_DAMON_PERF_SOURCE */
+#endif /* _DAMON_PERF_SOURCE_H */
diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c
index 1f2d923e569d6..3d6387d4ef7bb 100644
--- a/mm/damon/vaddr.c
+++ b/mm/damon/vaddr.c
@@ -552,6 +552,16 @@ static void damon_va_prep_probe_region(struct damon_ctx *ctx,
{
struct damon_prep *p;
+ /*
+ * Event-driven probes have no software prep action: their hits arrive
+ * asynchronously through the per-CPU report rings and are applied by
+ * the kdamond ring drain, so skip the software prep path here.
+ * Reaching this for an event-driven probe is normal (prep runs for
+ * every probe); it is silently skipped, not a warning.
+ */
+ if (probe->event_driven)
+ return;
+
damon_for_each_prep(p, probe) {
switch (p->action) {
case DAMON_PREP_SET_PGIDLE:
@@ -656,7 +666,13 @@ static void damon_va_probe_folio(struct damon_ctx *ctx,
int i = 0;
damon_for_each_probe(probe, ctx) {
- if (damon_va_filter_pass(folio, probe, pte, pmd, mm,
+ /*
+ * Event-driven probes skip this software apply path; their hits
+ * are counted asynchronously by the kdamond ring drain. Only
+ * sampling-based probes credit probe_hits[] here.
+ */
+ if (!probe->event_driven &&
+ damon_va_filter_pass(folio, probe, pte, pmd, mm,
r->sampling_addr))
r->probe_hits[i]++;
i++;
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 7/9] mm/damon/ops-common: use probe-weighted score when probe weights are set
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
` (5 preceding siblings ...)
2026-09-10 17:16 ` [RFC PATCH v2 6/9] mm/damon: add perf-event overflow handler feeding the report ring Ravi Jonnalagadda
@ 2026-09-10 17:16 ` Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 8/9] mm/damon: add perf_event prep for PMU-driven hotness probes Ravi Jonnalagadda
` (3 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Ravi Jonnalagadda @ 2026-09-10 17:16 UTC (permalink / raw)
To: sj, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc
Cc: akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
ravis.opensrc, rientjes, weixugc, jic23, gourry
When probe weights are configured (damon_has_probe_weights()), use
damon_probe_hits_wsum() for the frequency subscore in damon_hot_score()
instead of damon_nr_accesses_mvsum(). This routes hardware event counts
from a perf-event probe into the tiering decision.
When no probe weights are set the frequency subscore comes from
nr_accesses_mvsum, so page-table-only monitoring scores from the
page-table access rate.
Export damon_has_probe_weights() so ops-common.c can call it without a
static dependency on core.c internals.
Guard against integer overflow: clamp the probe-weighted subscore to
DAMON_MAX_SUBSCORE via min_t after the mult_frac scaling so a large
weighted-hit sum cannot overflow the subscore range.
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@gmail.com>
---
include/linux/damon.h | 1 +
mm/damon/core.c | 2 +-
mm/damon/ops-common.c | 21 ++++++++++++++++++---
3 files changed, 20 insertions(+), 4 deletions(-)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 88a459a60b296..3147ce30951f5 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -1307,6 +1307,7 @@ unsigned char damon_probe_hits_mvsum(int probe_idx, struct damon_region *r,
struct damon_ctx *ctx);
unsigned int damon_probe_hits_wsum(struct damon_region *r, bool last, bool mv,
struct damon_ctx *ctx);
+bool damon_has_probe_weights(struct damon_ctx *c);
int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
unsigned int nr_ranges, unsigned long min_region_sz);
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 850880f791c9e..6b3aa86386b76 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -391,7 +391,7 @@ static struct damon_probe *damon_nth_probe(int n, struct damon_ctx *ctx)
return NULL;
}
-static bool damon_has_probe_weights(struct damon_ctx *c)
+bool damon_has_probe_weights(struct damon_ctx *c)
{
struct damon_probe *p;
diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
index c36cc39cd2c70..3033a5c0ba690 100644
--- a/mm/damon/ops-common.c
+++ b/mm/damon/ops-common.c
@@ -170,9 +170,24 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r,
unsigned int age_weight = s->quota.weight_age;
int hotness;
- freq_subscore = mult_frac(damon_nr_accesses_mvsum(r, c),
- DAMON_MAX_SUBSCORE,
- damon_nr_samples_per_aggr(&c->attrs));
+ if (damon_has_probe_weights(c)) {
+ unsigned int wsum = damon_probe_hits_wsum(r, false, true, c);
+
+ /*
+ * Route perf-event hardware event counts into the score.
+ * Clamp to DAMON_MAX_SUBSCORE so a large weighted-hit sum
+ * cannot overflow the subscore range.
+ */
+ freq_subscore = min_t(int,
+ mult_frac(wsum, DAMON_MAX_SUBSCORE,
+ /* +1 guards divide-by-zero: samples-per-aggr can be 0 */
+ damon_nr_samples_per_aggr(&c->attrs) + 1),
+ DAMON_MAX_SUBSCORE);
+ } else {
+ freq_subscore = mult_frac(damon_nr_accesses_mvsum(r, c),
+ DAMON_MAX_SUBSCORE,
+ damon_nr_samples_per_aggr(&c->attrs));
+ }
age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000;
if (age_in_sec)
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 8/9] mm/damon: add perf_event prep for PMU-driven hotness probes
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
` (6 preceding siblings ...)
2026-09-10 17:16 ` [RFC PATCH v2 7/9] mm/damon/ops-common: use probe-weighted score when probe weights are set Ravi Jonnalagadda
@ 2026-09-10 17:16 ` Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 9/9] mm/damon/tests/drain-kunit: kunit for report rings and unified drain Ravi Jonnalagadda
` (2 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Ravi Jonnalagadda @ 2026-09-10 17:16 UTC (permalink / raw)
To: sj, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc
Cc: akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
ravis.opensrc, rientjes, weixugc, jic23, gourry
The prep interface exposes a single preparation action, set_pgidle, which
arms the page-idle scan used by the software access check. There is no
way for user space to configure a probe whose access information comes
from a hardware PMU sampling facility, even though the perf-event source
(CONFIG_DAMON_PERF_SOURCE) already provides the NMI-safe overflow
handler, the report ring, and the probe setup/teardown lifecycle to
consume such samples.
Add a DAMON_PREP_PERF_EVENT prep action carrying a subset of
perf_event_attr (type, config, config1, config2, sampling period or
frequency, wakeup_events, precise_ip, sample_phys_addr,
sample_weight_struct, exclude_kernel, exclude_hv), and wire it through
include/linux/damon.h, mm/damon/perf_source.h, mm/damon/sysfs.c and
mm/damon/core.c. A probe that has a perf_event prep is event driven:
DAMON does not walk it in the apply_probes vtable; instead the PMU samples
memory accesses and feeds region hit counters through the report ring.
The prep also carries how many counters the PMU needs. A per-CPU PMU is
armed by adding a cpuhp instance, which opens one kernel counter on every
online CPU. A system-wide PMU is a single hardware unit served by exactly
one counter, so a single_instance flag opens one kernel counter pinned to
a fixed online CPU and bypasses the cpuhp fan-out. The pin names a real
online CPU because such a PMU registers with perf_invalid_context, for
which cpu = -1 would route to task context. If that CPU is later
offlined the counter stops and is not migrated, which is acceptable for a
dedicated monitoring host.
The PMU is only armed when building the context that will actually run. A
param_ctx built for a commit carries the attributes but defers arming,
because arming a throwaway context would collide with the running
context's per-PMU ownership, and a commit_live flag distinguishes the
dry-run validation pass from the real commit. The commit path hands the
armed event between the running probe and the committed source probe,
keeping the running event untouched on the common weight-only commit and
re-arming only when the perf attributes change.
The armed event is released both when monitoring is turned off, from
kdamond_fn()'s exit path after damon_destroy_targets(), and when the
context is destroyed. Both run outside NMI context, so a provider's
sleeping teardown is safe, and a later turn-on re-arms the event from the
carried probe attributes. The teardown NMI barrier, an smp_store_release
of event->ctx = NULL, covers the single-instance and per-CPU paths alike;
the overflow handler reads its owning context with a matching
smp_load_acquire.
After this change user space can configure a PMU hotness probe entirely
through sysfs, for example:
echo perf_event > .../probes/0/preps/0/prep_action
echo 8 > .../probes/0/preps/0/type
echo 0x100 > .../probes/0/preps/0/sample_period
echo 1 > .../probes/0/preps/0/sample_phys_addr
Co-developed-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@gmail.com>
---
include/linux/damon.h | 29 ++++
mm/damon/core.c | 134 +++++++++++++++++-
mm/damon/perf_source.c | 195 ++++++++++++++++++--------
mm/damon/perf_source.h | 30 +++-
mm/damon/sysfs.c | 266 ++++++++++++++++++++++++++++++++++--
mm/damon/tests/core-kunit.h | 2 +-
6 files changed, 583 insertions(+), 73 deletions(-)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 3147ce30951f5..feafe58b60b24 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -876,18 +876,45 @@ struct damon_intervals_goal {
* enum damon_prep_action - DAMON probing preparation action.
*
* @DAMON_PREP_SET_PGIDLE: Set the probing memory as idle page.
+ * @DAMON_PREP_PERF_EVENT: Configure a perf-event hotness probe.
*/
enum damon_prep_action {
DAMON_PREP_SET_PGIDLE,
+ DAMON_PREP_PERF_EVENT,
};
/**
* struct damon_prep - DAMON probing preparation request.
*
* @action: Action to do to the probing memory for the preparation.
+ * @perf: perf_event_attr subset selecting the PMU and sampling
+ * parameters. Only valid when @action is DAMON_PREP_PERF_EVENT.
+ *
+ * A DAMON_PREP_PERF_EVENT prep turns the containing &struct damon_probe into
+ * an event-driven probe: instead of DAMON walking the address space each
+ * sampling interval, a per-CPU perf_event (e.g. AMD IBS Op, Intel PEBS)
+ * samples memory accesses and feeds them into the probe hit counters via the
+ * report ring. The @perf fields are copied into a perf_event_attr when the
+ * kdamond is turned on.
*/
struct damon_prep {
enum damon_prep_action action;
+ struct {
+ u32 type;
+ u64 config;
+ u64 config1;
+ u64 config2;
+ u64 sample_period;
+ u64 sample_freq;
+ u32 wakeup_events;
+ u32 precise_ip;
+ bool sample_phys_addr;
+ bool sample_weight_struct;
+ bool exclude_kernel;
+ bool exclude_hv;
+ bool freq;
+ bool single_instance;
+ } perf;
/* private: */
/* siblings list. */
struct list_head list;
@@ -936,6 +963,8 @@ struct damon_filter {
struct damon_probe {
unsigned int weight;
bool event_driven; /* hits arrive via ring drain, not apply_probes */
+ /* perf-event probe state (struct damon_perf_probe_event *) for teardown */
+ void *perf_priv;
/* private: */
/* Preparation actions to apply to each probing memory. */
struct list_head preps;
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 6b3aa86386b76..37909420910e2 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -18,6 +18,7 @@
/* for damon_get_folio() used by node eligible memory metrics */
#include "ops-common.h"
+#include "perf_source.h"
#define CREATE_TRACE_POINTS
#include <trace/events/damon.h>
@@ -345,6 +346,16 @@ struct damon_probe *damon_new_probe(void)
if (!p)
return NULL;
p->weight = 0;
+ p->event_driven = false;
+ /*
+ * Must be NULL: damon_destroy_ctx()/damon_commit_probes() test
+ * p->perf_priv to decide whether to call damon_perf_probe_teardown().
+ * An uninitialised (garbage) perf_priv makes the teardown kfree() a
+ * wild pointer when a probe is destroyed before it is ever armed
+ * (e.g. perf_event_create_kernel_counter() fails and damon_start()
+ * unwinds via damon_destroy_ctx()).
+ */
+ p->perf_priv = NULL;
INIT_LIST_HEAD(&p->preps);
INIT_LIST_HEAD(&p->filters);
INIT_LIST_HEAD(&p->list);
@@ -409,6 +420,8 @@ bool damon_has_probe_weights(struct damon_ctx *c)
* Event-driven probes (e.g. perf-event IBS/PEBS) populate probe_hits[] via
* the SPSC ring drain rather than the apply_probes vtable. Callers use this
* to decide whether to arm hardware sampling.
+ *
+ * @ctx: the DAMON context whose probes are inspected.
*/
bool damon_has_event_driven_probes(struct damon_ctx *ctx)
{
@@ -1216,8 +1229,20 @@ void damon_destroy_ctx(struct damon_ctx *ctx)
damon_for_each_scheme_safe(s, next_s, ctx)
damon_destroy_scheme(s);
- damon_for_each_probe_safe(p, next_p, ctx)
+ damon_for_each_probe_safe(p, next_p, ctx) {
+#ifdef CONFIG_DAMON_PERF_SOURCE
+ /*
+ * Release the PMU counters before freeing the probe.
+ * damon_perf_probe_teardown() owns and frees the event
+ * descriptor.
+ */
+ if (p->perf_priv) {
+ damon_perf_probe_teardown(ctx, p->perf_priv);
+ p->perf_priv = NULL;
+ }
+#endif
damon_destroy_probe(p);
+ }
damon_for_each_sample_filter_safe(f, next_f, &ctx->sample_control)
damon_destroy_sample_filter(f, &ctx->sample_control);
@@ -2038,6 +2063,7 @@ static int damon_commit_targets(
static void damon_commit_prep(struct damon_prep *dst, struct damon_prep *src)
{
dst->action = src->action;
+ dst->perf = src->perf;
}
static int damon_commit_preps(struct damon_probe *dst, struct damon_probe *src)
@@ -2114,7 +2140,63 @@ static int damon_commit_filters(struct damon_probe *dst,
return 0;
}
-static int damon_commit_probes(struct damon_ctx *dst, struct damon_ctx *src)
+#ifdef CONFIG_DAMON_PERF_SOURCE
+/*
+ * Hand off the armed perf event between a running probe (@dst, in the live
+ * @ctx) and the committed source probe (@src, from a discarded param_ctx that
+ * was built without arming). Keeps the running event untouched when the perf
+ * attributes are unchanged (the common weight-only commit); otherwise tears
+ * down the old event and re-arms from @src's carried attributes.
+ *
+ * @src->perf_priv is built by the sysfs layer from @src's DAMON_PREP_PERF_EVENT
+ * prep before the commit; ownership of that descriptor moves to @dst here.
+ */
+static int damon_commit_perf_probe(struct damon_ctx *ctx,
+ struct damon_probe *dst, struct damon_probe *src)
+{
+ struct damon_perf_probe_event *dst_ev = dst->perf_priv;
+ struct damon_perf_probe_event *src_ev = src->perf_priv;
+ int err;
+
+ if (!src_ev) {
+ /* Source has no perf probe: tear down any running event. */
+ if (dst_ev) {
+ damon_perf_probe_teardown(ctx, dst_ev);
+ dst->perf_priv = NULL;
+ }
+ return 0;
+ }
+
+ /* Already armed with identical attrs: keep the running event. */
+ if (dst_ev && dst_ev->priv &&
+ !memcmp(&dst_ev->attr, &src_ev->attr, sizeof(dst_ev->attr)))
+ return 0;
+
+ /* Attrs changed (or dst not armed): re-arm from src attributes. */
+ if (dst_ev) {
+ damon_perf_probe_teardown(ctx, dst_ev);
+ dst->perf_priv = NULL;
+ }
+ err = damon_perf_probe_setup(ctx, dst, src_ev);
+ if (err) {
+ /* dst old event already torn down; nothing armed now. */
+ dst->event_driven = false;
+ return err;
+ }
+ /* Ownership of src_ev moves to dst; the param_ctx must not free it. */
+ dst->perf_priv = src_ev;
+ src->perf_priv = NULL;
+ return 0;
+}
+#endif /* CONFIG_DAMON_PERF_SOURCE */
+
+/*
+ * @commit_live is false for the dry-run validation pass (a commit into a
+ * throwaway test_ctx) and true for the real commit into the running ctx.
+ * The PMU may only be armed or disarmed on the real commit.
+ */
+static int damon_commit_probes(struct damon_ctx *dst, struct damon_ctx *src,
+ bool commit_live)
{
struct damon_probe *dst_probe, *next, *src_probe, *new_probe;
int i = 0, j = 0, err;
@@ -2123,13 +2205,29 @@ static int damon_commit_probes(struct damon_ctx *dst, struct damon_ctx *src)
src_probe = damon_nth_probe(i++, src);
if (src_probe) {
dst_probe->weight = src_probe->weight;
+ dst_probe->event_driven = src_probe->event_driven;
err = damon_commit_preps(dst_probe, src_probe);
if (err)
return err;
err = damon_commit_filters(dst_probe, src_probe);
if (err)
return err;
+#ifdef CONFIG_DAMON_PERF_SOURCE
+ if (commit_live) {
+ err = damon_commit_perf_probe(dst, dst_probe,
+ src_probe);
+ if (err)
+ return err;
+ }
+#endif
} else {
+#ifdef CONFIG_DAMON_PERF_SOURCE
+ if (commit_live && dst_probe->perf_priv) {
+ damon_perf_probe_teardown(dst,
+ dst_probe->perf_priv);
+ dst_probe->perf_priv = NULL;
+ }
+#endif
damon_destroy_probe(dst_probe);
}
}
@@ -2143,12 +2241,23 @@ static int damon_commit_probes(struct damon_ctx *dst, struct damon_ctx *src)
return -ENOMEM;
damon_add_probe(dst, new_probe);
new_probe->weight = src_probe->weight;
+ new_probe->event_driven = src_probe->event_driven;
err = damon_commit_preps(new_probe, src_probe);
if (err)
return err;
err = damon_commit_filters(new_probe, src_probe);
if (err)
return err;
+#ifdef CONFIG_DAMON_PERF_SOURCE
+ if (commit_live && src_probe->perf_priv) {
+ err = damon_perf_probe_setup(dst, new_probe,
+ src_probe->perf_priv);
+ if (err)
+ return err;
+ new_probe->perf_priv = src_probe->perf_priv;
+ src_probe->perf_priv = NULL;
+ }
+#endif
}
return 0;
}
@@ -2293,7 +2402,7 @@ static int __damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src,
}
dst->pause = src->pause;
dst->ops = src->ops;
- err = damon_commit_probes(dst, src);
+ err = damon_commit_probes(dst, src, commit_live);
if (err)
return err;
err = damon_commit_sample_control(&dst->sample_control,
@@ -5064,6 +5173,25 @@ static int kdamond_fn(void *data)
done:
damon_destroy_targets(ctx);
+#ifdef CONFIG_DAMON_PERF_SOURCE
+ /*
+ * Release perf-event probes here so a stopped kdamond leaves no event
+ * firing overflows into its report ring, and holds no PMU ownership
+ * or provider module reference. This runs in kdamond context, so a
+ * provider's sleeping teardown is safe here.
+ */
+ {
+ struct damon_probe *p, *next_p;
+
+ damon_for_each_probe_safe(p, next_p, ctx) {
+ if (p->perf_priv) {
+ damon_perf_probe_teardown(ctx, p->perf_priv);
+ p->perf_priv = NULL;
+ }
+ }
+ }
+#endif
+
kfree(ctx->regions_score_histogram);
mutex_lock(&ctx->call_controls_lock);
ctx->call_controls_obsolete = true;
diff --git a/mm/damon/perf_source.c b/mm/damon/perf_source.c
index 1102426152eca..e3c6783f45a54 100644
--- a/mm/damon/perf_source.c
+++ b/mm/damon/perf_source.c
@@ -14,49 +14,49 @@
#include <linux/slab.h>
#include "perf_source.h"
-/* PMU event attribute for perf-event probe configuration */
-struct damon_perf_event_attr {
- u32 type;
- u64 config;
- u64 config1;
- u64 config2;
- bool sample_phys_addr;
- bool sample_weight_struct;
- bool exclude_kernel;
- bool exclude_hv;
- bool freq;
- u64 sample_freq;
- u64 sample_period;
- u32 wakeup_events;
- u32 precise_ip;
-};
-
-struct damon_perf_probe_event {
- struct damon_perf_event_attr attr;
- void *priv; /* struct damon_perf_probe_state * */
- struct hlist_node hlist_node;
- int probe_idx; /* index into probe_hits[]; set at registration */
-};
+/*
+ * struct damon_perf_event_attr and struct damon_perf_probe_event are defined
+ * in perf_source.h so that the sysfs configuration surface can build a probe
+ * event descriptor before handing it to damon_perf_probe_setup().
+ */
struct damon_perf_probe_state {
- struct perf_event * __percpu *event;
+ struct perf_event * __percpu *event; /* per-CPU probes (PEBS/IBS) */
+ struct perf_event *single_event; /* single-instance probes (system-wide PMU) */
};
static DEFINE_PER_CPU(unsigned long, damon_perf_samples_total);
static DEFINE_PER_CPU(unsigned long, damon_perf_samples_filtered);
static DEFINE_PER_CPU(unsigned long, damon_perf_samples_no_addr);
+
static void damon_perf_overflow(struct perf_event *perf_event,
struct perf_sample_data *data,
struct pt_regs *regs)
{
- int probe_idx = (int)(unsigned long)perf_event->overflow_handler_context;
+ struct damon_perf_probe_event *event =
+ (struct damon_perf_probe_event *)perf_event->overflow_handler_context;
+ int probe_idx;
+ struct damon_ctx *ctx;
struct damon_access_report report = {
- .probe_idx = probe_idx,
.size = PAGE_SIZE,
.cpu = smp_processor_id(),
};
+ /*
+ * Teardown NULLs event->ctx (with a release barrier) before releasing
+ * the per-CPU perf events, so an in-flight overflow racing the
+ * disable/release observes the torn-down state and drops the sample
+ * instead of reporting into a freed ctx. Pairs with the
+ * smp_store_release(&event->ctx, NULL) in damon_perf_probe_teardown().
+ */
+ ctx = smp_load_acquire(&event->ctx);
+ if (!ctx)
+ return;
+ probe_idx = event->probe_idx;
+ report.probe_idx = probe_idx;
+ report.ctx = ctx; /* route to this ctx's per-ctx perf ring */
+
/* probe_idx 0 is the zero-init sentinel; a valid index must be >= 1 */
if (WARN_ONCE(probe_idx == 0,
"damon-perf: overflow handler called with probe_idx=0\n"))
@@ -94,8 +94,7 @@ static enum cpuhp_state damon_perf_cpuhp_state;
/*
* Per-PMU exclusivity: each PMU type may be owned by at most one damon_ctx.
- * Multiple probes from the same ctx sharing a PMU type are allowed; a second
- * ctx attempting to grab a PMU type already owned returns -EBUSY.
+ * Multiple probes from the same ctx sharing a PMU type are allowed.
*/
struct damon_pmu_owner {
struct list_head node;
@@ -165,7 +164,7 @@ static int damon_perf_cpu_online(unsigned int cpu, struct hlist_node *node)
perf_event = perf_event_create_kernel_counter(&attr, cpu, NULL,
damon_perf_overflow,
- (void *)(unsigned long)event->probe_idx);
+ event);
if (IS_ERR(perf_event)) {
pr_warn_ratelimited("damon-perf: cpu %u event create failed: %ld\n",
cpu, PTR_ERR(perf_event));
@@ -220,6 +219,10 @@ int damon_perf_probe_setup(struct damon_ctx *ctx,
* Per-PMU exclusivity: find or create an owner slot for this PMU type.
* Multiple probes from the same ctx sharing a PMU type are allowed;
* a second ctx attempting the same PMU type returns -EBUSY.
+ *
+ * NOTE: damon_commit_perf_probe() updates perf_event parameters
+ * in-place and never calls damon_perf_probe_setup(), so the owner
+ * table is never touched on the commit path.
*/
spin_lock(&damon_pmu_owner_lock);
list_for_each_entry(owner, &damon_pmu_owner_list, node) {
@@ -271,16 +274,69 @@ int damon_perf_probe_setup(struct damon_ctx *ctx,
goto release_owner;
}
event->probe_idx = idx + 1; /* 1-based; 0 is reserved sentinel */
+ event->ctx = ctx; /* route overflow reports to this ctx */
+
+ /*
+ * Allocate the ctx's per-ctx perf report ring before arming any event,
+ * so the overflow handler always finds a ready ring. Idempotent across
+ * a ctx's multiple probes.
+ */
+ err = damon_ctx_alloc_perf_ring(ctx);
+ if (err)
+ goto release_owner;
perf = kzalloc_obj(*perf, GFP_KERNEL);
if (!perf)
goto release_owner;
+ event->priv = perf;
+
+ /*
+ * A system-wide PMU is a single hardware unit rather than a per-CPU
+ * counter, so it needs exactly one counter: the cpuhp fan-out below
+ * would run one redundant sampler per CPU against the one device and
+ * corrupt its shared state. Pin that counter to a fixed online CPU
+ * and bypass cpuhp.
+ *
+ * The pin must name a real CPU (>= 0) because the PMU is
+ * perf_invalid_context, for which cpu = -1 routes to task context. If
+ * that CPU is later offlined the counter stops and is not migrated,
+ * which is acceptable for a dedicated monitoring host that does not
+ * hotplug CPUs.
+ */
+ if (event->attr.single_instance) {
+ struct perf_event_attr attr;
+ int cpu = cpumask_first(cpu_online_mask);
+
+ damon_perf_event_init_attr(event, &attr);
+ /*
+ * Pass @event (not a probe_idx cookie) as the overflow context:
+ * damon_perf_overflow() casts it to damon_perf_probe_event* and
+ * reads event->ctx via smp_load_acquire() for the teardown race
+ * barrier, same as the per-CPU path (damon_perf_cpu_online()).
+ */
+ perf->single_event = perf_event_create_kernel_counter(&attr, cpu,
+ NULL, damon_perf_overflow, event);
+ if (IS_ERR(perf->single_event)) {
+ err = PTR_ERR(perf->single_event);
+ perf->single_event = NULL;
+ pr_warn("damon-perf: single-instance event create failed: %d\n",
+ err);
+ goto free_perf;
+ }
+ perf_event_enable(perf->single_event);
+ /*
+ * Ownership is already held via the per-PMU owner->refcount
+ * acquired at the top of setup; the single-instance path shares
+ * that slot, so no separate refcount is taken here. Teardown
+ * releases it through the same owner list as the per-CPU path.
+ */
+ return 0;
+ }
perf->event = alloc_percpu(typeof(*perf->event));
if (!perf->event)
goto free_perf;
- event->priv = perf;
INIT_HLIST_NODE(&event->hlist_node);
err = cpuhp_state_add_instance(damon_perf_cpuhp_state,
@@ -310,44 +366,75 @@ EXPORT_SYMBOL_GPL(damon_perf_probe_setup);
/**
* damon_perf_probe_teardown - disarm perf_events.
+ * @ctx: DAMON context that owns the probe (used to release per-PMU ownership).
* @event: perf event descriptor previously passed to damon_perf_probe_setup()
*/
void damon_perf_probe_teardown(struct damon_ctx *ctx,
struct damon_perf_probe_event *event)
{
struct damon_perf_probe_state *perf = event->priv;
- struct damon_pmu_owner *owner, *tmp;
- if (!perf)
- return;
+ if (perf) {
+ struct damon_pmu_owner *owner, *tmp;
- cpuhp_state_remove_instance(damon_perf_cpuhp_state,
- &event->hlist_node);
- free_percpu(perf->event);
- kfree(perf);
- event->priv = NULL;
+ /*
+ * Signal in-flight NMI overflow handlers to drop samples
+ * before tearing down the perf events and freeing their
+ * backing state. Pairs with smp_load_acquire(&event->ctx)
+ * in damon_perf_overflow(). This applies to both the per-CPU
+ * and single-instance paths: a single-instance counter can
+ * also deliver an overflow racing teardown.
+ */
+ smp_store_release(&event->ctx, NULL);
- /*
- * Release per-PMU ownership when the last probe for this
- * ctx/PMU-type pair is torn down.
- */
- spin_lock(&damon_pmu_owner_lock);
- list_for_each_entry_safe(owner, tmp, &damon_pmu_owner_list, node) {
- if (owner->pmu_type == event->attr.type &&
- atomic_long_read(&owner->owner_ctx) == (long)ctx) {
+ if (perf->single_event) {
+ /*
+ * Single-instance probe: no cpuhp instance was added, so
+ * just release the one counter (process context here, so
+ * disable+release is safe). disable() also quiesces any
+ * pending overflow before release.
+ */
+ perf_event_disable(perf->single_event);
+ perf_event_release_kernel(perf->single_event);
+ perf->single_event = NULL;
+ } else {
/*
- * Free under the lock so a concurrent same-PMU teardown
- * cannot observe and free the same owner. kfree() under
- * a non-irq spinlock in process context is safe.
+ * cpuhp_state_remove_instance() disables+releases each
+ * CPU's perf event; once it returns no new overflow can
+ * be delivered for this event.
*/
- if (atomic_dec_and_test(&owner->refcount)) {
- list_del(&owner->node);
- kfree(owner);
+ cpuhp_state_remove_instance(damon_perf_cpuhp_state,
+ &event->hlist_node);
+ free_percpu(perf->event);
+ }
+ kfree(perf);
+ event->priv = NULL;
+
+ /*
+ * Release per-PMU ownership when the last probe for this
+ * ctx/PMU-type pair is torn down.
+ */
+ spin_lock(&damon_pmu_owner_lock);
+ list_for_each_entry_safe(owner, tmp, &damon_pmu_owner_list, node) {
+ if (owner->pmu_type == event->attr.type &&
+ atomic_long_read(&owner->owner_ctx) == (long)ctx) {
+ /*
+ * Free under the lock so a concurrent same-PMU
+ * teardown cannot observe and free the same
+ * owner. kfree() under a non-irq spinlock in
+ * process context is safe.
+ */
+ if (atomic_dec_and_test(&owner->refcount)) {
+ list_del(&owner->node);
+ kfree(owner);
+ }
+ break;
}
- break;
}
+ spin_unlock(&damon_pmu_owner_lock);
}
- spin_unlock(&damon_pmu_owner_lock);
+ /* teardown owns the event allocation */
+ kfree(event);
}
EXPORT_SYMBOL_GPL(damon_perf_probe_teardown);
diff --git a/mm/damon/perf_source.h b/mm/damon/perf_source.h
index 5de55a29785f4..f2dec17a5e026 100644
--- a/mm/damon/perf_source.h
+++ b/mm/damon/perf_source.h
@@ -13,7 +13,35 @@
#include <linux/damon.h>
#include <linux/perf_event.h>
-struct damon_perf_probe_event;
+/*
+ * PMU event attributes for a perf-event probe. A subset of perf_event_attr
+ * chosen at probe creation time to select the PMU (AMD IBS, Intel PEBS, ...)
+ * and its sampling parameters.
+ */
+struct damon_perf_event_attr {
+ u32 type;
+ u64 config;
+ u64 config1;
+ u64 config2;
+ bool sample_phys_addr;
+ bool sample_weight_struct;
+ bool exclude_kernel;
+ bool exclude_hv;
+ bool freq;
+ bool single_instance; /* system-wide PMU: open one counter, not per-CPU */
+ u64 sample_freq;
+ u64 sample_period;
+ u32 wakeup_events;
+ u32 precise_ip;
+};
+
+struct damon_perf_probe_event {
+ struct damon_perf_event_attr attr;
+ struct damon_ctx *ctx; /* owning ctx for ring routing; set at setup */
+ void *priv; /* struct damon_perf_probe_state * */
+ struct hlist_node hlist_node;
+ int probe_idx; /* index into probe_hits[]; set at registration */
+};
int damon_perf_probe_setup(struct damon_ctx *ctx,
struct damon_probe *probe,
diff --git a/mm/damon/sysfs.c b/mm/damon/sysfs.c
index 768b6b49f0b0f..54ab3bbe8cdbb 100644
--- a/mm/damon/sysfs.c
+++ b/mm/damon/sysfs.c
@@ -7,6 +7,7 @@
#include <linux/slab.h>
#include "sysfs-common.h"
+#include "perf_source.h"
/*
* init region directory
@@ -756,6 +757,21 @@ static const struct kobj_type damon_sysfs_intervals_ktype = {
struct damon_sysfs_prep {
struct kobject kobj;
enum damon_prep_action action;
+ /* perf_event_attr subset; valid when action == DAMON_PREP_PERF_EVENT */
+ u32 perf_type;
+ u64 config;
+ u64 config1;
+ u64 config2;
+ u64 sample_period;
+ u64 sample_freq;
+ u32 wakeup_events;
+ u32 precise_ip;
+ bool sample_phys_addr;
+ bool sample_weight_struct;
+ bool exclude_kernel;
+ bool exclude_hv;
+ bool freq;
+ bool single_instance;
};
static struct damon_sysfs_prep *damon_sysfs_prep_alloc(void)
@@ -780,6 +796,10 @@ damon_sysfs_prep_action_names[] = {
.action = DAMON_PREP_SET_PGIDLE,
.name = "set_pgidle",
},
+ {
+ .action = DAMON_PREP_PERF_EVENT,
+ .name = "perf_event",
+ },
};
static ssize_t avail_prep_actions_show(struct kobject *kobj,
@@ -852,9 +872,123 @@ static struct kobj_attribute damon_sysfs_prep_avail_prep_actions_attr =
static struct kobj_attribute damon_sysfs_prep_prep_action_attr =
__ATTR_RW_MODE(prep_action, 0600);
+/*
+ * perf_event configuration attributes. These mirror a subset of
+ * perf_event_attr and are only meaningful when prep_action is "perf_event".
+ * They select the PMU (via type/config) and its sampling parameters, and are
+ * copied into the perf-event probe when the kdamond is turned on.
+ *
+ * The sysfs file names stay bare (type, config, ...) while the backing C
+ * symbols are prefixed to avoid clashing with identically named attributes
+ * elsewhere in this file.
+ */
+#define DAMON_SYSFS_PREP_PERF_U32(name, field) \
+static ssize_t damon_sysfs_prep_##name##_show(struct kobject *kobj, \
+ struct kobj_attribute *attr, char *buf) \
+{ \
+ struct damon_sysfs_prep *prep = container_of(kobj, \
+ struct damon_sysfs_prep, kobj); \
+ return sysfs_emit(buf, "%u\n", prep->field); \
+} \
+static ssize_t damon_sysfs_prep_##name##_store(struct kobject *kobj, \
+ struct kobj_attribute *attr, const char *buf, \
+ size_t count) \
+{ \
+ struct damon_sysfs_prep *prep = container_of(kobj, \
+ struct damon_sysfs_prep, kobj); \
+ u32 v; \
+ int err = kstrtou32(buf, 0, &v); \
+ if (err) \
+ return err; \
+ prep->field = v; \
+ return count; \
+} \
+static struct kobj_attribute damon_sysfs_prep_##name##_attr = __ATTR(name, \
+ 0600, damon_sysfs_prep_##name##_show, \
+ damon_sysfs_prep_##name##_store)
+
+#define DAMON_SYSFS_PREP_PERF_U64(name, field) \
+static ssize_t damon_sysfs_prep_##name##_show(struct kobject *kobj, \
+ struct kobj_attribute *attr, char *buf) \
+{ \
+ struct damon_sysfs_prep *prep = container_of(kobj, \
+ struct damon_sysfs_prep, kobj); \
+ return sysfs_emit(buf, "%llu\n", prep->field); \
+} \
+static ssize_t damon_sysfs_prep_##name##_store(struct kobject *kobj, \
+ struct kobj_attribute *attr, const char *buf, \
+ size_t count) \
+{ \
+ struct damon_sysfs_prep *prep = container_of(kobj, \
+ struct damon_sysfs_prep, kobj); \
+ u64 v; \
+ int err = kstrtou64(buf, 0, &v); \
+ if (err) \
+ return err; \
+ prep->field = v; \
+ return count; \
+} \
+static struct kobj_attribute damon_sysfs_prep_##name##_attr = __ATTR(name, \
+ 0600, damon_sysfs_prep_##name##_show, \
+ damon_sysfs_prep_##name##_store)
+
+#define DAMON_SYSFS_PREP_PERF_BOOL(name, field) \
+static ssize_t damon_sysfs_prep_##name##_show(struct kobject *kobj, \
+ struct kobj_attribute *attr, char *buf) \
+{ \
+ struct damon_sysfs_prep *prep = container_of(kobj, \
+ struct damon_sysfs_prep, kobj); \
+ return sysfs_emit(buf, "%u\n", prep->field); \
+} \
+static ssize_t damon_sysfs_prep_##name##_store(struct kobject *kobj, \
+ struct kobj_attribute *attr, const char *buf, \
+ size_t count) \
+{ \
+ struct damon_sysfs_prep *prep = container_of(kobj, \
+ struct damon_sysfs_prep, kobj); \
+ bool v; \
+ int err = kstrtobool(buf, &v); \
+ if (err) \
+ return err; \
+ prep->field = v; \
+ return count; \
+} \
+static struct kobj_attribute damon_sysfs_prep_##name##_attr = __ATTR(name, \
+ 0600, damon_sysfs_prep_##name##_show, \
+ damon_sysfs_prep_##name##_store)
+
+DAMON_SYSFS_PREP_PERF_U32(type, perf_type);
+DAMON_SYSFS_PREP_PERF_U64(config, config);
+DAMON_SYSFS_PREP_PERF_U64(config1, config1);
+DAMON_SYSFS_PREP_PERF_U64(config2, config2);
+DAMON_SYSFS_PREP_PERF_U64(sample_period, sample_period);
+DAMON_SYSFS_PREP_PERF_U64(sample_freq, sample_freq);
+DAMON_SYSFS_PREP_PERF_U32(wakeup_events, wakeup_events);
+DAMON_SYSFS_PREP_PERF_U32(precise_ip, precise_ip);
+DAMON_SYSFS_PREP_PERF_BOOL(sample_phys_addr, sample_phys_addr);
+DAMON_SYSFS_PREP_PERF_BOOL(sample_weight_struct, sample_weight_struct);
+DAMON_SYSFS_PREP_PERF_BOOL(exclude_kernel, exclude_kernel);
+DAMON_SYSFS_PREP_PERF_BOOL(exclude_hv, exclude_hv);
+DAMON_SYSFS_PREP_PERF_BOOL(freq, freq);
+DAMON_SYSFS_PREP_PERF_BOOL(single_instance, single_instance);
+
static struct attribute *damon_sysfs_prep_attrs[] = {
&damon_sysfs_prep_avail_prep_actions_attr.attr,
&damon_sysfs_prep_prep_action_attr.attr,
+ &damon_sysfs_prep_type_attr.attr,
+ &damon_sysfs_prep_config_attr.attr,
+ &damon_sysfs_prep_config1_attr.attr,
+ &damon_sysfs_prep_config2_attr.attr,
+ &damon_sysfs_prep_sample_period_attr.attr,
+ &damon_sysfs_prep_sample_freq_attr.attr,
+ &damon_sysfs_prep_wakeup_events_attr.attr,
+ &damon_sysfs_prep_precise_ip_attr.attr,
+ &damon_sysfs_prep_sample_phys_addr_attr.attr,
+ &damon_sysfs_prep_sample_weight_struct_attr.attr,
+ &damon_sysfs_prep_exclude_kernel_attr.attr,
+ &damon_sysfs_prep_exclude_hv_attr.attr,
+ &damon_sysfs_prep_freq_attr.attr,
+ &damon_sysfs_prep_single_instance_attr.attr,
NULL,
};
ATTRIBUTE_GROUPS(damon_sysfs_prep);
@@ -2252,9 +2386,40 @@ static int damon_sysfs_set_preps(struct damon_probe *probe,
struct damon_sysfs_prep *sys_prep = sys_preps->preps_arr[i];
struct damon_prep *prep;
+ /*
+ * period and freq are mutually exclusive perf sampling modes;
+ * reject a config that sets both before it can arm a counter.
+ */
+ if (sys_prep->action == DAMON_PREP_PERF_EVENT &&
+ sys_prep->sample_period && sys_prep->sample_freq)
+ return -EINVAL;
+
prep = damon_new_prep(sys_prep->action);
if (!prep)
return -ENOMEM;
+ if (sys_prep->action == DAMON_PREP_PERF_EVENT) {
+ /*
+ * damon_new_prep() does not zero prep->perf; clear it
+ * so any field not assigned below starts from a known
+ * zero rather than kmalloc garbage.
+ */
+ memset(&prep->perf, 0, sizeof(prep->perf));
+ prep->perf.type = sys_prep->perf_type;
+ prep->perf.config = sys_prep->config;
+ prep->perf.config1 = sys_prep->config1;
+ prep->perf.config2 = sys_prep->config2;
+ prep->perf.sample_period = sys_prep->sample_period;
+ prep->perf.sample_freq = sys_prep->sample_freq;
+ prep->perf.wakeup_events = sys_prep->wakeup_events;
+ prep->perf.precise_ip = sys_prep->precise_ip;
+ prep->perf.sample_phys_addr = sys_prep->sample_phys_addr;
+ prep->perf.sample_weight_struct =
+ sys_prep->sample_weight_struct;
+ prep->perf.exclude_kernel = sys_prep->exclude_kernel;
+ prep->perf.exclude_hv = sys_prep->exclude_hv;
+ prep->perf.freq = sys_prep->freq;
+ prep->perf.single_instance = sys_prep->single_instance;
+ }
damon_add_prep(probe, prep);
}
return 0;
@@ -2291,8 +2456,71 @@ static int damon_sysfs_set_filters(struct damon_probe *probe,
return 0;
}
-static int damon_sysfs_set_probe(struct damon_probe *probe,
- struct damon_sysfs_probe *sys_probe)
+#ifdef CONFIG_DAMON_PERF_SOURCE
+/*
+ * Build a perf-event probe descriptor from the probe's DAMON_PREP_PERF_EVENT
+ * prep and attach it to @probe. The descriptor is always carried (so the ring
+ * drain and the commit hand-off recognise the probe as event-driven), but the
+ * PMU counters are only armed when @arm is set.
+ *
+ * @arm is true when building the context that will actually run (turn-on
+ * path); it is false when building a param_ctx for a commit, which is
+ * discarded after validation. Arming a param_ctx would collide with the
+ * running context's perf-probe ownership and return -EBUSY, so the commit path
+ * defers arming to damon_commit_perf_probe().
+ */
+static int damon_sysfs_set_perf_probe(struct damon_ctx *ctx,
+ struct damon_probe *probe, bool arm)
+{
+ struct damon_prep *prep;
+
+ damon_for_each_prep(prep, probe) {
+ struct damon_perf_probe_event *event;
+ int err;
+
+ if (prep->action != DAMON_PREP_PERF_EVENT)
+ continue;
+
+ event = kzalloc_obj(*event, GFP_KERNEL);
+ if (!event)
+ return -ENOMEM;
+ event->attr.type = prep->perf.type;
+ event->attr.config = prep->perf.config;
+ event->attr.config1 = prep->perf.config1;
+ event->attr.config2 = prep->perf.config2;
+ event->attr.sample_period = prep->perf.sample_period;
+ event->attr.sample_freq = prep->perf.sample_freq;
+ event->attr.wakeup_events = prep->perf.wakeup_events;
+ event->attr.precise_ip = prep->perf.precise_ip;
+ event->attr.sample_phys_addr = prep->perf.sample_phys_addr;
+ event->attr.sample_weight_struct =
+ prep->perf.sample_weight_struct;
+ event->attr.exclude_kernel = prep->perf.exclude_kernel;
+ event->attr.exclude_hv = prep->perf.exclude_hv;
+ event->attr.freq = prep->perf.freq;
+ event->attr.single_instance = prep->perf.single_instance;
+
+ probe->perf_priv = event;
+ probe->event_driven = true;
+ if (arm) {
+ err = damon_perf_probe_setup(ctx, probe, event);
+ if (err) {
+ probe->perf_priv = NULL;
+ probe->event_driven = false;
+ kfree(event);
+ return err;
+ }
+ }
+ /* At most one perf-event prep per probe. */
+ break;
+ }
+ return 0;
+}
+#endif /* CONFIG_DAMON_PERF_SOURCE */
+
+static int damon_sysfs_set_probe(struct damon_ctx *ctx,
+ struct damon_probe *probe,
+ struct damon_sysfs_probe *sys_probe, bool arm)
{
struct damon_sysfs_filters *sys_filters;
struct damon_sysfs_preps *sys_preps;
@@ -2305,13 +2533,21 @@ static int damon_sysfs_set_probe(struct damon_probe *probe,
return err;
}
sys_filters = sys_probe->filters;
- if (!sys_filters)
- return 0;
- return damon_sysfs_set_filters(probe, sys_filters);
+ if (sys_filters) {
+ err = damon_sysfs_set_filters(probe, sys_filters);
+ if (err)
+ return err;
+ }
+#ifdef CONFIG_DAMON_PERF_SOURCE
+ err = damon_sysfs_set_perf_probe(ctx, probe, arm);
+ if (err)
+ return err;
+#endif
+ return 0;
}
static int damon_sysfs_set_probes(struct damon_ctx *ctx,
- struct damon_sysfs_probes *sys_probes)
+ struct damon_sysfs_probes *sys_probes, bool arm)
{
int i, err;
@@ -2325,9 +2561,10 @@ static int damon_sysfs_set_probes(struct damon_ctx *ctx,
damon_add_probe(ctx, p);
sys_probe = sys_probes->probes_arr[i];
p->weight = sys_probe->weight;
- err = damon_sysfs_set_probe(p, sys_probe);
+ err = damon_sysfs_set_probe(ctx, p, sys_probe, arm);
if (err)
return err;
+
}
return 0;
}
@@ -2426,7 +2663,7 @@ static inline bool damon_sysfs_kdamond_running(
}
static int damon_sysfs_apply_inputs(struct damon_ctx *ctx,
- struct damon_sysfs_context *sys_ctx)
+ struct damon_sysfs_context *sys_ctx, bool arm)
{
enum damon_ops_id ops_id;
int err;
@@ -2444,7 +2681,7 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx,
err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs);
if (err)
return err;
- err = damon_sysfs_set_probes(ctx, sys_ctx->attrs->probes);
+ err = damon_sysfs_set_probes(ctx, sys_ctx->attrs->probes, arm);
if (err)
return err;
err = damon_sysfs_set_sample_control(&ctx->sample_control,
@@ -2458,7 +2695,7 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx,
}
static struct damon_ctx *damon_sysfs_build_ctx(
- struct damon_sysfs_context *sys_ctx);
+ struct damon_sysfs_context *sys_ctx, bool arm);
/*
* damon_sysfs_commit_input() - Commit user inputs to a running kdamond.
@@ -2478,7 +2715,8 @@ static int damon_sysfs_commit_input(void *data)
if (kdamond->contexts->nr != 1)
return -EINVAL;
- param_ctx = damon_sysfs_build_ctx(kdamond->contexts->contexts_arr[0]);
+ param_ctx = damon_sysfs_build_ctx(kdamond->contexts->contexts_arr[0],
+ false);
if (IS_ERR(param_ctx))
return PTR_ERR(param_ctx);
err = damon_commit_ctx(kdamond->damon_ctx, param_ctx);
@@ -2536,7 +2774,7 @@ static int damon_sysfs_upd_tuned_intervals(void *data)
}
static struct damon_ctx *damon_sysfs_build_ctx(
- struct damon_sysfs_context *sys_ctx)
+ struct damon_sysfs_context *sys_ctx, bool arm)
{
struct damon_ctx *ctx = damon_new_ctx();
int err;
@@ -2544,7 +2782,7 @@ static struct damon_ctx *damon_sysfs_build_ctx(
if (!ctx)
return ERR_PTR(-ENOMEM);
- err = damon_sysfs_apply_inputs(ctx, sys_ctx);
+ err = damon_sysfs_apply_inputs(ctx, sys_ctx, arm);
if (err) {
damon_destroy_ctx(ctx);
return ERR_PTR(err);
@@ -2596,7 +2834,7 @@ static int damon_sysfs_turn_damon_on(struct damon_sysfs_kdamond *kdamond)
if (!repeat_call_control)
return -ENOMEM;
- ctx = damon_sysfs_build_ctx(kdamond->contexts->contexts_arr[0]);
+ ctx = damon_sysfs_build_ctx(kdamond->contexts->contexts_arr[0], true);
if (IS_ERR(ctx)) {
kfree(repeat_call_control);
return PTR_ERR(ctx);
diff --git a/mm/damon/tests/core-kunit.h b/mm/damon/tests/core-kunit.h
index 9ed182ff38804..78c2cb145edce 100644
--- a/mm/damon/tests/core-kunit.h
+++ b/mm/damon/tests/core-kunit.h
@@ -1477,7 +1477,7 @@ static void damon_test_commit_probes_for(struct kunit *test,
kunit_skip(test, "src alloc fail");
}
- err = damon_commit_probes(dst, src);
+ err = damon_commit_probes(dst, src, false);
KUNIT_EXPECT_EQ(test, err, 0);
if (err)
goto out;
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 9/9] mm/damon/tests/drain-kunit: kunit for report rings and unified drain
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
` (7 preceding siblings ...)
2026-09-10 17:16 ` [RFC PATCH v2 8/9] mm/damon: add perf_event prep for PMU-driven hotness probes Ravi Jonnalagadda
@ 2026-09-10 17:16 ` Ravi Jonnalagadda
2026-09-11 0:34 ` [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports SJ Park
2026-09-12 1:38 ` SJ Park
10 siblings, 0 replies; 12+ messages in thread
From: Ravi Jonnalagadda @ 2026-09-10 17:16 UTC (permalink / raw)
To: sj, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc
Cc: akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
ravis.opensrc, rientjes, weixugc, jic23, gourry
Add kunit coverage for the report rings (the global page-fault ring and
a per-context perf ring) and the unified drain: ring inject/drain,
overflow safety on wrap, the damon_report_access() producer + probe_idx
routing (page-fault reports to the global pf ring, perf reports to the
reporting ctx's own perf ring), ring partition, the pf-ring owner
-EBUSY guard, vaddr thread-group-id filtering, and paddr crediting left
unfiltered. Wire the suites into core.c (included after the drain and its
counters are defined) so CONFIG_DAMON_KUNIT_TEST=y builds them.
Cover the page-fault primitive and an event-driven probe sharing one
context: a report on each ring credits the access rate of the region
while only the probe report reaches the probe hits array feeding the
weighted sum. Cover the address-space match by injecting a report whose
virtual address falls inside a region of a physical address space target
and whose physical address falls outside it, which is not credited.
Cover the hot and cold sides of the page-fault signal by reporting an
access on one of two regions and aging both.
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@gmail.com>
---
mm/damon/core.c | 2 +
mm/damon/tests/.kunitconfig | 4 +
mm/damon/tests/drain-kunit.h | 1091 ++++++++++++++++++++++++++++++++++
mm/damon/tests/perf-kunit.h | 133 +++++
4 files changed, 1230 insertions(+)
create mode 100644 mm/damon/tests/drain-kunit.h
create mode 100644 mm/damon/tests/perf-kunit.h
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 37909420910e2..d14795993a888 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -5354,3 +5354,5 @@ struct damon_region *damon_search(unsigned long addr, struct pid *pid)
subsys_initcall(damon_init);
#include "tests/core-kunit.h"
+#include "tests/drain-kunit.h"
+#include "tests/perf-kunit.h"
diff --git a/mm/damon/tests/.kunitconfig b/mm/damon/tests/.kunitconfig
index 144d27e6ecc5c..8096aca135807 100644
--- a/mm/damon/tests/.kunitconfig
+++ b/mm/damon/tests/.kunitconfig
@@ -16,3 +16,7 @@ CONFIG_DAMON_SYSFS_KUNIT_TEST=y
# enable DAMON_DEBUG_SANITY to catch any bug
CONFIG_DAMON_DEBUG_SANITY=y
+
+# for global report rings (pf/perf) and per-PMU exclusivity tests
+CONFIG_PERF_EVENTS=y
+CONFIG_DAMON_PERF_SOURCE=y
diff --git a/mm/damon/tests/drain-kunit.h b/mm/damon/tests/drain-kunit.h
new file mode 100644
index 0000000000000..21ad90027eddb
--- /dev/null
+++ b/mm/damon/tests/drain-kunit.h
@@ -0,0 +1,1091 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * DAMON kunit tests for the unified paddr/vaddr report drain path.
+ *
+ * Included at the bottom of core.c (after kdamond_check_reported_accesses
+ * is defined) so the static function is visible.
+ */
+
+#ifdef CONFIG_DAMON_KUNIT_TEST
+
+#ifndef _DAMON_DRAIN_KUNIT_H
+#define _DAMON_DRAIN_KUNIT_H
+
+#include <kunit/test.h>
+#include <linux/damon.h>
+
+/*
+ * Reports are partitioned by probe_idx: probe_idx == DAMON_PROBE_IDX_NONE (0)
+ * lands in the global page_fault ring; probe_idx >= 1 lands in the owning
+ * context's per-context perf ring (ctx->perf_rings). The drain dispatcher
+ * kdamond_check_reported_accesses() drains the perf ring only for a ctx that
+ * has event-driven probes.
+ *
+ * Attach a dummy event-driven probe AND allocate the ctx's per-ctx perf ring
+ * so the ctx both drains the perf ring and has ring storage for injected
+ * probe_idx>=1 reports. In a live run damon_perf_probe_setup() allocates the
+ * ring; kunit has no real perf event, so it allocates directly. Returns
+ * 0/-ENOMEM.
+ */
+static int damon_test_attach_perf_probe(struct damon_ctx *ctx)
+{
+ struct damon_probe *p = damon_new_probe();
+ int err;
+
+ if (!p)
+ return -ENOMEM;
+ p->event_driven = true;
+ damon_add_probe(ctx, p);
+
+ err = damon_ctx_alloc_perf_ring(ctx);
+ if (err)
+ return err;
+ return 0;
+}
+
+/*
+ * Mark @ctx as monitoring the physical address space.
+ *
+ * The drain matches a report against the address space of the context, which
+ * damon_target_has_pid() derives from ctx->ops.id, so a context whose targets
+ * carry no pid needs the paddr id for its reports to be matched by paddr.
+ * Only the id is set: the drain reads no other operations field, and these
+ * tests call it directly rather than through a kdamond.
+ */
+static void damon_test_set_paddr_ctx(struct damon_ctx *ctx)
+{
+ ctx->ops.id = DAMON_OPS_PADDR;
+}
+
+/*
+ * Test A: vaddr entry with a matching thread group id drains correctly.
+ *
+ * Create a vaddr ctx with target pid=current, region [0x1000, 0x2000).
+ * Inject entry: paddr=0, vaddr=0x1500, tgid=current tgid, probe_idx=1, ctx=ctx.
+ * After drain: probe_hits[0]==1 (probe_idx 1 stored 0-based), samples_drained
+ * increments.
+ */
+static void damon_test_unified_vaddr_match(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_target *t;
+ struct damon_region *r;
+ struct damon_access_report rep = {
+ .paddr = 0,
+ .vaddr = 0x1500,
+ .probe_idx = 1,
+ .size = PAGE_SIZE,
+ };
+ unsigned long before, after;
+ int hits;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ if (damon_test_attach_perf_probe(ctx)) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "perf probe alloc failed");
+ }
+
+ t = damon_new_target();
+ if (!t) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "target alloc failed");
+ }
+ t->pid = get_pid(task_tgid(current));
+ if (!t->pid) {
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "pid alloc failed");
+ }
+ rep.tgid = task_tgid_vnr(current);
+ rep.ctx = ctx; /* route to this ctx's per-ctx perf ring */
+
+ /*
+ * Region must fully contain the report [vaddr, vaddr + size): a report
+ * straddling the region end is rejected by the drain (correctly). With
+ * vaddr=0x1500 and size=PAGE_SIZE the region must reach >= 0x2500.
+ */
+ r = damon_new_region(0x1000, 0x3000);
+ if (!r) {
+ put_pid(t->pid);
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "region alloc failed");
+ }
+ damon_add_region(r, t);
+ damon_add_target(ctx, t);
+
+ rep.report_jiffies = jiffies;
+ before = damon_get_samples_drained();
+ damon_report_access(&rep);
+ kdamond_check_reported_accesses(ctx);
+ after = damon_get_samples_drained();
+
+ hits = 0;
+ damon_for_each_region(r, t)
+ hits += r->probe_hits[0];
+
+ KUNIT_EXPECT_EQ(test, hits, 1);
+ KUNIT_EXPECT_GT(test, after, before);
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test B: a vaddr entry whose thread group id matches no target is dropped.
+ *
+ * Same setup but inject with a thread group id no target carries.
+ * probe_hits[0]==0 (probe_idx 1 stored 0-based), samples_no_region increments.
+ */
+static void damon_test_unified_vaddr_tgid_mismatch(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_target *t;
+ struct damon_region *r;
+ struct damon_access_report rep = {
+ .paddr = 0,
+ .vaddr = 0x1500,
+ .tgid = 9999, /* matches no target */
+ .probe_idx = 1,
+ .size = PAGE_SIZE,
+ };
+ unsigned long before, after;
+ int hits;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ if (damon_test_attach_perf_probe(ctx)) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "perf probe alloc failed");
+ }
+
+ t = damon_new_target();
+ if (!t) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "target alloc failed");
+ }
+ t->pid = get_pid(task_tgid(current));
+ if (!t->pid) {
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "pid alloc failed");
+ }
+ rep.ctx = ctx;
+
+ /* Wide enough to contain the report; the id mismatch is the sole reject reason. */
+ r = damon_new_region(0x1000, 0x3000);
+ if (!r) {
+ put_pid(t->pid);
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "region alloc failed");
+ }
+ damon_add_region(r, t);
+ damon_add_target(ctx, t);
+
+ rep.report_jiffies = jiffies;
+ before = damon_get_samples_no_region();
+ damon_report_access(&rep);
+ kdamond_check_reported_accesses(ctx);
+ after = damon_get_samples_no_region();
+
+ hits = 0;
+ damon_for_each_region(r, t)
+ hits += r->probe_hits[0];
+
+ KUNIT_EXPECT_EQ(test, hits, 0);
+ KUNIT_EXPECT_GT(test, after, before);
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test C: paddr entry drains correctly (no id filter for paddr ops).
+ *
+ * Create a paddr ctx (no pid), region [0x10000, 0x20000).
+ * Inject: paddr=0x15000, vaddr=0, probe_idx=1, ctx=ctx.
+ * After drain: probe_hits[0]==1 (probe_idx 1 stored 0-based).
+ */
+static void damon_test_unified_paddr_no_regression(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_target *t;
+ struct damon_region *r;
+ struct damon_access_report rep = {
+ .paddr = 0x15000,
+ .vaddr = 0,
+ .tid = 0,
+ .probe_idx = 1,
+ .size = PAGE_SIZE,
+ };
+ unsigned long before, after;
+ int hits;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ if (damon_test_attach_perf_probe(ctx)) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "perf probe alloc failed");
+ }
+
+ t = damon_new_target();
+ if (!t) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "target alloc failed");
+ }
+ t->pid = NULL; /* paddr target: no pid */
+ damon_test_set_paddr_ctx(ctx);
+ rep.ctx = ctx;
+
+ r = damon_new_region(0x10000, 0x20000);
+ if (!r) {
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "region alloc failed");
+ }
+ damon_add_region(r, t);
+ damon_add_target(ctx, t);
+
+ rep.report_jiffies = jiffies;
+ before = damon_get_samples_drained();
+ damon_report_access(&rep);
+ kdamond_check_reported_accesses(ctx);
+ after = damon_get_samples_drained();
+
+ hits = 0;
+ damon_for_each_region(r, t)
+ hits += r->probe_hits[0];
+
+ KUNIT_EXPECT_EQ(test, hits, 1);
+ KUNIT_EXPECT_GT(test, after, before);
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test pf-ring credit: a page_fault ctx drains the global pf ring.
+ *
+ * Create a paddr ctx with page_fault primitive enabled (no probes), region
+ * [0x10000, 0x20000). Inject probe_idx=0 (DAMON_PROBE_IDX_NONE) paddr entry;
+ * such reports carry no ctx and land in the global pf ring. After drain via
+ * the dispatcher: samples_drained increments (access rate credited) and
+ * probe_hits stays 0 (no probe attribution for probe_idx 0).
+ */
+static void damon_test_ring0_pagefault_credit(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_target *t;
+ struct damon_region *r;
+ struct damon_access_report rep = {
+ .paddr = 0x15000,
+ .vaddr = 0,
+ .tid = 0,
+ .probe_idx = 0, /* DAMON_PROBE_IDX_NONE -> global pf ring */
+ .size = PAGE_SIZE,
+ };
+ unsigned long before, after;
+ int hits;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ /* page_fault primitive: routes drain to the global pf ring. */
+ ctx->sample_control.primitives_enabled.page_table = false;
+ ctx->sample_control.primitives_enabled.page_fault = true;
+
+ t = damon_new_target();
+ if (!t) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "target alloc failed");
+ }
+ t->pid = NULL; /* paddr target: no pid */
+ damon_test_set_paddr_ctx(ctx);
+
+ r = damon_new_region(0x10000, 0x20000);
+ if (!r) {
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "region alloc failed");
+ }
+ damon_add_region(r, t);
+ damon_add_target(ctx, t);
+
+ rep.report_jiffies = jiffies;
+ before = damon_get_samples_drained();
+ damon_report_access(&rep);
+ kdamond_check_reported_accesses(ctx);
+ after = damon_get_samples_drained();
+
+ hits = 0;
+ damon_for_each_region(r, t)
+ hits += r->probe_hits[0];
+
+ KUNIT_EXPECT_EQ(test, hits, 0); /* probe_idx 0: no probe_hits */
+ KUNIT_EXPECT_GT(test, after, before);
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test pf-ring cold-demote signal: complements ring0_pagefault_credit
+ * (which proves the HOT side). This proves the COLD side.
+ *
+ * Create a paddr ctx with the page_fault primitive enabled (no probes) and one
+ * target with TWO regions: HOT [0x10000, 0x20000) and COLD [0x20000, 0x30000).
+ * Prime both regions to the same nonzero nr_accesses so aging is observable.
+ * Inject a single page_fault report (probe_idx=0 -> pf ring) hitting only the
+ * HOT region, then run the drain + zero-access-report aging pass:
+ *
+ * kdamond_check_reported_accesses() credits the HOT region: it gets a
+ * nr_accesses bump and access_reported=true; the COLD region is untouched.
+ *
+ * kdamond_apply_zero_access_report() clears access_reported on the HOT
+ * region (keeping its nr_accesses) and, because the COLD region was NOT
+ * reported this tick, calls damon_update_region_access_rate(r, false) on it
+ * (no credit) so the COLD region does not advance.
+ *
+ * After the cycle the HOT region's nr_accesses has pulled ahead of the COLD
+ * region's (HOT > COLD) -- the cold-demote signal a migrate_cold scheme acts
+ * on -- and access_reported is false on both regions again.
+ */
+static void damon_test_ring0_pagefault_cold_demote(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_target *t;
+ struct damon_region *hot, *cold;
+ struct damon_access_report rep = {
+ .paddr = 0x15000, /* inside HOT region */
+ .vaddr = 0,
+ .tid = 0,
+ .probe_idx = 0, /* DAMON_PROBE_IDX_NONE -> global pf ring */
+ .size = PAGE_SIZE,
+ };
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ /* page_fault primitive: routes drain to the global pf ring. */
+ ctx->sample_control.primitives_enabled.page_table = false;
+ ctx->sample_control.primitives_enabled.page_fault = true;
+
+ t = damon_new_target();
+ if (!t) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "target alloc failed");
+ }
+ t->pid = NULL; /* paddr target: no pid */
+ damon_test_set_paddr_ctx(ctx);
+
+ hot = damon_new_region(0x10000, 0x20000);
+ if (!hot) {
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "hot region alloc failed");
+ }
+ damon_add_region(hot, t);
+
+ cold = damon_new_region(0x20000, 0x30000);
+ if (!cold) {
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "cold region alloc failed");
+ }
+ damon_add_region(cold, t);
+ damon_add_target(ctx, t);
+
+ /* Prime both regions to the same nonzero access rate. */
+ hot->nr_accesses = 4;
+ cold->nr_accesses = 4;
+
+ /* Inject a pf-ring report hitting only the HOT region, then drain. */
+ rep.report_jiffies = jiffies;
+ damon_report_access(&rep);
+ kdamond_check_reported_accesses(ctx);
+
+ /* HOT reported this tick; COLD was not. */
+ KUNIT_EXPECT_TRUE(test, hot->access_reported);
+ KUNIT_EXPECT_FALSE(test, cold->access_reported);
+
+ /* Aging pass: HOT keeps its (credited) rate; COLD is not credited. */
+ kdamond_apply_zero_access_report(ctx);
+
+ /* Cold-demote signal: HOT pulled ahead of COLD. */
+ KUNIT_EXPECT_GT(test, hot->nr_accesses, cold->nr_accesses);
+ /* access_reported cleared on both after the zero-report pass. */
+ KUNIT_EXPECT_FALSE(test, hot->access_reported);
+ KUNIT_EXPECT_FALSE(test, cold->access_reported);
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test perf-ring credit: a perf ctx drains its own per-ctx perf ring.
+ *
+ * Create a paddr ctx with an event-driven probe (which allocates the ctx's
+ * per-ctx perf ring), region [0x10000, 0x20000). Inject a probe_idx=1 paddr
+ * entry tagged with ctx, so it lands in that ctx's perf ring. After drain:
+ * probe_hits[0]==1.
+ */
+static void damon_test_ring1_perf_credit(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_target *t;
+ struct damon_region *r;
+ struct damon_access_report rep = {
+ .paddr = 0x15000,
+ .vaddr = 0,
+ .tid = 0,
+ .probe_idx = 1, /* -> per-ctx perf ring */
+ .size = PAGE_SIZE,
+ };
+ unsigned long before, after;
+ int hits;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ if (damon_test_attach_perf_probe(ctx)) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "perf probe alloc failed");
+ }
+
+ t = damon_new_target();
+ if (!t) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "target alloc failed");
+ }
+ t->pid = NULL;
+ damon_test_set_paddr_ctx(ctx);
+ rep.ctx = ctx;
+
+ r = damon_new_region(0x10000, 0x20000);
+ if (!r) {
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "region alloc failed");
+ }
+ damon_add_region(r, t);
+ damon_add_target(ctx, t);
+
+ rep.report_jiffies = jiffies;
+ before = damon_get_samples_drained();
+ damon_report_access(&rep);
+ kdamond_check_reported_accesses(ctx);
+ after = damon_get_samples_drained();
+
+ hits = 0;
+ damon_for_each_region(r, t)
+ hits += r->probe_hits[0];
+
+ KUNIT_EXPECT_EQ(test, hits, 1);
+ KUNIT_EXPECT_GT(test, after, before);
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test ring partition: a perf-only ctx does NOT consume a pf-ring entry.
+ *
+ * Inject a probe_idx=0 (global pf ring) entry, then drain with a perf-only ctx
+ * (event probe, page_fault disabled). The perf ctx drains only its per-ctx
+ * perf ring, which is empty, so samples_drained must NOT change. A subsequent
+ * pf ctx drain consumes the entry (samples_drained increments), proving the
+ * entry was partitioned into the global pf ring and left untouched by the perf
+ * drain.
+ */
+static void damon_test_ring_partition(struct kunit *test)
+{
+ struct damon_ctx *perf_ctx, *pf_ctx;
+ struct damon_target *t_perf, *t_pf;
+ struct damon_region *r;
+ struct damon_access_report rep = {
+ .paddr = 0x15000,
+ .vaddr = 0,
+ .tid = 0,
+ .probe_idx = 0, /* DAMON_PROBE_IDX_NONE -> global pf ring */
+ .size = PAGE_SIZE,
+ };
+ unsigned long before, mid, after;
+
+ perf_ctx = damon_new_ctx();
+ if (!perf_ctx)
+ kunit_skip(test, "perf ctx alloc failed");
+ if (damon_test_attach_perf_probe(perf_ctx)) {
+ damon_destroy_ctx(perf_ctx);
+ kunit_skip(test, "perf probe alloc failed");
+ }
+ t_perf = damon_new_target();
+ if (!t_perf) {
+ damon_destroy_ctx(perf_ctx);
+ kunit_skip(test, "perf target alloc failed");
+ }
+ t_perf->pid = NULL;
+ damon_test_set_paddr_ctx(perf_ctx);
+ r = damon_new_region(0x10000, 0x20000);
+ if (!r) {
+ damon_free_target(t_perf);
+ damon_destroy_ctx(perf_ctx);
+ kunit_skip(test, "perf region alloc failed");
+ }
+ damon_add_region(r, t_perf);
+ damon_add_target(perf_ctx, t_perf);
+
+ pf_ctx = damon_new_ctx();
+ if (!pf_ctx) {
+ damon_destroy_ctx(perf_ctx);
+ kunit_skip(test, "pf ctx alloc failed");
+ }
+ pf_ctx->sample_control.primitives_enabled.page_table = false;
+ pf_ctx->sample_control.primitives_enabled.page_fault = true;
+ t_pf = damon_new_target();
+ if (!t_pf) {
+ damon_destroy_ctx(pf_ctx);
+ damon_destroy_ctx(perf_ctx);
+ kunit_skip(test, "pf target alloc failed");
+ }
+ t_pf->pid = NULL;
+ damon_test_set_paddr_ctx(pf_ctx);
+ r = damon_new_region(0x10000, 0x20000);
+ if (!r) {
+ damon_free_target(t_pf);
+ damon_destroy_ctx(pf_ctx);
+ damon_destroy_ctx(perf_ctx);
+ kunit_skip(test, "pf region alloc failed");
+ }
+ damon_add_region(r, t_pf);
+ damon_add_target(pf_ctx, t_pf);
+
+ rep.report_jiffies = jiffies;
+ before = damon_get_samples_drained();
+ damon_report_access(&rep); /* lands in the global pf ring */
+ kdamond_check_reported_accesses(perf_ctx); /* drains perf ring only */
+ mid = damon_get_samples_drained();
+ kdamond_check_reported_accesses(pf_ctx); /* drains pf ring */
+ after = damon_get_samples_drained();
+
+ KUNIT_EXPECT_EQ(test, mid, before); /* perf drain left pf entry */
+ KUNIT_EXPECT_GT(test, after, mid); /* pf drain consumed it */
+
+ damon_destroy_ctx(pf_ctx);
+ damon_destroy_ctx(perf_ctx);
+}
+
+/*
+ * Test per-context perf ring isolation.
+ *
+ * Two independent perf ctxs (each with its own event-driven probe and its own
+ * per-ctx perf ring) each receive one probe_idx=1 report tagged with their
+ * respective ctx. Each ctx must credit exactly its own report and see nothing
+ * from the other: proof that perf reports route to the owning ctx's ring, and
+ * that two perf-driven ctxs coexist without a shared ring or a cross-ctx owner
+ * guard (unlike the global perf ring, which allowed only one perf drainer).
+ */
+static void damon_test_perf_per_ctx_isolation(struct kunit *test)
+{
+ struct damon_ctx *ctx_a, *ctx_b;
+ struct damon_target *ta, *tb;
+ struct damon_region *ra, *rb;
+ struct damon_access_report rep_a = {
+ .paddr = 0x15000, .probe_idx = 1, .size = PAGE_SIZE,
+ };
+ struct damon_access_report rep_b = {
+ .paddr = 0x35000, .probe_idx = 1, .size = PAGE_SIZE,
+ };
+ int hits_a, hits_b;
+
+ ctx_a = damon_new_ctx();
+ ctx_b = damon_new_ctx();
+ if (!ctx_a || !ctx_b) {
+ if (ctx_a)
+ damon_destroy_ctx(ctx_a);
+ if (ctx_b)
+ damon_destroy_ctx(ctx_b);
+ kunit_skip(test, "ctx alloc failed");
+ }
+ if (damon_test_attach_perf_probe(ctx_a) ||
+ damon_test_attach_perf_probe(ctx_b)) {
+ damon_destroy_ctx(ctx_a);
+ damon_destroy_ctx(ctx_b);
+ kunit_skip(test, "perf probe alloc failed");
+ }
+
+ ta = damon_new_target();
+ tb = damon_new_target();
+ if (!ta || !tb) {
+ if (ta)
+ damon_free_target(ta);
+ if (tb)
+ damon_free_target(tb);
+ damon_destroy_ctx(ctx_a);
+ damon_destroy_ctx(ctx_b);
+ kunit_skip(test, "target alloc failed");
+ }
+ ta->pid = NULL;
+ tb->pid = NULL;
+ damon_test_set_paddr_ctx(ctx_a);
+ damon_test_set_paddr_ctx(ctx_b);
+
+ ra = damon_new_region(0x10000, 0x20000); /* holds rep_a paddr */
+ rb = damon_new_region(0x30000, 0x40000); /* holds rep_b paddr */
+ if (!ra || !rb) {
+ if (ra)
+ damon_free_region(ra);
+ if (rb)
+ damon_free_region(rb);
+ damon_free_target(ta);
+ damon_free_target(tb);
+ damon_destroy_ctx(ctx_a);
+ damon_destroy_ctx(ctx_b);
+ kunit_skip(test, "region alloc failed");
+ }
+ damon_add_region(ra, ta);
+ damon_add_target(ctx_a, ta);
+ damon_add_region(rb, tb);
+ damon_add_target(ctx_b, tb);
+
+ /* Each report is tagged with its owning ctx. */
+ rep_a.ctx = ctx_a;
+ rep_b.ctx = ctx_b;
+ rep_a.report_jiffies = jiffies;
+ rep_b.report_jiffies = jiffies;
+
+ /*
+ * Report into both ctx rings, then drain each ctx. ctx_a must credit
+ * only rep_a; ctx_b must credit only rep_b -- no cross-talk, and no
+ * -EBUSY from a second perf drainer.
+ */
+ damon_report_access(&rep_a);
+ damon_report_access(&rep_b);
+ kdamond_check_reported_accesses(ctx_a);
+ kdamond_check_reported_accesses(ctx_b);
+
+ hits_a = 0;
+ damon_for_each_region(ra, ta)
+ hits_a += ra->probe_hits[0];
+ hits_b = 0;
+ damon_for_each_region(rb, tb)
+ hits_b += rb->probe_hits[0];
+
+ KUNIT_EXPECT_EQ(test, hits_a, 1); /* ctx_a credited its own report */
+ KUNIT_EXPECT_EQ(test, hits_b, 1); /* ctx_b credited its own report */
+
+ damon_destroy_ctx(ctx_a);
+ damon_destroy_ctx(ctx_b);
+}
+
+/*
+ * Test pf-ring single-owner claim (page_fault only).
+ *
+ * The global pf ring is a destructive SPSC channel with a single owner:
+ * two ctxs draining it in one batch must be rejected with -EBUSY. The perf
+ * ring has no such batch-claim owner (it is per-ctx), so only the pf owner
+ * helper is exercised here.
+ */
+static void damon_test_pf_ring_owner_ebusy(struct kunit *test)
+{
+ struct damon_ctx *a, *b;
+ struct damon_ctx *arr[2];
+ struct damon_ctx *owner_pf;
+ int err;
+
+ a = damon_new_ctx();
+ b = damon_new_ctx();
+ if (!a || !b) {
+ if (a)
+ damon_destroy_ctx(a);
+ if (b)
+ damon_destroy_ctx(b);
+ kunit_skip(test, "ctx alloc failed");
+ }
+
+ /* Both drain the global pf ring: same-ring collision -> -EBUSY. */
+ a->sample_control.primitives_enabled.page_table = false;
+ a->sample_control.primitives_enabled.page_fault = true;
+ b->sample_control.primitives_enabled.page_table = false;
+ b->sample_control.primitives_enabled.page_fault = true;
+ arr[0] = a;
+ arr[1] = b;
+ owner_pf = NULL;
+ err = damon_claim_ring_owner_start(arr, 2, damon_drains_ring_pf,
+ &owner_pf);
+ KUNIT_EXPECT_EQ(test, err, -EBUSY);
+
+ /* Only a drains pf: single owner, claimed to a. */
+ b->sample_control.primitives_enabled.page_fault = false;
+ owner_pf = NULL;
+ err = damon_claim_ring_owner_start(arr, 2, damon_drains_ring_pf,
+ &owner_pf);
+ KUNIT_EXPECT_EQ(test, err, 0);
+ KUNIT_EXPECT_PTR_EQ(test, owner_pf, a);
+
+ damon_destroy_ctx(a);
+ damon_destroy_ctx(b);
+}
+
+/*
+ * Test the queued/dropped return value, and that a ring-full drop is counted
+ * as ring-full rather than busy-guard.
+ *
+ * A per-context perf ring is private to its ctx, so a freshly created ctx
+ * starts with an empty ring nobody else writes to and the counts are exact.
+ * Preemption is held across the loop so every report targets the same CPU's
+ * ring, per the SPSC invariant damon_report_access() documents.
+ *
+ * A ring holds DAMON_REPORT_RING_SIZE - 1 entries (one slot is kept empty to
+ * distinguish full from empty), so exactly that many reports are queued and
+ * every one after that is dropped.
+ */
+static void damon_test_report_return_value(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_access_report rep = {
+ .paddr = 0x15000, .probe_idx = 1, .size = PAGE_SIZE,
+ };
+ unsigned long full_before, busy_before;
+ unsigned int queued = 0, dropped = 0;
+ int i;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ if (damon_test_attach_perf_probe(ctx)) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "perf probe alloc failed");
+ }
+ rep.ctx = ctx;
+
+ preempt_disable();
+ full_before = damon_get_report_ring_full();
+ busy_before = damon_get_report_busy_drop();
+
+ /* One past capacity, so the last iteration must be a drop. */
+ for (i = 0; i < DAMON_REPORT_RING_SIZE; i++) {
+ if (damon_report_access(&rep))
+ queued++;
+ else
+ dropped++;
+ }
+ preempt_enable();
+
+ KUNIT_EXPECT_EQ(test, queued, (unsigned int)DAMON_REPORT_RING_SIZE - 1);
+ KUNIT_EXPECT_EQ(test, dropped, 1u);
+ /* No NMI nests here, so the drop must be the full ring. */
+ KUNIT_EXPECT_GT(test, damon_get_report_ring_full(), full_before);
+ KUNIT_EXPECT_EQ(test, damon_get_report_busy_drop(), busy_before);
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test that draining restores capacity: fill the ring, drain it via the
+ * dispatcher, then report again and expect the report to be queued.
+ */
+static void damon_test_report_drain_restores_capacity(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_target *t;
+ struct damon_region *r;
+ struct damon_access_report rep = {
+ .paddr = 0x15000, .probe_idx = 1, .size = PAGE_SIZE,
+ };
+ int i;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ if (damon_test_attach_perf_probe(ctx)) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "perf probe alloc failed");
+ }
+
+ t = damon_new_target();
+ if (!t) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "target alloc failed");
+ }
+ t->pid = NULL;
+ damon_test_set_paddr_ctx(ctx);
+ rep.ctx = ctx;
+
+ r = damon_new_region(0x10000, 0x20000);
+ if (!r) {
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "region alloc failed");
+ }
+ damon_add_region(r, t);
+ damon_add_target(ctx, t);
+
+ /* Fill the ring: the last report is dropped. */
+ preempt_disable();
+ for (i = 0; i < DAMON_REPORT_RING_SIZE; i++)
+ damon_report_access(&rep);
+ KUNIT_EXPECT_FALSE(test, damon_report_access(&rep));
+ preempt_enable();
+
+ kdamond_check_reported_accesses(ctx);
+
+ /* Capacity is back. */
+ preempt_disable();
+ KUNIT_EXPECT_TRUE(test, damon_report_access(&rep));
+ preempt_enable();
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test that a page_fault report and an event-driven probe report credit the
+ * same region in one context.
+ *
+ * Create a paddr ctx with the page_fault primitive enabled AND a weighted
+ * event-driven probe, region [0x10000, 0x20000). Inject one report on each
+ * ring: probe_idx=0 (pf ring) and probe_idx=1 (perf ring). The drain
+ * dispatcher drains both, so the access rate is credited twice while only the
+ * probe report lands in probe_hits[].
+ */
+static void damon_test_pf_and_probe_one_ctx(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_target *t;
+ struct damon_region *r;
+ struct damon_probe *p;
+ struct damon_access_report pf_rep = {
+ .paddr = 0x15000,
+ .probe_idx = 0, /* DAMON_PROBE_IDX_NONE -> global pf ring */
+ .size = PAGE_SIZE,
+ };
+ struct damon_access_report probe_rep = {
+ .paddr = 0x16000,
+ .probe_idx = 1, /* -> per-ctx perf ring */
+ .size = PAGE_SIZE,
+ };
+ unsigned long before, after;
+ int hits;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ if (damon_test_attach_perf_probe(ctx)) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "perf probe alloc failed");
+ }
+ /* A nonzero weight selects the probe-weighted score. */
+ damon_for_each_probe(p, ctx)
+ p->weight = 1;
+ ctx->sample_control.primitives_enabled.page_table = false;
+ ctx->sample_control.primitives_enabled.page_fault = true;
+ probe_rep.ctx = ctx;
+
+ t = damon_new_target();
+ if (!t) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "target alloc failed");
+ }
+ t->pid = NULL; /* paddr target: no pid */
+ damon_test_set_paddr_ctx(ctx);
+
+ r = damon_new_region(0x10000, 0x20000);
+ if (!r) {
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "region alloc failed");
+ }
+ damon_add_region(r, t);
+ damon_add_target(ctx, t);
+
+ pf_rep.report_jiffies = jiffies;
+ probe_rep.report_jiffies = jiffies;
+ before = damon_get_samples_drained();
+ damon_report_access(&pf_rep);
+ damon_report_access(&probe_rep);
+ kdamond_check_reported_accesses(ctx);
+ after = damon_get_samples_drained();
+
+ hits = 0;
+ damon_for_each_region(r, t)
+ hits += r->probe_hits[0];
+
+ /* Both reports credited the access rate. */
+ KUNIT_EXPECT_EQ(test, after - before, 2ul);
+ /* Only the probe report has a probe_hits[] slot. */
+ KUNIT_EXPECT_EQ(test, hits, 1);
+ /* The weighted sum sees the probe hit. */
+ damon_for_each_region(r, t)
+ KUNIT_EXPECT_EQ(test,
+ damon_probe_hits_wsum(r, false, false, ctx), 1u);
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test that a report is matched by the address space of the target rather than
+ * by which address it carries.
+ *
+ * Create a paddr ctx with the page_fault primitive, region
+ * [0x10000, 0x20000). Inject a report whose vaddr falls inside that region
+ * and whose paddr falls outside it. A paddr target matches the paddr, so the
+ * report finds no region and is counted as such.
+ */
+static void damon_test_report_addr_space_keyed(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_target *t;
+ struct damon_region *r;
+ struct damon_access_report rep = {
+ .paddr = 0x95000, /* outside the region */
+ .vaddr = 0x15000, /* inside the region */
+ .tid = 0,
+ .probe_idx = 0,
+ .size = PAGE_SIZE,
+ };
+ unsigned long before, after;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ ctx->sample_control.primitives_enabled.page_table = false;
+ ctx->sample_control.primitives_enabled.page_fault = true;
+
+ t = damon_new_target();
+ if (!t) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "target alloc failed");
+ }
+ t->pid = NULL; /* paddr target: no pid */
+ damon_test_set_paddr_ctx(ctx);
+
+ r = damon_new_region(0x10000, 0x20000);
+ if (!r) {
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "region alloc failed");
+ }
+ damon_add_region(r, t);
+ damon_add_target(ctx, t);
+
+ rep.report_jiffies = jiffies;
+ before = damon_get_samples_no_region();
+ damon_report_access(&rep);
+ kdamond_check_reported_accesses(ctx);
+ after = damon_get_samples_no_region();
+
+ /* The vaddr was not used to match a paddr target. */
+ KUNIT_EXPECT_GT(test, after, before);
+ damon_for_each_region(r, t)
+ KUNIT_EXPECT_EQ(test, r->nr_accesses, 0u);
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test the hot side of the page-fault signal: a reported region scores hotter
+ * than a region that reported nothing.
+ *
+ * Create a paddr ctx with the page_fault primitive and one target with two
+ * regions. Report an access on the first region only, over several sampling
+ * ticks. Each tick credits the reported region and withholds credit from the
+ * other one, so the access rate of the first grows while the second stays at
+ * zero, and the hot score of the first is higher while the cold score is
+ * reversed.
+ */
+static void damon_test_pagefault_hot_promote(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_target *t;
+ struct damon_region *hot, *cold;
+ struct damos scheme = {
+ .quota = { .weight_nr_accesses = 100, .weight_age = 0 },
+ };
+ struct damon_access_report rep = {
+ .paddr = 0x15000,
+ .probe_idx = 0,
+ .size = PAGE_SIZE,
+ };
+ int i;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ ctx->sample_control.primitives_enabled.page_table = false;
+ ctx->sample_control.primitives_enabled.page_fault = true;
+
+ t = damon_new_target();
+ if (!t) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "target alloc failed");
+ }
+ t->pid = NULL; /* paddr target: no pid */
+ damon_test_set_paddr_ctx(ctx);
+
+ hot = damon_new_region(0x10000, 0x20000);
+ cold = damon_new_region(0x20000, 0x30000);
+ if (!hot || !cold) {
+ if (hot)
+ damon_free_region(hot);
+ if (cold)
+ damon_free_region(cold);
+ damon_free_target(t);
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "region alloc failed");
+ }
+ damon_add_region(hot, t);
+ damon_add_region(cold, t);
+ damon_add_target(ctx, t);
+
+ /* Both regions start from a zero access rate. */
+ KUNIT_EXPECT_EQ(test, hot->nr_accesses, 0u);
+ KUNIT_EXPECT_EQ(test, cold->nr_accesses, 0u);
+
+ /*
+ * One sampling tick per iteration: report on the hot region, drain the
+ * ring, then run the zero-access pass over the regions that reported
+ * nothing.
+ */
+ for (i = 0; i < 3; i++) {
+ rep.report_jiffies = jiffies;
+ damon_report_access(&rep);
+ kdamond_check_reported_accesses(ctx);
+ kdamond_apply_zero_access_report(ctx);
+ }
+
+ KUNIT_EXPECT_EQ(test, hot->nr_accesses, 3u);
+ KUNIT_EXPECT_EQ(test, cold->nr_accesses, 0u);
+ KUNIT_EXPECT_GT(test, hot->nr_accesses, cold->nr_accesses);
+ KUNIT_EXPECT_GT(test, damon_hot_score(ctx, hot, &scheme),
+ damon_hot_score(ctx, cold, &scheme));
+ KUNIT_EXPECT_LT(test, damon_cold_score(ctx, hot, &scheme),
+ damon_cold_score(ctx, cold, &scheme));
+
+ damon_destroy_ctx(ctx);
+}
+
+static struct kunit_case damon_drain_test_cases[] = {
+ KUNIT_CASE(damon_test_unified_vaddr_match),
+ KUNIT_CASE(damon_test_unified_vaddr_tgid_mismatch),
+ KUNIT_CASE(damon_test_unified_paddr_no_regression),
+ KUNIT_CASE(damon_test_ring0_pagefault_credit),
+ KUNIT_CASE(damon_test_ring0_pagefault_cold_demote),
+ KUNIT_CASE(damon_test_ring1_perf_credit),
+ KUNIT_CASE(damon_test_ring_partition),
+ KUNIT_CASE(damon_test_perf_per_ctx_isolation),
+ KUNIT_CASE(damon_test_pf_ring_owner_ebusy),
+ KUNIT_CASE(damon_test_report_return_value),
+ KUNIT_CASE(damon_test_report_drain_restores_capacity),
+ KUNIT_CASE(damon_test_pf_and_probe_one_ctx),
+ KUNIT_CASE(damon_test_report_addr_space_keyed),
+ KUNIT_CASE(damon_test_pagefault_hot_promote),
+ {}
+};
+
+static struct kunit_suite damon_drain_test_suite = {
+ .name = "damon_drain",
+ .test_cases = damon_drain_test_cases,
+};
+kunit_test_suite(damon_drain_test_suite);
+
+#endif /* _DAMON_DRAIN_KUNIT_H */
+
+#endif /* CONFIG_DAMON_KUNIT_TEST */
diff --git a/mm/damon/tests/perf-kunit.h b/mm/damon/tests/perf-kunit.h
new file mode 100644
index 0000000000000..08e8b911f4d94
--- /dev/null
+++ b/mm/damon/tests/perf-kunit.h
@@ -0,0 +1,133 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * DAMON kunit tests for the per-context perf report ring.
+ *
+ * Included at the bottom of core.c, after tests/drain-kunit.h, whose
+ * damon_test_attach_perf_probe() helper these tests reuse.
+ */
+
+#ifdef CONFIG_DAMON_KUNIT_TEST
+
+#ifndef _DAMON_PERF_KUNIT_H
+#define _DAMON_PERF_KUNIT_H
+
+#include <kunit/test.h>
+#include <linux/damon.h>
+
+/*
+ * A report with probe_idx >= 1 is enqueued into the ring of the context named
+ * by report->ctx, so these tests build a context with an allocated perf ring
+ * and point the injected reports at it. A freshly allocated ring is empty,
+ * which makes the accepted and rejected counts below exact.
+ */
+
+/*
+ * Test A: perf ring basic write
+ *
+ * Inject reports into a context's perf ring via damon_report_access() and
+ * verify each one is accepted.
+ */
+static void damon_test_perf_ring_basic(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_access_report report = {
+ .paddr = 0x1000, .size = PAGE_SIZE, .probe_idx = 1,
+ };
+ int i;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ if (damon_test_attach_perf_probe(ctx)) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "perf ring alloc failed");
+ }
+ report.ctx = ctx;
+
+ for (i = 0; i < 3; i++)
+ KUNIT_EXPECT_TRUE(test, damon_report_access(&report));
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test B: ring overflow is reported and does not corrupt head/tail
+ *
+ * Fill a context's perf ring to capacity and verify that further writes are
+ * refused and counted rather than overwriting live entries. The ring holds
+ * DAMON_REPORT_RING_SIZE - 1 entries, one slot being reserved to distinguish
+ * full from empty, so the last two of the writes below must be refused.
+ */
+static void damon_test_perf_ring_overflow_safety(struct kunit *test)
+{
+ struct damon_ctx *ctx;
+ struct damon_access_report report = {
+ .paddr = 0x3000, .size = PAGE_SIZE, .probe_idx = 1,
+ };
+ unsigned long overflow_before, overflow_after;
+ int queued = 0, refused = 0;
+ int i;
+
+ ctx = damon_new_ctx();
+ if (!ctx)
+ kunit_skip(test, "ctx alloc failed");
+ if (damon_test_attach_perf_probe(ctx)) {
+ damon_destroy_ctx(ctx);
+ kunit_skip(test, "perf ring alloc failed");
+ }
+ report.ctx = ctx;
+
+ /* Pinned so every write lands in the same CPU's ring. */
+ preempt_disable();
+ overflow_before = damon_get_report_overflow();
+
+ for (i = 0; i < DAMON_REPORT_RING_SIZE + 1; i++) {
+ if (damon_report_access(&report))
+ queued++;
+ else
+ refused++;
+ }
+
+ overflow_after = damon_get_report_overflow();
+ preempt_enable();
+
+ KUNIT_EXPECT_EQ(test, queued, DAMON_REPORT_RING_SIZE - 1);
+ KUNIT_EXPECT_EQ(test, refused, 2);
+ KUNIT_EXPECT_GT(test, overflow_after, overflow_before);
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Test C: a perf report without an owning context is refused
+ *
+ * A report with probe_idx >= 1 but no ctx cannot be routed to a ring. Verify
+ * it is refused instead of dereferenced, which is what an overflow arriving
+ * after its context's ring was freed looks like.
+ */
+static void damon_test_perf_report_requires_ctx(struct kunit *test)
+{
+ struct damon_access_report report = {
+ .paddr = 0x5000, .size = PAGE_SIZE, .probe_idx = 1,
+ .ctx = NULL,
+ };
+
+ KUNIT_EXPECT_FALSE(test, damon_report_access(&report));
+}
+
+static struct kunit_case damon_perf_test_cases[] = {
+ KUNIT_CASE(damon_test_perf_ring_basic),
+ KUNIT_CASE(damon_test_perf_ring_overflow_safety),
+ KUNIT_CASE(damon_test_perf_report_requires_ctx),
+ {}
+};
+
+static struct kunit_suite damon_perf_test_suite = {
+ .name = "damon_perf",
+ .test_cases = damon_perf_test_cases,
+};
+kunit_test_suite(damon_perf_test_suite);
+
+#endif /* _DAMON_PERF_KUNIT_H */
+
+#endif /* CONFIG_DAMON_KUNIT_TEST */
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
` (8 preceding siblings ...)
2026-09-10 17:16 ` [RFC PATCH v2 9/9] mm/damon/tests/drain-kunit: kunit for report rings and unified drain Ravi Jonnalagadda
@ 2026-09-11 0:34 ` SJ Park
2026-09-12 1:38 ` SJ Park
10 siblings, 0 replies; 12+ messages in thread
From: SJ Park @ 2026-09-11 0:34 UTC (permalink / raw)
To: Ravi Jonnalagadda
Cc: SJ Park, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc,
akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
rientjes, weixugc, jic23, gourry
On Thu, 10 Sep 2026 10:16:14 -0700 Ravi Jonnalagadda <ravis.opensrc@gmail.com> wrote:
> This series lets DAMON take its access information from a hardware sampler
> instead of from a page-table scan, and lets a scheme's score be weighted by what
> that sampler reported.
>
> The change from v1 [1] is that it is now built on the data attribute probe
> infrastructure that has since landed in mm-new: a PMU is expressed as one more
> probe on a context, with its own weight and probe hits, rather than as a
> mechanism of its own beside the ops set. The rest of this posting is the same
> substrate reworked onto that, plus two consumers.
>
> This is not a merge request. The series is based on damon/next at the
> base-commit below, which moves, so the tree it was built and tested from is also
> on
>
> https://github.com/ravis-opensrc/linux/tree/damon/perf-rfc-v2-09-06-26
>
> It is posted for design feedback, on the substrate and on where it belongs in
> the roadmap for extending DAMON beyond the pte-accessed bit [2]. That roadmap's
> second milestone, now open, is a first data attribute monitored through
> `damon_report_access()`, and that is what a sampling PMU is here. So this
> series keeps that function and its callers and replaces its body: the reporting
> path a hardware sampler needs cannot take a mutex, and the drain has to reach a
> virtual-address context as well as a physical one. The shape of the ring, the
> drain and the sysfs surface are what is most useful to review.
Thank you for this series, Ravi! I will take time to thoughly read and reply
this series by the end of this weekend.
Thanks,
SJ
[...]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
` (9 preceding siblings ...)
2026-09-11 0:34 ` [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports SJ Park
@ 2026-09-12 1:38 ` SJ Park
10 siblings, 0 replies; 12+ messages in thread
From: SJ Park @ 2026-09-12 1:38 UTC (permalink / raw)
To: Ravi Jonnalagadda
Cc: SJ Park, akinobu.mita, damon, linux-mm, linux-kernel, linux-doc,
akpm, corbet, bijan311, ajayjoshi, honggyu.kim, yunjeong.mun,
rientjes, weixugc, jic23, gourry
Hello Ravi,
On Thu, 10 Sep 2026 10:16:14 -0700 Ravi Jonnalagadda <ravis.opensrc@gmail.com> wrote:
> This series lets DAMON take its access information from a hardware sampler
> instead of from a page-table scan, and lets a scheme's score be weighted by what
> that sampler reported.
>
> The change from v1 [1] is that it is now built on the data attribute probe
> infrastructure that has since landed in mm-new: a PMU is expressed as one more
> probe on a context, with its own weight and probe hits, rather than as a
> mechanism of its own beside the ops set. The rest of this posting is the same
> substrate reworked onto that, plus two consumers.
>
> This is not a merge request. The series is based on damon/next at the
> base-commit below, which moves, so the tree it was built and tested from is also
> on
>
> https://github.com/ravis-opensrc/linux/tree/damon/perf-rfc-v2-09-06-26
>
> It is posted for design feedback, on the substrate and on where it belongs in
> the roadmap for extending DAMON beyond the pte-accessed bit [2]. That roadmap's
> second milestone, now open, is a first data attribute monitored through
> `damon_report_access()`, and that is what a sampling PMU is here. So this
> series keeps that function and its callers and replaces its body: the reporting
> path a hardware sampler needs cannot take a mutex, and the drain has to reach a
> virtual-address context as well as a physical one. The shape of the ring, the
> drain and the sysfs surface are what is most useful to review.
First of all, thank you very much for sharing this great series.
I think the high level direction, especially utilizing data attributes
infrastructure and interface is good and aligned with our beyond-page table
accessed-bit plan.
Nevertheless, this version is highly ambitious, optimized and big. We will
land only essential parts in milestone 2. The goal of milestone 2 is only
essential functionality. We will further extend its functionality and optimize
its performance, in multiple and parallel steps, in the phase for the
milestone 3 (or, simply phase 3).
I understand that's also what you are aligned with. And you had to post this
big series mainly because we just started milestone 2, and therefore you have
nothing to really base on.
Let me further provide some comments including what looks especially big and
optimized to me, and how those could be split into small pieces.
>
> ## Why a unified perf-event substrate
>
> DAMON derives its access information from the PTE Accessed bit. A sampling PMU
> carries what that bit cannot: which addresses the hardware went to, and how
> often it went there. Many machines already have such a unit, and more than one
> kind of it, so what this series is after is letting DAMON's regions be tuned
> from whichever perf-based hardware source a machine offers rather than from the
> Accessed bit alone. A sampler does not arrive on a kdamond's terms, though: it
> delivers an address when the hardware decides to, in NMI context, with no
> relation to the sampling interval and no mm to walk.
>
> The alternative is a backend per PMU vendor, each owning its own configuration,
> sysfs knobs and lifecycle. The perf-event direction [3] avoids that: let DAMON
> register kernel-counter perf events and consume samples from any sampling PMU the
> perf core already knows about. This series follows it, adding one substrate below
> the ops sets rather than an ops set per PMU -- a report ring that any in-kernel
> access source can push into, and a drain that folds those reports into region
> probe hits on the aggregation boundary the kdamond already has.
This perfectly aligns with the plan. We will use perf event abstraction for
AMD IBS-like h/w features based access monitoring primitives.
> What running it
> across vendors needed on top of that direction is:
>
> - per-CPU lockless rings between the NMI sample handler and the kdamond drain,
I understand we need to make it lockless. I wonder if we have to make it
per-CPU. I understand it will be better in terms of performance, especially on
machines having many CPUs. That said, this feels like somewhat we can discuss
in phase 3. And it would deserve to have sufficient discussions and
performance evaluations.
> - a ring partitioned by probe class, so the page-fault primitive and a PMU can
> populate one context without either seeing the other's reports,
This also feels like something that we can discuss later in phase 3.
Particularly, page-fault primitive is out of the scope of milestone 2. The
path forward and the timeline for it are quite unclear as of now. It requires
alignments with other subsystems that would be challenging and taking time. I
have a very rough plan to try it again with read-write protection, though. I
think it might be safer to just keep it out of the scope of your project for
now.
> - per-CPU events that follow CPU hotplug, armed when the kdamond starts and
> disarmed and drained when it stops,
> - a per-PMU owner, so two contexts cannot claim the same PMU type,
> - whichever address a PMU does report carried on the report and matched
> against the context's own address space, so one source serves a paddr or a
> vaddr context without a backend per address space.
These all soudns making sense to me. Nonetheless, I think we can scope
milestone 2 to support only physical address and defer these things to the
phase 3.
>
> This is tested with PEBS on Intel and IBS on AMD, both configured as `perf_event`
> attributes on a probe and using the perf core's event plumbing rather than
> per-vendor MSR code. A third source has already been written against the same
> ring: Kunwu Chan's ARM SPE backend [5], which reaches it through an AUX buffer
> drained in process context instead of an overflow callback, and which the
> roadmap [2] places in its third milestone.
Awesome, appreciate your huge effort on this!
>
> The partitioning is what lets promotion and demotion run in one context. A
> sampler says which regions are hot; it says nothing about which are cold, because
> a sampler that reports nothing about a page cannot distinguish untouched from
> unsampled.
I'm not really sure. I think absence of samples for an address range can also
mean the address range is cold? Actually the page table accessed bit based
monitoring also use a sort of sampling, so I don't show real distinction.
Maybe you're right, but I think this deserves sufficient discussions and
testing that we could defer to the phase 3.
> Region age is what a demotion scheme matches on, and age comes from
> the page-fault primitive.
We would have age in perf event based mode, too. Isn't it?
> With the ring partitioned by class both are live at
> once: the probe supplies hotness, the primitive supplies age, and two schemes
> over the same regions can move memory in both directions under one kdamond.
Unless the needs are clearly confirmed, I'd prefer having single class for
simplicity.
>
> The two classes are not symmetric, and the asymmetry is in where a report comes
> from rather than in what it carries. A page fault has no monitoring context at
> report time, so a per-context ring is not expressible for it: those reports go to
> one global set of per-CPU rings, drained by the single context whose page-fault
> primitive is enabled. A PMU overflow handler does carry the context that armed
> the event, so those reports go to that context's own per-CPU rings and each
> context drains only what it owns. Both classes share the ring layout and the
> producer side, so the barrier pairing cannot drift between them.
>
> ```
> page-fault report PMU sample, NMI context
> no ctx at report time handler carries the owning ctx
> | |
> v v
> +----------------------------+ +----------------------------+
> | one global per-CPU ring | | per-CPU rings per context |
> | set, shared by every ctx | | that armed a perf event |
> +----------------------------+ +----------------------------+
> | drained by the one ctx | | each ctx drains only the |
> | with the pf primitive on | | rings it owns |
> +----------------------------+ +----------------------------+
> | |
> +------------------+------------------+
> |
> drain on the aggregation boundary,
> each report matched to a region
> |
> +---------------+---------------+
> | |
> region age region probe hits
> demotion scheme promotion scheme
> ```
Again, I'd suggest to simply ignore page-fault report for now.
>
> That is the property this series exists to enable, and three things follow from
> it:
>
> - One measurement drives both directions, at whichever scope the mode gives.
> Bandwidth comes from resctrl MBM, which is where the discussion on a
> platform-independent bandwidth signal for tiering has pointed [6]: a
> virtual-address context reads a monitoring group created for the processes
> it monitors, and a physical-address context reads the root monitoring
> group, so the same controller is scoped to a set of processes or to the
> whole machine without changing what it does. Reading it there also means a
> bandwidth allocation reserved for an application bounds what the controller
> optimises within. It spreads the hot set across nodes in the ratio that
> maximises achieved bandwidth when bandwidth is what the workload is short of,
> and it demotes cold pages off the near node in the same context at the same
> time -- either proactively under a fixed quota or only once DRAM is under
> memory pressure. Neither direction is configured as a target ratio; the
> proportion or weight the controller writes bounds how much memory moves while
> the score decides which pages move, and a hardware sample makes that score
> proportional to the traffic a region generates rather than one bit per page
> per scan. Earlier work on bandwidth-driven interleaving [4] measured the same
> effect from userspace.
>
> - Bandwidth and capacity are expanded at once rather than one being chosen.
> Static weighted interleaving expands bandwidth, but it is a placement rule
> rather than a decision about any particular page, so cold pages keep their
> share of DRAM. NUMA balancing expands usable DRAM by promoting whatever is
> hot, but it does so without reference to what the far node can deliver, so it
> keeps promoting after the near node has stopped being the better place to
> read from. Here the hot set's distribution comes from a bandwidth
> measurement and the cold set is demoted on region age, in one context, so the
> far node's bandwidth is used and DRAM is not held by memory nothing is
> touching. The decision itself is a quota goal the kernel already carries:
> `node_eligible_mem_bp` [8] bounds how much of a node a scheme may fill, so
> what the controller supplies is the bandwidth measurement, not the
> placement.
>
> - When bandwidth is not the constraint it reverts to latency-first tiering on
> its own. The search starts with everything on the near node and moves memory
> outward only when the measurement improves, so on a workload that does not
> saturate that node no step outward improves anything: the hot set stays in
> DRAM for the lower access latency and cold pages keep being demoted. The
> same configuration covers both cases, with nothing to switch on and no ratio
> to revise.
Sounds very interesting.
>
> ## What the series adds
>
> 1. `mm/damon/vaddr: support page fault access check primitive` -- implements
> the page-fault primitive for a virtual address space context; it was
> available for physical addresses only. The marker whose fault reports the
> access is installed through the target mm and the vma of the sampling
> address directly, and prepare_access_checks() dispatches on the enabled
> primitive as the physical address space one does.
As I mentioned above, I'd suggest to keep page fault primitive out of the scope
for now.
>
> 2. `mm/damon/core: read the CPU number with preemption disabled` -- the report
> path runs in fault context, which is preemptible. Read the CPU number with
> preemption disabled and keep it disabled across the report, so the recorded
> CPU stays consistent with the per-CPU ring the report is queued to.
Sounds good and will be needed for milestone 2, too. I will reuse or reference
this in my milestone 2 implemenation. This will be very helpful, thank you for
sharing.
>
> 3. `mm/damon/paddr: lock the folio for the page fault primitive rmap walk` --
> take the folio lock unconditionally around the walk, as the other reverse
> mapping walks in DAMON do, so every mapping type reaches it locked, and
> release the reference the lookup took on each exit path.
Again, let's keep page fault-based monitoring out of the scope for now.
>
> 4. `mm/damon: add damos_node_eligible_mem_bp tracepoint` -- a per-tick
> tracepoint over the node-eligible-memory quota goal evaluation, exposing
> the goal's target and current values, so the loop a bandwidth-driven
> controller steers is visible to a tracer.
This seems doesn't need to wait anything. If this turned out to be helpful,
please feel free to separately send patches for this.
>
> 5. `mm/damon/core: add per-probe-class report rings and unified drain` -- the
> substrate. Per-CPU SPSC rings an NMI-context source can publish into,
> partitioned by probe index into a global page-fault ring and a per-context
> perf ring, plus the kdamond-side drain that matches each report to a region
> by binary search over a per-target snapshot and credits it to that region's
> probe hits on the aggregation boundary. The address space of the target
> selects which address of a report is matched, and pid targets are filtered
> by thread group id.
Sounds like good optimizations that we can discuss with tests in the phase 3.
>
> 6. `mm/damon: add perf-event overflow handler feeding the report ring` -- an
> ops-agnostic perf-event source whose overflow handler turns a PMU sample
> into a report, setting each address field the PMU reported as valid. A PMU
> that reports a virtual address only can therefore drive a virtual-address
> context, one that reports a physical address a physical-address context, and
> the same source serves either without a backend per address space. Per-CPU
> events are armed and released through cpuhp callbacks, and a per-PMU owner
> keeps two contexts from claiming the same PMU type.
This should be what also essential for milestone 2. I will reuse or reference
this in my milestone 2 implemenation. This will be very helpful, thank you for
sharing.
Nonetheless, virtual-address context considered parts may be skipped in
milestone 2. We can discuss such things in the phase 3.
>
> 7. `mm/damon/ops-common: use probe-weighted score when probe weights are set`
> -- lets a scheme's frequency subscore come from the probe hits, weighted per
> probe class, so what the sampler reported reaches the tiering decision.
> With no weights set the subscore comes from the access rate as before.
I'm working on adding more attributes-only mode (probe weights are set) support
to DAMOS. Hopefully we can converge on this. This is not a part of our
planned milestone 2, but I will keep doing this in parallel, so hopefully full
DAMOS support for attributes-only mode will be done around end of milestone 2.
Let's keep discussion on the works.
>
> 8. `mm/damon: add perf_event prep for PMU-driven hotness probes` -- the sysfs
> surface and the event lifecycle: a perf_event prep action carrying the PMU
> type, the event config and the sample attributes per probe, with per-CPU or
> single-instance arming depending on how many counters the PMU needs.
> Arming is deferred on a context built for a commit, so a weight-only commit
> leaves the running event untouched.
This is also what would be essential for milestone 2. I will reuse or reference
this in my milestone 2 implemenation. This will be very helpful, thank you for
sharing.
>
> 9. `mm/damon/tests/drain-kunit: kunit for report rings and unified drain` --
> unit tests for the rings and the drain: inject and drain, overflow on wrap,
> producer routing by probe index, ring partitioning, pf-ring ownership,
> thread-group filtering, the address space match, and both primitives live
> on one context.
Testing is always important and nice. Thank you for doing this.
>
> Patches 1-4 apply standalone on damon/next and are useful without the rest;
> 5-8 are the substrate and its first two consumers, and 9 covers them. If the
> first four read right, they need not wait on the rest of this series -- happy
> to repost them on their own for damon/next if that is easier to take. Two of
> them, 2 and 3, correct code that is already in mm-new rather than adding
> anything new, so they could instead go as a small fixes series carrying
> Fixes: tags; guidance on which of the two routes is preferred is welcome.
>
> ## Userspace setup model
>
> The runs were driven by an `auto_tier` subcommand added to damo on the branch
> below, which reads achieved bandwidth from resctrl MBM and searches for the hot
> set's distribution that maximises it, while the demotion scheme runs alongside
> it in the same context. That tooling is not part of this posting; it is on
>
> https://github.com/ravis-opensrc/damo/tree/damo/auto-tier-bw-2026-09-08
>
> - `--bw_source resctrl` is the option this adds: the controller reads achieved
> bandwidth from an MBM monitoring group and searches, rather than being given
> a distribution to install.
> - Everything else -- intervals, schemes, filters, the probe's `perf_event`
> attributes -- is what the generator writes from the command line shown: 5 ms
> sampling, 100 ms aggregation, a 1 s ops update.
> - Both configurations select proactive demotion, so they demote continuously
> rather than waiting for pressure, and both were left closed loop throughout.
> - The search is the algorithm described in [4].
>
> ### Configuration A: AMD IBS Op, paddr ops, system-wide
>
> ```
> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
> $ sudo python3 tools/damon_tier_gen.py --hotness ibs \
> --near_node 0 --far_node 4 \
> --cold_demote --cold_demote_mode proactive \
> -o tier.yaml
> $ sudo damo auto_tier tier.yaml --bw_source resctrl --verbose
> ```
>
> - Scope is the machine, not a process set. The distribution is steered
> through `node_eligible_mem_bp` quota goals over each node's own physical
> ranges, which the generator reads from `/proc/iomem`.
> - AMD Turin, DRAM on node 0 and a CXL node. IBS Op at a `sample_period` of
> 262144 with `sample_phys_addr` set, which is what a physical-address context
> needs from the sample.
> - Workload is a hot-and-cold mix so both schemes have something to act on:
> `multiload` drives the bandwidth-hungry hot set, and a second process
> allocates on the near node, touches it once and goes idle, leaving pages
> that age out for the demotion scheme to find.
>
> ### Configuration B: Intel PEBS L3-miss, vaddr ops, per-PID
>
> ```
> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
> $ sudo python3 tools/damon_tier_gen.py --hotness pebs \
> --pid $HOT_PID --pid $COLD_PID \
> --near_node 0 --far_node 1 \
> --cold_demote --cold_demote_mode proactive \
> --min_nr_regions 1000 --max_nr_regions 20000 \
> -o tier.yaml
> $ sudo damo auto_tier tier.yaml --bw_source resctrl --verbose
> ```
>
> - Scope is the processes the monitoring group names and nothing else. The
> distribution is steered through the hot scheme's `DamosDest` weights [9].
> - Intel Granite Rapids, DRAM on node 0 and CXL on node 1. PEBS L3-miss at
> `sample_freq` 5003 with `precise_ip` 2.
> - Same workload shape.
>
> ## What the runs show
>
> Per-node reference, each figure measured by binding the same workload to one
> node:
>
> ```
> Granite Rapids, 64 threads x 8 GiB
> node 0 1.0 TB DRAM DRAM only 269,885 MB/s
> node 1 2.0 TB CXL CXL only 249,903 MB/s
>
> Turin, 32 threads x 4 GiB
> node 0 386 GB DRAM DRAM only 113,596 MB/s
> node 4 1.0 TB CXL CXL only 33,253 MB/s
> ```
>
> - The two machines differ in the way that matters: on the Turin the far node
> is 3.4x slower than the near one, on the Granite Rapids the two are within
> 8%.
> - Each configuration gets one graph, the climb: the run itself, one mark per
> decision taken from the bandwidth just measured, with the share it moved to
> under each mark. The settled line below it is the median over the samples
> the run went on to hold at that share.
>
> ### Configuration B, virtual-address mode
>
> ```
> MB/s climb
> 460k | *
> 440k | *
> 420k | *
> 400k |
> 380k | *
> 360k | *
> 340k | * *
> 320k | *
> 300k | * *
> 280k | *
> 260k | * *
> --------+------------+--------------+---------------+------------+--
> 1 4 7 10 13
> 92 80 68 56 44
> decision index, near-node share (%)
>
> settled share 44, 430,547 MB/s held
> ```
>
> - Thirteen decisions to settle at 44%, no target given: about 60% more than
> the better of the two nodes on its own, and the share does not change again
> over the 81 samples that follow.
> - Cold demotion runs over the same interval, in the same context, actuated by
> the same kdamond: 34 regions totalling 15.4 GiB applied, and the idle
> process ends with 4,099,980 pages -- 15.6 GiB -- on the CXL node having
> started on DRAM.
> - The promotion scheme's applied byte count is flat across the demotion ramp,
> 291.1 GiB before against 291.3 GiB after, so the recovered capacity is the
> demotion scheme's.
>
> ### Configuration A, physical-address mode
>
> ```
> MB/s climb
> 136k |
> 134k | * *
> 132k | * *
> 130k | *
> 128k |
> 126k | *
> 124k |
> 122k | *
> 120k |
> 118k | *
> +----+-----+-----+-----+-----+-----+-----+-----+--
> 1 2 3 4 5 6 7 8
> 92 88 84 80 76 78 80 78
> decision index, near-node share (%)
>
> settled share 78, 134,107 MB/s held
> ```
>
> - Eight decisions to settle at 78, from a start with almost everything on the
> near node: the same algorithm reaches the optimum share on its own from the
> bandwidth reading alone, and cold pages are demoted to the far node while it
> does so.
> - A distribution beats either end because both nodes supply at once: at the
> settled share the near node is supplying 104,618 MB/s against the 113,596 it
> reaches with the whole working set bound to it, so the further 37,669 MB/s
> arriving over CXL is bandwidth it could not have supplied at any share.
> - The idle process ends with 2,863,156 pages -- 10.9 GiB -- on the CXL node,
> out of the 32 GiB it allocated on DRAM and stopped touching. Residency is
> the attributable figure in this mode rather than a per-scheme applied count,
> because the schemes share physical ranges.
>
> ### What the runs are meant to establish
>
> Not the settled share and not the absolute bandwidth; those are properties of
> these machines and these workloads. What reproduces is:
>
> - A controller reading achieved bandwidth converges unattended, from a
> configuration naming no target, to a distribution that beats either node on
> its own, and holds it once found.
> - A demotion scheme recovers near-node capacity in the same context while that
> happens.
> - One code path does both, steering `node_eligible_mem_bp` over physical
> ranges system-wide on one machine and `DamosDest` weights over a named
> process group on the other.
Nice testing and measurement, thank you for sharing! I'm happy the future
DAMON could help this!
>
> ## Beyond a CPU PMU
>
> Nothing above is specific to PEBS or IBS. A source qualifies if it can report
> an accessed address to the ring, which is the only thing the substrate asks of
> it. A CXL device's Hotness Monitoring Unit, whose r3.2 perf driver has been
> posted [7] exposing its hotlist through an AUX buffer, or a custom monitoring
> unit on an accelerator or a memory expander, reports exactly that, and a
> backend delivering those reports through a perf event reaches the same drain,
> the same probe hits and the same schemes already in the tree. Such a unit sees
> the traffic that reaches its own device rather than a system-wide view, so what
> it contributes is the hot set resident on that device: the tiering logic can
> promote those pages to DRAM without the host having to detect that hotness
> itself, and without depending on what produced the reports. That posting names
> driving tiering from such a unit as the intent and the in-kernel step as future
> work, and consuming it as a DAMON access-check primitive was raised there as
> one way to get it; a backend on this substrate is that path.
That makes sense to me. Thank you for sharing this nice survey and theories!
>
> [1] v1 of this series
> https://lore.kernel.org/damon/20260529165640.820-1-ravis.opensrc@gmail.com/
> [2] Roadmap for extending DAMON beyond pte-accessed bit
> https://lore.kernel.org/damon/20260525225208.1179-1-sj@kernel.org/
> [3] mm/damon: introduce perf event based access check
> https://lore.kernel.org/damon/20260423004211.7037-1-akinobu.mita@gmail.com/
> [4] B. Tabatabai, R. Jonnalagadda et al., "Bandwidth Speaks, We Listen:
> Dynamic Interleaving for Tiered Memory", ISMM 2026.
> https://dl.acm.org/doi/10.1145/3814942.3816137
> [5] mm/damon/perf: add ARM SPE AUX backend
> https://lore.kernel.org/damon/20260816142222.689624-1-kunwu.chan@linux.dev/
> [6] A platform-independent subsystem for bandwidth information, and resctrl
> as that source
> https://lore.kernel.org/linux-mm/d952a84f-332e-8f7a-4816-2c1cbd8f5b00@google.com/
> [7] CXL Hotness Monitoring Unit perf driver
> https://lore.kernel.org/linux-mm/20241121101845.1815660-1-Jonathan.Cameron@huawei.com/
> [8] mm/damon: add node_eligible_mem_bp goal metric, merged for v7.2
> https://lore.kernel.org/linux-mm/20260428030520.701-1-ravis.opensrc@gmail.com/
> [9] mm/damon/vaddr: allow interleaving in migrate_{hot,cold} actions, merged
> for v6.17
> https://lore.kernel.org/linux-mm/20250709005952.17776-1-bijan311@gmail.com/
>
> Ravi Jonnalagadda (9):
> mm/damon/vaddr: support page fault access check primitive
> mm/damon/core: read the CPU number with preemption disabled
> mm/damon/paddr: lock the folio for the page fault primitive rmap walk
> mm/damon: add damos_node_eligible_mem_bp tracepoint
> mm/damon/core: add per-probe-class report rings and unified drain
> mm/damon: add perf-event overflow handler feeding the report ring
> mm/damon/ops-common: use probe-weighted score when probe weights are
> set
> mm/damon: add perf_event prep for PMU-driven hotness probes
> mm/damon/tests/drain-kunit: kunit for report rings and unified drain
As this series is for high level discussion, and the code will significantly
changed after rebasing on milestone 2 output, I wouldn't go deep into each line
of the code for now. Feel free to let me know if you want.
>
> include/linux/damon.h | 136 ++++-
> include/trace/events/damon.h | 32 +
> mm/damon/Kconfig | 18 +
> mm/damon/Makefile | 1 +
> mm/damon/core.c | 995 +++++++++++++++++++++++++++----
> mm/damon/ops-common.c | 21 +-
> mm/damon/paddr.c | 37 +-
> mm/damon/perf_source.c | 489 +++++++++++++++
> mm/damon/perf_source.h | 54 ++
> mm/damon/sysfs.c | 266 ++++++++-
> mm/damon/tests/.kunitconfig | 4 +
> mm/damon/tests/core-kunit.h | 2 +-
> mm/damon/tests/drain-kunit.h | 1091 ++++++++++++++++++++++++++++++++++
> mm/damon/tests/perf-kunit.h | 133 +++++
> mm/damon/vaddr.c | 83 ++-
> 15 files changed, 3220 insertions(+), 142 deletions(-)
> create mode 100644 mm/damon/perf_source.c
> create mode 100644 mm/damon/perf_source.h
> create mode 100644 mm/damon/tests/drain-kunit.h
> create mode 100644 mm/damon/tests/perf-kunit.h
>
>
> base-commit: e1f34dce183a96fc93bf9a42dcdc0ec0bf82a3d2
> --
> 2.43.0
>
>
Thanks,
SJ
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2026-09-12 1:38 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-09-10 17:16 [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 1/9] mm/damon/vaddr: support page fault access check primitive Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 2/9] mm/damon/core: read the CPU number with preemption disabled Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 3/9] mm/damon/paddr: lock the folio for the page fault primitive rmap walk Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 4/9] mm/damon: add damos_node_eligible_mem_bp tracepoint Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 5/9] mm/damon/core: add per-probe-class report rings and unified drain Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 6/9] mm/damon: add perf-event overflow handler feeding the report ring Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 7/9] mm/damon/ops-common: use probe-weighted score when probe weights are set Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 8/9] mm/damon: add perf_event prep for PMU-driven hotness probes Ravi Jonnalagadda
2026-09-10 17:16 ` [RFC PATCH v2 9/9] mm/damon/tests/drain-kunit: kunit for report rings and unified drain Ravi Jonnalagadda
2026-09-11 0:34 ` [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports SJ Park
2026-09-12 1:38 ` SJ Park
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®