From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl1-f41.google.com (mail-dl1-f41.google.com [74.125.82.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B14B3E638D for ; Thu, 18 Jun 2026 09:49:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.41 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781776158; cv=none; b=adTlKV4xWNeqdfAOI+SlBoeekJEIMPsNCnJwPDwR61OSdu+9QVG/wmNTQ+/9bZYVR7N+t7DUotPDHM8mwigqVbDwLBT1C8TJJ0J2XIloCZuu3nYjl1yruosphkiuSh3u6yEAKqAe3ZHVPiykxorj9a9Z73ICUUeBIwE2qw49Tgs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781776158; c=relaxed/simple; bh=RURShsfHHGYS/EyZ3xaM0+0qALq8ov9WvGfaWFHPmhY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=J+0ImWguQA5dE+yyTcB5rSldJMTJIhe/fCS7CivpvSRhS/19aa28lA6YodLFCfs0jGhiobZMQKwKRYsa/s97H0YjH2ZGJybRPMWF3lrRseWM0M83Jv5jRGAmjT3qRY/KcRcQwux1n25RGYlhT7f1zKLrQDUBY99BEn/7LwFVn0Y= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=A4N8NZ/n; arc=none smtp.client-ip=74.125.82.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="A4N8NZ/n" Received: by mail-dl1-f41.google.com with SMTP id a92af1059eb24-1363fe80fe8so985878c88.0 for ; Thu, 18 Jun 2026 02:49:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781776150; x=1782380950; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eZAtuUTHJDOiKLx4zrrhbntE6nPwfKSnV0j3f+6dKp0=; b=A4N8NZ/n1J8PjeXL3pPDUUIzOM6P3FlNTuqTdI5LbFmlqyKQX2+64OfGaPFrtdpPLJ xtBiPFCGM1/G6dNcEKzXRbwzcxUlcecepCAADPI29T2WU0RHOCSXLu4TjMxGrgbesAEW xz8hZWfB75dYk/vWNgkYyW/FJDBqFgqtXk14ImWZU4DYHsXG/O/YZbi/d0sJuzhbziMR yPtmJ3cOGp3oFSri+9oWOtradAI4osZjn2P9L//unNR/V/IgFAyApzZjMn11DqXeP9Ku QNOCjCTRSOv8XYSpeGK6D/XW6aBflFKmnDBJTyQ7PUWw8usWw9rwubtjGmFuBwpwOEPB B6Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781776150; x=1782380950; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=eZAtuUTHJDOiKLx4zrrhbntE6nPwfKSnV0j3f+6dKp0=; b=fSDKUhkDCiceoAhfB1sl/qih/S7lResMW+Plm/s74f2KskMVHfd+tPYaThKW8CRgQ0 Dz1M4htVnYGZ1rptd5NH93+ZdoA/fp8vj87oZDlPjIVQxn3HLwBLiECOFVk4z2/sWeKq uD3NCrj5Oo+yxAv69aBNbIKDiLPwcJxmFMFkwCN+u7SEHoq0XnEAMofkelE2IWcNE9sF 4UYUCdsbGptts7CV8nQoR3ItuphA7PsV4ke/9EbkcyBudDQNaZWGjSVBw3V/QAzf5Bu9 r9xpN0rVRxN3K8BUHTKtaJHVU7PsU9saE38w4yGEXyNAxaS4vXms/apqGIxwyJW8sfKa +OLQ== X-Forwarded-Encrypted: i=1; AFNElJ/7Bb4YhjOmzoaZEO2GWKc6H7DtY27TCNYV++ENf7JaBESKIwLgjOvYZL2+ary6BcaLhO1ET6K/xBVd/pk=@vger.kernel.org X-Gm-Message-State: AOJu0Yxt8juI8rdNZ5IidekNL1HzSj01jtXgADvkY1pbU4L0+y0pq/H+ 4Y/REMCl/Y5sdoHt6NiTnxfDdyiWCFoHDhY08y2MigrTp2FQh2RbU+Xp X-Gm-Gg: Acq92OHNLnmKCpwX/EMqsPMs1RUZqUlY+1SMb3CHGwIfsvlH9+EB/IvetEurLCqdLBF csih08FpEnVCERDdlx4iegte0e/6goAiHEGF5PbwQOJj7Cfg8ZCj9q6sX6qJqO2bHkjPTsfG/+3 JuqI475pIvBV6csyQRkfuGeV2e2osm0rI3rueEWI3nJqc3S3OiXSvyOYW0Hj2AgwsR7OyOsY3DZ +B36TyrKaaqUB0qxwnNMX2g5e5JofqVKM/6Dw2euXUltt17s33DjDUR5JBz0417G335NqWwTtX/ wqZ5AjAvGQ5d6+ALQurx9xTk3jYQNrE1bDV/qLEY38VtpO0tUryJfmJNYCRVY6UDiizkwdJrMnl CxaK9NFHewzyXS58XqJjItHx0P4XpAr5nI78G5UqKGdgLGgdm2SGRFNcpIwF73Ebuge9OJfE3+v EmLCWvTrT63s45HfgIkSYCSwSZ X-Received: by 2002:a05:7022:1b18:b0:136:ff02:d00b with SMTP id a92af1059eb24-1398f6de2c7mr3533585c88.25.1781776149844; Thu, 18 Jun 2026 02:49:09 -0700 (PDT) Received: from localhost.localdomain ([104.28.158.224]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-1384b91a4c7sm18069971c88.8.2026.06.18.02.49.05 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 18 Jun 2026 02:49:09 -0700 (PDT) From: Wang Lian To: sj@kernel.org, akpm@linux-foundation.org Cc: npache@redhat.com, gutierrez.asier@huawei-partners.com, daichaobing@sangfor.com.cn, linux-mm@kvack.org, linux-kernel@vger.kernel.org, lianux.mm@gmail.com, kunwu.chan@gmail.com Subject: [RFC PATCH 6/6] mm/damon: add SPE feedback for sub-THP split decisions Date: Thu, 18 Jun 2026 17:48:38 +0800 Message-ID: <20260618094838.32805-7-lianux.mm@gmail.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260618094838.32805-1-lianux.mm@gmail.com> References: <20260618094838.32805-1-lianux.mm@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a sub-THP access heatmap that enables data-driven split decisions in DAMOS_MTHP_SPLIT. The split handler queries damon_spe_hot_fraction() and compares against the scheme's configurable hot_threshold (default 30%, set in patch 4) to preserve genuinely hot THPs while splitting cold ones. Key data-driven design decisions from Kunpeng 920 SPE profiling: 1. Signal vs noise threshold (this patch): Raw SPE data shows most THPs have scattered 1-2 sample hits across many subpages — noise, not genuine access patterns. The heatmap now uses a two-pass signal threshold: a subpage chunk must have >= 1/10 of the peak chunk's access count to be considered hot. This reduces false hot classification from ~50% to <5% of subpages. 2. hot_threshold 30% (patch 4, sysfs-configurable): With the signal filter applied, 97% of THPs have <10% hot subpages (clearly cold), 1-2% have 10-30% (borderline), and <1% have >30% (genuinely hot). The 30% default catches hot THPs while allowing the vast majority to be split. Architecture (three-phase): Phase 2a (current fallback): Walk PTE access bits via folio_walk for THPs already split to PTEs. For PMD-mapped THPs (the common case), return -EOPNOTSUPP, which causes the split handler to split unconditionally. Phase 2b (userspace daemon -> kernel, ready for validation): Userspace SPE daemon decodes ARM SPE records, feeds PFNs via debugfs (/sys/kernel/debug/damon/spe_feed). The kernel aggregates accesses into a per-folio rbtree keyed by THP-aligned PFN. Phase 2c (kernel-native, future): perf_event_create_kernel_counter for ARM SPE. Overflow handler calls damon_spe_record_access() directly. Data structure (mm/damon/spe.c): - Per-folio rbtree keyed by PFN, storing access_count[512] (one counter per 4KB subpage) - Max 1024 entries, entries older than 30s are pruned periodically - Global spinlock-protected rbtree with GFP_ATOMIC allocation Debugfs interface: - /sys/kernel/debug/damon/spe_feed (write): accept one PFN per line - /sys/kernel/debug/damon/spe_stats (read): rbtree stats + top entries When CONFIG_DAMON_SPE is disabled, all SPE functions are empty stubs returning -EOPNOTSUPP, making the split unconditional. Co-developed-by: Kunwu Chan Signed-off-by: Kunwu Chan Signed-off-by: Wang Lian --- mm/damon/Kconfig | 12 ++ mm/damon/Makefile | 1 + mm/damon/core.c | 3 + mm/damon/spe.c | 505 ++++++++++++++++++++++++++++++++++++++++++++++ mm/damon/spe.h | 62 ++++++ mm/damon/vaddr.c | 16 +- 6 files changed, 597 insertions(+), 2 deletions(-) create mode 100644 mm/damon/spe.c create mode 100644 mm/damon/spe.h diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig index 34631a44cdec..ea75a8dab989 100644 --- a/mm/damon/Kconfig +++ b/mm/damon/Kconfig @@ -121,4 +121,16 @@ config DAMON_STAT_ENABLED_DEFAULT Whether to enable DAMON_STAT by default. Users can disable it in boot or runtime using its 'enabled' parameter. +config DAMON_SPE + bool "DAMON SPE feedback for sub-THP access monitoring (prototype)" + depends on DAMON_VADDR + help + Enable sub-THP access heatmap feedback for DAMOS_MTHP_SPLIT. + Currently a prototype: uses PTE access bits for THPs that have + been split to PTEs, returns "no data" for PMD-mapped THPs. + + On hardware with ARM SPE (e.g. Kunpeng 920), this will be + extended to provide per-subpage access data without needing to + split the PMD first, enabling precise mTHP split decisions. + endmenu diff --git a/mm/damon/Makefile b/mm/damon/Makefile index d8d6bf5f8bff..507b43a9f009 100644 --- a/mm/damon/Makefile +++ b/mm/damon/Makefile @@ -7,3 +7,4 @@ obj-$(CONFIG_DAMON_SYSFS) += sysfs-common.o sysfs-schemes.o sysfs.o obj-$(CONFIG_DAMON_RECLAIM) += modules-common.o reclaim.o obj-$(CONFIG_DAMON_LRU_SORT) += modules-common.o lru_sort.o obj-$(CONFIG_DAMON_STAT) += modules-common.o stat.o +obj-$(CONFIG_DAMON_SPE) += spe.o diff --git a/mm/damon/core.c b/mm/damon/core.c index 265d51ade25b..0805e71a90d8 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -20,6 +20,7 @@ /* for damon_get_folio() used by node eligible memory metrics */ #include "ops-common.h" +#include "spe.h" #define CREATE_TRACE_POINTS #include @@ -2987,6 +2988,8 @@ static void kdamond_apply_schemes(struct damon_ctx *c) if (!has_schemes_to_apply) return; + damon_spe_prune(); + max_region_sz = damon_region_sz_limit(c); mutex_lock(&c->walk_control_lock); damon_for_each_target(t, c) { diff --git a/mm/damon/spe.c b/mm/damon/spe.c new file mode 100644 index 000000000000..98f8d32053e4 --- /dev/null +++ b/mm/damon/spe.c @@ -0,0 +1,505 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * DAMON SPE (Statistical Profiling Extension) feedback + * + * Provides sub-THP access heatmap for intelligent split decisions. + * + * Architecture: + * Phase 2a (current): PTE access bits via folio_walk. + * Works only when a THP has been previously split to PTEs. + * Returns -EOPNOTSUPP for PMD-mapped THPs. + * + * Phase 2b (userspace): spe_hist daemon decodes SPE in userspace, + * feeds {pfn, subpage_idx} via debugfs/sysfs into the rbtree below. + * + * Phase 2c (kernel): perf_event_create_kernel_counter for ARM SPE, + * overflow handler aggregates into rbtree. Requires SPE hardware. + * + * Data structure: + * Per-folio rbtree keyed by PFN, storing per-subpage access counts. + * Entries are aged and pruned periodically. + * + * Copyright (C) 2026 Wang Lian + */ + +#define pr_fmt(fmt) "damon-spe: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "spe.h" + +/* Max sub-pages when querying at order 0 */ +#define DAMON_SPE_MAX_CHUNKS 512 + +/* Max folio entries in the rbtree (per-mm or global) */ +#define DAMON_SPE_MAX_ENTRIES 1024 + +/* Entry considered stale after this many jiffies (default: 30s) */ +#define DAMON_SPE_ENTRY_TTL (30 * HZ) + +/* + * Per-folio access histogram entry. + * Keyed by pfn in an rbtree. Each entry tracks access count per subpage. + * The access_count array is sized for PMD-order / 0 = 512 4KB subpages. + */ +struct damon_spe_entry { + struct rb_node node; + unsigned long pfn; /* THP-aligned PFN */ + pid_t pid; /* owner process */ + unsigned long access_count[DAMON_SPE_MAX_CHUNKS]; + unsigned long total_accesses; + unsigned long last_access; /* jiffies of last update */ +}; + +static struct rb_root spe_tree = RB_ROOT; +static DEFINE_SPINLOCK(spe_lock); +static unsigned int spe_nr_entries; + +/* Forward declarations */ +static void __spe_prune(void); + +/* + * Find an entry by PFN. Must be called with spe_lock held. + */ +static struct damon_spe_entry *spe_find(unsigned long pfn) +{ + struct rb_node *node = spe_tree.rb_node; + + while (node) { + struct damon_spe_entry *e = + rb_entry(node, struct damon_spe_entry, node); + + if (pfn < e->pfn) + node = node->rb_left; + else if (pfn > e->pfn) + node = node->rb_right; + else + return e; + } + return NULL; +} + +/* + * Insert a new entry. Must be called with spe_lock held. + * Returns the new entry, or NULL if the tree is full. + */ +static struct damon_spe_entry *spe_insert(unsigned long pfn, pid_t pid) +{ + struct rb_node **new = &spe_tree.rb_node, *parent = NULL; + struct damon_spe_entry *e; + + if (spe_nr_entries >= DAMON_SPE_MAX_ENTRIES) { + __spe_prune(); + if (spe_nr_entries >= DAMON_SPE_MAX_ENTRIES) + return NULL; + } + + e = kzalloc(sizeof(*e), GFP_ATOMIC); + if (!e) + return NULL; + + e->pfn = pfn; + e->pid = pid; + e->last_access = jiffies; + + while (*new) { + struct damon_spe_entry *this = + rb_entry(*new, struct damon_spe_entry, node); + + parent = *new; + if (pfn < this->pfn) + new = &((*new)->rb_left); + else if (pfn > this->pfn) + new = &((*new)->rb_right); + else { + /* Race: another CPU inserted the same PFN */ + kfree(e); + return this; + } + } + + rb_link_node(&e->node, parent, new); + rb_insert_color(&e->node, &spe_tree); + spe_nr_entries++; + return e; +} + +/* + * Prune entries that haven't been updated for DAMON_SPE_ENTRY_TTL. + * Must be called with spe_lock held. + */ +static void __spe_prune(void) +{ + struct rb_node *node, *next; + unsigned long deadline = jiffies - DAMON_SPE_ENTRY_TTL; + + node = rb_first(&spe_tree); + while (node) { + struct damon_spe_entry *e = + rb_entry(node, struct damon_spe_entry, node); + + next = rb_next(node); + + if (time_before(e->last_access, deadline)) { + rb_erase(&e->node, &spe_tree); + spe_nr_entries--; + kfree(e); + } + node = next; + } +} + +/** + * damon_spe_record_access() - Record a single subpage access + * @pfn: Physical page frame number (any page within a THP) + * @pid: Process ID that performed the access + * + * The PFN is automatically aligned to the THP base. The subpage index + * within the THP is derived from the low bits of the PFN. + * + * Context: Can be called from IRQ context. + */ +void damon_spe_record_access(unsigned long pfn, pid_t pid) +{ + unsigned long thp_pfn = pfn & ~(unsigned long)(DAMON_SPE_MAX_CHUNKS - 1); + unsigned int idx = pfn & (DAMON_SPE_MAX_CHUNKS - 1); + struct damon_spe_entry *e; + unsigned long flags; + + spin_lock_irqsave(&spe_lock, flags); + + e = spe_find(thp_pfn); + if (!e) + e = spe_insert(thp_pfn, pid); + + if (e) { + e->access_count[idx]++; + e->total_accesses++; + e->last_access = jiffies; + } + + spin_unlock_irqrestore(&spe_lock, flags); +} +EXPORT_SYMBOL_GPL(damon_spe_record_access); + +/** + * damon_spe_folio_heatmap() - Get sub-THP access bitmap for a folio + * @folio: The folio to query + * @vma: VMA containing the folio + * @addr: Virtual address of the folio start + * @target_order: Page order for each chunk in the bitmap + * @hot_bitmap: Output bitmap with one bit per chunk + * + * Queries the SPE rbtree first. Falls back to PTE access bits if no + * SPE data is available (requires the THP to be split to PTEs). + * + * Return: Number of chunks on success, negative error on failure. + */ +int damon_spe_folio_heatmap(struct folio *folio, struct vm_area_struct *vma, + unsigned long addr, unsigned int target_order, + unsigned long *hot_bitmap) +{ + unsigned long num_chunks = folio_nr_pages(folio) >> target_order; + unsigned long chunk_sz = PAGE_SIZE << target_order; + unsigned long pfn; + unsigned long flags; + struct damon_spe_entry *e; + struct folio_walk fw; + struct folio *sub_folio; + int i; + + if (!folio || !vma || !hot_bitmap) + return -EINVAL; + if (target_order >= folio_order(folio)) + return -EINVAL; + + pfn = folio_pfn(folio); + + /* + * Phase 2b/2c path: query the SPE rbtree. + * If we have aggregated SPE data for this folio, use it. + */ + spin_lock_irqsave(&spe_lock, flags); + e = spe_find(pfn); + if (e && e->total_accesses > 0) { + unsigned long max_sum = 0; + unsigned long sig_thresh; + unsigned int spp = chunk_sz >> PAGE_SHIFT; + + /* First pass: find peak chunk access count */ + for (i = 0; i < num_chunks; i++) { + unsigned long sum = 0; + int j; + + for (j = 0; j < spp; j++) { + unsigned int idx = i * spp + j; + + if (idx < DAMON_SPE_MAX_CHUNKS) + sum += e->access_count[idx]; + } + if (sum > max_sum) + max_sum = sum; + } + + /* + * Signal threshold: a chunk needs >= 1/10 of peak access + * count to be considered hot. This filters SPE noise — + * Kunpeng 920 data shows most THPs have scattered 1-2 + * sample hits across many subpages that don't represent + * genuine hot access patterns. + */ + sig_thresh = max(max_sum / 10, 1UL); + + /* Second pass: build hot bitmap using signal threshold */ + bitmap_zero(hot_bitmap, num_chunks); + for (i = 0; i < num_chunks; i++) { + unsigned long sum = 0; + int j; + + for (j = 0; j < spp; j++) { + unsigned int idx = i * spp + j; + + if (idx < DAMON_SPE_MAX_CHUNKS) + sum += e->access_count[idx]; + } + if (sum >= sig_thresh) + __set_bit(i, hot_bitmap); + } + + spin_unlock_irqrestore(&spe_lock, flags); + return (int)num_chunks; + } + spin_unlock_irqrestore(&spe_lock, flags); + + /* + * Phase 2a fallback: walk PTEs to check access bits. + * Only works when the THP has been split to PTEs. + */ + bitmap_zero(hot_bitmap, num_chunks); + + for (i = 0; i < num_chunks; i++) { + unsigned long chunk_addr = addr + i * chunk_sz; + + sub_folio = folio_walk_start(&fw, vma, chunk_addr, 0); + if (!sub_folio) + return -EOPNOTSUPP; + + if (fw.level == FW_LEVEL_PMD) { + folio_walk_end(&fw, vma); + return -EOPNOTSUPP; + } + + if (fw.level == FW_LEVEL_PTE && pte_young(fw.pte)) + __set_bit(i, hot_bitmap); + + folio_walk_end(&fw, vma); + } + + return (int)num_chunks; +} +EXPORT_SYMBOL_GPL(damon_spe_folio_heatmap); + +/** + * damon_spe_hot_fraction() - Return hot chunk percentage of a folio + * @folio: The folio to query + * @vma: VMA containing the folio + * @addr: Virtual address of the folio start + * @target_order: Page order for each chunk + * + * Return: Percentage (0-100) on success, negative error on failure. + */ +int damon_spe_hot_fraction(struct folio *folio, struct vm_area_struct *vma, + unsigned long addr, unsigned int target_order) +{ + unsigned long num_chunks = folio_nr_pages(folio) >> target_order; + DECLARE_BITMAP(hot_bitmap, DAMON_SPE_MAX_CHUNKS); + int ret, hot; + + if (num_chunks > DAMON_SPE_MAX_CHUNKS) + return -ERANGE; + + ret = damon_spe_folio_heatmap(folio, vma, addr, target_order, + hot_bitmap); + if (ret < 0) + return ret; + + hot = bitmap_weight(hot_bitmap, num_chunks); + return (hot * 100) / (int)num_chunks; +} +EXPORT_SYMBOL_GPL(damon_spe_hot_fraction); + +/** + * damon_spe_prune() - Remove stale entries from the SPE rbtree + * + * Called from DAMON's aggregation cycle. Removes entries not updated + * within DAMON_SPE_ENTRY_TTL jiffies. + */ +void damon_spe_prune(void) +{ + unsigned long flags; + + spin_lock_irqsave(&spe_lock, flags); + __spe_prune(); + spin_unlock_irqrestore(&spe_lock, flags); +} + +/** + * damon_spe_stats() - Return current SPE rbtree statistics + * @nr_entries: Output for number of entries, may be NULL + * @total_accesses: Output for total accumulated accesses, may be NULL + */ +void damon_spe_stats(unsigned int *nr_entries, unsigned long *total_accesses) +{ + struct rb_node *node; + unsigned long flags; + unsigned int count = 0; + unsigned long total = 0; + + spin_lock_irqsave(&spe_lock, flags); + for (node = rb_first(&spe_tree); node; node = rb_next(node)) { + struct damon_spe_entry *e = + rb_entry(node, struct damon_spe_entry, node); + count++; + total += e->total_accesses; + } + spin_unlock_irqrestore(&spe_lock, flags); + + if (nr_entries) + *nr_entries = count; + if (total_accesses) + *total_accesses = total; +} +EXPORT_SYMBOL_GPL(damon_spe_stats); + +/* ---- debugfs interface for Phase 2b (userspace daemon → kernel rbtree) ---- */ + +static struct dentry *damon_spe_dentry; + +/* + * spe_feed write: accept one PFN per line (hex or decimal). + * The PFN is recorded as an access via damon_spe_record_access(). + * + * Usage from userspace: + * echo 0x12345678 > /sys/kernel/debug/damon/spe_feed + * + * For bulk feed from SPE daemon: + * cat spe_pfns.txt > /sys/kernel/debug/damon/spe_feed + */ +static ssize_t spe_feed_write(struct file *file, const char __user *buf, + size_t count, loff_t *ppos) +{ + char line[32]; + size_t len = min(count, sizeof(line) - 1); + unsigned long pfn; + + if (copy_from_user(line, buf, len)) + return -EFAULT; + line[len] = '\0'; + + /* Strip trailing newline */ + if (len > 0 && line[len - 1] == '\n') + line[len - 1] = '\0'; + + if (kstrtoul(line, 0, &pfn) == 0 && pfn != 0) + damon_spe_record_access(pfn, 0); + + return count; +} + +/* + * spe_stats read: show current SPE rbtree statistics. + * + * Usage: + * cat /sys/kernel/debug/damon/spe_stats + */ +static int spe_stats_show(struct seq_file *m, void *v) +{ + struct rb_node *node; + unsigned long flags; + unsigned int count = 0; + unsigned long total = 0; + + spin_lock_irqsave(&spe_lock, flags); + for (node = rb_first(&spe_tree); node; node = rb_next(node)) { + struct damon_spe_entry *e = + rb_entry(node, struct damon_spe_entry, node); + count++; + total += e->total_accesses; + } + spin_unlock_irqrestore(&spe_lock, flags); + + seq_printf(m, "nr_entries=%u total_accesses=%lu\n", count, total); + + /* Show top entries (limit output) */ + spin_lock_irqsave(&spe_lock, flags); + count = 0; + for (node = rb_first(&spe_tree); node; node = rb_next(node)) { + struct damon_spe_entry *e = + rb_entry(node, struct damon_spe_entry, node); + unsigned int hot_pages = 0; + int i; + + for (i = 0; i < DAMON_SPE_MAX_CHUNKS; i++) + if (e->access_count[i]) + hot_pages++; + + seq_printf(m, " pfn=0x%lx pid=%d total=%lu hot_pages=%u/%d\n", + e->pfn, e->pid, e->total_accesses, + hot_pages, DAMON_SPE_MAX_CHUNKS); + if (++count >= 10) + break; + } + spin_unlock_irqrestore(&spe_lock, flags); + + return 0; +} + +static int spe_stats_open(struct inode *inode, struct file *file) +{ + return single_open(file, spe_stats_show, inode->i_private); +} + +static const struct file_operations spe_feed_fops = { + .write = spe_feed_write, +}; + +static const struct file_operations spe_stats_fops = { + .open = spe_stats_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +static int __init damon_spe_dbgfs_init(void) +{ + struct dentry *d; + + d = debugfs_lookup("damon", NULL); + if (!d) { + d = debugfs_create_dir("damon", NULL); + if (IS_ERR(d)) + return PTR_ERR(d); + } + damon_spe_dentry = d; + + debugfs_create_file("spe_feed", 0200, damon_spe_dentry, + NULL, &spe_feed_fops); + debugfs_create_file("spe_stats", 0400, damon_spe_dentry, + NULL, &spe_stats_fops); + + pr_info("debugfs interface ready: /sys/kernel/debug/damon/spe_{feed,stats}\n"); + return 0; +} + +late_initcall(damon_spe_dbgfs_init); diff --git a/mm/damon/spe.h b/mm/damon/spe.h new file mode 100644 index 000000000000..38799688b5af --- /dev/null +++ b/mm/damon/spe.h @@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * DAMON SPE (Statistical Profiling Extension) feedback + * + * Provides sub-THP access heatmap for intelligent split decisions. + * + * Three-phase architecture: + * Phase 2a: PTE access bits via folio_walk (current fallback) + * Phase 2b: Userspace SPE daemon feeds {pfn, subpage} via debugfs + * Phase 2c: Kernel perf_event_create_kernel_counter for ARM SPE + * + * Copyright (C) 2026 Wang Lian + */ + +#ifndef _DAMON_SPE_H +#define _DAMON_SPE_H + +#include +#include + +#ifdef CONFIG_DAMON_SPE + +/* ---- Sub-page heatmap query ---- */ + +int damon_spe_folio_heatmap(struct folio *folio, struct vm_area_struct *vma, + unsigned long addr, unsigned int target_order, + unsigned long *hot_bitmap); + +int damon_spe_hot_fraction(struct folio *folio, struct vm_area_struct *vma, + unsigned long addr, unsigned int target_order); + +/* ---- Recording (called from SPE event handler or userspace daemon) ---- */ + +void damon_spe_record_access(unsigned long pfn, pid_t pid); + +/* ---- Maintenance ---- */ + +void damon_spe_prune(void); +void damon_spe_stats(unsigned int *nr_entries, unsigned long *total_accesses); + +#else /* !CONFIG_DAMON_SPE */ + +static inline int damon_spe_folio_heatmap(struct folio *folio, + struct vm_area_struct *vma, unsigned long addr, + unsigned int target_order, unsigned long *hot_bitmap) +{ + return -EOPNOTSUPP; +} + +static inline int damon_spe_hot_fraction(struct folio *folio, + struct vm_area_struct *vma, unsigned long addr, + unsigned int target_order) +{ + return -EOPNOTSUPP; +} + +static inline void damon_spe_record_access(unsigned long pfn, pid_t pid) {} +static inline void damon_spe_prune(void) {} +static inline void damon_spe_stats(unsigned int *nr, unsigned long *total) {} + +#endif /* CONFIG_DAMON_SPE */ +#endif /* _DAMON_SPE_H */ diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 1957e390a277..cb3ea2766b9e 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -18,6 +18,7 @@ #include "../internal.h" #include "ops-common.h" +#include "spe.h" #ifdef CONFIG_DAMON_VADDR_KUNIT_TEST #undef DAMON_MIN_REGION_SZ @@ -945,6 +946,7 @@ static unsigned long damos_va_mthp_split(struct damon_target *target, struct vm_area_struct *vma; struct folio *folio; struct folio_walk fw; + int hot_pct; mm = damon_get_mm(target); if (!mm) @@ -979,8 +981,18 @@ static unsigned long damos_va_mthp_split(struct damon_target *target, folio_get(folio); folio_walk_end(&fw, vma); - if (!split_folio_to_order(folio, target_order)) - applied += chunk_sz; + hot_pct = damon_spe_hot_fraction(folio, vma, addr, + target_order); + /* + * hot_pct < 0: no heatmap data (no SPE, PMD-mapped), + * split unconditionally — DAMON access pattern already + * identified this region as cold. + */ + if (hot_pct < 0 || + (unsigned int)hot_pct < s->hot_threshold) { + if (!split_folio_to_order(folio, target_order)) + applied += chunk_sz; + } folio_unlock(folio); folio_put(folio); -- 2.50.1 (Apple Git-155)