* [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection
@ 2026-11-10 3:29 Yafang Shao
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 1/4] sched: add helpers for numa balancing Yafang Shao
` (4 more replies)
0 siblings, 5 replies; 7+ messages in thread
From: Yafang Shao @ 2026-11-10 3:29 UTC (permalink / raw)
To: ast, daniel, andrii, eddyz87, memxor, mingo, peterz, juri.lelli,
vincent.guittot, akpm, david, gourry, tj, hannes, mkoutny,
shakeel.butt, roman.gushchin, yu.c.chen, zhao1.liu, tim.c.chen,
lizhe.67, ziy, mgorman
Cc: bpf, linux-mm, cgroups, linux-kernel, Yafang Shao
Problem
=======
In our large fleet of Kubernetes-managed servers, NUMA balancing has
historically been disabled globally on each server. With the growing
deployment of multi-socket AMD EPYC servers, cross-NUMA access has become
a critical performance issue, which led us to reconsider enabling NUMA
balancing.
kernel.numa_balancing is a single system-wide sysctl. Latency-sensitive
workloads want NUMA balancing disabled, while NUMA-aware workloads want it
enabled. No single global setting serves both.
Design
======
Core structure
~~~~~~~~~~~~~~
This series follows Tejun's suggestion to store the existing NUMA
balancing mode bits in the cgroup's memcg and expose them through BPF
kfuncs.
A new field, mem_cgroup::numab_mode, holds the per-cgroup NUMA balancing
setting. Two kfuncs are added to set and read it:
bpf_numab_set_mode(): set the mode for a specific memory cgroup. The
update is rejected with -EOPNOTSUPP if the cgroup
has no visible, online memcg. Valid
values are the NUMAB_* mode enum and
NUMAB_FOLLOW_SYSTEM (-1); NUMAB_FOLLOW_SYSTEM
uses the global sysctl state.
bpf_numab_get_mode(): return the combined task-side request of the
cgroup's effective memcg and of all its ancestors.
NUMAB_FOLLOW_SYSTEM is returned when no request is
present. A cgroup without a memcg resolves to its
nearest ancestor memcg. The following section
describes how the request is combined across the
hierarchy.
Both kfuncs are registered for BPF_PROG_TYPE_SYSCALL. bpf_numab_set_mode()
is KF_SLEEPABLE because it takes the mode mutex and the jump-label locks;
bpf_numab_get_mode() does not sleep. The mode values are exposed as a BTF
enum, so a BPF program can use NUMAB_FOLLOW_SYSTEM and the NUMAB_* mode
names without hardcoding private integers.
Effective mode semantics
~~~~~~~~~~~~~~~~~~~~~~~~
In the current implementation, a memory cgroup can be set to the
following modes:
NUMAB_FOLLOW_SYSTEM (-1): use the global sysctl state
NUMAB_DISABLED (0): disabled
NUMAB_NORMAL (0x1): normal mode
NUMAB_TIERING (0x2): tiering mode
NUMAB_NORMAL_TIERING (0x3): normal mode plus memory tiering
The diagrams below assume that each cgroup shown has its own memcg. A
cgroup without one resolves to its nearest ancestor memcg, as described
in Storage of numab_mode.
Except for the sentinel and the opt-out described below, the effective
mode is the OR of this memcg's mode and that of all its ancestors.
- An opt-out wins for the task side and binds the subtree
A DISABLED mode anywhere in the cgroup's ancestry opts the cgroup's
tasks out entirely. Given:
Pods (normal)
/
Pod-A (disabled)
/ \
cgroup-A1(normal) cgroup-A2(tiering)
The effective task-side mode for tasks(*) in cgroup-A1 and cgroup-A2
is:
Pods (normal)
/
Pod-A (disabled)
/ \
cgroup-A1(disabled) cgroup-A2(disabled)
(*) The folio side is described below.
- NUMAB_FOLLOW_SYSTEM is skipped
Given:
Pods (normal)
/
Pod-A (-1)
/ \
cgroup-A1(none) cgroup-A2(none)
The effective mode for tasks in cgroup-A1 and cgroup-A2 is:
Pods (normal)
/
Pod-A (normal)
/ \
cgroup-A1(normal) cgroup-A2(normal)
- For all other modes, the effective value is the OR of this memcg's
mode and that of its ancestors
Given:
Pods (normal)
/
Pod-A (-1)
/ \
cgroup-A1(none) cgroup-A2(tiering)
The effective mode for tasks in cgroup-A1 and cgroup-A2 is:
Pods (normal)
/
Pod-A (-1)
/ \
cgroup-A1(normal) cgroup-A2(normal|tiering)
- The rationale behind these semantics
The cgroup hierarchy semantics invoked here are those of resource
limits: an ancestor's limit binds every descendant. That is the right
semantics for limits: a limit partitions a shared, finite resource,
and what a descendant consumes comes out of its ancestors' pool, so an
ancestor's limit must bind every level below it.
A policy request is a different kind of control: it is a preference
about how the kernel should treat the workload, not a quantity drawn
from a parent's pool. Expressing it consumes nothing, so the limit
semantics does not carry over; the question is only how the preferences
of several levels should combine.
We chose the additive interpretation: a descendant may ask for more on
top of what its ancestors asked for, while an opt-out at any level
still wins: DISABLED anywhere in the ancestry caps the whole subtree.
Task and folio keying
~~~~~~~~~~~~~~~~~~~~~
The NUMA balancing knob is consulted at some callsites with only a folio
and no task, and at others with only a task and no folio.
- Task only
kernel/sched/fair.c: task_numa_fault()
kernel/sched/fair.c: update_scan_period()
kernel/sched/fair.c: migrate_degrades_locality()
kernel/sched/fair.c: exceed_llc_capacity()
kernel/sched/fair.c: get_pref_llc()
kernel/sched/fair.c: get_scan_cpumasks()
- Task and folio
kernel/sched/fair.c: should_numa_migrate_memory()
- Folio only
mm/memory-tiers.c: folio_use_access_time()
mm/mempolicy.c: folio_can_map_prot_numa()
mm/migrate.c: folio_migrate_flags()
mm/migrate.c: migrate_misplaced_folio_prepare()
mm/migrate.c: migrate_misplaced_folio()
Therefore, the helpers are split by side:
folio_numab_effective_mode()
folio_numab_in_*_mode()
task_numab_enabled()
task_numab_get_modes()
Another concern is how to handle callsites that have both a task and a
folio. The following anchor resolves it. Both the user fault path and the
kswapd demotion path reach folio_migrate_flags():
kswapd demotion (no task):
kswapd
+ shrink_folio_list
+ demote_folio_list
+ migrate_pages
+ folio_migrate_flags
+ folio_numab_in_tiering_mode
Fault path (user task):
do_numa_page
+ mpol_misplaced
+ should_numa_migrate_memory(task-m, folio-n) (A)
+ task or folio?
+ migrate_misplaced_folio_prepare(folio-n)
+ migrate_misplaced_folio(folio-n)
+ migrate_pages
+ folio_migrate_flags(folio-n) (B)
+ folio_numab_in_tiering_mode
folio_migrate_flags() must key on the folio, since kswapd has no task to
consult. In the user fault path, both should_numa_migrate_memory() and
folio_numab_in_tiering_mode() are called. Since both refer to the same
folio in this context, keying on the folio at B but not at A would be
inconsistent. The rule is therefore simple: if a folio is available, key
on the folio; otherwise, key on the task. This also keeps the keying
robust to future refactoring: if either folio-keyed function is ever
reached by a kernel thread, the answer does not change.
Storage of numab_mode
~~~~~~~~~~~~~~~~~~~~~
Some cgroups do not have the memory controller enabled:
Given:
Pods (+memory)
/
Pod-A ()
/
cgroup-A1 ()
cgroup-A1 has no memcg. Given the task/folio split, where should the
mode for such a cgroup live?
- If the mode were stored in struct cgroup and setting it on cgroup-A1
were allowed, tasks in cgroup-A1 would use cgroup-A1's request while
its folios would use Pod-A's memcg request, which is inconsistent.
- Storing the mode in struct mem_cgroup avoids that: cgroup-A1 cannot be
set, and both its tasks and its folios resolve to the same ancestor
memcg (Pod-A).
For consistency, the mode is stored in mem_cgroup, setting it on a
cgroup without the memory controller enabled is rejected with
-EOPNOTSUPP, and both task and folio lookups resolve a cgroup without a
memcg to its nearest ancestor memcg. A request is effective only while
its memcg is visible and online. Disabling the memory controller or
offlining the memcg clears the request, and lookups skip such a memcg, so
a pinned memcg cannot keep applying a stale request. A re-enabled memcg
starts transparent.
Shared memory semantics
~~~~~~~~~~~~~~~~~~~~~~~
With the mode stored in the memcg, the longstanding shared-memory
problem in memcg has to be addressed for per-memcg NUMA balancing as
well.
Given:
cgroup-A cgroup-B
\ | (charge)
Memory_shared_by_cgroup-A_and_cgroup-B
A choice is required when the NUMA modes of cgroup-A and cgroup-B
differ. Since the folio-side lookup starts at the folio's memcg, a task
in cgroup-A operating on this shared memory uses cgroup-B's folio-side
mode. If DISABLED were applied to memory, cgroup-B's DISABLED would
prevent cgroup-A from promoting the shared page. We avoid that by making
DISABLED task-side only.
In our experience, NUMA balancing is mostly disabled to avoid the
potential latency spikes it can cause. The intent of disabling it for
cgroup-B is to avoid latency spikes for tasks in cgroup-B; it does not
matter whether tasks in other cgroups perform NUMA balancing. We
therefore disable the task side only, not the memory side: DISABLED
applies to tasks, not to memory.
Demotion is global reclaim, not a per-cgroup balancing request.
DISABLED only opts the cgroup's tasks out of NUMA balancing; it must
not become a memory-side reclaim veto, or a cgroup could keep cold
pages in the fast tier, leaving less fast-tier capacity for other
cgroups' hot pages and defeating promotion for those pages.
So, in the end, DISABLED is a task-side property and does not gate
memory-side operations.
Shared fast tier
~~~~~~~~~~~~~~~~
A promotion competes for capacity in the shared fast tier, but it
cannot consume more memory than the cgroup may already consume: it moves
the cgroup's own folios, within its own memory limit, from the slow tier
to the fast tier. Each contributor's own limit bounds its footprint,
but the fast tier is still a shared resource; demotion is the global
release valve that keeps cold pages from occupying it indefinitely.
So a child can still enable tiering mode if its ancestor has only set
normal.
Given:
Pod (normal)
/ \
cgroup-A (none) cgroup-B (tiering)
the effective mode is:
Pod (normal)
/ \
cgroup-A (normal) cgroup-B (normal|tiering)
What an ancestor cannot do now is cap only tiering while leaving normal
balancing available below. If that targeted cap is wanted, the natural
extension is a negative request: NO_TIERING would propagate down the
tree like the positive requests and mask TIERING out of the effective
mode. It would cap promotion, not demotion; demotion remains global
reclaim. With:
Pod (normal | no_tiering)
/ \
cgroup-A (none) cgroup-B (tiering)
the effective modes would be:
Pod (normal)
/ \
cgroup-A (normal) cgroup-B (normal)
For this version, I would keep the modes minimal and add NO_TIERING as a
follow-up once there is a real user for it.
Cost
====
On the task side, an ancestor can opt out the whole subtree with
DISABLED, but cannot cap only tiering while leaving normal balancing
available below. On the memory side there is no opt-out at all:
DISABLED is task-side only, so a descendant's TIERING request is enough
to enable promotion for its own memory, and promotion is the mode that
consumes the shared fast tier.
The per-cgroup override was measured with schbench, 384 workers, on a
two-node 384-core machine. In that configuration, a cgroup mode put the
cgroup at the throughput of the equivalent system-wide setting, within a
fraction of a percent. This is a throughput measurement, not a
microbenchmark of the scan path: while the static key is on, a folio
lookup is O(cgroup depth), and callers that need more than one bit now
share a single lookup. An effective-mode cache may be worthwhile for
very deep hierarchies; that is left for follow-up measurements.
Limitations
===========
The mode is per-memcg, not per-task. A cgroup without its own memcg
cannot be set; its tasks and folios resolve to the nearest ancestor
memcg. If multiple tasks in a single cgroup need different modes, place
them in separate child cgroups with the memory controller enabled.
For threaded cgroups, the mode must be set on the nearest domain
ancestor that has the memory controller enabled, because the memory
controller is not threaded.
Changelog
=========
RFC v3 -> RFC v4:
- Store numab_mode in mem_cgroup instead of struct cgroup
- Refine the commit log
RFC v3: https://lwn.net/Articles/1101442/
RFC v2 -> RFC v3:
- Replace the struct_ops interface with two kfuncs, following Tejun's
suggestion.
- Improve function naming and simplify the code (David)
RFC v2: https://lwn.net/Articles/1099962/
RFC v1 -> RFC v2:
- v1 keyed the hook on the memory cgroup and temporarily embedded the
cgroup id into the struct_ops value, because cgroup struct_ops support
was still under development.
- v2 attaches through the cgroup struct_ops support that has since landed
in bpf-next, keys the task side on the task's cgroup and the memory side
on the folio's memcg, and adds the memory-tiering mode alongside enable
and disable.
RFC v1: https://lwn.net/Articles/1054030/
Yafang Shao (4):
sched: add helpers for numa balancing
sched: sample NUMA balancing mode once per path
sched: add BPF-based per-cgroup NUMA balancing mode selection
selftests/bpf: add tests for the numab kfuncs
include/linux/memcontrol.h | 6 +
include/linux/sched/numa_balancing.h | 144 +++++++++
init/Kconfig | 23 ++
kernel/sched/bpf_numab.c | 261 +++++++++++++++++
kernel/sched/build_policy.c | 4 +
kernel/sched/fair.c | 99 +++++--
kernel/sched/sched.h | 1 -
mm/memcontrol.c | 14 +
mm/memory-tiers.c | 9 +-
mm/mempolicy.c | 9 +-
mm/migrate.c | 9 +-
mm/vmscan.c | 6 +-
tools/testing/selftests/bpf/config | 2 +
.../selftests/bpf/prog_tests/bpf_numab.c | 273 ++++++++++++++++++
tools/testing/selftests/bpf/progs/bpf_numab.c | 32 ++
15 files changed, 844 insertions(+), 48 deletions(-)
create mode 100644 kernel/sched/bpf_numab.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_numab.c
create mode 100644 tools/testing/selftests/bpf/progs/bpf_numab.c
--
2.52.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [RFC PATCH bpf-next v4 1/4] sched: add helpers for numa balancing
2026-11-10 3:29 [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection Yafang Shao
@ 2026-11-10 3:29 ` Yafang Shao
2026-11-10 3:37 ` Zi Yan
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 2/4] sched: sample NUMA balancing mode once per path Yafang Shao
` (3 subsequent siblings)
4 siblings, 1 reply; 7+ messages in thread
From: Yafang Shao @ 2026-11-10 3:29 UTC (permalink / raw)
To: ast, daniel, andrii, eddyz87, memxor, mingo, peterz, juri.lelli,
vincent.guittot, akpm, david, gourry, tj, hannes, mkoutny,
shakeel.butt, roman.gushchin, yu.c.chen, zhao1.liu, tim.c.chen,
lizhe.67, ziy, mgorman
Cc: bpf, linux-mm, cgroups, linux-kernel, Yafang Shao
Introduce task_numab_enabled(), numab_in_normal_mode() and
numab_in_tiering_mode() to replace the direct accesses to the
sched_numa_balancing static key and the sysctl_numa_balancing_mode
bitmask.
No functional change.
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
---
include/linux/sched/numa_balancing.h | 36 ++++++++++++++++++++++++++++
kernel/sched/fair.c | 22 ++++++++---------
kernel/sched/sched.h | 1 -
mm/memory-tiers.c | 3 ++-
mm/mempolicy.c | 3 +--
mm/migrate.c | 7 +++---
mm/vmscan.c | 6 ++---
7 files changed, 57 insertions(+), 21 deletions(-)
diff --git a/include/linux/sched/numa_balancing.h b/include/linux/sched/numa_balancing.h
index 52b22c5c396d..61ab161fb481 100644
--- a/include/linux/sched/numa_balancing.h
+++ b/include/linux/sched/numa_balancing.h
@@ -8,6 +8,7 @@
*/
#include <linux/sched.h>
+#include <linux/sched/sysctl.h>
#define TNF_MIGRATED 0x01
#define TNF_NO_GROUP 0x02
@@ -32,6 +33,26 @@ extern void set_numabalancing_state(bool enabled);
extern void task_numa_free(struct task_struct *p, bool final);
bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
int src_nid, int dst_cpu);
+
+extern struct static_key_false sched_numa_balancing;
+
+/* Whether NUMA balancing is enabled. */
+static inline bool task_numab_enabled(void)
+{
+ return static_branch_likely(&sched_numa_balancing);
+}
+
+/* Whether NUMA balancing is in normal mode. */
+static inline bool numab_in_normal_mode(void)
+{
+ return sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL;
+}
+
+/* Whether NUMA balancing is in memory tiering mode. */
+static inline bool numab_in_tiering_mode(void)
+{
+ return sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING;
+}
#else
static inline void task_numa_fault(int last_node, int node, int pages,
int flags)
@@ -52,6 +73,21 @@ static inline bool should_numa_migrate_memory(struct task_struct *p,
{
return true;
}
+
+static inline bool task_numab_enabled(void)
+{
+ return false;
+}
+
+static inline bool numab_in_normal_mode(void)
+{
+ return false;
+}
+
+static inline bool numab_in_tiering_mode(void)
+{
+ return false;
+}
#endif
#endif /* _LINUX_SCHED_NUMA_BALANCING_H */
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index cfc982205b98..672dda541458 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -39,6 +39,7 @@
#include <linux/sched/cputime.h>
#include <linux/sched/isolation.h>
#include <linux/sched/nohz.h>
+#include <linux/sched/numa_balancing.h>
#include <linux/sched/prio.h>
#include <linux/static_call.h>
@@ -1434,7 +1435,7 @@ static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
if (!sd)
return true;
- if (static_branch_likely(&sched_numa_balancing)) {
+ if (task_numab_enabled()) {
/*
* TBD: RDT exclusive LLC ways reserved should be
* excluded.
@@ -1822,7 +1823,7 @@ static int get_pref_llc(struct task_struct *p, struct sched_cache_group *grp)
* than sched_setnuma() at least -- and thus the
* conflict only exists for a short period of time.
*/
- if (static_branch_likely(&sched_numa_balancing) &&
+ if (task_numab_enabled() &&
p->numa_preferred_nid >= 0 &&
cpu_to_node(mm_sched_cpu) != p->numa_preferred_nid)
mm_sched_llc = -1;
@@ -1923,7 +1924,7 @@ static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p,
#ifdef CONFIG_NUMA_BALANCING
int cpu, curr_cpu, nid, pref_nid;
- if (!static_branch_likely(&sched_numa_balancing))
+ if (!task_numab_enabled())
goto out;
cpu = READ_ONCE(grp->cpu);
@@ -2906,8 +2907,8 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
last_cpupid = folio_xchg_last_cpupid(folio, this_cpupid);
- if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
- !node_is_toptier(src_nid) && !cpupid_valid(last_cpupid))
+ if (!numab_in_tiering_mode() && !node_is_toptier(src_nid) &&
+ !cpupid_valid(last_cpupid))
return false;
/*
@@ -4140,7 +4141,7 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
struct numa_group *ng;
int priv;
- if (!static_branch_likely(&sched_numa_balancing))
+ if (!task_numab_enabled())
return;
/* for example, ksmd faulting in a user's mm */
@@ -4152,8 +4153,7 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
* node for memory tiering mode.
*/
if (!node_is_toptier(mem_node) &&
- (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING ||
- !cpupid_valid(last_cpupid)))
+ (numab_in_tiering_mode() || !cpupid_valid(last_cpupid)))
return;
/* Allocate buffer to track faults on a per-node basis */
@@ -4611,7 +4611,7 @@ static void update_scan_period(struct task_struct *p, int new_cpu)
int src_nid = cpu_to_node(task_cpu(p));
int dst_nid = cpu_to_node(new_cpu);
- if (!static_branch_likely(&sched_numa_balancing))
+ if (!task_numab_enabled())
return;
if (!p->mm || !p->numa_faults || (p->flags & PF_EXITING))
@@ -10702,7 +10702,7 @@ static long migrate_degrades_locality(struct task_struct *p, struct lb_env *env)
unsigned long src_weight, dst_weight;
int src_nid, dst_nid, dist;
- if (!static_branch_likely(&sched_numa_balancing))
+ if (!task_numab_enabled())
return 0;
if (!p->numa_faults || !(env->sd->flags & SD_NUMA))
@@ -15362,7 +15362,7 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
if (queued)
return;
- if (static_branch_unlikely(&sched_numa_balancing))
+ if (task_numab_enabled())
task_tick_numa(rq, curr);
task_tick_cache(rq, curr);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 6c7c5633524a..9b616abf5c9c 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2496,7 +2496,6 @@ extern union sched_feat_key sched_feat_keys[__SCHED_FEAT_NR];
#endif /* !CONFIG_JUMP_LABEL */
-extern struct static_key_false sched_numa_balancing;
extern struct static_key_false sched_schedstats;
static inline u64 global_rt_period(void)
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 25e121851b58..c49071261078 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -6,6 +6,7 @@
#include <linux/memory.h>
#include <linux/memory-tiers.h>
#include <linux/notifier.h>
+#include <linux/sched/numa_balancing.h>
#include <linux/sched/sysctl.h>
#include "internal.h"
@@ -64,7 +65,7 @@ static const struct bus_type memory_tier_subsys = {
*/
bool folio_use_access_time(struct folio *folio)
{
- return (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+ return numab_in_tiering_mode() &&
!node_is_toptier(folio_nid(folio));
}
#endif
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 0884c629670d..991b80d5e899 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -887,8 +887,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
* Skip scanning top tier node if normal numa
* balancing is disabled
*/
- if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
- node_is_toptier(nid))
+ if (!numab_in_normal_mode() && node_is_toptier(nid))
return false;
if (folio_use_access_time(folio))
diff --git a/mm/migrate.c b/mm/migrate.c
index 7bdcdb57652f..4659ecdc4d0c 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -41,6 +41,7 @@
#include <linux/ptrace.h>
#include <linux/memory.h>
#include <linux/sched/sysctl.h>
+#include <linux/sched/numa_balancing.h>
#include <linux/memory-tiers.h>
#include <linux/pagewalk.h>
@@ -818,7 +819,7 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
* memory node, reset cpupid, because that is used to record
* page access time in slow memory node.
*/
- if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) {
+ if (numab_in_tiering_mode()) {
bool f_toptier = node_is_toptier(folio_nid(folio));
bool t_toptier = node_is_toptier(folio_nid(newfolio));
@@ -2717,7 +2718,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
if (!migrate_balanced_pgdat(pgdat, nr_pages)) {
int z;
- if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING))
+ if (!numab_in_tiering_mode())
return -EAGAIN;
for (z = pgdat->nr_zones - 1; z >= 0; z--) {
if (managed_zone(pgdat->node_zones + z))
@@ -2769,7 +2770,7 @@ int migrate_misplaced_folio(struct folio *folio, int node)
if (nr_succeeded) {
count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
- if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
+ if (numab_in_tiering_mode()
&& !node_is_toptier(folio_nid(folio))
&& node_is_toptier(node))
mod_lruvec_state(lruvec, PGPROMOTE_SUCCESS, nr_succeeded);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce8..7ef4205b6a4e 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -65,6 +65,7 @@
#include <linux/swapops.h>
#include <linux/sched/sysctl.h>
+#include <linux/sched/numa_balancing.h>
#include "internal.h"
#include "page_alloc.h"
@@ -5224,8 +5225,7 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc)
if (!current_is_kswapd() || sc->order)
return false;
- mark = sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING ?
- WMARK_PROMO : WMARK_HIGH;
+ mark = numab_in_tiering_mode() ? WMARK_PROMO : WMARK_HIGH;
for (i = 0; i <= sc->reclaim_idx; i++) {
struct zone *zone = lruvec_pgdat(lruvec)->node_zones + i;
@@ -7131,7 +7131,7 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
enum zone_stat_item item;
unsigned long free_pages;
- if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
+ if (numab_in_tiering_mode())
mark = promo_wmark_pages(zone);
else
mark = high_wmark_pages(zone);
--
2.52.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [RFC PATCH bpf-next v4 2/4] sched: sample NUMA balancing mode once per path
2026-11-10 3:29 [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection Yafang Shao
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 1/4] sched: add helpers for numa balancing Yafang Shao
@ 2026-11-10 3:29 ` Yafang Shao
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 3/4] sched: add BPF-based per-cgroup NUMA balancing mode selection Yafang Shao
` (2 subsequent siblings)
4 siblings, 0 replies; 7+ messages in thread
From: Yafang Shao @ 2026-11-10 3:29 UTC (permalink / raw)
To: ast, daniel, andrii, eddyz87, memxor, mingo, peterz, juri.lelli,
vincent.guittot, akpm, david, gourry, tj, hannes, mkoutny,
shakeel.butt, roman.gushchin, yu.c.chen, zhao1.liu, tim.c.chen,
lizhe.67, ziy, mgorman
Cc: bpf, linux-mm, cgroups, linux-kernel, Yafang Shao
Take one snapshot per path and pass it down.
No functional change.
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
kernel/sched/fair.c | 77 ++++++++++++++++++++++++++++++---------------
1 file changed, 51 insertions(+), 26 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 672dda541458..18b976a47afa 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1422,7 +1422,8 @@ static inline int get_sched_cache_scale(int mul)
return (1 + (tol - 1) * mul);
}
-static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
+static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu,
+ bool numab_enabled)
{
#ifdef CONFIG_NUMA_BALANCING
unsigned long llc, footprint;
@@ -1435,7 +1436,7 @@ static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
if (!sd)
return true;
- if (task_numab_enabled()) {
+ if (numab_enabled) {
/*
* TBD: RDT exclusive LLC ways reserved should be
* excluded.
@@ -1801,7 +1802,8 @@ static unsigned long fraction_mm_sched(struct rq *rq,
return div64_u64(NICE_0_LOAD * pcpu_sched->runtime, rq->cpu_runtime + 1);
}
-static int get_pref_llc(struct task_struct *p, struct sched_cache_group *grp)
+static int get_pref_llc(struct task_struct *p, struct sched_cache_group *grp,
+ bool numab_enabled)
{
int mm_sched_llc = -1, mm_sched_cpu;
@@ -1823,7 +1825,7 @@ static int get_pref_llc(struct task_struct *p, struct sched_cache_group *grp)
* than sched_setnuma() at least -- and thus the
* conflict only exists for a short period of time.
*/
- if (task_numab_enabled() &&
+ if (numab_enabled &&
p->numa_preferred_nid >= 0 &&
cpu_to_node(mm_sched_cpu) != p->numa_preferred_nid)
mm_sched_llc = -1;
@@ -1842,6 +1844,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
struct sched_cache_time *pcpu_sched;
int mm_sched_llc = -1;
unsigned long epoch;
+ bool numab_enabled;
if (!sched_cache_enabled())
return;
@@ -1875,14 +1878,19 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
* its preferred state.
*/
if ((long)(epoch - READ_ONCE(grp->epoch)) > llc_epoch_affinity_timeout ||
- invalid_llc_nr(grp, p, cpu_of(rq)) ||
- exceed_llc_capacity(grp, cpu_of(rq))) {
+ invalid_llc_nr(grp, p, cpu_of(rq))) {
if (READ_ONCE(grp->cpu) != -1)
WRITE_ONCE(grp->cpu, -1);
+ } else {
+ numab_enabled = task_numab_enabled();
+ if (exceed_llc_capacity(grp, cpu_of(rq), numab_enabled)) {
+ if (READ_ONCE(grp->cpu) != -1)
+ WRITE_ONCE(grp->cpu, -1);
+ } else {
+ mm_sched_llc = get_pref_llc(p, grp, numab_enabled);
+ }
}
- mm_sched_llc = get_pref_llc(p, grp);
-
/* task not on rq accounted later in account_entity_enqueue() */
if (task_running_on_cpu(rq->cpu, p) &&
READ_ONCE(p->preferred_llc) != mm_sched_llc) {
@@ -1919,12 +1927,12 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p)
}
static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p,
- struct sched_cache_group *grp)
+ struct sched_cache_group *grp, bool numab_enabled)
{
#ifdef CONFIG_NUMA_BALANCING
int cpu, curr_cpu, nid, pref_nid;
- if (!task_numab_enabled())
+ if (!numab_enabled)
goto out;
cpu = READ_ONCE(grp->cpu);
@@ -1991,6 +1999,7 @@ static void task_cache_work(struct callback_head *work)
struct task_struct *p = current, *cur;
unsigned long curr_m_a_occ = 0;
unsigned long m_a_occ = 0;
+ bool numab_enabled;
WARN_ON_ONCE(work != &p->cache_work);
@@ -2019,11 +2028,16 @@ static void task_cache_work(struct callback_head *work)
return;
curr_cpu = task_cpu(p);
- if (invalid_llc_nr(grp, p, curr_cpu) ||
- exceed_llc_capacity(grp, curr_cpu)) {
+ if (invalid_llc_nr(grp, p, curr_cpu)) {
if (READ_ONCE(grp->cpu) != -1)
WRITE_ONCE(grp->cpu, -1);
+ return;
+ }
+ numab_enabled = task_numab_enabled();
+ if (exceed_llc_capacity(grp, curr_cpu, numab_enabled)) {
+ if (READ_ONCE(grp->cpu) != -1)
+ WRITE_ONCE(grp->cpu, -1);
return;
}
@@ -2033,7 +2047,7 @@ static void task_cache_work(struct callback_head *work)
scoped_guard (cpus_read_lock) {
guard(rcu)();
- get_scan_cpumasks(cpus, p, grp);
+ get_scan_cpumasks(cpus, p, grp, numab_enabled);
for_each_cpu(cpu, cpus) {
/* XXX sched_cluster_active */
@@ -2138,7 +2152,8 @@ void init_sched_mm(struct task_struct *p) { }
static void task_tick_cache(struct rq *rq, struct task_struct *p) { }
static inline int get_pref_llc(struct task_struct *p,
- struct sched_cache_group *grp)
+ struct sched_cache_group *grp,
+ bool numab_enabled)
{
return -1;
}
@@ -2867,6 +2882,9 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
struct numa_group *ng = deref_curr_numa_group(p);
int dst_nid = cpu_to_node(dst_cpu);
int last_cpupid, this_cpupid;
+ bool tiering;
+
+ tiering = numab_in_tiering_mode();
/*
* Cannot migrate to memoryless nodes.
@@ -2878,7 +2896,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
* The pages in slow memory node should be migrated according
* to hot/cold instead of private/shared.
*/
- if (folio_use_access_time(folio)) {
+ if (tiering && !node_is_toptier(folio_nid(folio))) {
struct pglist_data *pgdat;
unsigned long rate_limit;
unsigned int latency, th, def_th;
@@ -2907,7 +2925,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
last_cpupid = folio_xchg_last_cpupid(folio, this_cpupid);
- if (!numab_in_tiering_mode() && !node_is_toptier(src_nid) &&
+ if (!tiering && !node_is_toptier(src_nid) &&
!cpupid_valid(last_cpupid))
return false;
@@ -10696,13 +10714,14 @@ static int task_hot(struct task_struct *p, struct lb_env *env)
* Returns 0, if task migration is not affected by locality.
* Returns a negative value, if task migration improves locality i.e migration preferred.
*/
-static long migrate_degrades_locality(struct task_struct *p, struct lb_env *env)
+static long migrate_degrades_locality(struct task_struct *p, struct lb_env *env,
+ bool numab_enabled)
{
struct numa_group *numa_group = rcu_dereference_all(p->numa_group);
unsigned long src_weight, dst_weight;
int src_nid, dst_nid, dist;
- if (!task_numab_enabled())
+ if (!numab_enabled)
return 0;
if (!p->numa_faults || !(env->sd->flags & SD_NUMA))
@@ -10744,7 +10763,8 @@ static long migrate_degrades_locality(struct task_struct *p, struct lb_env *env)
#else /* !CONFIG_NUMA_BALANCING: */
static inline long migrate_degrades_locality(struct task_struct *p,
- struct lb_env *env)
+ struct lb_env *env,
+ bool numab_enabled)
{
return 0;
}
@@ -10946,7 +10966,8 @@ static inline bool sched_asym(struct sched_domain *sd, int dst_cpu, int src_cpu)
* destination LLC in terms of cache aware load balance.
*/
static enum llc_mig can_migrate_llc_task(struct lb_env *env,
- struct task_struct *p)
+ struct task_struct *p,
+ bool numab_enabled)
{
struct sched_cache_group *grp;
bool to_pref;
@@ -10971,7 +10992,7 @@ static enum llc_mig can_migrate_llc_task(struct lb_env *env,
/* skip cache aware load balance for too many threads */
if (invalid_llc_nr(grp, p, dst_cpu) ||
- exceed_llc_capacity(grp, dst_cpu)) {
+ exceed_llc_capacity(grp, dst_cpu, numab_enabled)) {
if (READ_ONCE(grp->cpu) != -1)
WRITE_ONCE(grp->cpu, -1);
return mig_unrestricted;
@@ -11060,7 +11081,8 @@ migrate_llc_task_wrong_dst(struct task_struct *p, struct lb_env *env)
* Check if migrating task p from env->src_cpu to
* env->dst_cpu breaks LLC localiy.
*/
-static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env)
+static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env,
+ bool numab_enabled)
{
if (!sched_cache_enabled())
return false;
@@ -11087,7 +11109,7 @@ static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env)
if (migrate_llc_task_wrong_dst(p, env))
return true;
- if (can_migrate_llc_task(env, p) != mig_forbid)
+ if (can_migrate_llc_task(env, p, numab_enabled) != mig_forbid)
return false;
return true;
@@ -11113,7 +11135,8 @@ migrate_llc_task_wrong_dst(struct task_struct *p, struct lb_env *env)
}
static inline bool
-migrate_degrades_llc(struct task_struct *p, struct lb_env *env)
+migrate_degrades_llc(struct task_struct *p, struct lb_env *env,
+ bool numab_enabled)
{
return false;
}
@@ -11125,6 +11148,7 @@ static
int can_migrate_task(struct task_struct *p, struct lb_env *env)
{
long degrades, hot;
+ bool numab_enabled;
lockdep_assert_rq_held(env->src_rq);
if (p->sched_task_hot)
@@ -11213,14 +11237,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
if (env->flags & LBF_ACTIVE_LB)
return !migrate_llc_task_wrong_dst(p, env);
- degrades = migrate_degrades_locality(p, env);
+ numab_enabled = task_numab_enabled();
+ degrades = migrate_degrades_locality(p, env, numab_enabled);
if (!degrades) {
/*
* If the NUMA locality is not broken,
* further check if migration would hurt
* LLC locality.
*/
- if (migrate_degrades_llc(p, env)) {
+ if (migrate_degrades_llc(p, env, numab_enabled)) {
/*
* If regular load balancing fails to pull a task
* due to LLC locality, this is expected behavior
--
2.52.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [RFC PATCH bpf-next v4 3/4] sched: add BPF-based per-cgroup NUMA balancing mode selection
2026-11-10 3:29 [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection Yafang Shao
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 1/4] sched: add helpers for numa balancing Yafang Shao
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 2/4] sched: sample NUMA balancing mode once per path Yafang Shao
@ 2026-11-10 3:29 ` Yafang Shao
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 4/4] selftests/bpf: add tests for the numab kfuncs Yafang Shao
2026-11-10 3:35 ` [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection Zi Yan
4 siblings, 0 replies; 7+ messages in thread
From: Yafang Shao @ 2026-11-10 3:29 UTC (permalink / raw)
To: ast, daniel, andrii, eddyz87, memxor, mingo, peterz, juri.lelli,
vincent.guittot, akpm, david, gourry, tj, hannes, mkoutny,
shakeel.butt, roman.gushchin, yu.c.chen, zhao1.liu, tim.c.chen,
lizhe.67, ziy, mgorman
Cc: bpf, linux-mm, cgroups, linux-kernel, Yafang Shao
kernel.numa_balancing is a single system-wide sysctl. Latency-sensitive
workloads want NUMA balancing disabled, while NUMA-aware workloads want it
enabled. No single global setting serves both.
This commit adds a per-memcg numa balancing mode and two kfuncs to
manage it. The mode domain is a BTF enum so BPF programs do not have to
hardcode private integer values.
Store the request in the cgroup's memcg, not in the task or the cgroup
object. Memory is charged to a memcg, so a shared folio has exactly one
policy owner; both the task-side and folio-side lookups can resolve a
cgroup without its own memcg to the same ancestor memcg. This is also why
setting a mode on a cgroup without the memory controller is rejected:
there is no memcg to own the request, and accepting it would make the
task and folio views disagree.
DISABLED wins and binds the whole subtree; otherwise the requested bits
OR together. A nearest-request-wins rule would let a descendant undo an
ancestor's opt-out, and an opt-out is only a subtree guarantee if it
binds. Additive requests let a descendant ask for more without taking
away what an ancestor asked for.
DISABLED speaks for tasks only, never for the memcg's memory. Demotion
is global reclaim, gated by the global demotion toggle, not by
numab_mode. A task-side opt-out must not become a memory-side reclaim
veto: otherwise a cgroup could keep cold pages in the fast tier, leaving
less capacity for other cgroups' hot pages and defeating promotion for
them. The misplaced-migration fallback follows the same rule. kswapd has
no task context, so the folio's memcg is the only policy owner there.
The requests are combined at read time rather than propagated at write
time. A new memcg then starts transparent and sees its ancestors' current
state by construction; propagating a combined value would need write-side
hooks to stay coherent with concurrent updates and a second field for the
folio view, which reads opt-outs differently. The previous patch already
samples the effective mode once per path; this patch changes what that
sampled value means.
The hot paths may run in atomic context, so the lookup takes no
reference: the memcg chain is RCU-protected. The setter needs locks and
may sleep; the getter does not. A request is cleared when the memory
controller is disabled or the memcg is offlined. Lookups also skip
memcgs which are dying, offline or not visible, so a pinned memcg cannot
keep applying a stale request. The tick path uses the cheap global gate
and checks the per-task request only after the scan period expires.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
include/linux/memcontrol.h | 6 +
include/linux/sched/numa_balancing.h | 126 ++++++++++++-
init/Kconfig | 23 +++
kernel/sched/bpf_numab.c | 261 +++++++++++++++++++++++++++
kernel/sched/build_policy.c | 4 +
kernel/sched/fair.c | 26 ++-
mm/memcontrol.c | 14 ++
mm/memory-tiers.c | 8 +-
mm/mempolicy.c | 8 +-
mm/migrate.c | 8 +-
10 files changed, 456 insertions(+), 28 deletions(-)
create mode 100644 kernel/sched/bpf_numab.c
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 69e1bcc67c58..78ea21bfab95 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -342,6 +342,12 @@ struct mem_cgroup {
*/
bool zswap_writeback;
#endif
+
+#ifdef CONFIG_BPF_NUMAB
+ /* per-cgroup NUMA balancing mode request; -1 follows the global state. */
+ int numab_mode;
+#endif
+
/* Not padded: nodeinfo[] is read-mostly too, let it share the line. */
__cacheline_group_end(memcg_read_mostly);
diff --git a/include/linux/sched/numa_balancing.h b/include/linux/sched/numa_balancing.h
index 61ab161fb481..e10093386cf7 100644
--- a/include/linux/sched/numa_balancing.h
+++ b/include/linux/sched/numa_balancing.h
@@ -36,16 +36,56 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
extern struct static_key_false sched_numa_balancing;
-/* Whether NUMA balancing is enabled. */
-static inline bool task_numab_enabled(void)
+/*
+ * Mode values used by the NUMA balancing helpers.
+ */
+enum numab_mode {
+ NUMAB_FOLLOW_SYSTEM = -1,
+ NUMAB_DISABLED = NUMA_BALANCING_DISABLED,
+ NUMAB_NORMAL = NUMA_BALANCING_NORMAL,
+ NUMAB_TIERING = NUMA_BALANCING_MEMORY_TIERING,
+ NUMAB_NORMAL_TIERING = NUMA_BALANCING_NORMAL |
+ NUMA_BALANCING_MEMORY_TIERING,
+};
+
+#ifdef CONFIG_BPF_NUMAB
+extern struct static_key_false numab_cgroup_mode_set;
+
+int task_numab_mode(struct task_struct *p);
+int folio_numab_mode(struct folio *folio);
+void numab_memcg_reset(struct mem_cgroup *memcg);
+#endif
+
+/*
+ * Whether any task might need the NUMA scanner on this tick. This is
+ * only the cheap global gate; the per-task request is checked after the
+ * scan period in task_tick_numa().
+ */
+static inline bool task_numab_tick_possible(void)
{
- return static_branch_likely(&sched_numa_balancing);
+#ifdef CONFIG_BPF_NUMAB
+ if (static_branch_likely(&numab_cgroup_mode_set))
+ return true;
+#endif
+ return static_branch_unlikely(&sched_numa_balancing);
}
-/* Whether NUMA balancing is in normal mode. */
-static inline bool numab_in_normal_mode(void)
+/*
+ * Whether NUMA balancing applies to @p: the global sysctl state, possibly
+ * overridden by the requests of @p's memcg and of every ancestor.
+ * A memcg's request always takes priority over the system setting.
+ */
+static inline bool task_numab_enabled(struct task_struct *p)
{
- return sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL;
+#ifdef CONFIG_BPF_NUMAB
+ if (static_branch_unlikely(&numab_cgroup_mode_set)) {
+ int mode = task_numab_mode(p);
+
+ if (mode != NUMAB_FOLLOW_SYSTEM)
+ return mode != NUMAB_DISABLED;
+ }
+#endif
+ return static_branch_likely(&sched_numa_balancing);
}
/* Whether NUMA balancing is in memory tiering mode. */
@@ -53,6 +93,62 @@ static inline bool numab_in_tiering_mode(void)
{
return sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING;
}
+
+/*
+ * Return both task-side decisions from one request lookup. This is used
+ * by paths that need to know both whether to account the fault and
+ * whether slow-tier faults should be skipped.
+ */
+static inline void task_numab_get_modes(struct task_struct *p, bool *enabled,
+ bool *tiering)
+{
+#ifdef CONFIG_BPF_NUMAB
+ if (static_branch_unlikely(&numab_cgroup_mode_set)) {
+ int mode = task_numab_mode(p);
+
+ if (mode != NUMAB_FOLLOW_SYSTEM) {
+ *enabled = mode != NUMAB_DISABLED;
+ *tiering = mode & NUMAB_TIERING;
+ return;
+ }
+ }
+#endif
+ *enabled = static_branch_likely(&sched_numa_balancing);
+ *tiering = numab_in_tiering_mode();
+}
+
+/*
+ * The combined folio-side request, with the global sysctl state used when
+ * no memcg requests a mode. DISABLED never speaks for memory, so only
+ * NORMAL and TIERING are present here. Callers that need more than one
+ * bit should use this helper once rather than querying each bit
+ * separately.
+ */
+static inline int folio_numab_effective_mode(struct folio *folio)
+{
+#ifdef CONFIG_BPF_NUMAB
+ if (static_branch_unlikely(&numab_cgroup_mode_set)) {
+ int mode = folio_numab_mode(folio);
+
+ if (mode != NUMAB_FOLLOW_SYSTEM)
+ return mode;
+ }
+#endif
+ return sysctl_numa_balancing_mode;
+}
+
+/*
+ * Whether @folio's memory participates in tiering, i.e. whether its
+ * _last_cpupid carries an access time instead of a cpupid. The
+ * requests of the owner memcg and of every ancestor are combined.
+ * DISABLED never speaks for memory (it is a task-side request), so only
+ * NORMAL and TIERING are consulted here; without either, the global
+ * sysctl decides.
+ */
+static inline bool folio_numab_in_tiering_mode(struct folio *folio)
+{
+ return folio_numab_effective_mode(folio) & NUMAB_TIERING;
+}
#else
static inline void task_numa_fault(int last_node, int node, int pages,
int flags)
@@ -74,17 +170,29 @@ static inline bool should_numa_migrate_memory(struct task_struct *p,
return true;
}
-static inline bool task_numab_enabled(void)
+static inline bool task_numab_enabled(struct task_struct *p)
{
return false;
}
-static inline bool numab_in_normal_mode(void)
+static inline bool numab_in_tiering_mode(void)
{
return false;
}
-static inline bool numab_in_tiering_mode(void)
+static inline bool folio_numab_in_tiering_mode(struct folio *folio)
+{
+ return false;
+}
+
+static inline void task_numab_get_modes(struct task_struct *p, bool *enabled,
+ bool *tiering)
+{
+ *enabled = false;
+ *tiering = false;
+}
+
+static inline bool task_numab_tick_possible(void)
{
return false;
}
diff --git a/init/Kconfig b/init/Kconfig
index 4013781e0a02..3f38cf8077a3 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -827,6 +827,29 @@ config NUMA_BALANCING
This system will be inactive on UMA systems.
+config BPF_NUMAB
+ bool "Per-cgroup NUMA balancing mode selection"
+ default n
+ depends on NUMA_BALANCING && MEMCG && BPF_SYSCALL && DEBUG_INFO_BTF
+ help
+ Allow BPF programs to select the NUMA balancing mode (disabled,
+ normal, memory tiering, or both) on a per-cgroup basis, overriding the
+ global NUMA balancing sysctl. A latency-sensitive service can
+ opt its tasks out of balancing, or a batch service can have its
+ PMEM-resident memory promoted when hot, without changing the
+ mode for the rest of the system.
+
+ The request is written and read back through the
+ bpf_numab_set_mode() and bpf_numab_get_mode() kfuncs, callable
+ from BPF_PROG_TYPE_SYSCALL programs. The mode is stored in the
+ target cgroup's memcg; setting it on a cgroup without the memory
+ controller enabled is rejected. This interface only applies to
+ cgroups on the default hierarchy (cgroup v2). While no memcg
+ makes a request, the NUMA balancing hot paths pay a single
+ patched-out branch and nothing else.
+
+ If unsure, say N.
+
config SCHED_CACHE
bool "Cache aware load balance"
default y
diff --git a/kernel/sched/bpf_numab.c b/kernel/sched/bpf_numab.c
new file mode 100644
index 000000000000..c7f44bf383f6
--- /dev/null
+++ b/kernel/sched/bpf_numab.c
@@ -0,0 +1,261 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Per-cgroup NUMA balancing mode
+ *
+ * A cgroup overrides the global state through its memcg. Task and folio
+ * lookups walk the memcg chain; an opt-out binds the subtree, while the
+ * remaining bits OR together.
+ *
+ * Author: Yafang Shao <laoar.shao@gmail.com>
+ */
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/cgroup.h>
+#include <linux/memcontrol.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/sched.h>
+#include <linux/sched/numa_balancing.h>
+
+DEFINE_STATIC_KEY_FALSE(numab_cgroup_mode_set);
+
+/* Whether this memcg may still own a NUMA balancing request. */
+static bool numab_memcg_active(struct mem_cgroup *memcg)
+{
+ unsigned int flags;
+
+ if (mem_cgroup_is_root(memcg))
+ return true;
+
+ /* Take one snapshot so all three state bits describe the same moment. */
+ flags = READ_ONCE(memcg->css.flags);
+
+ return (flags & CSS_ONLINE) &&
+ !(flags & CSS_DYING) &&
+ (flags & CSS_VISIBLE);
+}
+
+/*
+ * Combine the requests of @memcg and of every ancestor. A task chain
+ * stops at an explicit opt-out: a task can never be forced into
+ * balancing. A folio chain skips opt-outs instead (a memcg's DISABLED
+ * request only speaks for its tasks, never for its memory) and keeps
+ * collecting the remaining requests.
+ */
+static int numab_memcg_mode(struct mem_cgroup *memcg, bool folio_side)
+{
+ int req = 0;
+
+ for (; memcg; memcg = parent_mem_cgroup(memcg)) {
+ int mode;
+
+ if (!numab_memcg_active(memcg))
+ continue;
+
+ /* Paired with WRITE_ONCE() under numab_mode_mutex. */
+ mode = data_race(READ_ONCE(memcg->numab_mode));
+ if (mode == NUMAB_FOLLOW_SYSTEM)
+ continue;
+ if (mode == NUMAB_DISABLED) {
+ if (!folio_side)
+ return NUMAB_DISABLED;
+ continue;
+ }
+ req |= mode;
+ }
+ return req ?: NUMAB_FOLLOW_SYSTEM;
+}
+
+/*
+ * The combined request for @p. Kernel threads have no user memory to
+ * balance, so they make no request. The walk needs no reference: the
+ * memcg chain is RCU-protected, so rcu_read_lock() keeps it alive for
+ * the whole walk.
+ */
+int task_numab_mode(struct task_struct *p)
+{
+ struct mem_cgroup *memcg;
+ int mode;
+
+ if (mem_cgroup_disabled() || !p->mm)
+ return NUMAB_FOLLOW_SYSTEM;
+
+ rcu_read_lock();
+ memcg = mem_cgroup_from_task(p);
+ mode = memcg ? numab_memcg_mode(memcg, false) : NUMAB_FOLLOW_SYSTEM;
+ rcu_read_unlock();
+ return mode;
+}
+
+/*
+ * The combined request for @folio's owner. An uncharged folio makes no
+ * request. A folio charged to a memcg whose cgroup the kfunc cannot
+ * reach carries no request and follows the global state.
+ */
+int folio_numab_mode(struct folio *folio)
+{
+ struct mem_cgroup *memcg;
+ int mode;
+
+ if (mem_cgroup_disabled())
+ return NUMAB_FOLLOW_SYSTEM;
+
+ rcu_read_lock();
+ memcg = folio_memcg(folio);
+ mode = memcg ? numab_memcg_mode(memcg, true) : NUMAB_FOLLOW_SYSTEM;
+ rcu_read_unlock();
+ return mode;
+}
+
+static DEFINE_MUTEX(numab_mode_mutex);
+
+/*
+ * Reset the request when the memory controller is disabled, the memcg is
+ * offlined or the memcg is pinned. A memcg may be made visible again, so
+ * it must start transparent.
+ */
+void numab_memcg_reset(struct mem_cgroup *memcg)
+{
+ mutex_lock(&numab_mode_mutex);
+ if (memcg->numab_mode != NUMAB_FOLLOW_SYSTEM) {
+ WRITE_ONCE(memcg->numab_mode, NUMAB_FOLLOW_SYSTEM);
+ static_branch_dec(&numab_cgroup_mode_set);
+ }
+ mutex_unlock(&numab_mode_mutex);
+}
+
+__bpf_kfunc_start_defs();
+
+/**
+ * bpf_numab_set_mode() - Set the NUMA balancing mode of a cgroup's memcg.
+ * @cgid: kernfs id of the cgroup whose memcg is updated
+ * @mode: NUMAB_FOLLOW_SYSTEM to defer to the global sysctl state, or a
+ * NUMAB_* mode. The kernel reads the bits independently, the way
+ * kernel.numa_balancing reads them, so
+ * NUMAB_NORMAL|NUMAB_TIERING engages both modes. A
+ * NUMAB_DISABLED request only speaks for the memcg's tasks, never
+ * for its memory, which follows the remaining requests of its
+ * ancestors.
+ *
+ * The cgroup must have a visible, online memory cgroup; otherwise the
+ * update is rejected.
+ *
+ * Return: 0 on success; %-EINVAL for a mode outside the domain;
+ * %-ENOENT for an id that resolves to no cgroup; %-EOPNOTSUPP
+ * for a cgroup without a visible, online memcg.
+ */
+__bpf_kfunc int bpf_numab_set_mode(u64 cgid, enum numab_mode mode)
+{
+ struct cgroup_subsys_state *css;
+ struct mem_cgroup *memcg;
+ struct cgroup *cgrp;
+ int old;
+
+ if (mode != NUMAB_FOLLOW_SYSTEM &&
+ (mode < 0 || mode > NUMAB_NORMAL_TIERING))
+ return -EINVAL;
+
+ cgrp = __cgroup_get_from_id(cgid);
+ if (IS_ERR(cgrp))
+ return PTR_ERR(cgrp);
+
+ if (mem_cgroup_disabled()) {
+ cgroup_put(cgrp);
+ return -EOPNOTSUPP;
+ }
+
+ rcu_read_lock();
+ css = cgroup_css(cgrp, &memory_cgrp_subsys);
+ memcg = mem_cgroup_from_css(css);
+ if (memcg && !mem_cgroup_tryget_online(memcg))
+ memcg = NULL;
+ rcu_read_unlock();
+
+ if (!memcg) {
+ cgroup_put(cgrp);
+ return -EOPNOTSUPP;
+ }
+
+ mutex_lock(&numab_mode_mutex);
+ if (!numab_memcg_active(memcg)) {
+ mutex_unlock(&numab_mode_mutex);
+ mem_cgroup_put(memcg);
+ cgroup_put(cgrp);
+ return -EOPNOTSUPP;
+ }
+
+ old = READ_ONCE(memcg->numab_mode);
+ if (old == NUMAB_FOLLOW_SYSTEM && mode != NUMAB_FOLLOW_SYSTEM)
+ static_branch_inc(&numab_cgroup_mode_set);
+ else if (old != NUMAB_FOLLOW_SYSTEM && mode == NUMAB_FOLLOW_SYSTEM)
+ static_branch_dec(&numab_cgroup_mode_set);
+ WRITE_ONCE(memcg->numab_mode, (int)mode);
+ mutex_unlock(&numab_mode_mutex);
+
+ mem_cgroup_put(memcg);
+ cgroup_put(cgrp);
+ return 0;
+}
+
+/**
+ * bpf_numab_get_mode() - Read the NUMA balancing mode that applies to a cgroup.
+ * @cgid: kernfs id of the cgroup
+ *
+ * The combined task-side request of the cgroup's effective memcg and of
+ * every ancestor: an opt-out wins, otherwise the requested bits OR
+ * together, and the kernel reads them independently, as
+ * kernel.numa_balancing does, so a combined
+ * NUMAB_NORMAL|NUMAB_TIERING engages both modes. NUMAB_FOLLOW_SYSTEM
+ * when nobody makes a request. If the cgroup
+ * has no memcg, the nearest ancestor memcg is used. The folio-side view
+ * stays internal to the kernel.
+ *
+ * Return: the combined mode, NUMAB_FOLLOW_SYSTEM (-1) included, or
+ * %-ENOENT for an id that resolves to no cgroup.
+ */
+__bpf_kfunc s64 bpf_numab_get_mode(u64 cgid)
+{
+ struct mem_cgroup *memcg;
+ struct cgroup *cgrp;
+ s64 mode;
+
+ cgrp = __cgroup_get_from_id(cgid);
+ if (IS_ERR(cgrp))
+ return PTR_ERR(cgrp);
+
+ if (mem_cgroup_disabled()) {
+ cgroup_put(cgrp);
+ return NUMAB_FOLLOW_SYSTEM;
+ }
+
+ rcu_read_lock();
+ memcg = mem_cgroup_from_css(cgroup_e_css(cgrp, &memory_cgrp_subsys));
+ mode = memcg ? numab_memcg_mode(memcg, false) : NUMAB_FOLLOW_SYSTEM;
+ rcu_read_unlock();
+
+ cgroup_put(cgrp);
+ return mode;
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(numab_kfunc_ids)
+BTF_ID_FLAGS(func, bpf_numab_set_mode, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_numab_get_mode)
+BTF_KFUNCS_END(numab_kfunc_ids)
+
+static const struct btf_kfunc_id_set numab_kfunc_set = {
+ .owner = THIS_MODULE,
+ .set = &numab_kfunc_ids,
+};
+
+static int __init numab_kfunc_init(void)
+{
+ int ret;
+
+ ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &numab_kfunc_set);
+ if (ret)
+ pr_warn("numab: failed to register kfuncs: %d\n", ret);
+ return ret;
+}
+late_initcall(numab_kfunc_init);
diff --git a/kernel/sched/build_policy.c b/kernel/sched/build_policy.c
index 2a828725a7f9..3746c6675ff3 100644
--- a/kernel/sched/build_policy.c
+++ b/kernel/sched/build_policy.c
@@ -75,4 +75,8 @@
# include "ext/sub.c"
#endif
+#ifdef CONFIG_BPF_NUMAB
+#include "bpf_numab.c"
+#endif
+
#include "syscalls.c"
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 18b976a47afa..6c2a7550eef5 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1882,7 +1882,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
if (READ_ONCE(grp->cpu) != -1)
WRITE_ONCE(grp->cpu, -1);
} else {
- numab_enabled = task_numab_enabled();
+ numab_enabled = task_numab_enabled(p);
if (exceed_llc_capacity(grp, cpu_of(rq), numab_enabled)) {
if (READ_ONCE(grp->cpu) != -1)
WRITE_ONCE(grp->cpu, -1);
@@ -2034,7 +2034,7 @@ static void task_cache_work(struct callback_head *work)
return;
}
- numab_enabled = task_numab_enabled();
+ numab_enabled = task_numab_enabled(p);
if (exceed_llc_capacity(grp, curr_cpu, numab_enabled)) {
if (READ_ONCE(grp->cpu) != -1)
WRITE_ONCE(grp->cpu, -1);
@@ -2884,7 +2884,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
int last_cpupid, this_cpupid;
bool tiering;
- tiering = numab_in_tiering_mode();
+ tiering = folio_numab_in_tiering_mode(folio);
/*
* Cannot migrate to memoryless nodes.
@@ -4154,12 +4154,14 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
{
struct task_struct *p = current;
bool migrated = flags & TNF_MIGRATED;
+ bool numab_enabled, numab_tiering;
int cpu_node = task_node(current);
int local = !!(flags & TNF_FAULT_LOCAL);
struct numa_group *ng;
int priv;
- if (!task_numab_enabled())
+ task_numab_get_modes(p, &numab_enabled, &numab_tiering);
+ if (!numab_enabled)
return;
/* for example, ksmd faulting in a user's mm */
@@ -4171,7 +4173,7 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
* node for memory tiering mode.
*/
if (!node_is_toptier(mem_node) &&
- (numab_in_tiering_mode() || !cpupid_valid(last_cpupid)))
+ (numab_tiering || !cpupid_valid(last_cpupid)))
return;
/* Allocate buffer to track faults on a per-node basis */
@@ -4315,6 +4317,9 @@ static void task_numa_work(struct callback_head *work)
if (p->flags & PF_EXITING)
return;
+ if (!task_numab_enabled(p))
+ return;
+
/*
* Memory is pinned to only one NUMA node via cpuset.mems, naturally
* no page can be migrated.
@@ -4619,6 +4624,11 @@ static void task_tick_numa(struct rq *rq, struct task_struct *curr)
curr->numa_scan_period = task_scan_start(curr);
curr->node_stamp += period;
+ if (!task_numab_enabled(curr)) {
+ curr->node_stamp = now;
+ return;
+ }
+
if (!time_before(jiffies, curr->mm->numa_next_scan))
task_work_add(curr, work, TWA_RESUME);
}
@@ -4629,7 +4639,7 @@ static void update_scan_period(struct task_struct *p, int new_cpu)
int src_nid = cpu_to_node(task_cpu(p));
int dst_nid = cpu_to_node(new_cpu);
- if (!task_numab_enabled())
+ if (!task_numab_enabled(p))
return;
if (!p->mm || !p->numa_faults || (p->flags & PF_EXITING))
@@ -11237,7 +11247,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
if (env->flags & LBF_ACTIVE_LB)
return !migrate_llc_task_wrong_dst(p, env);
- numab_enabled = task_numab_enabled();
+ numab_enabled = task_numab_enabled(p);
degrades = migrate_degrades_locality(p, env, numab_enabled);
if (!degrades) {
/*
@@ -15387,7 +15397,7 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
if (queued)
return;
- if (task_numab_enabled())
+ if (task_numab_tick_possible())
task_tick_numa(rq, curr);
task_tick_cache(rq, curr);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index a05d9a8baa1b..3971b5b3360b 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -31,6 +31,7 @@
#include <linux/cgroup.h>
#include <linux/cpuset.h>
#include <linux/sched/mm.h>
+#include <linux/sched/numa_balancing.h>
#include <linux/shmem_fs.h>
#include <linux/hugetlb.h>
#include <linux/pagemap.h>
@@ -4266,6 +4267,9 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent)
if (!memcg)
return ERR_PTR(-ENOMEM);
+#ifdef CONFIG_BPF_NUMAB
+ memcg->numab_mode = NUMAB_FOLLOW_SYSTEM;
+#endif
error = xa_alloc(&mem_cgroup_private_ids, &memcg->private_id, NULL,
XA_LIMIT(1, MEM_CGROUP_ID_MAX), GFP_KERNEL);
if (error)
@@ -4459,6 +4463,9 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
{
struct mem_cgroup *memcg = mem_cgroup_from_css(css);
+#ifdef CONFIG_BPF_NUMAB
+ numab_memcg_reset(memcg);
+#endif
memcg1_css_offline(memcg);
page_counter_set_min(&memcg->memory, 0);
@@ -4532,6 +4539,9 @@ static void mem_cgroup_css_reset(struct cgroup_subsys_state *css)
{
struct mem_cgroup *memcg = mem_cgroup_from_css(css);
+#ifdef CONFIG_BPF_NUMAB
+ numab_memcg_reset(memcg);
+#endif
page_counter_set_max(&memcg->memory, PAGE_COUNTER_MAX);
page_counter_set_max(&memcg->swap, PAGE_COUNTER_MAX);
WRITE_ONCE(memcg->oom_group, false);
@@ -5917,6 +5927,10 @@ static void __init memcg_struct_check(void)
kmemcg_id);
CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_read_mostly,
oom_group);
+#ifdef CONFIG_BPF_NUMAB
+ CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_read_mostly,
+ numab_mode);
+#endif
CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup_per_node,
memcg_pn_read_mostly, memcg);
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index c49071261078..7bfb92657e24 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -57,15 +57,15 @@ static const struct bus_type memory_tier_subsys = {
* folio_use_access_time - check if a folio reuses cpupid for page access time
* @folio: folio to check
*
- * folio's _last_cpupid field is repurposed by memory tiering. In memory
- * tiering mode, cpupid of slow memory folio (not toptier memory) is used to
- * record page access time.
+ * folio's _last_cpupid field is repurposed by memory tiering. When the
+ * owner of @folio participates in tiering, cpupid of slow memory folio
+ * (not toptier memory) is used to record page access time.
*
* Return: the folio _last_cpupid is used to record page access time
*/
bool folio_use_access_time(struct folio *folio)
{
- return numab_in_tiering_mode() &&
+ return folio_numab_in_tiering_mode(folio) &&
!node_is_toptier(folio_nid(folio));
}
#endif
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 991b80d5e899..3534dc63b5a9 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -854,7 +854,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
bool is_private_single_threaded)
{
- int nid;
+ int nid, mode;
if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
return false;
@@ -883,14 +883,16 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
if (is_private_single_threaded && (nid == numa_node_id()))
return false;
+ mode = folio_numab_effective_mode(folio);
+
/*
* Skip scanning top tier node if normal numa
* balancing is disabled
*/
- if (!numab_in_normal_mode() && node_is_toptier(nid))
+ if (!(mode & NUMAB_NORMAL) && node_is_toptier(nid))
return false;
- if (folio_use_access_time(folio))
+ if ((mode & NUMAB_TIERING) && !node_is_toptier(nid))
folio_xchg_access_time(folio, jiffies_to_msecs(jiffies));
return true;
diff --git a/mm/migrate.c b/mm/migrate.c
index 4659ecdc4d0c..f3a84a5903aa 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -819,7 +819,7 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
* memory node, reset cpupid, because that is used to record
* page access time in slow memory node.
*/
- if (numab_in_tiering_mode()) {
+ if (folio_numab_in_tiering_mode(folio)) {
bool f_toptier = node_is_toptier(folio_nid(folio));
bool t_toptier = node_is_toptier(folio_nid(newfolio));
@@ -2718,7 +2718,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
if (!migrate_balanced_pgdat(pgdat, nr_pages)) {
int z;
- if (!numab_in_tiering_mode())
+ if (!folio_numab_in_tiering_mode(folio))
return -EAGAIN;
for (z = pgdat->nr_zones - 1; z >= 0; z--) {
if (managed_zone(pgdat->node_zones + z))
@@ -2760,6 +2760,7 @@ int migrate_misplaced_folio(struct folio *folio, int node)
LIST_HEAD(migratepages);
struct mem_cgroup *memcg = get_mem_cgroup_from_folio(folio);
struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
+ bool folio_tiering = folio_numab_in_tiering_mode(folio);
list_add(&folio->lru, &migratepages);
nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
@@ -2770,8 +2771,7 @@ int migrate_misplaced_folio(struct folio *folio, int node)
if (nr_succeeded) {
count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
- if (numab_in_tiering_mode()
- && !node_is_toptier(folio_nid(folio))
+ if (folio_tiering && !node_is_toptier(folio_nid(folio))
&& node_is_toptier(node))
mod_lruvec_state(lruvec, PGPROMOTE_SUCCESS, nr_succeeded);
}
--
2.52.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [RFC PATCH bpf-next v4 4/4] selftests/bpf: add tests for the numab kfuncs
2026-11-10 3:29 [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection Yafang Shao
` (2 preceding siblings ...)
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 3/4] sched: add BPF-based per-cgroup NUMA balancing mode selection Yafang Shao
@ 2026-11-10 3:29 ` Yafang Shao
2026-11-10 3:35 ` [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection Zi Yan
4 siblings, 0 replies; 7+ messages in thread
From: Yafang Shao @ 2026-11-10 3:29 UTC (permalink / raw)
To: ast, daniel, andrii, eddyz87, memxor, mingo, peterz, juri.lelli,
vincent.guittot, akpm, david, gourry, tj, hannes, mkoutny,
shakeel.butt, roman.gushchin, yu.c.chen, zhao1.liu, tim.c.chen,
lizhe.67, ziy, mgorman
Cc: bpf, linux-mm, cgroups, linux-kernel, Yafang Shao
Exercise the bpf_numab_set_mode() and bpf_numab_get_mode() kfuncs
through BPF_PROG_TYPE_SYSCALL programs: the set/get roundtrip, the
mode domain (NUMAB_FOLLOW_SYSTEM and the NUMA_BALANCING_* bitmasks
accepted, anything else rejected with -EINVAL), unknown cgroup ids
rejected with -ENOENT, and the hierarchy combination the memcg walk
produces (inheritance, OR-ing, and an opt-out winning over every
request in the chain). Also verify the memory-controller requirement:
setting on a cgroup without a memcg is rejected with -EOPNOTSUPP,
while bpf_numab_get_mode() resolves it to the nearest ancestor memcg.
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
tools/testing/selftests/bpf/config | 2 +
.../selftests/bpf/prog_tests/bpf_numab.c | 273 ++++++++++++++++++
tools/testing/selftests/bpf/progs/bpf_numab.c | 32 ++
3 files changed, 307 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_numab.c
create mode 100644 tools/testing/selftests/bpf/progs/bpf_numab.c
diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index 49dfe939467c..e56c8872e51a 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -7,6 +7,7 @@ CONFIG_BPF_JIT=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_BPF_LIRC_MODE2=y
CONFIG_BPF_LSM=y
+CONFIG_BPF_NUMAB=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set
@@ -78,6 +79,7 @@ CONFIG_NET_CLS_FLOWER=y
CONFIG_NET_CLS_MATCHALL=y
CONFIG_NET_FOU=y
CONFIG_NET_FOU_IP_TUNNELS=y
+CONFIG_NUMA_BALANCING=y
CONFIG_NET_IPGRE=y
CONFIG_NET_IPGRE_DEMUX=y
CONFIG_NET_IPIP=y
diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_numab.c b/tools/testing/selftests/bpf/prog_tests/bpf_numab.c
new file mode 100644
index 000000000000..6fb6a4d9f291
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_numab.c
@@ -0,0 +1,273 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+#include "cgroup_helpers.h"
+#include "bpf_numab.skel.h"
+
+#define CGROUP_PATH "/bpf_numab"
+#define CHILD_PATH CGROUP_PATH "/child"
+#define HIER_PATH "/bpf_numab_hier"
+#define HIER_CHILD_PATH HIER_PATH "/child"
+#define NOMEM_PATH "/bpf_numab_nomem"
+#define NOMEM_CHILD_PATH NOMEM_PATH "/child"
+
+/* The per-cgroup request domain, mirroring the kernel constants. */
+#define NUMAB_FOLLOW_SYSTEM (-1)
+#define NUMAB_DISABLED 0
+#define NUMAB_NORMAL 1
+#define NUMAB_TIERING 2
+#define NUMAB_NORMAL_TIERING 3
+
+static int numab_set(struct bpf_numab *skel, u64 cgid,
+ long mode)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, opts);
+ int ret;
+
+ skel->bss->set_cgid = cgid;
+ skel->bss->set_mode = mode;
+ ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.numab_set),
+ &opts);
+ if (!ASSERT_OK(ret, "set run"))
+ return -1;
+ return skel->bss->set_ret;
+}
+
+static long numab_get(struct bpf_numab *skel, u64 cgid)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, opts);
+ int ret;
+
+ skel->bss->get_cgid = cgid;
+ ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.numab_get),
+ &opts);
+ if (!ASSERT_OK(ret, "get run"))
+ return -EIO;
+ return skel->bss->get_mode;
+}
+
+static void test_set_get(void)
+{
+ struct bpf_numab *skel;
+ int cgroup_fd;
+ u64 cgid;
+
+ cgroup_fd = test__join_cgroup(CGROUP_PATH);
+ if (!ASSERT_GE(cgroup_fd, 0, "join_cgroup"))
+ return;
+
+ skel = bpf_numab__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ goto cleanup;
+
+ cgid = get_cgroup_id(CGROUP_PATH);
+ if (!ASSERT_GT(cgid, 0, "get_cgroup_id"))
+ goto cleanup;
+
+ ASSERT_EQ(numab_set(skel, cgid, NUMAB_NORMAL), 0, "set normal");
+ ASSERT_EQ(numab_get(skel, cgid), NUMAB_NORMAL, "get normal");
+
+ ASSERT_EQ(numab_set(skel, cgid, NUMAB_TIERING), 0, "set tiering");
+ ASSERT_EQ(numab_get(skel, cgid), NUMAB_TIERING, "get tiering");
+
+ ASSERT_EQ(numab_set(skel, cgid, NUMAB_NORMAL_TIERING), 0, "set both");
+ ASSERT_EQ(numab_get(skel, cgid), NUMAB_NORMAL_TIERING, "get both");
+
+ ASSERT_EQ(numab_set(skel, cgid, NUMAB_DISABLED), 0, "set disabled");
+ ASSERT_EQ(numab_get(skel, cgid), NUMAB_DISABLED, "get disabled");
+
+ /* Back to the system setting. */
+ ASSERT_EQ(numab_set(skel, cgid, NUMAB_FOLLOW_SYSTEM), 0, "unset");
+ ASSERT_EQ(numab_get(skel, cgid), NUMAB_FOLLOW_SYSTEM, "get unset");
+
+cleanup:
+ bpf_numab__destroy(skel);
+ close(cgroup_fd);
+}
+
+static void test_invalid_mode(void)
+{
+ struct bpf_numab *skel;
+ int cgroup_fd;
+ u64 cgid;
+
+ cgroup_fd = test__join_cgroup(CGROUP_PATH);
+ if (!ASSERT_GE(cgroup_fd, 0, "join_cgroup"))
+ return;
+
+ skel = bpf_numab__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ goto cleanup;
+
+ cgid = get_cgroup_id(CGROUP_PATH);
+ if (!ASSERT_GT(cgid, 0, "get_cgroup_id"))
+ goto cleanup;
+
+ ASSERT_EQ(numab_set(skel, cgid, -2), -EINVAL, "set below domain");
+ ASSERT_EQ(numab_set(skel, cgid, 4), -EINVAL, "set above domain");
+ ASSERT_EQ(numab_set(skel, cgid, 0x1000), -EINVAL, "set far above");
+
+ /* The rejected requests must not have stuck. */
+ ASSERT_EQ(numab_get(skel, cgid), NUMAB_FOLLOW_SYSTEM, "get unchanged");
+
+cleanup:
+ bpf_numab__destroy(skel);
+ close(cgroup_fd);
+}
+
+static void test_bad_cgid(void)
+{
+ struct bpf_numab *skel;
+
+ skel = bpf_numab__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+
+ ASSERT_EQ(numab_set(skel, 0, NUMAB_NORMAL), -ENOENT, "set cgid 0");
+ ASSERT_EQ(numab_get(skel, 0), -ENOENT, "get cgid 0");
+ ASSERT_EQ(numab_set(skel, 0xdeadbeef, NUMAB_NORMAL), -ENOENT,
+ "set bad cgid");
+ ASSERT_EQ(numab_get(skel, 0xdeadbeef), -ENOENT, "get bad cgid");
+
+ bpf_numab__destroy(skel);
+}
+
+static void test_hierarchy(void)
+{
+ u64 parent_cgid, child_cgid;
+ int runner_fd, parent_fd, child_fd;
+ struct bpf_numab *skel;
+
+ runner_fd = test__join_cgroup("/bpf_numab_runner");
+ if (!ASSERT_GE(runner_fd, 0, "join_runner_cgroup"))
+ return;
+
+ parent_fd = create_and_get_cgroup(HIER_PATH);
+ if (!ASSERT_GE(parent_fd, 0, "create_parent_cgroup"))
+ goto cleanup_runner;
+
+ if (!ASSERT_OK(enable_controllers(HIER_PATH, "memory"),
+ "enable_memory_controller"))
+ goto cleanup_parent;
+
+ child_fd = create_and_get_cgroup(HIER_CHILD_PATH);
+ if (!ASSERT_GE(child_fd, 0, "create_child_cgroup"))
+ goto cleanup_parent;
+
+ skel = bpf_numab__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ goto cleanup;
+
+ parent_cgid = get_cgroup_id(HIER_PATH);
+ child_cgid = get_cgroup_id(HIER_CHILD_PATH);
+ if (!ASSERT_GT(parent_cgid, 0, "get_parent_cgid") ||
+ !ASSERT_GT(child_cgid, 0, "get_child_cgid"))
+ goto cleanup;
+
+ /* The child inherits an ancestor's request untouched. */
+ ASSERT_EQ(numab_set(skel, parent_cgid, NUMAB_TIERING), 0,
+ "set parent tiering");
+ ASSERT_EQ(numab_get(skel, child_cgid), NUMAB_TIERING,
+ "child inherits");
+
+ /* A request ORs with the ancestor's; NORMAL does not undo TIERING. */
+ ASSERT_EQ(numab_set(skel, child_cgid, NUMAB_NORMAL), 0,
+ "set child normal");
+ ASSERT_EQ(numab_get(skel, child_cgid), NUMAB_NORMAL_TIERING,
+ "or both");
+
+ /* The child's own opt-out wins. */
+ ASSERT_EQ(numab_set(skel, child_cgid, NUMAB_DISABLED), 0,
+ "set child disabled");
+ ASSERT_EQ(numab_get(skel, child_cgid), NUMAB_DISABLED,
+ "own opt-out wins");
+
+ /* So does an ancestor's: a task is never forced into balancing. */
+ ASSERT_EQ(numab_set(skel, child_cgid, NUMAB_TIERING), 0,
+ "set child tiering");
+ ASSERT_EQ(numab_set(skel, parent_cgid, NUMAB_DISABLED), 0,
+ "set parent disabled");
+ ASSERT_EQ(numab_get(skel, child_cgid), NUMAB_DISABLED,
+ "ancestor opt-out wins");
+
+ /* Unsetting the parent re-exposes the child's own request. */
+ ASSERT_EQ(numab_set(skel, parent_cgid, NUMAB_FOLLOW_SYSTEM), 0,
+ "unset parent");
+ ASSERT_EQ(numab_get(skel, child_cgid), NUMAB_TIERING,
+ "child request again");
+
+ /* Unsetting the child leaves the chain with no request. */
+ ASSERT_EQ(numab_set(skel, child_cgid, NUMAB_FOLLOW_SYSTEM), 0,
+ "unset child");
+ ASSERT_EQ(numab_get(skel, child_cgid), NUMAB_FOLLOW_SYSTEM,
+ "no request");
+
+cleanup:
+ bpf_numab__destroy(skel);
+ close(child_fd);
+ remove_cgroup(HIER_CHILD_PATH);
+cleanup_parent:
+ close(parent_fd);
+ remove_cgroup(HIER_PATH);
+cleanup_runner:
+ close(runner_fd);
+}
+
+static void test_no_memcg(void)
+{
+ struct bpf_numab *skel;
+ int parent_fd, child_fd;
+ u64 parent_cgid, child_cgid;
+
+ parent_fd = test__join_cgroup(NOMEM_PATH);
+ if (!ASSERT_GE(parent_fd, 0, "join_nomem_parent"))
+ return;
+
+ child_fd = create_and_get_cgroup(NOMEM_CHILD_PATH);
+ if (!ASSERT_GE(child_fd, 0, "create_nomem_child"))
+ goto cleanup_parent;
+
+ skel = bpf_numab__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ goto cleanup;
+
+ parent_cgid = get_cgroup_id(NOMEM_PATH);
+ child_cgid = get_cgroup_id(NOMEM_CHILD_PATH);
+ if (!ASSERT_GT(parent_cgid, 0, "get_nomem_parent_cgid") ||
+ !ASSERT_GT(child_cgid, 0, "get_nomem_child_cgid"))
+ goto cleanup;
+
+ /*
+ * The parent has a memcg; the child was created without enabling
+ * the memory controller in the parent's subtree, so it has none.
+ */
+ ASSERT_EQ(numab_set(skel, parent_cgid, NUMAB_NORMAL), 0,
+ "set parent normal");
+ ASSERT_EQ(numab_get(skel, child_cgid), NUMAB_NORMAL,
+ "child resolves to parent memcg");
+ ASSERT_EQ(numab_set(skel, child_cgid, NUMAB_TIERING), -EOPNOTSUPP,
+ "child set rejected");
+ ASSERT_EQ(numab_get(skel, child_cgid), NUMAB_NORMAL,
+ "child get unchanged");
+
+cleanup:
+ bpf_numab__destroy(skel);
+ close(child_fd);
+ remove_cgroup(NOMEM_CHILD_PATH);
+cleanup_parent:
+ close(parent_fd);
+}
+
+void test_bpf_numab(void)
+{
+ if (test__start_subtest("set_get"))
+ test_set_get();
+ if (test__start_subtest("invalid_mode"))
+ test_invalid_mode();
+ if (test__start_subtest("bad_cgid"))
+ test_bad_cgid();
+ if (test__start_subtest("hierarchy"))
+ test_hierarchy();
+ if (test__start_subtest("no_memcg"))
+ test_no_memcg();
+}
diff --git a/tools/testing/selftests/bpf/progs/bpf_numab.c b/tools/testing/selftests/bpf/progs/bpf_numab.c
new file mode 100644
index 000000000000..25da97bce82f
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/bpf_numab.c
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+/* Inputs, set from userspace before each run. */
+u64 set_cgid;
+long set_mode;
+u64 get_cgid;
+
+/* Results, read back after each run. */
+long set_ret;
+long get_mode;
+
+extern int bpf_numab_set_mode(u64 cgid, enum numab_mode mode) __ksym;
+extern s64 bpf_numab_get_mode(u64 cgid) __ksym;
+
+SEC("syscall")
+int numab_set(void *ctx)
+{
+ set_ret = bpf_numab_set_mode(set_cgid, (enum numab_mode)set_mode);
+ return 0;
+}
+
+SEC("syscall")
+int numab_get(void *ctx)
+{
+ get_mode = bpf_numab_get_mode(get_cgid);
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
--
2.52.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection
2026-11-10 3:29 [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection Yafang Shao
` (3 preceding siblings ...)
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 4/4] selftests/bpf: add tests for the numab kfuncs Yafang Shao
@ 2026-11-10 3:35 ` Zi Yan
4 siblings, 0 replies; 7+ messages in thread
From: Zi Yan @ 2026-11-10 3:35 UTC (permalink / raw)
To: Yafang Shao, Ying Huang
Cc: ast, daniel, andrii, eddyz87, memxor, mingo, peterz, juri.lelli,
vincent.guittot, akpm, david, gourry, tj, hannes, mkoutny,
shakeel.butt, roman.gushchin, yu.c.chen, zhao1.liu, tim.c.chen,
lizhe.67, mgorman, bpf, linux-mm, cgroups, linux-kernel
+Ying, who did a lot of work on NUMA balancing
On 9 Nov 2026, at 22:29, Yafang Shao wrote:
> Problem
> =======
>
> In our large fleet of Kubernetes-managed servers, NUMA balancing has
> historically been disabled globally on each server. With the growing
> deployment of multi-socket AMD EPYC servers, cross-NUMA access has become
> a critical performance issue, which led us to reconsider enabling NUMA
> balancing.
>
> kernel.numa_balancing is a single system-wide sysctl. Latency-sensitive
> workloads want NUMA balancing disabled, while NUMA-aware workloads want it
> enabled. No single global setting serves both.
>
> Design
> ======
>
> Core structure
> ~~~~~~~~~~~~~~
>
> This series follows Tejun's suggestion to store the existing NUMA
> balancing mode bits in the cgroup's memcg and expose them through BPF
> kfuncs.
>
> A new field, mem_cgroup::numab_mode, holds the per-cgroup NUMA balancing
> setting. Two kfuncs are added to set and read it:
>
> bpf_numab_set_mode(): set the mode for a specific memory cgroup. The
> update is rejected with -EOPNOTSUPP if the cgroup
> has no visible, online memcg. Valid
> values are the NUMAB_* mode enum and
> NUMAB_FOLLOW_SYSTEM (-1); NUMAB_FOLLOW_SYSTEM
> uses the global sysctl state.
>
> bpf_numab_get_mode(): return the combined task-side request of the
> cgroup's effective memcg and of all its ancestors.
> NUMAB_FOLLOW_SYSTEM is returned when no request is
> present. A cgroup without a memcg resolves to its
> nearest ancestor memcg. The following section
> describes how the request is combined across the
> hierarchy.
>
> Both kfuncs are registered for BPF_PROG_TYPE_SYSCALL. bpf_numab_set_mode()
> is KF_SLEEPABLE because it takes the mode mutex and the jump-label locks;
> bpf_numab_get_mode() does not sleep. The mode values are exposed as a BTF
> enum, so a BPF program can use NUMAB_FOLLOW_SYSTEM and the NUMAB_* mode
> names without hardcoding private integers.
>
> Effective mode semantics
> ~~~~~~~~~~~~~~~~~~~~~~~~
>
> In the current implementation, a memory cgroup can be set to the
> following modes:
>
> NUMAB_FOLLOW_SYSTEM (-1): use the global sysctl state
> NUMAB_DISABLED (0): disabled
> NUMAB_NORMAL (0x1): normal mode
> NUMAB_TIERING (0x2): tiering mode
> NUMAB_NORMAL_TIERING (0x3): normal mode plus memory tiering
>
> The diagrams below assume that each cgroup shown has its own memcg. A
> cgroup without one resolves to its nearest ancestor memcg, as described
> in Storage of numab_mode.
>
> Except for the sentinel and the opt-out described below, the effective
> mode is the OR of this memcg's mode and that of all its ancestors.
>
> - An opt-out wins for the task side and binds the subtree
>
> A DISABLED mode anywhere in the cgroup's ancestry opts the cgroup's
> tasks out entirely. Given:
> Pods (normal)
> /
> Pod-A (disabled)
> / \
> cgroup-A1(normal) cgroup-A2(tiering)
>
> The effective task-side mode for tasks(*) in cgroup-A1 and cgroup-A2
> is:
>
> Pods (normal)
> /
> Pod-A (disabled)
> / \
> cgroup-A1(disabled) cgroup-A2(disabled)
>
> (*) The folio side is described below.
>
> - NUMAB_FOLLOW_SYSTEM is skipped
>
> Given:
>
> Pods (normal)
> /
> Pod-A (-1)
> / \
> cgroup-A1(none) cgroup-A2(none)
>
> The effective mode for tasks in cgroup-A1 and cgroup-A2 is:
>
> Pods (normal)
> /
> Pod-A (normal)
> / \
> cgroup-A1(normal) cgroup-A2(normal)
>
> - For all other modes, the effective value is the OR of this memcg's
> mode and that of its ancestors
>
> Given:
>
> Pods (normal)
> /
> Pod-A (-1)
> / \
> cgroup-A1(none) cgroup-A2(tiering)
>
> The effective mode for tasks in cgroup-A1 and cgroup-A2 is:
>
> Pods (normal)
> /
> Pod-A (-1)
> / \
> cgroup-A1(normal) cgroup-A2(normal|tiering)
>
> - The rationale behind these semantics
>
> The cgroup hierarchy semantics invoked here are those of resource
> limits: an ancestor's limit binds every descendant. That is the right
> semantics for limits: a limit partitions a shared, finite resource,
> and what a descendant consumes comes out of its ancestors' pool, so an
> ancestor's limit must bind every level below it.
>
> A policy request is a different kind of control: it is a preference
> about how the kernel should treat the workload, not a quantity drawn
> from a parent's pool. Expressing it consumes nothing, so the limit
> semantics does not carry over; the question is only how the preferences
> of several levels should combine.
>
> We chose the additive interpretation: a descendant may ask for more on
> top of what its ancestors asked for, while an opt-out at any level
> still wins: DISABLED anywhere in the ancestry caps the whole subtree.
>
> Task and folio keying
> ~~~~~~~~~~~~~~~~~~~~~
>
> The NUMA balancing knob is consulted at some callsites with only a folio
> and no task, and at others with only a task and no folio.
>
> - Task only
>
> kernel/sched/fair.c: task_numa_fault()
> kernel/sched/fair.c: update_scan_period()
> kernel/sched/fair.c: migrate_degrades_locality()
> kernel/sched/fair.c: exceed_llc_capacity()
> kernel/sched/fair.c: get_pref_llc()
> kernel/sched/fair.c: get_scan_cpumasks()
>
> - Task and folio
>
> kernel/sched/fair.c: should_numa_migrate_memory()
>
> - Folio only
>
> mm/memory-tiers.c: folio_use_access_time()
> mm/mempolicy.c: folio_can_map_prot_numa()
> mm/migrate.c: folio_migrate_flags()
> mm/migrate.c: migrate_misplaced_folio_prepare()
> mm/migrate.c: migrate_misplaced_folio()
>
> Therefore, the helpers are split by side:
>
> folio_numab_effective_mode()
> folio_numab_in_*_mode()
> task_numab_enabled()
> task_numab_get_modes()
>
> Another concern is how to handle callsites that have both a task and a
> folio. The following anchor resolves it. Both the user fault path and the
> kswapd demotion path reach folio_migrate_flags():
>
> kswapd demotion (no task):
>
> kswapd
> + shrink_folio_list
> + demote_folio_list
> + migrate_pages
> + folio_migrate_flags
> + folio_numab_in_tiering_mode
>
> Fault path (user task):
>
> do_numa_page
> + mpol_misplaced
> + should_numa_migrate_memory(task-m, folio-n) (A)
> + task or folio?
> + migrate_misplaced_folio_prepare(folio-n)
> + migrate_misplaced_folio(folio-n)
> + migrate_pages
> + folio_migrate_flags(folio-n) (B)
> + folio_numab_in_tiering_mode
>
> folio_migrate_flags() must key on the folio, since kswapd has no task to
> consult. In the user fault path, both should_numa_migrate_memory() and
> folio_numab_in_tiering_mode() are called. Since both refer to the same
> folio in this context, keying on the folio at B but not at A would be
> inconsistent. The rule is therefore simple: if a folio is available, key
> on the folio; otherwise, key on the task. This also keeps the keying
> robust to future refactoring: if either folio-keyed function is ever
> reached by a kernel thread, the answer does not change.
>
> Storage of numab_mode
> ~~~~~~~~~~~~~~~~~~~~~
>
> Some cgroups do not have the memory controller enabled:
>
> Given:
> Pods (+memory)
> /
> Pod-A ()
> /
> cgroup-A1 ()
>
> cgroup-A1 has no memcg. Given the task/folio split, where should the
> mode for such a cgroup live?
>
> - If the mode were stored in struct cgroup and setting it on cgroup-A1
> were allowed, tasks in cgroup-A1 would use cgroup-A1's request while
> its folios would use Pod-A's memcg request, which is inconsistent.
>
> - Storing the mode in struct mem_cgroup avoids that: cgroup-A1 cannot be
> set, and both its tasks and its folios resolve to the same ancestor
> memcg (Pod-A).
>
> For consistency, the mode is stored in mem_cgroup, setting it on a
> cgroup without the memory controller enabled is rejected with
> -EOPNOTSUPP, and both task and folio lookups resolve a cgroup without a
> memcg to its nearest ancestor memcg. A request is effective only while
> its memcg is visible and online. Disabling the memory controller or
> offlining the memcg clears the request, and lookups skip such a memcg, so
> a pinned memcg cannot keep applying a stale request. A re-enabled memcg
> starts transparent.
>
> Shared memory semantics
> ~~~~~~~~~~~~~~~~~~~~~~~
>
> With the mode stored in the memcg, the longstanding shared-memory
> problem in memcg has to be addressed for per-memcg NUMA balancing as
> well.
>
> Given:
>
> cgroup-A cgroup-B
> \ | (charge)
> Memory_shared_by_cgroup-A_and_cgroup-B
>
> A choice is required when the NUMA modes of cgroup-A and cgroup-B
> differ. Since the folio-side lookup starts at the folio's memcg, a task
> in cgroup-A operating on this shared memory uses cgroup-B's folio-side
> mode. If DISABLED were applied to memory, cgroup-B's DISABLED would
> prevent cgroup-A from promoting the shared page. We avoid that by making
> DISABLED task-side only.
>
> In our experience, NUMA balancing is mostly disabled to avoid the
> potential latency spikes it can cause. The intent of disabling it for
> cgroup-B is to avoid latency spikes for tasks in cgroup-B; it does not
> matter whether tasks in other cgroups perform NUMA balancing. We
> therefore disable the task side only, not the memory side: DISABLED
> applies to tasks, not to memory.
>
> Demotion is global reclaim, not a per-cgroup balancing request.
> DISABLED only opts the cgroup's tasks out of NUMA balancing; it must
> not become a memory-side reclaim veto, or a cgroup could keep cold
> pages in the fast tier, leaving less fast-tier capacity for other
> cgroups' hot pages and defeating promotion for those pages.
>
> So, in the end, DISABLED is a task-side property and does not gate
> memory-side operations.
>
> Shared fast tier
> ~~~~~~~~~~~~~~~~
>
> A promotion competes for capacity in the shared fast tier, but it
> cannot consume more memory than the cgroup may already consume: it moves
> the cgroup's own folios, within its own memory limit, from the slow tier
> to the fast tier. Each contributor's own limit bounds its footprint,
> but the fast tier is still a shared resource; demotion is the global
> release valve that keeps cold pages from occupying it indefinitely.
>
> So a child can still enable tiering mode if its ancestor has only set
> normal.
>
> Given:
>
> Pod (normal)
> / \
> cgroup-A (none) cgroup-B (tiering)
>
> the effective mode is:
>
> Pod (normal)
> / \
> cgroup-A (normal) cgroup-B (normal|tiering)
>
> What an ancestor cannot do now is cap only tiering while leaving normal
> balancing available below. If that targeted cap is wanted, the natural
> extension is a negative request: NO_TIERING would propagate down the
> tree like the positive requests and mask TIERING out of the effective
> mode. It would cap promotion, not demotion; demotion remains global
> reclaim. With:
>
> Pod (normal | no_tiering)
> / \
> cgroup-A (none) cgroup-B (tiering)
>
> the effective modes would be:
>
> Pod (normal)
> / \
> cgroup-A (normal) cgroup-B (normal)
>
> For this version, I would keep the modes minimal and add NO_TIERING as a
> follow-up once there is a real user for it.
>
> Cost
> ====
>
> On the task side, an ancestor can opt out the whole subtree with
> DISABLED, but cannot cap only tiering while leaving normal balancing
> available below. On the memory side there is no opt-out at all:
> DISABLED is task-side only, so a descendant's TIERING request is enough
> to enable promotion for its own memory, and promotion is the mode that
> consumes the shared fast tier.
>
> The per-cgroup override was measured with schbench, 384 workers, on a
> two-node 384-core machine. In that configuration, a cgroup mode put the
> cgroup at the throughput of the equivalent system-wide setting, within a
> fraction of a percent. This is a throughput measurement, not a
> microbenchmark of the scan path: while the static key is on, a folio
> lookup is O(cgroup depth), and callers that need more than one bit now
> share a single lookup. An effective-mode cache may be worthwhile for
> very deep hierarchies; that is left for follow-up measurements.
>
> Limitations
> ===========
>
> The mode is per-memcg, not per-task. A cgroup without its own memcg
> cannot be set; its tasks and folios resolve to the nearest ancestor
> memcg. If multiple tasks in a single cgroup need different modes, place
> them in separate child cgroups with the memory controller enabled.
>
> For threaded cgroups, the mode must be set on the nearest domain
> ancestor that has the memory controller enabled, because the memory
> controller is not threaded.
>
> Changelog
> =========
>
> RFC v3 -> RFC v4:
> - Store numab_mode in mem_cgroup instead of struct cgroup
> - Refine the commit log
>
> RFC v3: https://lwn.net/Articles/1101442/
>
> RFC v2 -> RFC v3:
> - Replace the struct_ops interface with two kfuncs, following Tejun's
> suggestion.
> - Improve function naming and simplify the code (David)
>
> RFC v2: https://lwn.net/Articles/1099962/
>
> RFC v1 -> RFC v2:
> - v1 keyed the hook on the memory cgroup and temporarily embedded the
> cgroup id into the struct_ops value, because cgroup struct_ops support
> was still under development.
> - v2 attaches through the cgroup struct_ops support that has since landed
> in bpf-next, keys the task side on the task's cgroup and the memory side
> on the folio's memcg, and adds the memory-tiering mode alongside enable
> and disable.
>
> RFC v1: https://lwn.net/Articles/1054030/
>
> Yafang Shao (4):
> sched: add helpers for numa balancing
> sched: sample NUMA balancing mode once per path
> sched: add BPF-based per-cgroup NUMA balancing mode selection
> selftests/bpf: add tests for the numab kfuncs
>
> include/linux/memcontrol.h | 6 +
> include/linux/sched/numa_balancing.h | 144 +++++++++
> init/Kconfig | 23 ++
> kernel/sched/bpf_numab.c | 261 +++++++++++++++++
> kernel/sched/build_policy.c | 4 +
> kernel/sched/fair.c | 99 +++++--
> kernel/sched/sched.h | 1 -
> mm/memcontrol.c | 14 +
> mm/memory-tiers.c | 9 +-
> mm/mempolicy.c | 9 +-
> mm/migrate.c | 9 +-
> mm/vmscan.c | 6 +-
> tools/testing/selftests/bpf/config | 2 +
> .../selftests/bpf/prog_tests/bpf_numab.c | 273 ++++++++++++++++++
> tools/testing/selftests/bpf/progs/bpf_numab.c | 32 ++
> 15 files changed, 844 insertions(+), 48 deletions(-)
> create mode 100644 kernel/sched/bpf_numab.c
> create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_numab.c
> create mode 100644 tools/testing/selftests/bpf/progs/bpf_numab.c
>
> --
> 2.52.0
Best Regards,
Yan, Zi
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC PATCH bpf-next v4 1/4] sched: add helpers for numa balancing
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 1/4] sched: add helpers for numa balancing Yafang Shao
@ 2026-11-10 3:37 ` Zi Yan
0 siblings, 0 replies; 7+ messages in thread
From: Zi Yan @ 2026-11-10 3:37 UTC (permalink / raw)
To: Yafang Shao, ast, daniel, andrii, eddyz87, memxor, mingo, peterz,
juri.lelli, vincent.guittot, akpm, david, gourry, tj, hannes,
mkoutny, shakeel.butt, roman.gushchin, yu.c.chen, zhao1.liu,
tim.c.chen, lizhe.67, mgorman
Cc: bpf, linux-mm, cgroups, linux-kernel
On Mon Nov 9, 2026 at 10:29 PM EST, Yafang Shao wrote:
> Introduce task_numab_enabled(), numab_in_normal_mode() and
> numab_in_tiering_mode() to replace the direct accesses to the
> sched_numa_balancing static key and the sysctl_numa_balancing_mode
> bitmask.
>
> No functional change.
>
> Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
> Cc: David Hildenbrand (Arm) <david@kernel.org>
> ---
> include/linux/sched/numa_balancing.h | 36 ++++++++++++++++++++++++++++
> kernel/sched/fair.c | 22 ++++++++---------
> kernel/sched/sched.h | 1 -
> mm/memory-tiers.c | 3 ++-
> mm/mempolicy.c | 3 +--
> mm/migrate.c | 7 +++---
> mm/vmscan.c | 6 ++---
> 7 files changed, 57 insertions(+), 21 deletions(-)
>
This looks like a nice cleanup and can be picked up separately.
Acked-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-11-10 3:37 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-10 3:29 [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection Yafang Shao
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 1/4] sched: add helpers for numa balancing Yafang Shao
2026-11-10 3:37 ` Zi Yan
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 2/4] sched: sample NUMA balancing mode once per path Yafang Shao
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 3/4] sched: add BPF-based per-cgroup NUMA balancing mode selection Yafang Shao
2026-11-10 3:29 ` [RFC PATCH bpf-next v4 4/4] selftests/bpf: add tests for the numab kfuncs Yafang Shao
2026-11-10 3:35 ` [RFC PATCH bpf-next v4 0/4] BPF-based per-cgroup NUMA balancing mode selection Zi Yan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®