[PATCH 0/3] mm/mempolicy: add per-cgroup weighted interleave controls

[PATCH 0/3] mm/mempolicy: add per-cgroup weighted interleave controls

[Li Zhe]

MPOL_WEIGHTED_INTERLEAVE currently has one system-wide weight table,
configured through sysfs or auto-derived from node bandwidth. This series
adds an optional BPF control plane that lets privileged BPF syscall
programs install manual weighted-interleave overrides for individual
cgroups. The per-cgroup override is supported only for the default
cgroup-v2 hierarchy.

Different workloads may prefer different DRAM/CXL memory ratios depending
on latency and bandwidth sensitivity. Per-cgroup weights let an operator
tune co-located jobs independently while leaving the system-wide default
unchanged.

There is no new user-visible Kconfig option. The BPF control plane is
compiled behind a hidden derived CONFIG_BPF_WEIGHTED_INTERLEAVE symbol
when the required NUMA, cgroup, BPF syscall and BTF prerequisites are
present. The existing weighted-interleave mempolicy remains
unconditional, as it is today.

Each cgroup stores only an explicit weight override. Cgroups without
their own weights inherit the nearest ancestor override and otherwise
fall back to the system-wide weighted-interleave weights. No per-cgroup
weight table is allocated when a cgroup is created. The first BPF update
for a cgroup materializes a complete local weight table by copying the
currently effective parent or global weights. Nodes not named by that
first update become cgroup-local values and do not continue to track
ancestor changes. This keeps cgroup creation unchanged and keeps the
common no-override case behind a static key.

For simplicity, this series keeps the existing per-task
weighted-interleave cursor behavior. A task still caches the current
node and remaining dynamic weight quota. Weight changes and cgroup
migration are observed when that cached quota is refilled; the hot
allocation path does not revalidate a partially consumed quota on every
allocation.

Readers use the same lifetime model as the upstream weighted-interleave
table. Normal allocation paths select the effective table under RCU.
The bulk allocation path may sleep in the page allocator, so it first
enters wi_srcu, briefly uses RCU to choose the current task/cgroup
effective table, and then keeps wi_srcu held while walking the table
through bulk allocation. Replaced cgroup and global tables are retired
with wi_state_retire(), preserving both the SRCU sleeping-reader and RCU
non-sleeping-reader grace periods used by the upstream code.

This patchset provides three BPF kfuncs: bpf_wi_set_weight() creates or
updates a cgroup weight override, bpf_wi_clear_weights() removes it, and
bpf_wi_get_weight() returns the effective weight. All three kfuncs
require host CAP_SYS_ADMIN.

The cgid argument must name a cgroup from the default cgroup-v2
hierarchy. Overrides are cgroup kernel state: they persist after the BPF
program that installed them exits and are removed by
bpf_wi_clear_weights() or by cgroup deletion.

The existing sysfs interface remains global. It still shows and updates
only the system-wide weighted-interleave state; cgroup overrides affect
new allocations by tasks in the target cgroup and descendants that do
not have a nearer override.

Li Zhe (3):
  mm/mempolicy: factor weighted interleave state helpers
  mm/mempolicy: add cgroup weighted interleave plumbing
  mm/mempolicy: add BPF kfuncs for cgroup interleave weights

 include/linux/cgroup-defs.h |   5 +
 include/linux/mempolicy.h   |  10 +
 kernel/cgroup/cgroup.c      |   2 +
 mm/Kconfig                  |   6 +
 mm/mempolicy.c              | 357 ++++++++++++++++++++++++++++++++----
 5 files changed, 348 insertions(+), 32 deletions(-)

-- 
2.20.1

[PATCH 1/3] mm/mempolicy: factor weighted interleave state helpers

[Li Zhe]

Weighted interleave updates repeat the same allocation, default table
initialization, table copy and default table setup patterns in the sysfs
and auto-weighting paths.

Factor those pieces into small helpers so later changes can reuse one
state lifecycle for both the global table and cgroup-local override
tables. Keep the existing upstream wi_srcu retirement path unchanged.

No functional change is intended.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 mm/mempolicy.c | 54 ++++++++++++++++++++++++++++++--------------------
 1 file changed, 32 insertions(+), 22 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 0884c629670d..fb9962302664 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -188,6 +188,30 @@ static void wi_state_retire(struct weighted_interleave_state *state)
 		call_srcu(&wi_srcu, &state->rcu, wi_state_free_rcu);
 }
 
+static struct weighted_interleave_state *wi_state_alloc(void)
+{
+	struct weighted_interleave_state *state;
+
+	return kzalloc_flex(*state, iw_table, nr_node_ids, GFP_KERNEL);
+}
+
+static void wi_state_init(struct weighted_interleave_state *state)
+{
+	for (int i = 0; i < nr_node_ids; i++)
+		state->iw_table[i] = 1;
+}
+
+#if defined(CONFIG_SYSFS) || defined(CONFIG_BPF_WEIGHTED_INTERLEAVE)
+static void wi_state_init_from(struct weighted_interleave_state *dst,
+			       struct weighted_interleave_state *src)
+{
+	if (src)
+		memcpy(dst->iw_table, src->iw_table, nr_node_ids * sizeof(u8));
+	else
+		wi_state_init(dst);
+}
+#endif
+
 static u8 get_il_weight(int node)
 {
 	struct weighted_interleave_state *state;
@@ -243,22 +267,17 @@ int mempolicy_set_node_perf(unsigned int node, struct access_coordinate *coords)
 	struct weighted_interleave_state *new_wi_state, *old_wi_state = NULL;
 	unsigned int *old_bw, *new_bw;
 	unsigned int bw_val;
-	int i;
 
 	bw_val = min(coords->read_bandwidth, coords->write_bandwidth);
 	new_bw = kcalloc(nr_node_ids, sizeof(unsigned int), GFP_KERNEL);
 	if (!new_bw)
 		return -ENOMEM;
 
-	new_wi_state = kmalloc_flex(*new_wi_state, iw_table, nr_node_ids);
+	new_wi_state = wi_state_alloc();
 	if (!new_wi_state) {
 		kfree(new_bw);
 		return -ENOMEM;
 	}
-	new_wi_state->mode_auto = true;
-	for (i = 0; i < nr_node_ids; i++)
-		new_wi_state->iw_table[i] = 1;
-
 	/*
 	 * Update bandwidth info, even in manual mode. That way, when switching
 	 * to auto mode in the future, iw_table can be overwritten using
@@ -282,6 +301,8 @@ int mempolicy_set_node_perf(unsigned int node, struct access_coordinate *coords)
 	}
 
 	/* NULL wi_state assumes auto=true; reduce weights and update wi_state*/
+	wi_state_init(new_wi_state);
+	new_wi_state->mode_auto = true;
 	reduce_interleave_weights(new_bw, new_wi_state->iw_table);
 	rcu_assign_pointer(wi_state, new_wi_state);
 
@@ -3672,27 +3693,20 @@ static ssize_t node_store(struct kobject *kobj, struct kobj_attribute *attr,
 	struct weighted_interleave_state *new_wi_state, *old_wi_state = NULL;
 	struct iw_node_attr *node_attr;
 	u8 weight = 0;
-	int i;
 
 	node_attr = container_of(attr, struct iw_node_attr, kobj_attr);
 	if (count == 0 || sysfs_streq(buf, "") ||
 	    kstrtou8(buf, 0, &weight) || weight == 0)
 		return -EINVAL;
 
-	new_wi_state = kzalloc_flex(*new_wi_state, iw_table, nr_node_ids);
+	new_wi_state = wi_state_alloc();
 	if (!new_wi_state)
 		return -ENOMEM;
 
 	mutex_lock(&wi_state_lock);
 	old_wi_state = rcu_dereference_protected(wi_state,
 					lockdep_is_held(&wi_state_lock));
-	if (old_wi_state) {
-		memcpy(new_wi_state->iw_table, old_wi_state->iw_table,
-					nr_node_ids * sizeof(u8));
-	} else {
-		for (i = 0; i < nr_node_ids; i++)
-			new_wi_state->iw_table[i] = 1;
-	}
+	wi_state_init_from(new_wi_state, old_wi_state);
 	new_wi_state->iw_table[node_attr->nid] = weight;
 	new_wi_state->mode_auto = false;
 
@@ -3723,16 +3737,13 @@ static ssize_t weighted_interleave_auto_store(struct kobject *kobj,
 	struct weighted_interleave_state *new_wi_state, *old_wi_state = NULL;
 	unsigned int *bw;
 	bool input;
-	int i;
 
 	if (kstrtobool(buf, &input))
 		return -EINVAL;
 
-	new_wi_state = kzalloc_flex(*new_wi_state, iw_table, nr_node_ids);
+	new_wi_state = wi_state_alloc();
 	if (!new_wi_state)
 		return -ENOMEM;
-	for (i = 0; i < nr_node_ids; i++)
-		new_wi_state->iw_table[i] = 1;
 
 	mutex_lock(&wi_state_lock);
 	old_wi_state = rcu_dereference_protected(wi_state,
@@ -3745,9 +3756,7 @@ static ssize_t weighted_interleave_auto_store(struct kobject *kobj,
 	}
 
 	if (!input) {
-		if (old_wi_state)
-			memcpy(new_wi_state->iw_table, old_wi_state->iw_table,
-						       nr_node_ids * sizeof(u8));
+		wi_state_init_from(new_wi_state, old_wi_state);
 		goto update_wi_state;
 	}
 
@@ -3758,6 +3767,7 @@ static ssize_t weighted_interleave_auto_store(struct kobject *kobj,
 		return -ENODEV;
 	}
 
+	wi_state_init(new_wi_state);
 	new_wi_state->mode_auto = true;
 	reduce_interleave_weights(bw, new_wi_state->iw_table);
 
-- 
2.20.1

[PATCH 2/3] mm/mempolicy: add cgroup weighted interleave plumbing

[Li Zhe]

Add the kernel-side storage and lookup plumbing for per-cgroup weighted
interleave overrides. Each cgroup stores only an explicit override
table. Cgroups without one inherit the nearest ancestor override and
otherwise use the existing global weighted-interleave state.

The BPF control plane is built through a hidden derived Kconfig symbol,
so it follows the BPF syscall and BTF prerequisites without adding a
user-visible weighted-interleave knob.

Per-task weighted-interleave state continues to cache the current node
and the remaining dynamic weight quota. Weight changes and cgroup
migration are observed when that quota is refilled; the hot allocation
path does not revalidate the cached quota on every allocation.

Bulk weighted-interleave allocations continue to use wi_srcu for
concurrency protection.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/cgroup-defs.h |   5 ++
 include/linux/mempolicy.h   |  10 ++++
 kernel/cgroup/cgroup.c      |   2 +
 mm/Kconfig                  |   6 +++
 mm/mempolicy.c              | 104 ++++++++++++++++++++++++++++++++----
 5 files changed, 117 insertions(+), 10 deletions(-)

diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index cbfe8937647c..2dc1cb7336a5 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -29,6 +29,7 @@ struct cgroup;
 struct cgroup_root;
 struct cgroup_subsys;
 struct cgroup_taskset;
+struct weighted_interleave_state;
 struct kernfs_node;
 struct kernfs_ops;
 struct kernfs_open_file;
@@ -647,6 +648,10 @@ struct cgroup {
 #ifdef CONFIG_EXT_SUB_SCHED
 	struct scx_sched __rcu *scx_sched;
 #endif
+#ifdef CONFIG_BPF_WEIGHTED_INTERLEAVE
+	/* Explicit weighted-interleave override for this cgroup. */
+	struct weighted_interleave_state __rcu *wi_state;
+#endif
 
 	/* All ancestors including self */
 	union {
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index 65c732d440d2..59bfb272da31 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -16,6 +16,7 @@
 #include <linux/pagemap.h>
 #include <uapi/linux/mempolicy.h>
 
+struct cgroup;
 struct mm_struct;
 
 #define NO_INTERLEAVE_INDEX (-1UL)	/* use task il_prev for interleaving */
@@ -304,4 +305,13 @@ static inline bool mpol_is_preferred_many(struct mempolicy *pol)
 }
 
 #endif /* CONFIG_NUMA */
+
+#ifdef CONFIG_BPF_WEIGHTED_INTERLEAVE
+void weighted_interleave_cgroup_free(struct cgroup *cgrp);
+#else
+static inline void weighted_interleave_cgroup_free(struct cgroup *cgrp)
+{
+}
+#endif
+
 #endif
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index a284225f8dc9..705bb2e85571 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -35,6 +35,7 @@
 #include <linux/magic.h>
 #include <linux/mutex.h>
 #include <linux/mount.h>
+#include <linux/mempolicy.h>
 #include <linux/pagemap.h>
 #include <linux/proc_fs.h>
 #include <linux/rcupdate.h>
@@ -5733,6 +5734,7 @@ static void css_free_rwork_fn(struct work_struct *work)
 			cgroup1_pidlist_destroy_all(cgrp);
 		cancel_work_sync(&cgrp->release_agent_work);
 		bpf_cgrp_storage_free(cgrp);
+		weighted_interleave_cgroup_free(cgrp);
 
 		if (cgroup_parent(cgrp)) {
 			/*
diff --git a/mm/Kconfig b/mm/Kconfig
index afff1b25b543..43190060f25f 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1530,6 +1530,12 @@ config ARCH_HAS_PTDUMP
 config PTDUMP
 	bool
 
+config BPF_WEIGHTED_INTERLEAVE
+	def_bool y
+	depends on NUMA
+	depends on CGROUPS
+	depends on BPF_SYSCALL
+	depends on DEBUG_INFO_BTF
 
 source "mm/damon/Kconfig"
 
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index fb9962302664..751a5e90a4e1 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -96,6 +96,7 @@
 #include <linux/interrupt.h>
 #include <linux/init.h>
 #include <linux/compat.h>
+#include <linux/cgroup.h>
 #include <linux/ptrace.h>
 #include <linux/swap.h>
 #include <linux/seq_file.h>
@@ -112,6 +113,7 @@
 #include <linux/printk.h>
 #include <linux/leafops.h>
 #include <linux/gcd.h>
+#include <linux/jump_label.h>
 #include <linux/srcu.h>
 
 #include <asm/tlbflush.h>
@@ -170,6 +172,11 @@ static unsigned int *node_bw_table;
  */
 static DEFINE_MUTEX(wi_state_lock);
 
+#ifdef CONFIG_BPF_WEIGHTED_INTERLEAVE
+static DEFINE_STATIC_KEY_FALSE(wi_cgroup_weights_set);
+static DEFINE_MUTEX(wi_cgroup_lock);
+#endif
+
 /* Readers that sleep while walking iw_table hold this instead */
 DEFINE_STATIC_SRCU_FAST(wi_srcu);
 
@@ -212,18 +219,90 @@ static void wi_state_init_from(struct weighted_interleave_state *dst,
 }
 #endif
 
-static u8 get_il_weight(int node)
+static inline u8 wi_state_weight(const struct weighted_interleave_state *state,
+				 int node)
+{
+	return state ? state->iw_table[node] : 1;
+}
+
+#ifdef CONFIG_BPF_WEIGHTED_INTERLEAVE
+static struct weighted_interleave_state *wi_cgroup_state_rcu(struct cgroup *cgrp)
+{
+	struct weighted_interleave_state *state;
+
+	for (; cgrp; cgrp = cgroup_parent(cgrp)) {
+		state = rcu_dereference(cgrp->wi_state);
+		if (state)
+			return state;
+	}
+	return NULL;
+}
+
+static struct weighted_interleave_state *wi_task_state_rcu(void)
+{
+	if (static_branch_unlikely(&wi_cgroup_weights_set) && in_task())
+		return wi_cgroup_state_rcu(task_dfl_cgroup(current));
+	return NULL;
+}
+
+void weighted_interleave_cgroup_free(struct cgroup *cgrp)
+{
+	struct weighted_interleave_state *old_wi_state;
+
+	mutex_lock(&wi_cgroup_lock);
+	old_wi_state = rcu_dereference_protected(cgrp->wi_state,
+						 lockdep_is_held(&wi_cgroup_lock));
+	if (old_wi_state) {
+		rcu_assign_pointer(cgrp->wi_state, NULL);
+		static_branch_dec(&wi_cgroup_weights_set);
+	}
+	mutex_unlock(&wi_cgroup_lock);
+
+	wi_state_retire(old_wi_state);
+}
+#else
+static struct weighted_interleave_state *wi_task_state_rcu(void)
+{
+	return NULL;
+}
+
+#endif
+
+static struct weighted_interleave_state *wi_effective_state_rcu(void)
+{
+	struct weighted_interleave_state *state;
+
+	state = wi_task_state_rcu();
+	if (state)
+		return state;
+	return rcu_dereference(wi_state);
+}
+
+static u8 get_effective_il_weight(int node)
 {
 	struct weighted_interleave_state *state;
-	u8 weight = 1;
+	u8 weight;
+
+	rcu_read_lock();
+	state = wi_effective_state_rcu();
+	weight = wi_state_weight(state, node);
+	rcu_read_unlock();
+	return weight;
+}
+
+#ifdef CONFIG_SYSFS
+static u8 get_global_il_weight(int node)
+{
+	struct weighted_interleave_state *state;
+	u8 weight;
 
 	rcu_read_lock();
 	state = rcu_dereference(wi_state);
-	if (state)
-		weight = state->iw_table[node];
+	weight = wi_state_weight(state, node);
 	rcu_read_unlock();
 	return weight;
 }
+#endif
 
 /*
  * Convert bandwidth values into weighted interleave weights.
@@ -2144,7 +2223,7 @@ static unsigned int weighted_interleave_nodes(struct mempolicy *policy)
 		if (node == MAX_NUMNODES)
 			return node;
 		current->il_prev = node;
-		current->il_weight = get_il_weight(node);
+		current->il_weight = get_effective_il_weight(node);
 	}
 	current->il_weight--;
 	return node;
@@ -2228,8 +2307,8 @@ static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
 
 	rcu_read_lock();
 
-	state = rcu_dereference(wi_state);
-	/* Uninitialized wi_state means we should assume all weights are 1 */
+	state = wi_effective_state_rcu();
+	/* Uninitialized effective state means we should assume all weights are 1 */
 	if (state)
 		table = state->iw_table;
 
@@ -2734,10 +2813,15 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
 	me->il_weight = 0;
 	prev_node = node;
 
-	/* The page allocator may sleep, pin the weight table with SRCU */
+	/*
+	 * The page allocator may sleep. Pin the selected weight table with
+	 * SRCU, but choose the task/cgroup effective table under RCU.
+	 */
 	scp = srcu_read_lock_fast(&wi_srcu);
-	state = srcu_dereference(wi_state, &wi_srcu);
+	rcu_read_lock();
+	state = wi_effective_state_rcu();
 	table = state ? state->iw_table : NULL;
+	rcu_read_unlock();
 
 	/* calculate total, detect system default usage */
 	for_each_node_mask(node, pol->nodes)
@@ -3683,7 +3767,7 @@ static ssize_t node_show(struct kobject *kobj, struct kobj_attribute *attr,
 	u8 weight;
 
 	node_attr = container_of(attr, struct iw_node_attr, kobj_attr);
-	weight = get_il_weight(node_attr->nid);
+	weight = get_global_il_weight(node_attr->nid);
 	return sysfs_emit(buf, "%d\n", weight);
 }
 
-- 
2.20.1

[PATCH 3/3] mm/mempolicy: add BPF kfuncs for cgroup interleave weights

[Li Zhe]

Expose bpf_wi_set_weight(), bpf_wi_clear_weights(), and
bpf_wi_get_weight() as a BPF syscall control plane for cgroup
weighted-interleave overrides. The kfuncs let a privileged program
update one node weight, remove an override, or read the effective
weight.

All three kfuncs are sleepable, require CAP_SYS_ADMIN in the initial
user namespace, and accept only cgroup ids from the default cgroup-v2
hierarchy. Updates materialize a complete weight table and publish it
with RCU.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 mm/mempolicy.c | 201 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 200 insertions(+), 1 deletion(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 751a5e90a4e1..7521542712cb 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -78,6 +78,8 @@
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
 #include <linux/mempolicy.h>
+#include <linux/capability.h>
+#include <linux/bpf.h>
 #include <linux/pagewalk.h>
 #include <linux/highmem.h>
 #include <linux/hugetlb.h>
@@ -103,6 +105,7 @@
 #include <linux/proc_fs.h>
 #include <linux/memory-tiers.h>
 #include <linux/migrate.h>
+#include <linux/module.h>
 #include <linux/ksm.h>
 #include <linux/rmap.h>
 #include <linux/security.h>
@@ -245,7 +248,7 @@ static struct weighted_interleave_state *wi_task_state_rcu(void)
 	return NULL;
 }
 
-void weighted_interleave_cgroup_free(struct cgroup *cgrp)
+static void wi_cgroup_clear_weights(struct cgroup *cgrp)
 {
 	struct weighted_interleave_state *old_wi_state;
 
@@ -260,6 +263,202 @@ void weighted_interleave_cgroup_free(struct cgroup *cgrp)
 
 	wi_state_retire(old_wi_state);
 }
+
+void weighted_interleave_cgroup_free(struct cgroup *cgrp)
+{
+	wi_cgroup_clear_weights(cgrp);
+}
+
+static struct weighted_interleave_state *
+wi_effective_state_cgroup_rcu(struct cgroup *cgrp)
+{
+	struct weighted_interleave_state *state;
+
+	state = wi_cgroup_state_rcu(cgrp);
+	if (state)
+		return state;
+	return rcu_dereference(wi_state);
+}
+
+static void wi_cgroup_set_state(struct cgroup *cgrp,
+				struct weighted_interleave_state *new_wi_state,
+				u32 nid, u8 weight)
+{
+	struct weighted_interleave_state *old_wi_state;
+
+	mutex_lock(&wi_cgroup_lock);
+	old_wi_state = rcu_dereference_protected(cgrp->wi_state,
+						 lockdep_is_held(&wi_cgroup_lock));
+	rcu_read_lock();
+	wi_state_init_from(new_wi_state, wi_effective_state_cgroup_rcu(cgrp));
+	rcu_read_unlock();
+
+	new_wi_state->mode_auto = false;
+	new_wi_state->iw_table[nid] = weight;
+	rcu_assign_pointer(cgrp->wi_state, new_wi_state);
+	if (!old_wi_state)
+		static_branch_inc(&wi_cgroup_weights_set);
+	mutex_unlock(&wi_cgroup_lock);
+
+	wi_state_retire(old_wi_state);
+}
+
+static bool wi_valid_nid(u32 nid)
+{
+	return nid < nr_node_ids && node_state(nid, N_MEMORY);
+}
+
+static int wi_cgroup_get(u64 cgid, struct cgroup **cgrpp)
+{
+	struct cgroup *cgrp;
+
+	cgrp = __cgroup_get_from_id(cgid);
+	if (IS_ERR(cgrp))
+		return PTR_ERR(cgrp);
+
+	if (!cgroup_on_dfl(cgrp)) {
+		cgroup_put(cgrp);
+		return -ENOENT;
+	}
+
+	*cgrpp = cgrp;
+	return 0;
+}
+
+__bpf_kfunc_start_defs();
+
+/**
+ * bpf_wi_set_weight - Set one weighted-interleave node weight for a cgroup.
+ * @cgid: cgroup id that must name a cgroup in default cgroup-v2 hierarchy
+ * @nid: memory NUMA node id to update
+ * @weight: new weight in range [1, 255]
+ *
+ * The first update creates an explicit override for @cgid by copying the
+ * nearest ancestor override, or the system-wide weighted-interleave weights
+ * if no ancestor has one. Nodes not named by the first update become local
+ * values in that table and do not keep tracking ancestor changes. Later
+ * updates modify the cgroup-local table. The override affects new
+ * MPOL_WEIGHTED_INTERLEAVE allocations made by tasks in the cgroup and
+ * descendants that do not have a nearer override.
+ *
+ * Return: 0 on success, -EINVAL for an invalid node or weight, -ENOENT
+ * when @cgid does not name a cgroup in the default cgroup-v2 hierarchy,
+ * -EPERM for insufficient privilege, or -ENOMEM when the override table
+ * cannot be allocated.
+ */
+__bpf_kfunc int bpf_wi_set_weight(u64 cgid, u32 nid, u32 weight)
+{
+	struct weighted_interleave_state *new_wi_state;
+	struct cgroup *cgrp;
+	int ret;
+
+	if (!wi_valid_nid(nid) || weight == 0 || weight > U8_MAX)
+		return -EINVAL;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	new_wi_state = wi_state_alloc();
+	if (!new_wi_state)
+		return -ENOMEM;
+
+	ret = wi_cgroup_get(cgid, &cgrp);
+	if (ret) {
+		kfree(new_wi_state);
+		return ret;
+	}
+
+	wi_cgroup_set_state(cgrp, new_wi_state, nid, weight);
+	cgroup_put(cgrp);
+	return 0;
+}
+
+/**
+ * bpf_wi_clear_weights - Remove a cgroup weighted-interleave override.
+ * @cgid: cgroup id that must name a cgroup in default cgroup-v2 hierarchy
+ *
+ * After clearing, the cgroup inherits the nearest ancestor override, or the
+ * system-wide weighted-interleave weights if no ancestor has an override.
+ *
+ * Return: 0 on success, -ENOENT when @cgid does not name a cgroup in
+ * the default cgroup-v2 hierarchy, or -EPERM for insufficient
+ * privilege.
+ */
+__bpf_kfunc int bpf_wi_clear_weights(u64 cgid)
+{
+	struct cgroup *cgrp;
+	int ret;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	ret = wi_cgroup_get(cgid, &cgrp);
+	if (ret)
+		return ret;
+
+	wi_cgroup_clear_weights(cgrp);
+	cgroup_put(cgrp);
+	return 0;
+}
+
+/**
+ * bpf_wi_get_weight - Read the effective weighted-interleave node weight.
+ * @cgid: cgroup id that must name a cgroup in default cgroup-v2 hierarchy
+ * @nid: memory NUMA node id to query
+ *
+ * Return: the effective weight, -EINVAL for an invalid node, -ENOENT when
+ * @cgid does not name a cgroup in the default cgroup-v2 hierarchy, or
+ * -EPERM for insufficient privilege.
+ */
+__bpf_kfunc s64 bpf_wi_get_weight(u64 cgid, u32 nid)
+{
+	struct weighted_interleave_state *state;
+	struct cgroup *cgrp;
+	s64 weight;
+	int ret;
+
+	if (!wi_valid_nid(nid))
+		return -EINVAL;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	ret = wi_cgroup_get(cgid, &cgrp);
+	if (ret)
+		return ret;
+
+	rcu_read_lock();
+	state = wi_effective_state_cgroup_rcu(cgrp);
+	weight = wi_state_weight(state, nid);
+	rcu_read_unlock();
+
+	cgroup_put(cgrp);
+	return weight;
+}
+
+__bpf_kfunc_end_defs();
+
+__ADDRESSABLE(bpf_wi_set_weight)
+__ADDRESSABLE(bpf_wi_clear_weights)
+__ADDRESSABLE(bpf_wi_get_weight)
+
+BTF_KFUNCS_START(wi_kfunc_ids)
+BTF_ID_FLAGS(func, bpf_wi_set_weight, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_wi_clear_weights, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_wi_get_weight, KF_SLEEPABLE)
+BTF_KFUNCS_END(wi_kfunc_ids)
+
+static const struct btf_kfunc_id_set wi_kfunc_set = {
+	.owner = THIS_MODULE,
+	.set = &wi_kfunc_ids,
+};
+
+static int __init wi_kfunc_init(void)
+{
+	return register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL,
+					 &wi_kfunc_set);
+}
+late_initcall(wi_kfunc_init);
 #else
 static struct weighted_interleave_state *wi_task_state_rcu(void)
 {
-- 
2.20.1

Re: [PATCH 0/3] mm/mempolicy: add per-cgroup weighted interleave controls

[Tejun Heo]

Hello, Zhe.

On Tue, Nov 03, 2026 at 08:28:13PM +0800, Li Zhe wrote:
> MPOL_WEIGHTED_INTERLEAVE currently has one system-wide weight table,
> configured through sysfs or auto-derived from node bandwidth. This series
> adds an optional BPF control plane that lets privileged BPF syscall
> programs install manual weighted-interleave overrides for individual
> cgroups.
...

I don't think this has much to do with cgroups. Weighted interleave is
already a per-thread/per-VMA policy chosen through set_mempolicy() and
mbind(), and the weights are parameters of that policy. I think this
should be tied to the memory policy instead. This is different from
the per-cgroup NUMA balancing kfuncs. NUMA balancing has no per-task
opt-in and acts on memory with no task context.

Gregory Price's mempolicy2 series already proposed set_mempolicy2() and
mbind2() with task-local weights:

  https://lore.kernel.org/all/20240103224209.2541-1-gregory.price@memverge.com/

Thanks.

--
tejun

Re: [PATCH 0/3] mm/mempolicy: add per-cgroup weighted interleave controls

[Li Zhe]

On 11/3/26 11:35 PM, Tejun Heo wrote:
> Hello, Zhe.
>
> On Tue, Nov 03, 2026 at 08:28:13PM +0800, Li Zhe wrote:
>> MPOL_WEIGHTED_INTERLEAVE currently has one system-wide weight table,
>> configured through sysfs or auto-derived from node bandwidth. This series
>> adds an optional BPF control plane that lets privileged BPF syscall
>> programs install manual weighted-interleave overrides for individual
>> cgroups.
> ...
>
> I don't think this has much to do with cgroups. Weighted interleave is
> already a per-thread/per-VMA policy chosen through set_mempolicy() and
> mbind(), and the weights are parameters of that policy. I think this
> should be tied to the memory policy instead. This is different from
> the per-cgroup NUMA balancing kfuncs. NUMA balancing has no per-task
> opt-in and acts on memory with no task context.
>
> Gregory Price's mempolicy2 series already proposed set_mempolicy2() and
> mbind2() with task-local weights:
>
>    https://lore.kernel.org/all/20240103224209.2541-1-gregory.price@memverge.com/


Hi Tejun. Thanks for taking a look.

I agree that MPOL_WEIGHTED_INTERLEAVE itself is selected by the task or
VMA through the existing mempolicy interfaces. This series is not trying
to make cgroups select that policy for tasks. A task still has to opt in
to MPOL_WEIGHTED_INTERLEAVE explicitly.

The part this series tries to make cgroup-scoped is the weight table.
Today that table is already outside struct mempolicy and is configured
as a system-wide administrative parameter. The problem we are trying to
solve is that different co-located workloads may need different DRAM/CXL
ratios while using the same MPOL_WEIGHTED_INTERLEAVE policy.

I have looked at the mempolicy2/mbind2 direction. It would model the
weights as task-local or VMA-local policy parameters, but it also requires
new syscall ABI and a much larger mempolicy UAPI extension. This BPF kfunc
interface was intended as a smaller administrative control plane for the
existing global-weight model: keep the existing mempolicy ABI, keep the
global sysfs table as the default, and allow a privileged manager to
install an override for a cgroup in the default hierarchy.

Does that distinction make the cgroup-scoped override acceptable? If not,
could you suggest a better way to support this kind of administrative
weight override without adding new mempolicy syscalls?

Thanks,
Zhe

>
> Thanks.
>
> --
> tejun

Re: [PATCH 0/3] mm/mempolicy: add per-cgroup weighted interleave controls

[Gregory Price]

On Wed, Nov 04, 2026 at 10:44:46AM +0800, Li Zhe wrote:
> 
> Does that distinction make the cgroup-scoped override acceptable? If not,
> could you suggest a better way to support this kind of administrative
> weight override without adding new mempolicy syscalls?
>

I'm a little confused why cgroup-anything is needed here when you could
just override the weight table / interleave functions.

I don't particularly like the idea of bpf hooks for a single mempolicy
mode as opposed to something more generally useful - and I question
how useful task-mempolicies are in general for tiered memory systems
(they also affect kernel allocation placement, which is not great).

You have many patch proposals recently trying to adjust the mempolicy
components and I think it might be helpful if we could chat a bit
about what big problems you're actually trying to solve.

I get that you want better control over task/task-groups, i think that
can be useful - but I also know that demotion immediately mucks all
this placement logic up (and we can't check vma / task policies at
demotion time) so I question how realisitic the value of these
extensions will be.

~Gregory

Re: [PATCH 0/3] mm/mempolicy: add per-cgroup weighted interleave controls

[Li Zhe]

On 11/4/26 2:42 PM, Gregory Price wrote:
> On Wed, Nov 04, 2026 at 10:44:46AM +0800, Li Zhe wrote:
>> Does that distinction make the cgroup-scoped override acceptable? If not,
>> could you suggest a better way to support this kind of administrative
>> weight override without adding new mempolicy syscalls?
>>
> I'm a little confused why cgroup-anything is needed here when you could
> just override the weight table / interleave functions.
>
> I don't particularly like the idea of bpf hooks for a single mempolicy
> mode as opposed to something more generally useful - and I question
> how useful task-mempolicies are in general for tiered memory systems
> (they also affect kernel allocation placement, which is not great).
>
> You have many patch proposals recently trying to adjust the mempolicy
> components and I think it might be helpful if we could chat a bit
> about what big problems you're actually trying to solve.
>
> I get that you want better control over task/task-groups, i think that
> can be useful - but I also know that demotion immediately mucks all
> this placement logic up (and we can't check vma / task policies at
> demotion time) so I question how realisitic the value of these
> extensions will be.


Thanks for the feedback.

Our target environment is a class of machines with both DRAM and CXL
memory. On these machines, different services run together in different
cgroups. For us, CXL memory has two main values: it increases the total
memory capacity of the machine, and it lowers the memory cost for
workloads that need a large resident memory footprint. DRAM is the
faster but also more expensive tier, so the practical question becomes:
how much DRAM does each workload really need, and how much of the rest
can be placed on CXL memory?

 From our earlier testing, different workloads have different suitable
DRAM/CXL ratios. By "suitable", I do not mean maintaining a fixed ratio
for its own sake. I mean that, after using some CXL memory, the
workload's important metrics, such as throughput, average latency, tail
latency, or other SLOs, are still within an acceptable range.

So the core problem I am trying to solve is: on a mixed DRAM/CXL memory
machine, each workload in each cgroup should be able to run with its own
suitable DRAM/CXL ratio, so that it can stay near its own
performance/cost balance point.

This is why a single system-wide weighted interleave table is not enough
for our use case. Two co-located services may have completely different
suitable DRAM/CXL ratios. The cgroup is also the more natural management
unit for us, because resource isolation, charging, SLO monitoring, and
userspace control logic are all organized by cgroup, not by individual
task.

I understand that Joshua's approach can solve part of this problem.
However, based on the current discussion, that direction still seems to
need more discussion before it can move forward upstream.

Of course, initial allocation is not the whole story. Demotion and
promotion will change the actual page placement over time, so I do not
expect weighted interleave alone to provide long-term placement control.
For long-term adjustment, using a userspace controller with
memory.reclaim is a more natural and convenient way to adjust placement
over time.

The value of weighted interleave here is in the initial allocation
phase. First, for bandwidth-sensitive workloads, it allows DRAM and CXL
bandwidth to be used together from startup, instead of concentrating
allocations on one tier first and relying on later reclaim or demotion to
reshape the distribution. Second, it gives the container scheduler a
more deterministic view of memory capacity, so the scheduler can place
workloads based on the total memory capacity provided by both DRAM and
CXL, rather than treating only DRAM as the schedulable capacity.

In the long term, we want to maintain the target ratio through a dynamic
control loop. Memory tiering promotes hot pages back to DRAM. A
userspace controller observes each cgroup's per-tier memory footprint
and workload metrics, decides whether the current ratio has deviated
from the target, and then uses memory.reclaim to apply reclaim pressure
to the source tier that should be relieved [1].

The target ratio is not meant to be a strict hard limit. The userspace
controller can adjust the policy based on workload metrics: when cost is
the priority, it can keep more cold memory on CXL; when performance or
SLO pressure becomes high, it can temporarily relax the ratio and allow
the workload to use more DRAM. In other words, I am not trying to make
demotion reinterpret task policy or VMA policy. I am trying to adjust
the actual resident distribution at cgroup granularity through a
userspace feedback loop.

This is also the background for my other patchsets. The memory.reclaim
source selector is intended to give the userspace controller a more
precise actuator [1]. In our testing, this is an important capability for
building the feedback loop.

The sched/numa memory tiering patchset came from issues we found while
testing this model [2]. Its goal is to make tiering promotion behavior
adapt better to different scenarios, and to avoid tiering-only feedback
being misused by the normal AutoNUMA task placement heuristic.

I am not attached to the exact API shape in this RFC. If the
per-cgroup weighted interleave BPF kfuncs are too narrow, or if you
think there is a more general control plane, or a better control
granularity, I am happy to adjust the direction.

[1] 
https://lore.kernel.org/all/20261102070440.16262-1-lizhe.67@bytedance.com/
[2] 
https://lore.kernel.org/all/20261028041253.74839-1-lizhe.67@bytedance.com/

Thanks,
Zhe

>
> ~Gregory