[PATCH 0/3] mm/mempolicy: add per-cgroup weighted interleave controls

[PATCH 0/3] mm/mempolicy: add per-cgroup weighted interleave controls

[Li Zhe]

MPOL_WEIGHTED_INTERLEAVE currently has one system-wide weight table,
configured through sysfs or auto-derived from node bandwidth. This series
adds an optional BPF control plane that lets privileged BPF syscall
programs install manual weighted-interleave overrides for individual
cgroups. The per-cgroup override is supported only for the default
cgroup-v2 hierarchy.

Different workloads may prefer different DRAM/CXL memory ratios depending
on latency and bandwidth sensitivity. Per-cgroup weights let an operator
tune co-located jobs independently while leaving the system-wide default
unchanged.

There is no new user-visible Kconfig option. The BPF control plane is
compiled behind a hidden derived CONFIG_BPF_WEIGHTED_INTERLEAVE symbol
when the required NUMA, cgroup, BPF syscall and BTF prerequisites are
present. The existing weighted-interleave mempolicy remains
unconditional, as it is today.

Each cgroup stores only an explicit weight override. Cgroups without
their own weights inherit the nearest ancestor override and otherwise
fall back to the system-wide weighted-interleave weights. No per-cgroup
weight table is allocated when a cgroup is created. The first BPF update
for a cgroup materializes a complete local weight table by copying the
currently effective parent or global weights. Nodes not named by that
first update become cgroup-local values and do not continue to track
ancestor changes. This keeps cgroup creation unchanged and keeps the
common no-override case behind a static key.

For simplicity, this series keeps the existing per-task
weighted-interleave cursor behavior. A task still caches the current
node and remaining dynamic weight quota. Weight changes and cgroup
migration are observed when that cached quota is refilled; the hot
allocation path does not revalidate a partially consumed quota on every
allocation.

Readers use the same lifetime model as the upstream weighted-interleave
table. Normal allocation paths select the effective table under RCU.
The bulk allocation path may sleep in the page allocator, so it first
enters wi_srcu, briefly uses RCU to choose the current task/cgroup
effective table, and then keeps wi_srcu held while walking the table
through bulk allocation. Replaced cgroup and global tables are retired
with wi_state_retire(), preserving both the SRCU sleeping-reader and RCU
non-sleeping-reader grace periods used by the upstream code.

This patchset provides three BPF kfuncs: bpf_wi_set_weight() creates or
updates a cgroup weight override, bpf_wi_clear_weights() removes it, and
bpf_wi_get_weight() returns the effective weight. All three kfuncs
require host CAP_SYS_ADMIN.

The cgid argument must name a cgroup from the default cgroup-v2
hierarchy. Overrides are cgroup kernel state: they persist after the BPF
program that installed them exits and are removed by
bpf_wi_clear_weights() or by cgroup deletion.

The existing sysfs interface remains global. It still shows and updates
only the system-wide weighted-interleave state; cgroup overrides affect
new allocations by tasks in the target cgroup and descendants that do
not have a nearer override.

Li Zhe (3):
  mm/mempolicy: factor weighted interleave state helpers
  mm/mempolicy: add cgroup weighted interleave plumbing
  mm/mempolicy: add BPF kfuncs for cgroup interleave weights

 include/linux/cgroup-defs.h |   5 +
 include/linux/mempolicy.h   |  10 +
 kernel/cgroup/cgroup.c      |   2 +
 mm/Kconfig                  |   6 +
 mm/mempolicy.c              | 357 ++++++++++++++++++++++++++++++++----
 5 files changed, 348 insertions(+), 32 deletions(-)

-- 
2.20.1

[PATCH 1/3] mm/mempolicy: factor weighted interleave state helpers

[Li Zhe]

Weighted interleave updates repeat the same allocation, default table
initialization, table copy and default table setup patterns in the sysfs
and auto-weighting paths.

Factor those pieces into small helpers so later changes can reuse one
state lifecycle for both the global table and cgroup-local override
tables. Keep the existing upstream wi_srcu retirement path unchanged.

No functional change is intended.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 mm/mempolicy.c | 54 ++++++++++++++++++++++++++++++--------------------
 1 file changed, 32 insertions(+), 22 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 0884c629670d..fb9962302664 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -188,6 +188,30 @@ static void wi_state_retire(struct weighted_interleave_state *state)
 		call_srcu(&wi_srcu, &state->rcu, wi_state_free_rcu);
 }
 
+static struct weighted_interleave_state *wi_state_alloc(void)
+{
+	struct weighted_interleave_state *state;
+
+	return kzalloc_flex(*state, iw_table, nr_node_ids, GFP_KERNEL);
+}
+
+static void wi_state_init(struct weighted_interleave_state *state)
+{
+	for (int i = 0; i < nr_node_ids; i++)
+		state->iw_table[i] = 1;
+}
+
+#if defined(CONFIG_SYSFS) || defined(CONFIG_BPF_WEIGHTED_INTERLEAVE)
+static void wi_state_init_from(struct weighted_interleave_state *dst,
+			       struct weighted_interleave_state *src)
+{
+	if (src)
+		memcpy(dst->iw_table, src->iw_table, nr_node_ids * sizeof(u8));
+	else
+		wi_state_init(dst);
+}
+#endif
+
 static u8 get_il_weight(int node)
 {
 	struct weighted_interleave_state *state;
@@ -243,22 +267,17 @@ int mempolicy_set_node_perf(unsigned int node, struct access_coordinate *coords)
 	struct weighted_interleave_state *new_wi_state, *old_wi_state = NULL;
 	unsigned int *old_bw, *new_bw;
 	unsigned int bw_val;
-	int i;
 
 	bw_val = min(coords->read_bandwidth, coords->write_bandwidth);
 	new_bw = kcalloc(nr_node_ids, sizeof(unsigned int), GFP_KERNEL);
 	if (!new_bw)
 		return -ENOMEM;
 
-	new_wi_state = kmalloc_flex(*new_wi_state, iw_table, nr_node_ids);
+	new_wi_state = wi_state_alloc();
 	if (!new_wi_state) {
 		kfree(new_bw);
 		return -ENOMEM;
 	}
-	new_wi_state->mode_auto = true;
-	for (i = 0; i < nr_node_ids; i++)
-		new_wi_state->iw_table[i] = 1;
-
 	/*
 	 * Update bandwidth info, even in manual mode. That way, when switching
 	 * to auto mode in the future, iw_table can be overwritten using
@@ -282,6 +301,8 @@ int mempolicy_set_node_perf(unsigned int node, struct access_coordinate *coords)
 	}
 
 	/* NULL wi_state assumes auto=true; reduce weights and update wi_state*/
+	wi_state_init(new_wi_state);
+	new_wi_state->mode_auto = true;
 	reduce_interleave_weights(new_bw, new_wi_state->iw_table);
 	rcu_assign_pointer(wi_state, new_wi_state);
 
@@ -3672,27 +3693,20 @@ static ssize_t node_store(struct kobject *kobj, struct kobj_attribute *attr,
 	struct weighted_interleave_state *new_wi_state, *old_wi_state = NULL;
 	struct iw_node_attr *node_attr;
 	u8 weight = 0;
-	int i;
 
 	node_attr = container_of(attr, struct iw_node_attr, kobj_attr);
 	if (count == 0 || sysfs_streq(buf, "") ||
 	    kstrtou8(buf, 0, &weight) || weight == 0)
 		return -EINVAL;
 
-	new_wi_state = kzalloc_flex(*new_wi_state, iw_table, nr_node_ids);
+	new_wi_state = wi_state_alloc();
 	if (!new_wi_state)
 		return -ENOMEM;
 
 	mutex_lock(&wi_state_lock);
 	old_wi_state = rcu_dereference_protected(wi_state,
 					lockdep_is_held(&wi_state_lock));
-	if (old_wi_state) {
-		memcpy(new_wi_state->iw_table, old_wi_state->iw_table,
-					nr_node_ids * sizeof(u8));
-	} else {
-		for (i = 0; i < nr_node_ids; i++)
-			new_wi_state->iw_table[i] = 1;
-	}
+	wi_state_init_from(new_wi_state, old_wi_state);
 	new_wi_state->iw_table[node_attr->nid] = weight;
 	new_wi_state->mode_auto = false;
 
@@ -3723,16 +3737,13 @@ static ssize_t weighted_interleave_auto_store(struct kobject *kobj,
 	struct weighted_interleave_state *new_wi_state, *old_wi_state = NULL;
 	unsigned int *bw;
 	bool input;
-	int i;
 
 	if (kstrtobool(buf, &input))
 		return -EINVAL;
 
-	new_wi_state = kzalloc_flex(*new_wi_state, iw_table, nr_node_ids);
+	new_wi_state = wi_state_alloc();
 	if (!new_wi_state)
 		return -ENOMEM;
-	for (i = 0; i < nr_node_ids; i++)
-		new_wi_state->iw_table[i] = 1;
 
 	mutex_lock(&wi_state_lock);
 	old_wi_state = rcu_dereference_protected(wi_state,
@@ -3745,9 +3756,7 @@ static ssize_t weighted_interleave_auto_store(struct kobject *kobj,
 	}
 
 	if (!input) {
-		if (old_wi_state)
-			memcpy(new_wi_state->iw_table, old_wi_state->iw_table,
-						       nr_node_ids * sizeof(u8));
+		wi_state_init_from(new_wi_state, old_wi_state);
 		goto update_wi_state;
 	}
 
@@ -3758,6 +3767,7 @@ static ssize_t weighted_interleave_auto_store(struct kobject *kobj,
 		return -ENODEV;
 	}
 
+	wi_state_init(new_wi_state);
 	new_wi_state->mode_auto = true;
 	reduce_interleave_weights(bw, new_wi_state->iw_table);
 
-- 
2.20.1

[PATCH 2/3] mm/mempolicy: add cgroup weighted interleave plumbing

[Li Zhe]

Add the kernel-side storage and lookup plumbing for per-cgroup weighted
interleave overrides. Each cgroup stores only an explicit override
table. Cgroups without one inherit the nearest ancestor override and
otherwise use the existing global weighted-interleave state.

The BPF control plane is built through a hidden derived Kconfig symbol,
so it follows the BPF syscall and BTF prerequisites without adding a
user-visible weighted-interleave knob.

Per-task weighted-interleave state continues to cache the current node
and the remaining dynamic weight quota. Weight changes and cgroup
migration are observed when that quota is refilled; the hot allocation
path does not revalidate the cached quota on every allocation.

Bulk weighted-interleave allocations continue to use wi_srcu for
concurrency protection.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/cgroup-defs.h |   5 ++
 include/linux/mempolicy.h   |  10 ++++
 kernel/cgroup/cgroup.c      |   2 +
 mm/Kconfig                  |   6 +++
 mm/mempolicy.c              | 104 ++++++++++++++++++++++++++++++++----
 5 files changed, 117 insertions(+), 10 deletions(-)

diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index cbfe8937647c..2dc1cb7336a5 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -29,6 +29,7 @@ struct cgroup;
 struct cgroup_root;
 struct cgroup_subsys;
 struct cgroup_taskset;
+struct weighted_interleave_state;
 struct kernfs_node;
 struct kernfs_ops;
 struct kernfs_open_file;
@@ -647,6 +648,10 @@ struct cgroup {
 #ifdef CONFIG_EXT_SUB_SCHED
 	struct scx_sched __rcu *scx_sched;
 #endif
+#ifdef CONFIG_BPF_WEIGHTED_INTERLEAVE
+	/* Explicit weighted-interleave override for this cgroup. */
+	struct weighted_interleave_state __rcu *wi_state;
+#endif
 
 	/* All ancestors including self */
 	union {
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index 65c732d440d2..59bfb272da31 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -16,6 +16,7 @@
 #include <linux/pagemap.h>
 #include <uapi/linux/mempolicy.h>
 
+struct cgroup;
 struct mm_struct;
 
 #define NO_INTERLEAVE_INDEX (-1UL)	/* use task il_prev for interleaving */
@@ -304,4 +305,13 @@ static inline bool mpol_is_preferred_many(struct mempolicy *pol)
 }
 
 #endif /* CONFIG_NUMA */
+
+#ifdef CONFIG_BPF_WEIGHTED_INTERLEAVE
+void weighted_interleave_cgroup_free(struct cgroup *cgrp);
+#else
+static inline void weighted_interleave_cgroup_free(struct cgroup *cgrp)
+{
+}
+#endif
+
 #endif
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index a284225f8dc9..705bb2e85571 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -35,6 +35,7 @@
 #include <linux/magic.h>
 #include <linux/mutex.h>
 #include <linux/mount.h>
+#include <linux/mempolicy.h>
 #include <linux/pagemap.h>
 #include <linux/proc_fs.h>
 #include <linux/rcupdate.h>
@@ -5733,6 +5734,7 @@ static void css_free_rwork_fn(struct work_struct *work)
 			cgroup1_pidlist_destroy_all(cgrp);
 		cancel_work_sync(&cgrp->release_agent_work);
 		bpf_cgrp_storage_free(cgrp);
+		weighted_interleave_cgroup_free(cgrp);
 
 		if (cgroup_parent(cgrp)) {
 			/*
diff --git a/mm/Kconfig b/mm/Kconfig
index afff1b25b543..43190060f25f 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1530,6 +1530,12 @@ config ARCH_HAS_PTDUMP
 config PTDUMP
 	bool
 
+config BPF_WEIGHTED_INTERLEAVE
+	def_bool y
+	depends on NUMA
+	depends on CGROUPS
+	depends on BPF_SYSCALL
+	depends on DEBUG_INFO_BTF
 
 source "mm/damon/Kconfig"
 
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index fb9962302664..751a5e90a4e1 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -96,6 +96,7 @@
 #include <linux/interrupt.h>
 #include <linux/init.h>
 #include <linux/compat.h>
+#include <linux/cgroup.h>
 #include <linux/ptrace.h>
 #include <linux/swap.h>
 #include <linux/seq_file.h>
@@ -112,6 +113,7 @@
 #include <linux/printk.h>
 #include <linux/leafops.h>
 #include <linux/gcd.h>
+#include <linux/jump_label.h>
 #include <linux/srcu.h>
 
 #include <asm/tlbflush.h>
@@ -170,6 +172,11 @@ static unsigned int *node_bw_table;
  */
 static DEFINE_MUTEX(wi_state_lock);
 
+#ifdef CONFIG_BPF_WEIGHTED_INTERLEAVE
+static DEFINE_STATIC_KEY_FALSE(wi_cgroup_weights_set);
+static DEFINE_MUTEX(wi_cgroup_lock);
+#endif
+
 /* Readers that sleep while walking iw_table hold this instead */
 DEFINE_STATIC_SRCU_FAST(wi_srcu);
 
@@ -212,18 +219,90 @@ static void wi_state_init_from(struct weighted_interleave_state *dst,
 }
 #endif
 
-static u8 get_il_weight(int node)
+static inline u8 wi_state_weight(const struct weighted_interleave_state *state,
+				 int node)
+{
+	return state ? state->iw_table[node] : 1;
+}
+
+#ifdef CONFIG_BPF_WEIGHTED_INTERLEAVE
+static struct weighted_interleave_state *wi_cgroup_state_rcu(struct cgroup *cgrp)
+{
+	struct weighted_interleave_state *state;
+
+	for (; cgrp; cgrp = cgroup_parent(cgrp)) {
+		state = rcu_dereference(cgrp->wi_state);
+		if (state)
+			return state;
+	}
+	return NULL;
+}
+
+static struct weighted_interleave_state *wi_task_state_rcu(void)
+{
+	if (static_branch_unlikely(&wi_cgroup_weights_set) && in_task())
+		return wi_cgroup_state_rcu(task_dfl_cgroup(current));
+	return NULL;
+}
+
+void weighted_interleave_cgroup_free(struct cgroup *cgrp)
+{
+	struct weighted_interleave_state *old_wi_state;
+
+	mutex_lock(&wi_cgroup_lock);
+	old_wi_state = rcu_dereference_protected(cgrp->wi_state,
+						 lockdep_is_held(&wi_cgroup_lock));
+	if (old_wi_state) {
+		rcu_assign_pointer(cgrp->wi_state, NULL);
+		static_branch_dec(&wi_cgroup_weights_set);
+	}
+	mutex_unlock(&wi_cgroup_lock);
+
+	wi_state_retire(old_wi_state);
+}
+#else
+static struct weighted_interleave_state *wi_task_state_rcu(void)
+{
+	return NULL;
+}
+
+#endif
+
+static struct weighted_interleave_state *wi_effective_state_rcu(void)
+{
+	struct weighted_interleave_state *state;
+
+	state = wi_task_state_rcu();
+	if (state)
+		return state;
+	return rcu_dereference(wi_state);
+}
+
+static u8 get_effective_il_weight(int node)
 {
 	struct weighted_interleave_state *state;
-	u8 weight = 1;
+	u8 weight;
+
+	rcu_read_lock();
+	state = wi_effective_state_rcu();
+	weight = wi_state_weight(state, node);
+	rcu_read_unlock();
+	return weight;
+}
+
+#ifdef CONFIG_SYSFS
+static u8 get_global_il_weight(int node)
+{
+	struct weighted_interleave_state *state;
+	u8 weight;
 
 	rcu_read_lock();
 	state = rcu_dereference(wi_state);
-	if (state)
-		weight = state->iw_table[node];
+	weight = wi_state_weight(state, node);
 	rcu_read_unlock();
 	return weight;
 }
+#endif
 
 /*
  * Convert bandwidth values into weighted interleave weights.
@@ -2144,7 +2223,7 @@ static unsigned int weighted_interleave_nodes(struct mempolicy *policy)
 		if (node == MAX_NUMNODES)
 			return node;
 		current->il_prev = node;
-		current->il_weight = get_il_weight(node);
+		current->il_weight = get_effective_il_weight(node);
 	}
 	current->il_weight--;
 	return node;
@@ -2228,8 +2307,8 @@ static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
 
 	rcu_read_lock();
 
-	state = rcu_dereference(wi_state);
-	/* Uninitialized wi_state means we should assume all weights are 1 */
+	state = wi_effective_state_rcu();
+	/* Uninitialized effective state means we should assume all weights are 1 */
 	if (state)
 		table = state->iw_table;
 
@@ -2734,10 +2813,15 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
 	me->il_weight = 0;
 	prev_node = node;
 
-	/* The page allocator may sleep, pin the weight table with SRCU */
+	/*
+	 * The page allocator may sleep. Pin the selected weight table with
+	 * SRCU, but choose the task/cgroup effective table under RCU.
+	 */
 	scp = srcu_read_lock_fast(&wi_srcu);
-	state = srcu_dereference(wi_state, &wi_srcu);
+	rcu_read_lock();
+	state = wi_effective_state_rcu();
 	table = state ? state->iw_table : NULL;
+	rcu_read_unlock();
 
 	/* calculate total, detect system default usage */
 	for_each_node_mask(node, pol->nodes)
@@ -3683,7 +3767,7 @@ static ssize_t node_show(struct kobject *kobj, struct kobj_attribute *attr,
 	u8 weight;
 
 	node_attr = container_of(attr, struct iw_node_attr, kobj_attr);
-	weight = get_il_weight(node_attr->nid);
+	weight = get_global_il_weight(node_attr->nid);
 	return sysfs_emit(buf, "%d\n", weight);
 }
 
-- 
2.20.1

[PATCH 3/3] mm/mempolicy: add BPF kfuncs for cgroup interleave weights

[Li Zhe]

Expose bpf_wi_set_weight(), bpf_wi_clear_weights(), and
bpf_wi_get_weight() as a BPF syscall control plane for cgroup
weighted-interleave overrides. The kfuncs let a privileged program
update one node weight, remove an override, or read the effective
weight.

All three kfuncs are sleepable, require CAP_SYS_ADMIN in the initial
user namespace, and accept only cgroup ids from the default cgroup-v2
hierarchy. Updates materialize a complete weight table and publish it
with RCU.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 mm/mempolicy.c | 201 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 200 insertions(+), 1 deletion(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 751a5e90a4e1..7521542712cb 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -78,6 +78,8 @@
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
 #include <linux/mempolicy.h>
+#include <linux/capability.h>
+#include <linux/bpf.h>
 #include <linux/pagewalk.h>
 #include <linux/highmem.h>
 #include <linux/hugetlb.h>
@@ -103,6 +105,7 @@
 #include <linux/proc_fs.h>
 #include <linux/memory-tiers.h>
 #include <linux/migrate.h>
+#include <linux/module.h>
 #include <linux/ksm.h>
 #include <linux/rmap.h>
 #include <linux/security.h>
@@ -245,7 +248,7 @@ static struct weighted_interleave_state *wi_task_state_rcu(void)
 	return NULL;
 }
 
-void weighted_interleave_cgroup_free(struct cgroup *cgrp)
+static void wi_cgroup_clear_weights(struct cgroup *cgrp)
 {
 	struct weighted_interleave_state *old_wi_state;
 
@@ -260,6 +263,202 @@ void weighted_interleave_cgroup_free(struct cgroup *cgrp)
 
 	wi_state_retire(old_wi_state);
 }
+
+void weighted_interleave_cgroup_free(struct cgroup *cgrp)
+{
+	wi_cgroup_clear_weights(cgrp);
+}
+
+static struct weighted_interleave_state *
+wi_effective_state_cgroup_rcu(struct cgroup *cgrp)
+{
+	struct weighted_interleave_state *state;
+
+	state = wi_cgroup_state_rcu(cgrp);
+	if (state)
+		return state;
+	return rcu_dereference(wi_state);
+}
+
+static void wi_cgroup_set_state(struct cgroup *cgrp,
+				struct weighted_interleave_state *new_wi_state,
+				u32 nid, u8 weight)
+{
+	struct weighted_interleave_state *old_wi_state;
+
+	mutex_lock(&wi_cgroup_lock);
+	old_wi_state = rcu_dereference_protected(cgrp->wi_state,
+						 lockdep_is_held(&wi_cgroup_lock));
+	rcu_read_lock();
+	wi_state_init_from(new_wi_state, wi_effective_state_cgroup_rcu(cgrp));
+	rcu_read_unlock();
+
+	new_wi_state->mode_auto = false;
+	new_wi_state->iw_table[nid] = weight;
+	rcu_assign_pointer(cgrp->wi_state, new_wi_state);
+	if (!old_wi_state)
+		static_branch_inc(&wi_cgroup_weights_set);
+	mutex_unlock(&wi_cgroup_lock);
+
+	wi_state_retire(old_wi_state);
+}
+
+static bool wi_valid_nid(u32 nid)
+{
+	return nid < nr_node_ids && node_state(nid, N_MEMORY);
+}
+
+static int wi_cgroup_get(u64 cgid, struct cgroup **cgrpp)
+{
+	struct cgroup *cgrp;
+
+	cgrp = __cgroup_get_from_id(cgid);
+	if (IS_ERR(cgrp))
+		return PTR_ERR(cgrp);
+
+	if (!cgroup_on_dfl(cgrp)) {
+		cgroup_put(cgrp);
+		return -ENOENT;
+	}
+
+	*cgrpp = cgrp;
+	return 0;
+}
+
+__bpf_kfunc_start_defs();
+
+/**
+ * bpf_wi_set_weight - Set one weighted-interleave node weight for a cgroup.
+ * @cgid: cgroup id that must name a cgroup in default cgroup-v2 hierarchy
+ * @nid: memory NUMA node id to update
+ * @weight: new weight in range [1, 255]
+ *
+ * The first update creates an explicit override for @cgid by copying the
+ * nearest ancestor override, or the system-wide weighted-interleave weights
+ * if no ancestor has one. Nodes not named by the first update become local
+ * values in that table and do not keep tracking ancestor changes. Later
+ * updates modify the cgroup-local table. The override affects new
+ * MPOL_WEIGHTED_INTERLEAVE allocations made by tasks in the cgroup and
+ * descendants that do not have a nearer override.
+ *
+ * Return: 0 on success, -EINVAL for an invalid node or weight, -ENOENT
+ * when @cgid does not name a cgroup in the default cgroup-v2 hierarchy,
+ * -EPERM for insufficient privilege, or -ENOMEM when the override table
+ * cannot be allocated.
+ */
+__bpf_kfunc int bpf_wi_set_weight(u64 cgid, u32 nid, u32 weight)
+{
+	struct weighted_interleave_state *new_wi_state;
+	struct cgroup *cgrp;
+	int ret;
+
+	if (!wi_valid_nid(nid) || weight == 0 || weight > U8_MAX)
+		return -EINVAL;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	new_wi_state = wi_state_alloc();
+	if (!new_wi_state)
+		return -ENOMEM;
+
+	ret = wi_cgroup_get(cgid, &cgrp);
+	if (ret) {
+		kfree(new_wi_state);
+		return ret;
+	}
+
+	wi_cgroup_set_state(cgrp, new_wi_state, nid, weight);
+	cgroup_put(cgrp);
+	return 0;
+}
+
+/**
+ * bpf_wi_clear_weights - Remove a cgroup weighted-interleave override.
+ * @cgid: cgroup id that must name a cgroup in default cgroup-v2 hierarchy
+ *
+ * After clearing, the cgroup inherits the nearest ancestor override, or the
+ * system-wide weighted-interleave weights if no ancestor has an override.
+ *
+ * Return: 0 on success, -ENOENT when @cgid does not name a cgroup in
+ * the default cgroup-v2 hierarchy, or -EPERM for insufficient
+ * privilege.
+ */
+__bpf_kfunc int bpf_wi_clear_weights(u64 cgid)
+{
+	struct cgroup *cgrp;
+	int ret;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	ret = wi_cgroup_get(cgid, &cgrp);
+	if (ret)
+		return ret;
+
+	wi_cgroup_clear_weights(cgrp);
+	cgroup_put(cgrp);
+	return 0;
+}
+
+/**
+ * bpf_wi_get_weight - Read the effective weighted-interleave node weight.
+ * @cgid: cgroup id that must name a cgroup in default cgroup-v2 hierarchy
+ * @nid: memory NUMA node id to query
+ *
+ * Return: the effective weight, -EINVAL for an invalid node, -ENOENT when
+ * @cgid does not name a cgroup in the default cgroup-v2 hierarchy, or
+ * -EPERM for insufficient privilege.
+ */
+__bpf_kfunc s64 bpf_wi_get_weight(u64 cgid, u32 nid)
+{
+	struct weighted_interleave_state *state;
+	struct cgroup *cgrp;
+	s64 weight;
+	int ret;
+
+	if (!wi_valid_nid(nid))
+		return -EINVAL;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	ret = wi_cgroup_get(cgid, &cgrp);
+	if (ret)
+		return ret;
+
+	rcu_read_lock();
+	state = wi_effective_state_cgroup_rcu(cgrp);
+	weight = wi_state_weight(state, nid);
+	rcu_read_unlock();
+
+	cgroup_put(cgrp);
+	return weight;
+}
+
+__bpf_kfunc_end_defs();
+
+__ADDRESSABLE(bpf_wi_set_weight)
+__ADDRESSABLE(bpf_wi_clear_weights)
+__ADDRESSABLE(bpf_wi_get_weight)
+
+BTF_KFUNCS_START(wi_kfunc_ids)
+BTF_ID_FLAGS(func, bpf_wi_set_weight, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_wi_clear_weights, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_wi_get_weight, KF_SLEEPABLE)
+BTF_KFUNCS_END(wi_kfunc_ids)
+
+static const struct btf_kfunc_id_set wi_kfunc_set = {
+	.owner = THIS_MODULE,
+	.set = &wi_kfunc_ids,
+};
+
+static int __init wi_kfunc_init(void)
+{
+	return register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL,
+					 &wi_kfunc_set);
+}
+late_initcall(wi_kfunc_init);
 #else
 static struct weighted_interleave_state *wi_task_state_rcu(void)
 {
-- 
2.20.1

Re: [PATCH 0/3] mm/mempolicy: add per-cgroup weighted interleave controls

[Tejun Heo]

Hello, Zhe.

On Tue, Nov 03, 2026 at 08:28:13PM +0800, Li Zhe wrote:
> MPOL_WEIGHTED_INTERLEAVE currently has one system-wide weight table,
> configured through sysfs or auto-derived from node bandwidth. This series
> adds an optional BPF control plane that lets privileged BPF syscall
> programs install manual weighted-interleave overrides for individual
> cgroups.
...

I don't think this has much to do with cgroups. Weighted interleave is
already a per-thread/per-VMA policy chosen through set_mempolicy() and
mbind(), and the weights are parameters of that policy. I think this
should be tied to the memory policy instead. This is different from
the per-cgroup NUMA balancing kfuncs. NUMA balancing has no per-task
opt-in and acts on memory with no task context.

Gregory Price's mempolicy2 series already proposed set_mempolicy2() and
mbind2() with task-local weights:

  https://lore.kernel.org/all/20240103224209.2541-1-gregory.price@memverge.com/

Thanks.

--
tejun