[RFC PATCH 0/4] memcg_ext: memcg policy through cgroup-attached struct_ops

[RFC PATCH 0/4] memcg_ext: memcg policy through cgroup-attached struct_ops

[Shakeel Butt]

This is the first series of memcg_ext, proposed at [1].  It adds
bpf_memcg_ops, a struct_ops type through which memory controller policy is
attached to a cgroup.  A charge runs the policies of its cgroup and of
every ancestor, and the kernel combines what they return.  BPF picks
between things the kernel already does; it never does the work itself and
never touches a page counter.

The plan is to grow this one member at a time, and to add a member only
when there is a concrete problem it solves and a measurement showing it
does.  Nothing is added because it might be useful later.  So this first
series adds one member, for one problem.

The problem
===========

try_charge_memcg() calls __mem_cgroup_handle_over_high() before it returns,
which reclaims and can throttle the task.  That happens wherever the charge
happens, so a task holding a kernel lock can be stuck there, and everything
waiting on that lock is stuck behind it.

One concrete scenario which can be resolved by this new feature is the
kernfs notify worker. It delivers notifications with the cgroup2
kernfs_rwsem held for read, and the charge for the delivery allocation goes
to the cgroup that set the watch, usually one already under pressure. So
the worker reclaims while holding the lock, a waiting writer blocks every
later reader, and anything touching cgroupfs stalls for seconds. The patch
proposed in [2] fixes that one path in the kernel only for kernfs_rwsem.
The same path still takes kernfs_supers_rwsem.

One can make an argument that if we know the source of the issue in the
kernel, why not fix it similarly to [2] instead of having a generic
solution? The reason is that it will be an uphill and continuous battle as
the kernel keeps evolving and new sources of lock holders doing allocations
keep coming up. In addition, there will be cases where it might not be
possible to move the allocations out of locks, or where doing so would
make the code really ugly [3].

high_policy() lets a policy say where memory.high should be enforced
instead.  Its one request, BPF_MEMCG_HIGH_DEFER_INLINE, skips the inline
call, leaving the debt to be paid at the return to userspace.

Results
=======

Measured on a kernel without [2], with a policy that marks cgroupfs kernfs
lock holders.  Reproducer at [4].

                                      baseline    policy    upstream fix
  max kernfs_rwsem hold, worker        2.049 s     199 us   not taken
  max kernfs_supers_rwsem hold         2.049 s     4.8 ms   2.056 s
  max kernfs_rwsem write wait          4.096 s     6.2 ms   8.7 ms
  walker passes over /sys/fs/cgroup        352       5711   6700
  churn mkdir+rmdir ops                 13 059    364 832   415 488
  churn max latency                     30.7 s    15.1 ms   12.7 ms

The policy matches the fix on the bystander numbers, and does better on
kernfs_supers_rwsem, which the fix does not help: it drops kernfs_rwsem
from the delivery loop, while the policy stops the worker stalling at all,
so every lock it holds benefits.

The patches
===========

Patch 1 is a build fix.  __cgroup_bpf_query() only works because
CGROUP_TCP_SOCK_OPS is the one struct_ops attach type; a second one breaks
the build and makes a query for type 0 return the wrong thing.

Patch 2 adds the type and its registration, with no members, so nothing is
dispatched and behaviour does not change.  It also moves the attach type
enum out of CONFIG_CGROUP_BPF and fixes the register_bpf_struct_ops() no-op
stub, which never compiled because no caller reached it.

Patch 3 adds high_policy() and wires it into try_charge_memcg().

Patch 4 is the sample.  It tracks the two cgroupfs kernfs locks by hooking
the rwsem primitives rather than the forty-odd functions that take them.

Known open questions
====================

The semantics of memory.high for remote chargers or kernel threads is a
grey area and this series does not aim to resolve that.

Another open question is whether a bound on debt deferral is needed. At
the moment, we think that rather than putting a limit on deferral for
memory.high, it will be better to handle that through an async worker like
memcg->high_work.  We aim to introduce that later, along with the right CPU
accounting for that async work.

There is no prog_tests harness, so the numbers above come from the
reproducer and not from the test suite.

The sample's lock tracking costs about 8% throughput, because it
instruments every rwsem operation on the system.  That is fine for a sample
that has to cover every path, but it argues for a cheaper way to track
locks that cause isolation problems between unrelated workloads.

Future work
===========

Asking the kernel to reclaim on the policy's behalf, so a task that cannot
be throttled still pays. More members, each with a use case: hard-limit
policy, reclaim shaping, dynamic protection.

This builds on "bpf: A common way to attach struct_ops to a cgroup", which
supplies attach, detach, ordering, update, query and the RCU rules.
bpf_memcg_ops is its second user.

[1] https://lore.kernel.org/20260307182424.2889780-1-shakeel.butt@linux.dev/
[2] https://lore.kernel.org/20260910045406.485295-1-shakeel.butt@linux.dev/
[3] https://lore.kernel.org/20260917-wehten-achtfach-getarnt-c85a4812337d@brauner/
[4] https://github.com/shakeelb/mempressure-repros/tree/main/kernfs-notify-memcg

Shakeel Butt (4):
  bpf, cgroup: fix cgroup struct_ops query for a second attach type
  memcg_ext: add cgroup-attached bpf_memcg_ops
  memcg_ext: allow BPF to defer memory.high enforcement
  selftests/bpf: add a cgroupfs lock-holder bpf_memcg_ops sample

 MAINTAINERS                                   |   1 +
 include/linux/bpf-cgroup-defs.h               |  22 +-
 include/linux/bpf-cgroup.h                    |  14 +-
 include/linux/bpf.h                           |   2 +-
 include/linux/bpf_memcontrol.h                |  66 ++++++
 include/linux/cgroup.h                        |   7 +
 include/linux/sched.h                         |   4 +
 kernel/bpf/cgroup.c                           |   3 +-
 mm/bpf_memcontrol.c                           | 188 ++++++++++++++-
 mm/memcontrol.c                               |  31 ++-
 .../bpf/progs/memcg_ops_lockholder.c          | 222 ++++++++++++++++++
 11 files changed, 546 insertions(+), 14 deletions(-)
 create mode 100644 include/linux/bpf_memcontrol.h
 create mode 100644 tools/testing/selftests/bpf/progs/memcg_ops_lockholder.c


base-commit: 6e36e099b15bed1e8e5b3e3136c5e3eb56a15aa7
-- 
2.53.0-Meta

[RFC PATCH 1/4] bpf, cgroup: fix cgroup struct_ops query for a second attach type

[Shakeel Butt]

Two things in __cgroup_bpf_query() work only because CGROUP_TCP_SOCK_OPS is
the only one struct_ops attach type.

It calls cgroup_bpf_enabled(atype) with an atype that
find_atype_by_struct_ops_id() works out at runtime.  That macro is an asm
goto and needs a constant.  Today the compiler can see there is only one
value; add a second type and the build breaks with "impossible constraint in
'asm'".  Add cgroup_bpf_enabled_runtime(), which reads the key instead, and
use it here.  This is a syscall path, so the cost does not matter.

And find_atype_by_struct_ops_id() matches on type_id alone.  An attach type
whose subsystem is not built keeps type_id 0, so a query for type 0 finds it
and returns success with nothing instead of -ENOENT.  Skip such slots.

Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
 include/linux/bpf-cgroup.h | 9 +++++++++
 kernel/bpf/cgroup.c        | 3 ++-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/include/linux/bpf-cgroup.h b/include/linux/bpf-cgroup.h
index 8a75a6cd7309..3b2c127d401d 100644
--- a/include/linux/bpf-cgroup.h
+++ b/include/linux/bpf-cgroup.h
@@ -76,6 +76,14 @@ to_cgroup_bpf_attach_type(enum bpf_attach_type attach_type)
 
 extern struct static_key_false cgroup_bpf_enabled_key[MAX_CGROUP_BPF_ATTACH_TYPE];
 #define cgroup_bpf_enabled(atype) static_branch_unlikely(&cgroup_bpf_enabled_key[atype])
+/*
+ * Same test when @atype is not a constant.  cgroup_bpf_enabled() uses
+ * static_branch_unlikely which creates jump-label site and requires statically
+ * selected key. Since key is selected dynamically, use static_key_enabled here
+ * and keep it off fast paths.
+ */
+#define cgroup_bpf_enabled_runtime(atype) \
+	static_key_enabled(&cgroup_bpf_enabled_key[atype])
 
 struct bpf_cgroup_storage_map;
 
@@ -508,6 +516,7 @@ static inline int cgroup_bpf_struct_ops_attach(struct bpf_map *map,
 }
 
 #define cgroup_bpf_enabled(atype) (0)
+#define cgroup_bpf_enabled_runtime(atype) (0)
 #define BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, atype, t_ctx) ({ 0; })
 #define BPF_CGROUP_RUN_SA_PROG(sk, uaddr, uaddrlen, atype) ({ 0; })
 #define BPF_CGROUP_PRE_CONNECT_ENABLED(sk) (0)
diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c
index 696b27383974..99b4e96f5db7 100644
--- a/kernel/bpf/cgroup.c
+++ b/kernel/bpf/cgroup.c
@@ -45,6 +45,7 @@ static enum cgroup_bpf_attach_type find_atype_by_struct_ops_id(u32 type_id)
 
 	for (atype = 0; atype < MAX_CGROUP_BPF_ATTACH_TYPE; atype++) {
 		if (cgroup_bpf_is_struct_ops_atype(atype) &&
+		    cgroup_struct_ops[atype].type_id &&
 		    cgroup_struct_ops[atype].type_id == type_id)
 			return atype;
 	}
@@ -1448,7 +1449,7 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
 			return -ENOENT;
 		from_atype = to_atype = atype;
 		flags = 0;
-		if (!cgroup_bpf_enabled(atype))
+		if (!cgroup_bpf_enabled_runtime(atype))
 			goto skip_count;
 	} else if (type == BPF_LSM_CGROUP) {
 		if (!effective_query && attr->query.prog_cnt &&
-- 
2.53.0-Meta

[RFC PATCH 2/4] memcg_ext: add cgroup-attached bpf_memcg_ops

[Shakeel Butt]

Add an empty bpf_memcg_ops type for attaching memory policies to
cgroups.  This is the first patch of memcg_ext, which makes memcg
enforcement programmable one hook at a time.  Policy hooks will be added
later.

Move the cgroup BPF attach types outside CONFIG_CGROUP_BPF and fix the
struct_ops registration stub so all supported configurations build.

This does not change behavior yet.

Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
 include/linux/bpf-cgroup-defs.h |  22 ++++---
 include/linux/bpf-cgroup.h      |   2 +-
 include/linux/bpf.h             |   2 +-
 include/linux/bpf_memcontrol.h  |  17 +++++
 mm/bpf_memcontrol.c             | 108 +++++++++++++++++++++++++++++++-
 5 files changed, 140 insertions(+), 11 deletions(-)
 create mode 100644 include/linux/bpf_memcontrol.h

diff --git a/include/linux/bpf-cgroup-defs.h b/include/linux/bpf-cgroup-defs.h
index 0147b8bec973..53d2853535c6 100644
--- a/include/linux/bpf-cgroup-defs.h
+++ b/include/linux/bpf-cgroup-defs.h
@@ -2,14 +2,6 @@
 #ifndef _BPF_CGROUP_DEFS_H
 #define _BPF_CGROUP_DEFS_H
 
-#ifdef CONFIG_CGROUP_BPF
-
-#include <linux/list.h>
-#include <linux/percpu-refcount.h>
-#include <linux/workqueue.h>
-
-struct bpf_prog_array;
-
 #ifdef CONFIG_BPF_LSM
 /* Maximum number of concurrently attachable per-cgroup LSM hooks. */
 #define CGROUP_LSM_NUM 10
@@ -17,6 +9,10 @@ struct bpf_prog_array;
 #define CGROUP_LSM_NUM 0
 #endif
 
+/*
+ * Plain constants, so a subsystem can name its attach type without
+ * depending on CONFIG_CGROUP_BPF.
+ */
 enum cgroup_bpf_attach_type {
 	CGROUP_BPF_ATTACH_TYPE_INVALID = -1,
 	CGROUP_INET_INGRESS = 0,
@@ -48,11 +44,21 @@ enum cgroup_bpf_attach_type {
 	CGROUP_UNIX_GETSOCKNAME,
 	CGROUP_INET_SOCK_RELEASE,
 	CGROUP_TCP_SOCK_OPS,
+	CGROUP_MEMCG_OPS,
 	CGROUP_LSM_START,
 	CGROUP_LSM_END = CGROUP_LSM_START + CGROUP_LSM_NUM - 1,
 	MAX_CGROUP_BPF_ATTACH_TYPE
 };
 
+#ifdef CONFIG_CGROUP_BPF
+
+#include <linux/list.h>
+#include <linux/percpu-refcount.h>
+#include <linux/workqueue.h>
+
+struct bpf_prog_array;
+
+
 struct cgroup_bpf {
 	/* array of effective progs in this cgroup */
 	struct bpf_prog_array __rcu *effective[MAX_CGROUP_BPF_ATTACH_TYPE];
diff --git a/include/linux/bpf-cgroup.h b/include/linux/bpf-cgroup.h
index 3b2c127d401d..4e8150848bd2 100644
--- a/include/linux/bpf-cgroup.h
+++ b/include/linux/bpf-cgroup.h
@@ -127,7 +127,7 @@ struct bpf_prog_list {
 
 static inline bool cgroup_bpf_is_struct_ops_atype(enum cgroup_bpf_attach_type atype)
 {
-	return atype == CGROUP_TCP_SOCK_OPS;
+	return atype == CGROUP_TCP_SOCK_OPS || atype == CGROUP_MEMCG_OPS;
 }
 void cgroup_bpf_struct_ops_register(int atype, u32 type_id, void *cfi_stubs, bool mult_trace);
 int cgroup_bpf_struct_ops_attach(struct bpf_map *map, const union bpf_attr *attr);
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 5033b934ffd9..f8eb102e7fc4 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2331,7 +2331,7 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
 void bpf_map_struct_ops_info_fill(struct bpf_map_info *info, struct bpf_map *map);
 void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_ops_desc);
 #else
-#define register_bpf_struct_ops(st_ops, type) ({ (void *)(st_ops); 0; })
+#define register_bpf_struct_ops(st_ops, type) ({ (void)(st_ops); 0; })
 static inline bool bpf_try_module_get(const void *data, struct module *owner)
 {
 	return try_module_get(owner);
diff --git a/include/linux/bpf_memcontrol.h b/include/linux/bpf_memcontrol.h
new file mode 100644
index 000000000000..8204d894761e
--- /dev/null
+++ b/include/linux/bpf_memcontrol.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * BPF policy hooks for the memory controller.
+ *
+ * A bpf_memcg_ops is attached to a cgroup.  A charge runs the policies of
+ * that cgroup and of every ancestor, and the kernel combines what they
+ * return.  BPF only picks between things the kernel already does.
+ *
+ * The type has no members yet; they come with the policies that use them.
+ */
+#ifndef _LINUX_BPF_MEMCONTROL_H
+#define _LINUX_BPF_MEMCONTROL_H
+
+struct bpf_memcg_ops {
+};
+
+#endif /* _LINUX_BPF_MEMCONTROL_H */
diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c
index d8f579c28560..fd6dff150f01 100644
--- a/mm/bpf_memcontrol.c
+++ b/mm/bpf_memcontrol.c
@@ -7,6 +7,12 @@
 
 #include <linux/memcontrol.h>
 #include <linux/bpf.h>
+#include <linux/bpf-cgroup.h>
+#include <linux/bpf_memcontrol.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf_ids.h>
+#include <linux/cgroup.h>
+#include <linux/sched.h>
 
 #include "internal.h"
 
@@ -235,6 +241,100 @@ static const struct btf_kfunc_id_set bpf_memcontrol_reclaim_kfunc_set = {
 	.set            = &bpf_memcontrol_reclaim_kfuncs,
 };
 
+/*
+ * bpf_memcg_ops: memcg policy attached to a cgroup.  A program returns a
+ * request and the kernel acts on it.  Nothing here reclaims or sleeps.
+ */
+
+/* CFI stubs.  A slot points at these while its policy is being detached. */
+static struct bpf_memcg_ops __bpf_memcg_ops = {
+};
+
+static const struct bpf_func_proto *
+bpf_memcg_get_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
+{
+	/*
+	 * The base set is all a policy needs today, and none of it sleeps.
+	 * Anything added here must be safe from the charge path.
+	 */
+	return bpf_base_func_proto(func_id, prog);
+}
+
+static bool bpf_memcg_is_valid_access(int off, int size,
+				      enum bpf_access_type type,
+				      const struct bpf_prog *prog,
+				      struct bpf_insn_access_aux *info)
+{
+	/* The context is read-only. */
+	if (type != BPF_READ)
+		return false;
+
+	return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
+}
+
+static int bpf_memcg_init_member(const struct btf_type *t,
+				 const struct btf_member *member,
+				 void *kdata, const void *udata)
+{
+	/* Mandatory: the core calls it without a NULL check. */
+	return 0;
+}
+
+static int bpf_memcg_check_member(const struct btf_type *t,
+				  const struct btf_member *member,
+				  const struct bpf_prog *prog)
+{
+	/* Members run from the charge path, which cannot sleep. */
+	if (prog->sleepable)
+		return -EINVAL;
+
+	return 0;
+}
+
+static int bpf_memcg_init(struct btf *btf)
+{
+	return 0;
+}
+
+static int bpf_memcg_validate(void *kdata)
+{
+	return 0;
+}
+
+static const struct bpf_verifier_ops bpf_memcg_verifier_ops = {
+	.get_func_proto		= bpf_memcg_get_func_proto,
+	.is_valid_access	= bpf_memcg_is_valid_access,
+};
+
+static struct bpf_struct_ops bpf_memcg_ops_desc = {
+	.verifier_ops	= &bpf_memcg_verifier_ops,
+	.init		= bpf_memcg_init,
+	.init_member	= bpf_memcg_init_member,
+	.check_member	= bpf_memcg_check_member,
+	.validate	= bpf_memcg_validate,
+	.name		= "bpf_memcg_ops",
+	.cgroup_atype	= CGROUP_MEMCG_OPS,
+	.cfi_stubs	= &__bpf_memcg_ops,
+	.owner		= THIS_MODULE,
+	/*
+	 * .reg/.unreg stay NULL: the cgroup layer does attach and detach, and
+	 * registration fails if a cgroup_atype comes with either.
+	 *
+	 * .free_after_mult_rcu_gp stays false while no member sleeps.  A
+	 * sleepable one would also need a tasks-trace RCU version of
+	 * bpf_cgroup_struct_ops_foreach().
+	 */
+};
+
+static int __init bpf_memcg_ops_register(void)
+{
+	/*
+	 * register_bpf_struct_ops() is a no-op without struct_ops support, so
+	 * this needs no guard of its own.
+	 */
+	return register_bpf_struct_ops(&bpf_memcg_ops_desc, bpf_memcg_ops);
+}
+
 static int __init bpf_memcontrol_init(void)
 {
 	int err;
@@ -248,8 +348,14 @@ static int __init bpf_memcontrol_init(void)
 
 	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL,
 					&bpf_memcontrol_reclaim_kfunc_set);
-	if (err)
+	if (err) {
 		pr_warn("error registering bpf reclaim kfuncs: %d\n", err);
+		return err;
+	}
+
+	err = bpf_memcg_ops_register();
+	if (err)
+		pr_warn("error while registering bpf_memcg_ops: %d", err);
 
 	return err;
 }
-- 
2.53.0-Meta

[RFC PATCH 3/4] memcg_ext: allow BPF to defer memory.high enforcement

[Shakeel Butt]

Before it returns, try_charge_memcg() calls __mem_cgroup_handle_over_high(),
which reclaims and can throttle the task.  That happens wherever the charge
happens, so a task holding a kernel lock can be stuck there, and everything
waiting on the lock is stuck behind it.

Let's add high_policy ops through which a program gets a read-only snapshot
of the charge and returns a request.  There is one so far:
BPF_MEMCG_HIGH_DEFER_INLINE skips the inline call.

A charge runs the policies of its cgroup and of every ancestor.
task_struct::in_bpf_memcg stops a program that allocates from re-entering
the charge path and the dispatcher with it.

A memcg outlives its cgroup while it has charges, and cgroup_bpf_release()
frees the arrays when the cgroup goes. Take the reference for the walk.

Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
 MAINTAINERS                    |  1 +
 include/linux/bpf-cgroup.h     |  3 ++
 include/linux/bpf_memcontrol.h | 53 +++++++++++++++++++++-
 include/linux/cgroup.h         |  7 +++
 include/linux/sched.h          |  4 ++
 mm/bpf_memcontrol.c            | 82 +++++++++++++++++++++++++++++++++-
 mm/memcontrol.c                | 31 ++++++++++++-
 7 files changed, 176 insertions(+), 5 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 6215fcb07770..0c84beab396f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -5032,6 +5032,7 @@ L:	bpf@vger.kernel.org
 L:	linux-mm@kvack.org
 S:	Maintained
 F:	mm/bpf_memcontrol.c
+F:	include/linux/bpf_memcontrol.h
 
 BPF [MISC]
 L:	bpf@vger.kernel.org
diff --git a/include/linux/bpf-cgroup.h b/include/linux/bpf-cgroup.h
index 4e8150848bd2..e19cf83e58f3 100644
--- a/include/linux/bpf-cgroup.h
+++ b/include/linux/bpf-cgroup.h
@@ -517,6 +517,9 @@ static inline int cgroup_bpf_struct_ops_attach(struct bpf_map *map,
 
 #define cgroup_bpf_enabled(atype) (0)
 #define cgroup_bpf_enabled_runtime(atype) (0)
+/* Nothing can be attached, so the walk has nothing to walk. */
+#define bpf_cgroup_struct_ops_foreach(var, item, cgrp, atype)		\
+	for ((void)(cgrp), (item) = NULL, (var) = NULL; 0; )
 #define BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, atype, t_ctx) ({ 0; })
 #define BPF_CGROUP_RUN_SA_PROG(sk, uaddr, uaddrlen, atype) ({ 0; })
 #define BPF_CGROUP_PRE_CONNECT_ENABLED(sk) (0)
diff --git a/include/linux/bpf_memcontrol.h b/include/linux/bpf_memcontrol.h
index 8204d894761e..76ea5d1c3d32 100644
--- a/include/linux/bpf_memcontrol.h
+++ b/include/linux/bpf_memcontrol.h
@@ -5,13 +5,62 @@
  * A bpf_memcg_ops is attached to a cgroup.  A charge runs the policies of
  * that cgroup and of every ancestor, and the kernel combines what they
  * return.  BPF only picks between things the kernel already does.
- *
- * The type has no members yet; they come with the policies that use them.
  */
 #ifndef _LINUX_BPF_MEMCONTROL_H
 #define _LINUX_BPF_MEMCONTROL_H
 
+#include <linux/types.h>
+#include <linux/gfp_types.h>
+
+struct mem_cgroup;
+struct task_struct;
+
+/*
+ * What a policy can ask for when a cgroup is over memory.high.  The kernel
+ * ORs them, so one policy cannot undo another.
+ */
+enum bpf_memcg_high_request {
+	BPF_MEMCG_HIGH_NO_OPINION	= 0,
+	/*
+	 * Skip the inline reclaim and throttle.  The debt is kept and paid on
+	 * the way back to userspace, where no kernel locks are held.
+	 */
+	BPF_MEMCG_HIGH_DEFER_INLINE	= 1U << 0,
+};
+
+#define BPF_MEMCG_HIGH_VALID_MASK	BPF_MEMCG_HIGH_DEFER_INLINE
+
+/* Read-only snapshot.  Only values the caller already has. */
+struct bpf_memcg_ctx {
+	struct mem_cgroup	*memcg;			/* charged memcg */
+	struct mem_cgroup	*memcg_over_limit;	/* NULL if none found */
+	struct task_struct	*task;			/* current */
+	u64			cgroup_id;
+	u64			over_limit_cgroup_id;	/* 0 if none */
+	u64			nr_pages_over_high;
+	u32			gfp_flags;
+};
+
 struct bpf_memcg_ops {
+	/**
+	 * high_policy - say where memory.high should be enforced
+	 * @ctx: snapshot of the charge
+	 *
+	 * Return: bits from enum bpf_memcg_high_request, or 0.  Other bits
+	 * are dropped.
+	 */
+	u32 (*high_policy)(const struct bpf_memcg_ctx *ctx);
 };
 
+/*
+ * Run every high_policy on @memcg's cgroup and its ancestors, and return the
+ * combined request for the caller to act on.
+ *
+ * @memcg:	the memcg being charged, never NULL
+ * @over_limit:	first memcg found over memory.high or swap.high, or NULL
+ * @gfp_mask:	the charge's gfp mask
+ */
+u32 bpf_memcg_high_policy(struct mem_cgroup *memcg,
+			  struct mem_cgroup *over_limit, gfp_t gfp_mask);
+
 #endif /* _LINUX_BPF_MEMCONTROL_H */
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index 5dfa915a630e..cf92b6cec819 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -959,10 +959,17 @@ static inline void cgroup_bpf_put(struct cgroup *cgrp)
 	percpu_ref_put(&cgrp->bpf.refcnt);
 }
 
+/* Fails once the cgroup is gone and its bpf state has been freed. */
+static inline bool cgroup_bpf_tryget_live(struct cgroup *cgrp)
+{
+	return percpu_ref_tryget_live_rcu(&cgrp->bpf.refcnt);
+}
+
 #else /* CONFIG_CGROUP_BPF */
 
 static inline void cgroup_bpf_get(struct cgroup *cgrp) {}
 static inline void cgroup_bpf_put(struct cgroup *cgrp) {}
+static inline bool cgroup_bpf_tryget_live(struct cgroup *cgrp) { return false; }
 
 #endif /* CONFIG_CGROUP_BPF */
 
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 8b3d47a325cc..4b20a346aaa8 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1031,6 +1031,10 @@ struct task_struct {
 #ifdef CONFIG_MEMCG_V1
 	unsigned			in_user_fault:1;
 #endif
+#ifdef CONFIG_MEMCG
+	/* A bpf_memcg_ops program is running; do not recurse into policy */
+	unsigned			in_bpf_memcg:1;
+#endif
 #ifdef CONFIG_LRU_GEN
 	/* whether the LRU algorithm may apply to this access */
 	unsigned			in_lru_fault:1;
diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c
index fd6dff150f01..cfd0f1d443c9 100644
--- a/mm/bpf_memcontrol.c
+++ b/mm/bpf_memcontrol.c
@@ -246,8 +246,17 @@ static const struct btf_kfunc_id_set bpf_memcontrol_reclaim_kfunc_set = {
  * request and the kernel acts on it.  Nothing here reclaims or sleeps.
  */
 
-/* CFI stubs.  A slot points at these while its policy is being detached. */
+/*
+ * CFI stubs.  These really run: a slot points at them while its policy is
+ * being detached.  Return 0, the identity for the kernel's OR.
+ */
+static u32 high_policy_stub(const struct bpf_memcg_ctx *ctx)
+{
+	return BPF_MEMCG_HIGH_NO_OPINION;
+}
+
 static struct bpf_memcg_ops __bpf_memcg_ops = {
+	.high_policy = high_policy_stub,
 };
 
 static const struct bpf_func_proto *
@@ -326,6 +335,77 @@ static struct bpf_struct_ops bpf_memcg_ops_desc = {
 	 */
 };
 
+static void bpf_memcg_ctx_init(struct bpf_memcg_ctx *ctx,
+			       struct mem_cgroup *memcg,
+			       struct mem_cgroup *over_limit, gfp_t gfp_mask)
+{
+	ctx->memcg = memcg;
+	ctx->memcg_over_limit = over_limit;
+	ctx->task = current;
+	ctx->cgroup_id = cgroup_id(memcg->css.cgroup);
+	ctx->over_limit_cgroup_id = over_limit ?
+		cgroup_id(over_limit->css.cgroup) : 0;
+	ctx->nr_pages_over_high = current->memcg_nr_pages_over_high;
+	ctx->gfp_flags = (__force u32)gfp_mask;
+}
+
+u32 bpf_memcg_high_policy(struct mem_cgroup *memcg,
+			  struct mem_cgroup *over_limit, gfp_t gfp_mask)
+{
+	const struct bpf_prog_array_item *item;
+	const struct bpf_memcg_ops *ops;
+	struct bpf_memcg_ctx ctx;
+	u32 acc = BPF_MEMCG_HIGH_NO_OPINION;
+	struct cgroup *cgrp;
+
+	if (!cgroup_bpf_enabled(CGROUP_MEMCG_OPS))
+		return acc;
+
+	/*
+	 * Only the default hierarchy has a cgroup_bpf, and the static key is
+	 * global, so one policy anywhere turns this on for v1 memcgs too.  A
+	 * v1 memcg still cannot get here, because memory.high and swap.high
+	 * are both v2-only and so it never builds the debt that leads to this
+	 * call.  A hook on a path v1 can reach needs its own cgroup_on_dfl()
+	 * test: a v1 cgroup has no effective array and an uninitialised
+	 * cgrp->bpf.refcnt.
+	 */
+	cgrp = memcg->css.cgroup;
+
+	/*
+	 * A program can allocate and re-enter the charge path.  Skip the
+	 * nested call.  This guards the callbacks only.
+	 */
+	if (current->in_bpf_memcg)
+		return acc;
+	current->in_bpf_memcg = 1;
+
+	rcu_read_lock_dont_migrate();
+
+	/*
+	 * A memcg outlives its cgroup while it has charges, and
+	 * cgroup_bpf_release() frees the arrays when the cgroup goes.
+	 */
+	if (!cgroup_bpf_tryget_live(cgrp))
+		goto out;
+
+	bpf_memcg_ctx_init(&ctx, memcg, over_limit, gfp_mask);
+
+	bpf_cgroup_struct_ops_foreach(ops, item, cgrp, CGROUP_MEMCG_OPS) {
+		if (ops->high_policy)
+			acc |= ops->high_policy(&ctx) &
+			       BPF_MEMCG_HIGH_VALID_MASK;
+	}
+
+	cgroup_bpf_put(cgrp);
+out:
+	rcu_read_unlock_migrate();
+
+	current->in_bpf_memcg = 0;
+
+	return acc;
+}
+
 static int __init bpf_memcg_ops_register(void)
 {
 	/*
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1271d390b617..bc283680640b 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -28,6 +28,7 @@
 #include <linux/cgroup-defs.h>
 #include <linux/page_counter.h>
 #include <linux/memcontrol.h>
+#include <linux/bpf_memcontrol.h>
 #include <linux/cgroup.h>
 #include <linux/cpuset.h>
 #include <linux/sched/mm.h>
@@ -2642,9 +2643,26 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
 	css_put(&memcg->css);
 }
 
+/*
+ * Ask the attached bpf_memcg_ops whether to skip the inline memory.high
+ * reclaim and throttle.
+ *
+ * @memcg:	the memcg being charged
+ * @over_limit:	first memcg found over memory.high or swap.high, starting at
+ *		the charged one, or NULL if the walk found none
+ */
+static bool bpf_memcg_high_defer(struct mem_cgroup *memcg,
+				 struct mem_cgroup *over_limit, gfp_t gfp_mask)
+{
+	u32 req = bpf_memcg_high_policy(memcg, over_limit, gfp_mask);
+
+	return req & BPF_MEMCG_HIGH_DEFER_INLINE;
+}
+
 static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 			    unsigned int nr_pages)
 {
+	struct mem_cgroup *leaf_memcg = memcg;
 	unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages);
 	int nr_retries = MAX_RECLAIM_RETRIES;
 	struct mem_cgroup *mem_over_limit;
@@ -2846,8 +2864,17 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	 */
 	if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH &&
 	    !(current->flags & PF_MEMALLOC) &&
-	    gfpflags_allow_blocking(gfp_mask))
-		__mem_cgroup_handle_over_high(gfp_mask);
+	    gfpflags_allow_blocking(gfp_mask)) {
+		/*
+		 * The loop above left @memcg as the first memcg it found over
+		 * memory.high or swap.high -- possibly the charged one itself
+		 * -- or NULL if it found none.  Note the debt can be left over
+		 * from an earlier charge, so NULL does not mean no pressure.
+		 * The policy wants the memcg we charged.
+		 */
+		if (!bpf_memcg_high_defer(leaf_memcg, memcg, gfp_mask))
+			__mem_cgroup_handle_over_high(gfp_mask);
+	}
 	return 0;
 }
 
-- 
2.53.0-Meta

[RFC PATCH 4/4] selftests/bpf: add a cgroupfs lock-holder bpf_memcg_ops sample

[Shakeel Butt]

A bpf_memcg_ops policy that keeps tasks holding a cgroupfs kernfs lock out
of inline memory.high enforcement.

About forty functions take that lock, one of them through guard(rwsem_read),
so the sample tracks the lock rather than its callers.  A SEC("syscall")
program finds kernfs_rwsem and kernfs_supers_rwsem inside
cgrp_dfl_root.kf_root, and fentry/fexit on the rwsem primitives keeps a
per-task depth for those two addresses.  Every other kernfs user has its own
kernfs_root, so its own locks, which keeps this to cgroupfs.

down_*() return once the lock is held, so an acquire is counted on their
exit and a release on the entry of up_*().  The killable, interruptible and
trylock forms are covered too, checking the return so a failed acquire does
not count.  downgrade_write() needs no hook.  The depth is clamped at zero
for tasks already holding the lock when the hooks went on.

Two counters guard against passing for the wrong reason: one counts a holder
that reached inline enforcement anyway, and one counts a holder that could
not be tracked.  Both must be 0.

Measured with a reproducer [1] for the bug fixed by the patch proposed
in [2], on a kernel without that fix:

                                      baseline    policy    upstream fix
  max kernfs_rwsem hold, worker        2.049 s     199 us   not taken
  max kernfs_supers_rwsem hold         2.049 s     4.8 ms   2.056 s
  max kernfs_rwsem write wait          4.096 s     6.2 ms   8.7 ms
  walker passes over /sys/fs/cgroup        352       5711   6700
  churn mkdir+rmdir ops                 13 059    364 832   415 488
  churn max latency                     30.7 s    15.1 ms   12.7 ms

The policy does better than the fix on kernfs_supers_rwsem, which the fix
does not help: it drops kernfs_rwsem from the delivery loop, while the
policy stops the worker stalling at all.

The throughput columns are about 8% under what the same policy reaches
hooking one function.  That is the cost of the tracking and motivates
more optimized way to track the shared locks which can cause isolation
issues between unrelated workloads.

There is no harness yet, so these numbers come from the reproducer and not
from the test suite.

[1] https://github.com/shakeelb/mempressure-repros/tree/main/kernfs-notify-memcg
[2] https://lore.kernel.org/20260910045406.485295-1-shakeel.butt@linux.dev/

Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
 .../bpf/progs/memcg_ops_lockholder.c          | 222 ++++++++++++++++++
 1 file changed, 222 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/memcg_ops_lockholder.c

diff --git a/tools/testing/selftests/bpf/progs/memcg_ops_lockholder.c b/tools/testing/selftests/bpf/progs/memcg_ops_lockholder.c
new file mode 100644
index 000000000000..09a74f21f3da
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/memcg_ops_lockholder.c
@@ -0,0 +1,222 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Keep tasks holding a cgroupfs kernfs lock out of inline memory.high
+ * enforcement.
+ */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_core_read.h>
+
+char _license[] SEC("license") = "GPL";
+
+/* cgrp_dfl_root has no BTF variable, so take it as an untyped ksym. */
+extern const void cgrp_dfl_root __ksym;
+
+/* Set by resolve_locks() below, before any of the hooks are attached. */
+__u64 kernfs_rwsem_addr;
+__u64 kernfs_supers_rwsem_addr;
+
+struct lock_state {
+	__s32 depth;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_TASK_STORAGE);
+	__uint(map_flags, BPF_F_NO_PREALLOC);
+	__type(key, int);
+	__type(value, struct lock_state);
+} lock_state SEC(".maps");
+
+/* Charges from a task holding one of the locks: each is a stall avoided. */
+__u64 deferrals;
+/* Charges from everyone else, where we said nothing. */
+__u64 passthroughs;
+/* A holder reached inline enforcement anyway.  Must stay zero. */
+__u64 violations;
+/*
+ * A holder went untracked because task storage could not be created.  That
+ * looks just like a pass, and gets likelier under the pressure we are
+ * testing.  Must stay zero too.
+ */
+__u64 mark_failures;
+
+/*
+ * Find the two rw_semaphores in the cgroup2 kernfs_root.  The loader runs this
+ * once before attaching the hooks, so none of them sees a zero address.  Every
+ * other kernfs user has its own kernfs_root, hence its own locks, which is
+ * what keeps this to cgroupfs.
+ */
+SEC("syscall")
+int resolve_locks(void *ctx)
+{
+	struct cgroup_root *root = (struct cgroup_root *)&cgrp_dfl_root;
+	struct kernfs_root *kf;
+
+	kf = BPF_CORE_READ(root, kf_root);
+	if (!kf)
+		return 1;
+
+	kernfs_rwsem_addr = (__u64)kf +
+		bpf_core_field_offset(struct kernfs_root, kernfs_rwsem);
+	kernfs_supers_rwsem_addr = (__u64)kf +
+		bpf_core_field_offset(struct kernfs_root, kernfs_supers_rwsem);
+	return 0;
+}
+
+static __always_inline bool is_cgroup_kernfs_lock(const void *sem)
+{
+	__u64 addr = (__u64)sem;
+
+	return addr && (addr == kernfs_rwsem_addr ||
+			addr == kernfs_supers_rwsem_addr);
+}
+
+static __always_inline void note_acquire(const void *sem)
+{
+	struct lock_state *st;
+
+	if (!is_cgroup_kernfs_lock(sem))
+		return;
+
+	st = bpf_task_storage_get(&lock_state, bpf_get_current_task_btf(), NULL,
+				  BPF_LOCAL_STORAGE_GET_F_CREATE);
+	if (!st) {
+		__sync_fetch_and_add(&mark_failures, 1);
+		return;
+	}
+	st->depth++;
+}
+
+static __always_inline void note_release(const void *sem)
+{
+	struct lock_state *st;
+
+	if (!is_cgroup_kernfs_lock(sem))
+		return;
+
+	st = bpf_task_storage_get(&lock_state, bpf_get_current_task_btf(), NULL, 0);
+	if (!st)
+		return;
+	/* A task already holding it when we attached has no acquire to match. */
+	if (st->depth > 0)
+		st->depth--;
+}
+
+/*
+ * down_*() return once the lock is held, so the hold runs from their exit to
+ * the entry of up_*().  That excludes the wait.
+ *
+ * downgrade_write() needs no hook: the lock stays held, and the up_read()
+ * after it pairs with the original down_write().
+ */
+
+SEC("fexit/down_read")
+int BPF_PROG(down_read_exit, struct rw_semaphore *sem)
+{
+	note_acquire(sem);
+	return 0;
+}
+
+SEC("fexit/down_write")
+int BPF_PROG(down_write_exit, struct rw_semaphore *sem)
+{
+	note_acquire(sem);
+	return 0;
+}
+
+/* The killable and interruptible forms return 0 when they got the lock. */
+SEC("fexit/down_read_killable")
+int BPF_PROG(down_read_killable_exit, struct rw_semaphore *sem, int ret)
+{
+	if (!ret)
+		note_acquire(sem);
+	return 0;
+}
+
+SEC("fexit/down_read_interruptible")
+int BPF_PROG(down_read_interruptible_exit, struct rw_semaphore *sem, int ret)
+{
+	if (!ret)
+		note_acquire(sem);
+	return 0;
+}
+
+SEC("fexit/down_write_killable")
+int BPF_PROG(down_write_killable_exit, struct rw_semaphore *sem, int ret)
+{
+	if (!ret)
+		note_acquire(sem);
+	return 0;
+}
+
+/* The trylocks return 1 when they got the lock. */
+SEC("fexit/down_read_trylock")
+int BPF_PROG(down_read_trylock_exit, struct rw_semaphore *sem, int ret)
+{
+	if (ret == 1)
+		note_acquire(sem);
+	return 0;
+}
+
+SEC("fexit/down_write_trylock")
+int BPF_PROG(down_write_trylock_exit, struct rw_semaphore *sem, int ret)
+{
+	if (ret == 1)
+		note_acquire(sem);
+	return 0;
+}
+
+SEC("fentry/up_read")
+int BPF_PROG(up_read_enter, struct rw_semaphore *sem)
+{
+	note_release(sem);
+	return 0;
+}
+
+SEC("fentry/up_write")
+int BPF_PROG(up_write_enter, struct rw_semaphore *sem)
+{
+	note_release(sem);
+	return 0;
+}
+
+/* If a holder gets here, the policy did not work. */
+SEC("fentry/__mem_cgroup_handle_over_high")
+int BPF_PROG(over_high_enter)
+{
+	struct lock_state *st;
+
+	st = bpf_task_storage_get(&lock_state, bpf_get_current_task_btf(), NULL, 0);
+	if (st && st->depth > 0)
+		__sync_fetch_and_add(&violations, 1);
+	return 0;
+}
+
+/* BPF_PROG() uses "ctx" for the raw argument array, hence "mctx" here. */
+SEC("struct_ops")
+__u32 BPF_PROG(kernfs_high_policy, const struct bpf_memcg_ctx *mctx)
+{
+	struct lock_state *st;
+
+	/*
+	 * mctx->task is the charging task.  No F_CREATE, so this is just an
+	 * RCU read, and the counters are atomics on .bss.  Both are fine here.
+	 */
+	st = bpf_task_storage_get(&lock_state, mctx->task, NULL, 0);
+	if (!st || st->depth <= 0) {
+		__sync_fetch_and_add(&passthroughs, 1);
+		return BPF_MEMCG_HIGH_NO_OPINION;
+	}
+
+	__sync_fetch_and_add(&deferrals, 1);
+
+	/* The kernel ORs this with what the other policies return. */
+	return BPF_MEMCG_HIGH_DEFER_INLINE;
+}
+
+SEC(".struct_ops.link")
+struct bpf_memcg_ops kernfs_lockholder = {
+	.high_policy = (void *)kernfs_high_policy,
+};
-- 
2.53.0-Meta

Re: [RFC PATCH 1/4] bpf, cgroup: fix cgroup struct_ops query for a second attach type

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1336 bytes --]

> bpf, cgroup: fix cgroup struct_ops query for a second attach type
>
> Two things in __cgroup_bpf_query() work only because CGROUP_TCP_SOCK_OPS is
> the only one struct_ops attach type.
>
> It calls cgroup_bpf_enabled(atype) with an atype that
> find_atype_by_struct_ops_id() works out at runtime.  That macro is an asm
> goto and needs a constant.  Today the compiler can see there is only one
> value; add a second type and the build breaks with "impossible constraint in
> 'asm'".  Add cgroup_bpf_enabled_runtime(), which reads the key instead, and
> use it here.  This is a syscall path, so the cost does not matter.
>
> And find_atype_by_struct_ops_id() matches on type_id alone.  An attach type
> whose subsystem is not built keeps type_id 0, so a query for type 0 finds it
> and returns success with nothing instead of -ENOENT.  Skip such slots.
>
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>

This looks like a bug fix for the struct_ops attach infrastructure.
Should this include:

  Fixes: 369d9dcd8fb8 ("bpf: Add infrastructure to support attaching struct_ops to cgroups")


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35646140734