[PATCH v2 bpf-next 00/18] Auto-detect BPF fmodret targets

[PATCH v2 bpf-next 00/18] Auto-detect BPF fmodret targets

[Anton Protopopov]

Due to recent changes in software development, security fixes and attack-surface
hardening increasingly need to land on production systems _now_. BPF has proven
really useful for producing mitigations, but its error-injection capabilities
remain limited: the list of possible fmodret hooks is static, which no longer
matches reality.

This series allows more kernel functions to be targeted by fmodret BPF
programs. New functions are marked as fmodret-attachable _automatically_, if
CONFIG_CLANG_PLUGIN_BPF_SCAN_IR is on. The latter is off by default, but can be
enabled with CLANG >= 19 with a matching llvm-config and host C++ compiler.
Supported archs are x86_64 and arm64. (The existing static lists are, of
course, still supported.)

The decision on "if this function is safe to inject error" left to BPF program
writers.  The motivation/discussion can be found in the recent talk [1].  The
safety/operation model of this new functionality can be briefly stated as
"a trusted operator can already do this with live patch". However, the BPF
approach is more flexible. See the recent talk on shirudo [2] for more details
on how to build a trusted system managing such policies.

The changes introduced are as follows.

The first part of the series changes the way the lists of fmodret-attachable
functions are passed to kernel, namely, now a function is considered as a
fmodret target if it has a bpf_fmodret BTF tag. This tag is generated by the
resolve_btfids tool. The sleepable hooks are in addition tagged with
bpf_fmodret_sleepable. The old-style static hooks, explicitly defined in
kernel/modules, also get the bpf_fmodret tag. The security_*, syscall, and
error-injection hooks are _not_ all marked with the tag, and any new rules do
not apply to them.

Currently static hooks are defined using sets (BTF_FMODRET_START/END).  When
the minimal versions of GCC and pahole will support tags, the sets will go away
in favour of just declaring a proper tag.

The second part enables automatic search and usage of fmodret targets:

  * The search is performed by a new compiler pass via a compiler plugin.
    The candidates are written into a new .BTF_fmodret_candidates section which
    is dropped later in the final object.  Roughly speaking, a candidate is a
    function with a clean (side-effectless) path to "return -errno".

  * After the compilation is done, the resolve_btfids tool scans the candidates,
    drops some of them (redefined weak, non-ftracable, etc.), and assigns proper
    BTF tags: bpf_fmodret/bpf_fmodret_sleepable and bpf_fmodret_retval. The latter
    lists all possible return values the analysis found. The verifier rejects
    the values outside this list. (An alternative approach, maybe a followup,
    is to auto-convert all negative error values returned by a BPF program with
    a value from this list.)

The following selftests are added:
  
  * Test fmod_ret BTF decl tags and attach permission: the declared targets get
    the right tags, and fmod_ret programs attach only to tagged functions.

  * Test fmod_ret candidates found by bpf_scan_ir: functions in bpf_testmod
    annotated with the values the analysis must find, checked against the
    generated tags. The verifier allows only these values on every path and
    rejects extensions and tail calls of such programs.

Finally, the last patch adds a new `bpftool fmodret` command to simplify listing
the functions having bpf_fmodret* tags.

Planned follow ups, more or less in order:

  * LLVM LTO support
  * GCC support
  * Allow injecting errors into functions returning pointers
  * More complex logic to gather more candidates (mainly, error propagation,
    left out for now to reduce the number of commits in the series)

Build times with and without plugin enabled are nearly the same:

  with:                   without:
    real    5m48.377s       real    5m47.543s
    user    136m7.002s      user    135m57.994s
    sys     10m13.739s      sys     10m12.635s

Links:

  [1] Scaling BPF LSM hooks and error injection across the kernel,
      https://lpc.events/event/20/contributions/2435/

  [2] Shirudo: BPF live-patching infra for the agentic era,
      https://lpc.events/event/20/contributions/2437/

v1 -> v2:
  * Rebase onto bpf-next, as upstream removed mark_chain_precision()
  * Move tests into their own bpf_test_retval.ko from bpf_testmod.ko.
    The problem was with pahole, which can't properly find symbols in
    modules build from two or more objects (was only prominent with GCC,
    which reorders sections while linking objects together)
  * Fix the fmodret/extension test for GCC (force a const return)
  * resolve_btfids: dedup and cleanup code (bpf-ci bot)
  * bpftool: drop dead code (bpf-ci bot)
  * Rename two tests properly, remove dead code (bpf-ci bot)
  * Exclude KCSAN and UBSAN instrumentation from tests (sashiko)

v1: https://lore.kernel.org/bpf/20261106210314.1767267-1-a.s.protopopov@gmail.com/T/#t

Anton Protopopov (18):
  bpf: Add BTF ID sets for fmod_ret targets
  bpf: Add a btf_id_set8_entry helper
  bpf: Build fmod_ret target sets from BTF decl tags
  selftests/bpf: Declare bpf_fentry_test1 in tracing tests
  bpf: Declare fmod_ret targets with BTF_FMODRET sets
  selftests/bpf: Test fmod_ret BTF decl tags and attach permission
  scripts: Add LLVM plugin infrastructure
  bpf: Add the bpf_scan_ir LLVM plugin
  bpf: Find fmod_ret candidates in bpf_scan_ir
  bpf: Allow writes to private stack objects in bpf_scan_ir
  bpf: Follow PHIs in bpf_scan_ir
  bpf: Follow integer casts in bpf_scan_ir
  bpf: Check the return values of fmod_ret programs on found targets
  bpf: Require the JIT for fmod_ret programs on found 64-bit targets
  resolve_btfids: Tag fmod_ret candidates as targets
  selftests/bpf: Test fmod_ret candidates found by bpf_scan_ir
  selftests/bpf: Enable bpf_scan_ir in the config
  bpftool: List and show fmod_ret targets

 MAINTAINERS                                   |   10 +
 Makefile                                      |    6 +-
 arch/Kconfig                                  |    1 +
 arch/arm64/kernel/pi/Makefile                 |    1 +
 arch/x86/boot/startup/Makefile                |    2 +
 arch/x86/um/vdso/Makefile                     |    2 +-
 drivers/firmware/efi/libstub/Makefile         |    2 +-
 include/asm-generic/vmlinux.lds.h             |    7 +-
 include/linux/bpf.h                           |    6 +
 include/linux/btf.h                           |    9 +-
 include/linux/btf_ids.h                       |   24 +-
 include/linux/compiler-version.h              |    5 +
 kernel/bpf/btf.c                              |  124 +-
 kernel/bpf/syscall.c                          |    1 +
 kernel/bpf/verifier.c                         |  143 ++
 lib/vdso/Makefile.include                     |    3 +
 net/bpf/test_run.c                            |   12 +-
 net/mptcp/bpf.c                               |   15 +-
 scripts/Makefile                              |    1 +
 scripts/Makefile.btf                          |    1 +
 scripts/Makefile.llvm-plugins                 |   10 +
 scripts/Makefile.modfinal                     |    1 +
 scripts/Makefile.vmlinux                      |    3 +
 scripts/bpf/fmodret.lds                       |    6 +
 scripts/bpf/fmodret_metadata.h                |   29 +
 scripts/gen-btf.sh                            |    4 +-
 scripts/link-vmlinux.sh                       |    7 +
 scripts/llvm-plugins/Kconfig                  |   33 +
 scripts/llvm-plugins/Makefile                 |   21 +
 scripts/llvm-plugins/bpf_scan_ir.cpp          |  374 +++++
 scripts/mod/modpost.c                         |    1 +
 scripts/module.lds.S                          |    5 +
 .../bpftool/Documentation/bpftool-fmodret.rst |  111 ++
 tools/bpf/bpftool/Documentation/bpftool.rst   |    2 +-
 tools/bpf/bpftool/bash-completion/bpftool     |   31 +
 tools/bpf/bpftool/fmodret.c                   |  896 ++++++++++++
 tools/bpf/bpftool/main.c                      |    3 +-
 tools/bpf/bpftool/main.h                      |    1 +
 tools/bpf/resolve_btfids/Build                |    1 +
 tools/bpf/resolve_btfids/fmodret.c            | 1205 +++++++++++++++++
 tools/bpf/resolve_btfids/fmodret.h            |   28 +
 tools/bpf/resolve_btfids/main.c               |  185 ++-
 tools/include/linux/btf_ids.h                 |   24 +-
 tools/objtool/check.c                         |    1 +
 tools/testing/selftests/bpf/Makefile          |    2 +-
 tools/testing/selftests/bpf/config            |    2 +
 .../selftests/bpf/prog_tests/fmodret_retval.c |  211 +++
 .../selftests/bpf/prog_tests/fmodret_tags.c   |  155 +++
 .../selftests/bpf/progs/fmodret_retval.c      |  196 +++
 .../selftests/bpf/progs/fmodret_retval_ext.c  |   53 +
 .../selftests/bpf/progs/fsession_test.c       |    2 +
 .../bpf/progs/get_func_ip_fsession_test.c     |    2 +
 .../selftests/bpf/progs/tracing_multi_check.c |    2 +-
 .../testing/selftests/bpf/test_kmods/Makefile |   15 +-
 .../bpf/test_kmods/bpf_test_retval_main.c     |   12 +
 .../selftests/bpf/test_kmods/bpf_testmod.c    |   11 +-
 .../bpf/test_kmods/bpf_testmod_retval.c       |  311 +++++
 .../test_kmods/bpf_testmod_retval_strong.c    |   28 +
 58 files changed, 4277 insertions(+), 82 deletions(-)
 create mode 100644 scripts/Makefile.llvm-plugins
 create mode 100644 scripts/bpf/fmodret.lds
 create mode 100644 scripts/bpf/fmodret_metadata.h
 create mode 100644 scripts/llvm-plugins/Kconfig
 create mode 100644 scripts/llvm-plugins/Makefile
 create mode 100644 scripts/llvm-plugins/bpf_scan_ir.cpp
 create mode 100644 tools/bpf/bpftool/Documentation/bpftool-fmodret.rst
 create mode 100644 tools/bpf/bpftool/fmodret.c
 create mode 100644 tools/bpf/resolve_btfids/fmodret.c
 create mode 100644 tools/bpf/resolve_btfids/fmodret.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/fmodret_retval.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/fmodret_tags.c
 create mode 100644 tools/testing/selftests/bpf/progs/fmodret_retval.c
 create mode 100644 tools/testing/selftests/bpf/progs/fmodret_retval_ext.c
 create mode 100644 tools/testing/selftests/bpf/test_kmods/bpf_test_retval_main.c
 create mode 100644 tools/testing/selftests/bpf/test_kmods/bpf_testmod_retval.c
 create mode 100644 tools/testing/selftests/bpf/test_kmods/bpf_testmod_retval_strong.c

-- 
2.43.0

[PATCH v2 bpf-next 01/18] bpf: Add BTF ID sets for fmod_ret targets

[Anton Protopopov]

Add BTF_SET8_FMODRET and the BTF_FMODRET_START() and BTF_FMODRET_END()
macros to declare the functions BPF fmod_ret programs may attach to.
They work like kfunc sets: each entry is a BTF_ID_FLAGS() line and
KF_SLEEPABLE marks targets sleepable programs may attach to.

Teach resolve_btfids to find these sets and tag every listed function in
BTF with a "bpf_fmodret" decl tag, plus "bpf_fmodret_sleepable" when
KF_SLEEPABLE is set. The sleepable tag never comes without "bpf_fmodret"
and readers of the tags may rely on that. Share the set walk with the
kfunc collection. Any other flag is an error. Nothing in the code refers
to the sets, so keep .BTF_ids in the linker script, or --gc-sections may
discard them.

Nothing declares such sets yet and the kernel does not read the tags yet.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 include/asm-generic/vmlinux.lds.h |   2 +-
 include/linux/btf_ids.h           |  14 ++++
 tools/bpf/resolve_btfids/main.c   | 121 +++++++++++++++++++++++++-----
 tools/include/linux/btf_ids.h     |  14 ++++
 4 files changed, 130 insertions(+), 21 deletions(-)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 3eb732942d69..f8254a88e05d 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -714,7 +714,7 @@
 	BTF_INLINE_LINK						\
 	. = ALIGN(PAGE_SIZE);						\
 	.BTF_ids : AT(ADDR(.BTF_ids) - LOAD_OFFSET) {			\
-		*(.BTF_ids)						\
+		KEEP(*(.BTF_ids))					\
 	}
 #else
 #define BTF
diff --git a/include/linux/btf_ids.h b/include/linux/btf_ids.h
index 8b5a9ee92513..f27b7db81100 100644
--- a/include/linux/btf_ids.h
+++ b/include/linux/btf_ids.h
@@ -13,6 +13,12 @@ struct btf_id_set {
 /* This flag implies BTF_SET8 holds kfunc(s) */
 #define BTF_SET8_KFUNCS		(1 << 0)
 
+/*
+ * This flag implies BTF_SET8 holds functions which BPF fmod_ret, fentry and
+ * fexit programs may attach to. Per-function flags: KF_SLEEPABLE only.
+ */
+#define BTF_SET8_FMODRET	(1 << 1)
+
 struct btf_id_set8 {
 	u32 cnt;
 	u32 flags;
@@ -215,6 +221,12 @@ __BTF_SET8_START(name, local, BTF_SET8_KFUNCS)
 #define BTF_KFUNCS_END(name)				\
 BTF_SET8_END(name)
 
+#define BTF_FMODRET_START(name)				\
+__BTF_SET8_START(name, local, BTF_SET8_FMODRET)
+
+#define BTF_FMODRET_END(name)				\
+BTF_SET8_END(name)
+
 #else
 
 #define BTF_ID_LIST(name) static u32 __maybe_unused name[128];
@@ -231,6 +243,8 @@ BTF_SET8_END(name)
 #define BTF_SET8_END(name)
 #define BTF_KFUNCS_START(name) static struct btf_id_set8 __maybe_unused name = { .flags = BTF_SET8_KFUNCS };
 #define BTF_KFUNCS_END(name)
+#define BTF_FMODRET_START(name) static struct btf_id_set8 __maybe_unused name = { .flags = BTF_SET8_FMODRET };
+#define BTF_FMODRET_END(name)
 
 #endif /* CONFIG_DEBUG_INFO_BTF */
 
diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c
index c16dd93fa6d4..e0fb7525a5ea 100644
--- a/tools/bpf/resolve_btfids/main.c
+++ b/tools/bpf/resolve_btfids/main.c
@@ -69,6 +69,12 @@
  *   - rewrites the prototype of KF_IMPLICIT_ARGS kfuncs.
  *
  * These kfunc annotations were historically produced by pahole.
+ *
+ * Functions found in BTF_SET8_FMODRET sets are the targets of BPF fmod_ret
+ * programs. For each of them resolve_btfids emits a "bpf_fmodret" decl tag,
+ * and also "bpf_fmodret_sleepable" when KF_SLEEPABLE is set, so the sleepable
+ * tag never comes alone. The kernel builds the set of fmod_ret targets from
+ * these tags when it loads BTF.
  */
 
 #define  _GNU_SOURCE
@@ -197,7 +203,10 @@ struct btf_links {
 
 #define DECL_TAG_FASTCALL "bpf_fastcall"
 #define DECL_TAG_KFUNC "bpf_kfunc"
+#define DECL_TAG_FMODRET "bpf_fmodret"
+#define DECL_TAG_FMODRET_SLEEPABLE "bpf_fmodret_sleepable"
 
+#define KF_SLEEPABLE	(1 << 5)
 #define KF_FASTCALL	(1 << 12)
 #define KF_ARENA_RET	(1 << 13)
 #define KF_ARENA_ARG1	(1 << 14)
@@ -208,6 +217,7 @@ struct btf_links {
 #define PARAM_SUFFIX_ARENA "__arena"
 #define PARAM_SUFFIX_ARENA_NULLABLE "__arena__nullable"
 
+/* A function found in a BTF ID set: a kfunc or an fmod_ret target */
 struct kfunc {
 	struct rb_node rb_node;
 	const char *name;
@@ -221,6 +231,7 @@ struct btf2btf_context {
 	u32 nr_decl_tags;
 	u32 max_decl_tags;
 	struct rb_root kfuncs;
+	struct rb_root fmodrets;
 };
 
 static int verbose;
@@ -1256,16 +1267,16 @@ static int push_decl_tag_id(struct btf2btf_context *ctx, u32 decl_tag_id)
 	return 0;
 }
 
-static int push_kfunc(struct btf2btf_context *ctx, struct kfunc *kfunc)
+static int push_set_func(struct rb_root *root, const char *kind, struct kfunc *kfunc)
 {
-	struct rb_node **p = &ctx->kfuncs.rb_node;
+	struct rb_node **p = &root->rb_node;
 	struct rb_node *parent = NULL;
 	struct kfunc *k;
 
 	/*
-	 * Dedup by BTF ID: collecting the same kfunc twice is a no-op,
-	 * UNLESS the kfunc flags are inconsistent, in which case we
-	 * fail hard because it indicates a bug in a kfunc set declaration.
+	 * Dedup by BTF ID: collecting the same function twice is a no-op,
+	 * UNLESS its flags are inconsistent, in which case we fail hard
+	 * because it indicates a bug in a BTF ID set declaration.
 	 */
 	while (*p) {
 		parent = *p;
@@ -1278,8 +1289,8 @@ static int push_kfunc(struct btf2btf_context *ctx, struct kfunc *kfunc)
 		} else if (k->flags == kfunc->flags) {
 			return 0;
 		} else {
-			pr_err("ERROR: resolve_btfids: kfunc %s has inconsistent flags across BTF ID sets: 0x%x != 0x%x\n",
-			       kfunc->name, k->flags, kfunc->flags);
+			pr_err("ERROR: resolve_btfids: %s %s has inconsistent flags across BTF ID sets: 0x%x != 0x%x\n",
+			       kind, kfunc->name, k->flags, kfunc->flags);
 			return -EINVAL;
 		}
 	}
@@ -1290,7 +1301,7 @@ static int push_kfunc(struct btf2btf_context *ctx, struct kfunc *kfunc)
 
 	*k = *kfunc;
 	rb_link_node(&k->rb_node, parent, p);
-	rb_insert_color(&k->rb_node, &ctx->kfuncs);
+	rb_insert_color(&k->rb_node, root);
 
 	return 0;
 }
@@ -1344,7 +1355,16 @@ static bool is_arena_param(const struct btf *btf, const struct btf_param *param)
 	       param_name_has_suffix(name, PARAM_SUFFIX_ARENA_NULLABLE);
 }
 
-static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx)
+typedef int (*set_func_fn)(struct btf2btf_context *ctx, const char *name,
+			   u32 btf_id, u32 flags);
+
+/*
+ * Call FN for each function in the BTF ID sets which have SET_FLAG set. Reject
+ * an entry with flags other than ALLOWED_FLAGS, also if its function is missing.
+ */
+static int collect_set_funcs(struct object *obj, struct btf2btf_context *ctx,
+			     u32 set_flag, u32 allowed_flags, const char *kind,
+			     set_func_fn fn)
 {
 	Elf_Data *idlist = obj->efile.idlist;
 	struct btf *btf = ctx->btf;
@@ -1356,24 +1376,39 @@ static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx)
 	for (next = rb_first(&obj->sets); next; next = rb_next(next)) {
 		struct btf_id_set8 *set8;
 		struct btf_id *set_id;
-		u64 set_addr;
+		u64 set_addr, off;
 
 		set_id = rb_entry(next, struct btf_id, rb_node);
 		if (set_id->kind != BTF_ID_KIND_SET8 || set_id->addr_cnt != 1)
 			continue;
 
 		set_addr = set_id->addr[0];
-		set8 = idlist->d_buf + (set_addr - obj->efile.idlist_addr);
-		if (!(set8->flags & BTF_SET8_KFUNCS))
+		off = set_addr - obj->efile.idlist_addr;
+		/* A malformed set must not reach out of the section */
+		if (set_addr < obj->efile.idlist_addr || off > idlist->d_size ||
+		    idlist->d_size - off < sizeof(*set8) || set_id->cnt < 0 ||
+		    (u64)set_id->cnt > (idlist->d_size - off - sizeof(*set8)) /
+				       sizeof(set8->pairs[0])) {
+			pr_err("ERROR: resolve_btfids: set %s is out of %s\n",
+			       set_id->name, BTF_IDS_SECTION);
+			return -EINVAL;
+		}
+		set8 = idlist->d_buf + off;
+		if (!(set8->flags & set_flag))
 			continue;
 
 		for (u32 i = 0; i < set_id->cnt; i++) {
 			size_t off = (char *)&set8->pairs[i] - (char *)set8;
 			const char *name = find_name_by_addr(obj, set_addr + off);
-			struct kfunc kfunc;
+			u32 flags = set8->pairs[i].flags;
 			s32 func_id;
 			int err;
 
+			if (flags & ~allowed_flags) {
+				pr_err("ERROR: resolve_btfids: %s %s in %s has unsupported flags 0x%x\n",
+				       kind, name ?: "?", set_id->name, flags & ~allowed_flags);
+				return -EINVAL;
+			}
 			if (!name) {
 				pr_err("WARN: resolve_btfids: no BTF ID symbol for %s entry %u\n",
 				       set_id->name, i);
@@ -1383,18 +1418,15 @@ static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx)
 
 			func_id = btf__find_by_name_kind_own(btf, name, BTF_KIND_FUNC);
 			if (func_id < 0) {
-				pr_err("WARN: resolve_btfids: no BTF func for kfunc %s in %s\n",
-				       name, set_id->name);
+				pr_err("WARN: resolve_btfids: no BTF func for %s %s in %s\n",
+				       kind, name, set_id->name);
 				warnings++;
 				continue;
 			}
 
-			pr_debug("found kfunc %s in %s\n", name, set_id->name);
+			pr_debug("found %s %s in %s\n", kind, name, set_id->name);
 
-			kfunc.name = name;
-			kfunc.btf_id = func_id;
-			kfunc.flags = set8->pairs[i].flags;
-			err = push_kfunc(ctx, &kfunc);
+			err = fn(ctx, name, func_id, flags);
 			if (err)
 				return err;
 		}
@@ -1403,6 +1435,33 @@ static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx)
 	return 0;
 }
 
+static int push_kfunc(struct btf2btf_context *ctx, const char *name,
+		      u32 btf_id, u32 flags)
+{
+	struct kfunc kfunc = { .name = name, .btf_id = btf_id, .flags = flags };
+
+	return push_set_func(&ctx->kfuncs, "kfunc", &kfunc);
+}
+
+static int push_fmodret(struct btf2btf_context *ctx, const char *name,
+			u32 btf_id, u32 flags)
+{
+	struct kfunc fmodret = { .name = name, .btf_id = btf_id, .flags = flags };
+
+	return push_set_func(&ctx->fmodrets, "fmod_ret target", &fmodret);
+}
+
+static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx)
+{
+	return collect_set_funcs(obj, ctx, BTF_SET8_KFUNCS, ~0U, "kfunc", push_kfunc);
+}
+
+static int collect_fmodrets(struct object *obj, struct btf2btf_context *ctx)
+{
+	return collect_set_funcs(obj, ctx, BTF_SET8_FMODRET, KF_SLEEPABLE,
+				 "fmod_ret target", push_fmodret);
+}
+
 static int build_btf2btf_context(struct object *obj, struct btf2btf_context *ctx)
 {
 	int err;
@@ -1421,6 +1480,12 @@ static int build_btf2btf_context(struct object *obj, struct btf2btf_context *ctx
 		return err;
 	}
 
+	err = collect_fmodrets(obj, ctx);
+	if (err) {
+		pr_err("ERROR: resolve_btfids: failed to collect fmod_ret targets from BTF\n");
+		return err;
+	}
+
 	return 0;
 }
 
@@ -1761,10 +1826,26 @@ static int btf2btf(struct object *obj)
 			goto out;
 	}
 
+	for (next = rb_first(&ctx.fmodrets); next; next = rb_next(next)) {
+		struct kfunc *fmodret = rb_entry(next, struct kfunc, rb_node);
+
+		err = add_decl_tag(&ctx, DECL_TAG_FMODRET, fmodret->btf_id, -1);
+		if (err)
+			goto out;
+
+		if (fmodret->flags & KF_SLEEPABLE) {
+			err = add_decl_tag(&ctx, DECL_TAG_FMODRET_SLEEPABLE,
+					   fmodret->btf_id, -1);
+			if (err)
+				goto out;
+		}
+	}
+
 	err = 0;
 out:
 	free(ctx.decl_tags);
 	free_kfuncs(&ctx.kfuncs);
+	free_kfuncs(&ctx.fmodrets);
 
 	return err;
 }
diff --git a/tools/include/linux/btf_ids.h b/tools/include/linux/btf_ids.h
index 4fe5c5f1558c..b371ef7241b9 100644
--- a/tools/include/linux/btf_ids.h
+++ b/tools/include/linux/btf_ids.h
@@ -13,6 +13,12 @@ struct btf_id_set {
 /* This flag implies BTF_SET8 holds kfunc(s) */
 #define BTF_SET8_KFUNCS		(1 << 0)
 
+/*
+ * This flag implies BTF_SET8 holds functions which BPF fmod_ret, fentry and
+ * fexit programs may attach to. Per-function flags: KF_SLEEPABLE only.
+ */
+#define BTF_SET8_FMODRET	(1 << 1)
+
 struct btf_id_set8 {
 	u32 cnt;
 	u32 flags;
@@ -214,6 +220,12 @@ __BTF_SET8_START(name, local, BTF_SET8_KFUNCS)
 #define BTF_KFUNCS_END(name)				\
 BTF_SET8_END(name)
 
+#define BTF_FMODRET_START(name)				\
+__BTF_SET8_START(name, local, BTF_SET8_FMODRET)
+
+#define BTF_FMODRET_END(name)				\
+BTF_SET8_END(name)
+
 #else
 
 #define BTF_ID_LIST(name) static u32 __maybe_unused name[128];
@@ -230,6 +242,8 @@ BTF_SET8_END(name)
 #define BTF_SET8_END(name)
 #define BTF_KFUNCS_START(name) static struct btf_id_set8 __maybe_unused name = { .flags = BTF_SET8_KFUNCS };
 #define BTF_KFUNCS_END(name)
+#define BTF_FMODRET_START(name) static struct btf_id_set8 __maybe_unused name = { .flags = BTF_SET8_FMODRET };
+#define BTF_FMODRET_END(name)
 
 #endif /* CONFIG_DEBUG_INFO_BTF */
 
-- 
2.43.0

[PATCH v2 bpf-next 02/18] bpf: Add a btf_id_set8_entry helper

[Anton Protopopov]

The pairs of a struct btf_id_set8 have an anonymous type, and the
btf_id_set8_contains() helper returns a void pointer to the pair it
finds. All its users but one only check if a set contains an ID. The
one which needs the flags, btf_kfunc_id_set_contains(), finds them as
the word next to the ID.

Name the type struct btf_id_set8_pair, and add a btf_id_set8_entry()
helper which returns a pointer to it. Make btf_id_set8_contains()
return bool, like btf_id_set_contains() does, and use the new helper
in btf_kfunc_id_set_contains().

The layout of the set doesn't change. Update the tools copy of
btf_ids.h as well. A later patch uses the new helper to set the flags
of a pair.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 include/linux/btf.h           |  8 +++++++-
 include/linux/btf_ids.h       | 10 ++++++----
 kernel/bpf/btf.c              |  9 ++++-----
 tools/include/linux/btf_ids.h | 10 ++++++----
 4 files changed, 23 insertions(+), 14 deletions(-)

diff --git a/include/linux/btf.h b/include/linux/btf.h
index 87fad4076f0e..00495e7833b6 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -558,11 +558,17 @@ static inline bool btf_id_set_contains(const struct btf_id_set *set, u32 id)
 	return bsearch(&id, set->ids, set->cnt, sizeof(u32), btf_id_cmp_func) != NULL;
 }
 
-static inline void *btf_id_set8_contains(const struct btf_id_set8 *set, u32 id)
+static inline struct btf_id_set8_pair *
+btf_id_set8_entry(const struct btf_id_set8 *set, u32 id)
 {
 	return bsearch(&id, set->pairs, set->cnt, sizeof(set->pairs[0]), btf_id_cmp_func);
 }
 
+static inline bool btf_id_set8_contains(const struct btf_id_set8 *set, u32 id)
+{
+	return btf_id_set8_entry(set, id) != NULL;
+}
+
 bool btf_param_match_suffix(const struct btf *btf,
 			    const struct btf_param *arg,
 			    const char *suffix);
diff --git a/include/linux/btf_ids.h b/include/linux/btf_ids.h
index f27b7db81100..f5985acf661b 100644
--- a/include/linux/btf_ids.h
+++ b/include/linux/btf_ids.h
@@ -19,13 +19,15 @@ struct btf_id_set {
  */
 #define BTF_SET8_FMODRET	(1 << 1)
 
+struct btf_id_set8_pair {
+	u32 id;
+	u32 flags;
+};
+
 struct btf_id_set8 {
 	u32 cnt;
 	u32 flags;
-	struct {
-		u32 id;
-		u32 flags;
-	} pairs[];
+	struct btf_id_set8_pair pairs[];
 };
 
 #ifdef CONFIG_DEBUG_INFO_BTF
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 228b93d4f8c9..61d2df6776a9 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -9973,8 +9973,8 @@ static u32 *btf_kfunc_id_set_contains(const struct btf *btf,
 				      enum btf_kfunc_hook hook,
 				      u32 kfunc_btf_id)
 {
+	struct btf_id_set8_pair *pair;
 	struct btf_id_set8 *set;
-	u32 *id;
 
 	if (hook >= BTF_KFUNC_HOOK_MAX)
 		return NULL;
@@ -9983,11 +9983,10 @@ static u32 *btf_kfunc_id_set_contains(const struct btf *btf,
 	set = btf->kfunc_set_tab->sets[hook];
 	if (!set)
 		return NULL;
-	id = btf_id_set8_contains(set, kfunc_btf_id);
-	if (!id)
+	pair = btf_id_set8_entry(set, kfunc_btf_id);
+	if (!pair)
 		return NULL;
-	/* The flags for BTF ID are located next to it */
-	return id + 1;
+	return &pair->flags;
 }
 
 static bool __btf_kfunc_is_allowed(const struct btf *btf,
diff --git a/tools/include/linux/btf_ids.h b/tools/include/linux/btf_ids.h
index b371ef7241b9..3b5b2856c628 100644
--- a/tools/include/linux/btf_ids.h
+++ b/tools/include/linux/btf_ids.h
@@ -19,13 +19,15 @@ struct btf_id_set {
  */
 #define BTF_SET8_FMODRET	(1 << 1)
 
+struct btf_id_set8_pair {
+	u32 id;
+	u32 flags;
+};
+
 struct btf_id_set8 {
 	u32 cnt;
 	u32 flags;
-	struct {
-		u32 id;
-		u32 flags;
-	} pairs[];
+	struct btf_id_set8_pair pairs[];
 };
 
 #ifdef CONFIG_DEBUG_INFO_BTF
-- 
2.43.0

[PATCH v2 bpf-next 03/18] bpf: Build fmod_ret target sets from BTF decl tags

[Anton Protopopov]

Mark functions tagged with the "bpf_fmodret{,_sleepable}" decl tags
(added by resolve_btfids) as fmodret targets. Add the KF_SLEEPABLE flag
for the latter.

Do this for vmlinux at fs_initcall and for a module before its BTF
becomes visible to anyone else, so nothing can look at the set while it
is built.

The IDs come from the BTF after relocation, so they are final. Unlike
the IDs of a set registered by a module, they must not be relocated, and
the set is installed directly instead of through btf_populate_kfunc_set().
(The register_btf_fmodret_id_set() keeps working, it will be removed by
a later patch.)

The set shares kfunc_set_tab with the kfunc sets, but its flags only say
how a target may be attached. Skip it in btf_kfunc_check_flag(), so a
kfunc which is also an fmod_ret target keeps its own flags, such as
KF_IMPLICIT_ARGS. Otherwise btf_attach_func_proto() finds the flags of
such a kfunc inconsistent and attaching to it fails.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 kernel/bpf/btf.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 113 insertions(+), 1 deletion(-)

diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 61d2df6776a9..2c45e5fa1d7b 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -9307,6 +9307,8 @@ enum {
 };
 
 #ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+static int btf_register_fmodret_tags(struct btf *btf);
+
 struct btf_module {
 	struct list_head list;
 	struct module *module;
@@ -9476,6 +9478,15 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 			}
 			goto out;
 		}
+		/*
+		 * Do this before the BTF is visible to anyone else. A failure
+		 * is not fatal: the module loads, but without fmod_ret targets.
+		 */
+		err = btf_register_fmodret_tags(btf);
+		if (err)
+			pr_warn("failed to register fmod_ret targets of module [%s]: %d\n",
+				mod->name, err);
+
 		err = btf_alloc_id(btf);
 		if (err) {
 			btf_free(btf);
@@ -10100,7 +10111,9 @@ u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_p
 }
 
 /*
- * Check a single KF_* @flag on a kfunc across all of its hook sets.
+ * Check a single KF_* @flag on a kfunc across all of its hook sets. The set of
+ * fmod_ret targets is not one of them: its flags only say how a target may be
+ * attached, and a kfunc which is also a target keeps its own flags.
  * Returns:
  *   * 1 if @flag is set
  *   * 0 if @flag is not set
@@ -10115,6 +10128,8 @@ int btf_kfunc_check_flag(const struct btf *btf, u32 kfunc_btf_id, u32 flag)
 	u32 *flags;
 
 	for (hook = 0; hook < BTF_KFUNC_HOOK_MAX; hook++) {
+		if (hook == BTF_KFUNC_HOOK_FMODRET)
+			continue;
 		flags = btf_kfunc_id_set_contains(btf, hook, kfunc_btf_id);
 		if (!flags)
 			continue;
@@ -10182,6 +10197,103 @@ int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,
 }
 EXPORT_SYMBOL_GPL(register_btf_kfunc_id_set);
 
+/*
+ * Targets of fmod_ret programs are marked in BTF by resolve_btfids, see
+ * BTF_SET8_FMODRET. A "bpf_fmodret" decl tag on a function makes it a target.
+ * A target which sleepable programs may attach to also has a
+ * "bpf_fmodret_sleepable" tag, which never comes without "bpf_fmodret".
+ */
+#define BTF_FMODRET_TAG			"bpf_fmodret"
+#define BTF_FMODRET_SLEEPABLE_TAG	"bpf_fmodret_sleepable"
+
+/* Return the function which T tags with decl tag NAME, or 0 */
+static u32 btf_fmodret_tag_func(const struct btf *btf, const struct btf_type *t,
+				const char *name)
+{
+	const struct btf_type *func;
+
+	if (!btf_type_is_decl_tag(t) || btf_type_decl_tag(t)->component_idx != -1)
+		return 0;
+	if (strcmp(btf_name_by_offset(btf, t->name_off), name))
+		return 0;
+	func = btf_type_by_id(btf, t->type);
+	if (!func || !btf_type_is_func(func))
+		return 0;
+	return t->type;
+}
+
+/* Build the set of fmod_ret targets of BTF from its decl tags */
+static int btf_register_fmodret_tags(struct btf *btf)
+{
+	u32 id, func_id, nr_types = btf_nr_types(btf), cnt = 0;
+	struct btf_kfunc_set_tab *tab;
+	struct btf_id_set8 *set;
+
+	for (id = btf_start_id(btf); id < nr_types; id++)
+		cnt += !!btf_fmodret_tag_func(btf, btf_type_by_id(btf, id), BTF_FMODRET_TAG);
+	if (!cnt)
+		return 0;
+
+	set = kmalloc(struct_size(set, pairs, cnt), GFP_KERNEL | __GFP_NOWARN);
+	if (!set)
+		return -ENOMEM;
+	set->flags = 0;
+	set->cnt = 0;
+	for (id = btf_start_id(btf); id < nr_types; id++) {
+		func_id = btf_fmodret_tag_func(btf, btf_type_by_id(btf, id), BTF_FMODRET_TAG);
+		if (!func_id)
+			continue;
+		set->pairs[set->cnt].id = func_id;
+		set->pairs[set->cnt].flags = 0;
+		set->cnt++;
+	}
+	sort(set->pairs, set->cnt, sizeof(set->pairs[0]), btf_id_cmp_func, NULL);
+
+	/* Each sleepable tag marks a function which is already in the set */
+	for (id = btf_start_id(btf); id < nr_types; id++) {
+		struct btf_id_set8_pair *pair;
+
+		func_id = btf_fmodret_tag_func(btf, btf_type_by_id(btf, id),
+					       BTF_FMODRET_SLEEPABLE_TAG);
+		if (!func_id)
+			continue;
+		pair = btf_id_set8_entry(set, func_id);
+		if (pair)
+			pair->flags |= KF_SLEEPABLE;
+	}
+
+	tab = btf->kfunc_set_tab;
+	if (!tab) {
+		tab = kzalloc_obj(*tab, GFP_KERNEL | __GFP_NOWARN);
+		if (!tab) {
+			kfree(set);
+			return -ENOMEM;
+		}
+		btf->kfunc_set_tab = tab;
+	}
+	if (tab->sets[BTF_KFUNC_HOOK_FMODRET]) {
+		kfree(set);
+		return -EEXIST;
+	}
+	tab->sets[BTF_KFUNC_HOOK_FMODRET] = set;
+	return 0;
+}
+
+static int __init btf_fmodret_tags_init(void)
+{
+	struct btf *btf;
+	int err;
+
+	btf = btf_get_module_btf(NULL);
+	if (IS_ERR_OR_NULL(btf))
+		return PTR_ERR_OR_ZERO(btf);
+
+	err = btf_register_fmodret_tags(btf);
+	btf_put(btf);
+	return err;
+}
+fs_initcall(btf_fmodret_tags_init);
+
 /* This function must be invoked only from initcalls/module init functions */
 int register_btf_fmodret_id_set(const struct btf_kfunc_id_set *kset)
 {
-- 
2.43.0

[PATCH v2 bpf-next 04/18] selftests/bpf: Declare bpf_fentry_test1 in tracing tests

[Anton Protopopov]

bpftool puts bpf_fentry_test1 into vmlinux.h only because the function
is in a kfunc set, which exists to let fmod_ret programs attach to it.
A later patch moves it to an fmod_ret set, which does not make a
function a kfunc, and the declaration disappears.

The tests only compare its address. Declare it explicitly, as
get_func_ip_test.c does. The declaration matches the one in vmlinux.h,
so it builds both before and after.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 tools/testing/selftests/bpf/progs/fsession_test.c             | 2 ++
 tools/testing/selftests/bpf/progs/get_func_ip_fsession_test.c | 2 ++
 tools/testing/selftests/bpf/progs/tracing_multi_check.c       | 2 +-
 3 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/bpf/progs/fsession_test.c b/tools/testing/selftests/bpf/progs/fsession_test.c
index 86e8a2fe467e..4295f6a78dd1 100644
--- a/tools/testing/selftests/bpf/progs/fsession_test.c
+++ b/tools/testing/selftests/bpf/progs/fsession_test.c
@@ -4,6 +4,8 @@
 #include <bpf/bpf_helpers.h>
 #include <bpf/bpf_tracing.h>
 
+extern int bpf_fentry_test1(int a) __ksym;
+
 char _license[] SEC("license") = "GPL";
 
 __u64 test1_entry_result = 0;
diff --git a/tools/testing/selftests/bpf/progs/get_func_ip_fsession_test.c b/tools/testing/selftests/bpf/progs/get_func_ip_fsession_test.c
index bbeea0d512e3..51a7f605f9c5 100644
--- a/tools/testing/selftests/bpf/progs/get_func_ip_fsession_test.c
+++ b/tools/testing/selftests/bpf/progs/get_func_ip_fsession_test.c
@@ -3,6 +3,8 @@
 #include <bpf/bpf_helpers.h>
 #include <bpf/bpf_tracing.h>
 
+extern int bpf_fentry_test1(int a) __ksym;
+
 char _license[] SEC("license") = "GPL";
 
 __u64 test1_entry_result = 0;
diff --git a/tools/testing/selftests/bpf/progs/tracing_multi_check.c b/tools/testing/selftests/bpf/progs/tracing_multi_check.c
index b2959ba71179..9b008ecd6565 100644
--- a/tools/testing/selftests/bpf/progs/tracing_multi_check.c
+++ b/tools/testing/selftests/bpf/progs/tracing_multi_check.c
@@ -8,7 +8,7 @@ char _license[] SEC("license") = "GPL";
 int pid = 0;
 bool test_cookies = false;
 
-/* bpf_fentry_test1 is exported as kfunc via vmlinux.h */
+extern int bpf_fentry_test1(int a) __ksym;
 extern const void bpf_fentry_test2 __ksym;
 extern const void bpf_fentry_test3 __ksym;
 extern const void bpf_fentry_test4 __ksym;
-- 
2.43.0

[PATCH v2 bpf-next 05/18] bpf: Declare fmod_ret targets with BTF_FMODRET sets

[Anton Protopopov]

Move the fmod_ret targets of the BPF test functions, MPTCP and bpf_testmod
from registered kfunc sets to BTF_FMODRET sets. Attachment flags stay the
same: bpf_fentry_test1 is still the sleepable target.

These functions are no longer kfuncs, so the bpf_kfunc decl tag, and
with it the vmlinux.h declaration, is gone. Only
bpf_modify_return_test_tp() stays a kfunc, as it is also in the
common_btf_ids kfunc set. Selftests which need bpf_fentry_test1 declare
it themselves.

No code registers an fmod_ret set anymore, as the kernel builds them from
BTF. Remove register_btf_fmodret_id_set(), so a module cannot register a
second set next to the one from its BTF.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 include/linux/btf.h                               |  1 -
 kernel/bpf/btf.c                                  |  7 -------
 net/bpf/test_run.c                                | 12 +++---------
 net/mptcp/bpf.c                                   | 15 ++-------------
 .../selftests/bpf/test_kmods/bpf_testmod.c        | 11 ++---------
 5 files changed, 7 insertions(+), 39 deletions(-)

diff --git a/include/linux/btf.h b/include/linux/btf.h
index 00495e7833b6..5522c7fe1373 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -632,7 +632,6 @@ u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id,
 				const struct bpf_prog *prog);
 int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,
 			      const struct btf_kfunc_id_set *s);
-int register_btf_fmodret_id_set(const struct btf_kfunc_id_set *kset);
 s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id);
 int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,
 				struct module *owner);
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 2c45e5fa1d7b..f81a8ae9b92c 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -10294,13 +10294,6 @@ static int __init btf_fmodret_tags_init(void)
 }
 fs_initcall(btf_fmodret_tags_init);
 
-/* This function must be invoked only from initcalls/module init functions */
-int register_btf_fmodret_id_set(const struct btf_kfunc_id_set *kset)
-{
-	return __register_btf_kfunc_id_set(BTF_KFUNC_HOOK_FMODRET, kset);
-}
-EXPORT_SYMBOL_GPL(register_btf_fmodret_id_set);
-
 s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id)
 {
 	struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab;
diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c
index d6c9dc62182e..d63c3b697552 100644
--- a/net/bpf/test_run.c
+++ b/net/bpf/test_run.c
@@ -643,17 +643,12 @@ CFI_NOSEAL(bpf_kfunc_call_memb_release_dtor);
 
 __bpf_kfunc_end_defs();
 
-BTF_KFUNCS_START(bpf_test_modify_return_ids)
+BTF_FMODRET_START(bpf_test_modify_return_ids)
 BTF_ID_FLAGS(func, bpf_modify_return_test)
 BTF_ID_FLAGS(func, bpf_modify_return_test2)
 BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
 BTF_ID_FLAGS(func, bpf_fentry_test1, KF_SLEEPABLE)
-BTF_KFUNCS_END(bpf_test_modify_return_ids)
-
-static const struct btf_kfunc_id_set bpf_test_modify_return_set = {
-	.owner = THIS_MODULE,
-	.set   = &bpf_test_modify_return_ids,
-};
+BTF_FMODRET_END(bpf_test_modify_return_ids)
 
 BTF_KFUNCS_START(test_sk_check_kfunc_ids)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_release, KF_RELEASE)
@@ -1905,8 +1900,7 @@ static int __init bpf_prog_test_run_init(void)
 	};
 	int ret;
 
-	ret = register_btf_fmodret_id_set(&bpf_test_modify_return_set);
-	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_prog_test_kfunc_set);
+	ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_prog_test_kfunc_set);
 	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_prog_test_kfunc_set);
 	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_prog_test_kfunc_set);
 	return ret ?: register_btf_id_dtor_kfuncs(bpf_prog_test_dtor_kfunc,
diff --git a/net/mptcp/bpf.c b/net/mptcp/bpf.c
index 4cc16cbeb328..d0d489624689 100644
--- a/net/mptcp/bpf.c
+++ b/net/mptcp/bpf.c
@@ -20,17 +20,6 @@ struct mptcp_sock *bpf_mptcp_sock_from_subflow(struct sock *sk)
 	return NULL;
 }
 
-BTF_SET8_START(bpf_mptcp_fmodret_ids)
+BTF_FMODRET_START(bpf_mptcp_fmodret_ids)
 BTF_ID_FLAGS(func, update_socket_protocol)
-BTF_SET8_END(bpf_mptcp_fmodret_ids)
-
-static const struct btf_kfunc_id_set bpf_mptcp_fmodret_set = {
-	.owner = THIS_MODULE,
-	.set   = &bpf_mptcp_fmodret_ids,
-};
-
-static int __init bpf_mptcp_kfunc_init(void)
-{
-	return register_btf_fmodret_id_set(&bpf_mptcp_fmodret_set);
-}
-late_initcall(bpf_mptcp_kfunc_init);
+BTF_FMODRET_END(bpf_mptcp_fmodret_ids)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 41ff36f0ca77..91cb311a2dc8 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -2607,15 +2607,9 @@ struct bpf_struct_ops testmod_multi_st_ops = {
 
 extern int bpf_fentry_test1(int a);
 
-BTF_KFUNCS_START(bpf_testmod_trampoline_count_ids)
+BTF_FMODRET_START(bpf_testmod_trampoline_count_ids)
 BTF_ID_FLAGS(func, bpf_testmod_trampoline_count_test)
-BTF_KFUNCS_END(bpf_testmod_trampoline_count_ids)
-
-static const struct
-btf_kfunc_id_set bpf_testmod_trampoline_count_fmodret_set = {
-	.owner = THIS_MODULE,
-	.set = &bpf_testmod_trampoline_count_ids,
-};
+BTF_FMODRET_END(bpf_testmod_trampoline_count_ids)
 
 static int bpf_testmod_init(void)
 {
@@ -2633,7 +2627,6 @@ static int bpf_testmod_init(void)
 	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_testmod_kfunc_set);
 	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_testmod_kfunc_set);
 	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &bpf_testmod_kfunc_set);
-	ret = ret ?: register_btf_fmodret_id_set(&bpf_testmod_trampoline_count_fmodret_set);
 	ret = ret ?: register_bpf_struct_ops(&bpf_bpf_testmod_ops, bpf_testmod_ops);
 	ret = ret ?: register_bpf_struct_ops(&bpf_testmod_ops2, bpf_testmod_ops2);
 	ret = ret ?: register_bpf_struct_ops(&bpf_testmod_ops3, bpf_testmod_ops3);
-- 
2.43.0

[PATCH v2 bpf-next 06/18] selftests/bpf: Test fmod_ret BTF decl tags and attach permission

[Anton Protopopov]

Check that resolve_btfids tags the fmod_ret targets of vmlinux and of
bpf_testmod with "bpf_fmodret", that only bpf_fentry_test1 has the
"bpf_fmodret_sleepable" tag, and that a function which is not a target
has none.

Load minimal fmod_ret programs to check that the kernel builds its
target set from the tags: a target loads, a function which is not a
target does not, and sleepable programs load only for the sleepable
target.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 .../selftests/bpf/prog_tests/fmodret_tags.c   | 155 ++++++++++++++++++
 1 file changed, 155 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/fmodret_tags.c

diff --git a/tools/testing/selftests/bpf/prog_tests/fmodret_tags.c b/tools/testing/selftests/bpf/prog_tests/fmodret_tags.c
new file mode 100644
index 000000000000..62efe11af826
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/fmodret_tags.c
@@ -0,0 +1,155 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+#include <linux/filter.h>
+#include <bpf/btf.h>
+
+#define TAG_FMODRET		"bpf_fmodret"
+#define TAG_FMODRET_SLEEPABLE	"bpf_fmodret_sleepable"
+
+static bool func_has_tag(const struct btf *btf, const char *func, const char *tag)
+{
+	int func_id, id, nr_types;
+
+	func_id = btf__find_by_name_kind(btf, func, BTF_KIND_FUNC);
+	if (func_id < 0)
+		return false;
+
+	nr_types = btf__type_cnt(btf);
+	for (id = 1; id < nr_types; id++) {
+		const struct btf_type *t = btf__type_by_id(btf, id);
+
+		if (!btf_is_decl_tag(t) || t->type != func_id ||
+		    btf_decl_tag(t)->component_idx != -1)
+			continue;
+		if (!strcmp(btf__name_by_offset(btf, t->name_off), tag))
+			return true;
+	}
+
+	return false;
+}
+
+static void test_vmlinux_tags(void)
+{
+	struct btf *btf;
+
+	btf = btf__load_vmlinux_btf();
+	if (!ASSERT_OK_PTR(btf, "btf__load_vmlinux_btf"))
+		return;
+
+	ASSERT_TRUE(func_has_tag(btf, "bpf_modify_return_test", TAG_FMODRET), "modify_return_test");
+	ASSERT_TRUE(func_has_tag(btf, "bpf_modify_return_test2", TAG_FMODRET),
+		    "modify_return_test2");
+	ASSERT_TRUE(func_has_tag(btf, "bpf_modify_return_test_tp", TAG_FMODRET),
+		    "modify_return_test_tp");
+	ASSERT_TRUE(func_has_tag(btf, "bpf_fentry_test1", TAG_FMODRET), "fentry_test1");
+	ASSERT_TRUE(func_has_tag(btf, "update_socket_protocol", TAG_FMODRET),
+		    "update_socket_protocol");
+
+	/* Only bpf_fentry_test1 is a sleepable target */
+	ASSERT_TRUE(func_has_tag(btf, "bpf_fentry_test1", TAG_FMODRET_SLEEPABLE),
+		    "fentry_test1_sleepable");
+	ASSERT_FALSE(func_has_tag(btf, "bpf_modify_return_test", TAG_FMODRET_SLEEPABLE),
+		     "modify_return_test_sleepable");
+	ASSERT_FALSE(func_has_tag(btf, "bpf_modify_return_test2", TAG_FMODRET_SLEEPABLE),
+		     "modify_return_test2_sleepable");
+	ASSERT_FALSE(func_has_tag(btf, "bpf_modify_return_test_tp", TAG_FMODRET_SLEEPABLE),
+		     "modify_return_test_tp_sleepable");
+	ASSERT_FALSE(func_has_tag(btf, "update_socket_protocol", TAG_FMODRET_SLEEPABLE),
+		     "update_socket_protocol_sleepable");
+
+	/* A function which is not a target has no tag */
+	ASSERT_GT(btf__find_by_name_kind(btf, "bpf_fentry_test2", BTF_KIND_FUNC), 0,
+		  "fentry_test2_exists");
+	ASSERT_FALSE(func_has_tag(btf, "bpf_fentry_test2", TAG_FMODRET), "fentry_test2");
+	ASSERT_FALSE(func_has_tag(btf, "bpf_fentry_test2", TAG_FMODRET_SLEEPABLE),
+		     "fentry_test2_sleepable");
+
+	btf__free(btf);
+}
+
+static void test_module_tags(void)
+{
+	struct btf *vmlinux_btf, *module_btf;
+
+	if (!env.has_testmod) {
+		test__skip();
+		return;
+	}
+
+	vmlinux_btf = btf__load_vmlinux_btf();
+	if (!ASSERT_OK_PTR(vmlinux_btf, "btf__load_vmlinux_btf"))
+		return;
+	module_btf = btf__load_module_btf("bpf_testmod", vmlinux_btf);
+	if (!module_btf && errno == ENOENT) {
+		printf("%s:SKIP:bpf_testmod has no BTF\n", __func__);
+		test__skip();
+		goto out;
+	}
+	if (!ASSERT_OK_PTR(module_btf, "btf__load_module_btf"))
+		goto out;
+
+	ASSERT_TRUE(func_has_tag(module_btf, "bpf_testmod_trampoline_count_test", TAG_FMODRET),
+		    "trampoline_count_test");
+	ASSERT_FALSE(func_has_tag(module_btf, "bpf_testmod_trampoline_count_test",
+				  TAG_FMODRET_SLEEPABLE), "trampoline_count_test_sleepable");
+	ASSERT_FALSE(func_has_tag(module_btf, "bpf_testmod_fentry_test1", TAG_FMODRET),
+		     "testmod_fentry_test1");
+
+	btf__free(module_btf);
+out:
+	btf__free(vmlinux_btf);
+}
+
+/* Load a minimal fmod_ret program which attaches to function BTF_ID */
+static int load_fmod_ret(int btf_id, __u32 prog_flags)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	LIBBPF_OPTS(bpf_prog_load_opts, opts,
+		.expected_attach_type = BPF_MODIFY_RETURN,
+		.prog_flags = prog_flags,
+		.attach_btf_id = btf_id,
+	);
+
+	return bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL", insns, ARRAY_SIZE(insns), &opts);
+}
+
+static void check_load(const char *func, __u32 prog_flags, bool allowed, const char *name)
+{
+	int btf_id, fd;
+
+	btf_id = libbpf_find_vmlinux_btf_id(func, BPF_MODIFY_RETURN);
+	if (!ASSERT_GT(btf_id, 0, func))
+		return;
+	fd = load_fmod_ret(btf_id, prog_flags);
+	if (allowed)
+		ASSERT_GE(fd, 0, name);
+	else
+		ASSERT_LT(fd, 0, name);
+	if (fd >= 0)
+		close(fd);
+}
+
+static void test_attach_permission(void)
+{
+	check_load("bpf_modify_return_test", 0, true, "target");
+	check_load("bpf_fentry_test1", 0, true, "sleepable_target");
+	check_load("bpf_fentry_test2", 0, false, "not_a_target");
+
+	check_load("bpf_fentry_test1", BPF_F_SLEEPABLE, true, "sleepable_prog_sleepable_target");
+	check_load("bpf_modify_return_test", BPF_F_SLEEPABLE, false,
+		   "sleepable_prog_nonsleepable_target");
+}
+
+void test_fmodret_tags(void)
+{
+	if (test__start_subtest("vmlinux"))
+		test_vmlinux_tags();
+	if (test__start_subtest("module"))
+		test_module_tags();
+	if (test__start_subtest("attach_permission"))
+		test_attach_permission();
+}
-- 
2.43.0

[PATCH v2 bpf-next 07/18] scripts: Add LLVM plugin infrastructure

[Anton Protopopov]

Add scripts/llvm-plugins and the corresponding Makefiles and Kconfig for
LLVM IR passes loaded with -fpass-plugin, similar to the GCC plugins.
Individual plugins register their configuration and shared objects
separately. No plugin is added yet.

Use LLVM_CONFIG to obtain the host compiler and linker flags. Require
llvm-config of the same major version as clang, which loads the plugins:
unlike GCC, clang does not provide the headers for its plugins. Require
also a host C++ compiler which accepts the headers and flags of
llvm-config.

Build plugins without linking LLVM component libraries; their LLVM
symbols are resolved by the compiler loading the shared library. Include
a generated header so objects are rebuilt when enabled plugins change.
Track the headers of a plugin, LLVM ones included, with a dependency
file, as the GCC plugins do.

Export LLVM_PLUGINS_CFLAGS so vDSO, EFI stub and early boot builds can
exclude the plugins, as they exclude the GCC plugins.

Add the plugin infrastructure to the CLANG/LLVM BUILD SUPPORT entry in
MAINTAINERS.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 MAINTAINERS                           |  2 ++
 Makefile                              |  6 +++++-
 arch/Kconfig                          |  1 +
 arch/arm64/kernel/pi/Makefile         |  1 +
 arch/x86/boot/startup/Makefile        |  2 ++
 arch/x86/um/vdso/Makefile             |  2 +-
 drivers/firmware/efi/libstub/Makefile |  2 +-
 include/linux/compiler-version.h      |  5 +++++
 lib/vdso/Makefile.include             |  3 +++
 scripts/Makefile                      |  1 +
 scripts/Makefile.llvm-plugins         |  8 ++++++++
 scripts/llvm-plugins/Kconfig          | 17 +++++++++++++++++
 scripts/llvm-plugins/Makefile         | 21 +++++++++++++++++++++
 13 files changed, 68 insertions(+), 3 deletions(-)
 create mode 100644 scripts/Makefile.llvm-plugins
 create mode 100644 scripts/llvm-plugins/Kconfig
 create mode 100644 scripts/llvm-plugins/Makefile

diff --git a/MAINTAINERS b/MAINTAINERS
index 93abbb315065..02b6c8963c63 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6562,7 +6562,9 @@ C:	irc://irc.libera.chat/clangbuiltlinux
 F:	Documentation/kbuild/llvm.rst
 F:	include/linux/compiler-clang.h
 F:	scripts/Makefile.clang
+F:	scripts/Makefile.llvm-plugins
 F:	scripts/clang-tools/
+F:	scripts/llvm-plugins/
 K:	\b(?i:clang|llvm)\b
 
 CLK API
diff --git a/Makefile b/Makefile
index f895752d4b05..2fcad8109572 100644
--- a/Makefile
+++ b/Makefile
@@ -457,10 +457,13 @@ endif
 
 HOSTCC	= $(LLVM_PREFIX)clang$(LLVM_SUFFIX)
 HOSTCXX	= $(LLVM_PREFIX)clang++$(LLVM_SUFFIX)
+LLVM_CONFIG ?= $(LLVM_PREFIX)llvm-config$(LLVM_SUFFIX)
 else
 HOSTCC	= gcc
 HOSTCXX	= g++
+LLVM_CONFIG ?= llvm-config
 endif
+export LLVM_CONFIG
 HOSTRUSTC = rustc
 HOSTPKG_CONFIG	= pkg-config
 
@@ -1211,10 +1214,11 @@ include-$(CONFIG_AUTOFDO_CLANG)	+= scripts/Makefile.autofdo
 include-$(CONFIG_PROPELLER_CLANG)	+= scripts/Makefile.propeller
 include-$(CONFIG_WARN_CONTEXT_ANALYSIS) += scripts/Makefile.context-analysis
 include-$(CONFIG_GCC_PLUGINS)	+= scripts/Makefile.gcc-plugins
+include-$(CONFIG_LLVM_PLUGINS)	+= scripts/Makefile.llvm-plugins
 
 include $(addprefix $(srctree)/, $(include-y))
 
-# scripts/Makefile.gcc-plugins is intentionally included last.
+# The compiler plugin Makefiles are intentionally included last.
 # Do not add $(call cc-option,...) below this line. When you build the kernel
 # from the clean source tree, the GCC plugins do not exist at this point.
 
diff --git a/arch/Kconfig b/arch/Kconfig
index 1163736de1f9..4f1f36d3e82d 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -1772,6 +1772,7 @@ config HAVE_GENERIC_TIF_BITS
 source "kernel/gcov/Kconfig"
 
 source "scripts/gcc-plugins/Kconfig"
+source "scripts/llvm-plugins/Kconfig"
 
 config FUNCTION_ALIGNMENT_4B
 	bool
diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile
index a273ab5af805..5813bb75c992 100644
--- a/arch/arm64/kernel/pi/Makefile
+++ b/arch/arm64/kernel/pi/Makefile
@@ -10,6 +10,7 @@ KBUILD_CFLAGS	:= $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \
 		   -D__DISABLE_EXPORTS -ffreestanding -D__NO_FORTIFY \
 		   -fno-asynchronous-unwind-tables -fno-unwind-tables \
 		   $(CONFIG_CC_OPT_NO_ADDRSIG)
+KBUILD_CFLAGS	:= $(filter-out $(LLVM_PLUGINS_CFLAGS),$(KBUILD_CFLAGS))
 
 # this code may run with the MMU off so disable unaligned accesses
 CFLAGS_map_range.o += -mstrict-align
diff --git a/arch/x86/boot/startup/Makefile b/arch/x86/boot/startup/Makefile
index 5e499cfb29b5..7036e939097e 100644
--- a/arch/x86/boot/startup/Makefile
+++ b/arch/x86/boot/startup/Makefile
@@ -12,6 +12,8 @@ KBUILD_CFLAGS		+= -D__DISABLE_EXPORTS -mcmodel=small -fPIC \
 # disable ftrace hooks and LTO
 KBUILD_CFLAGS	:= $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS))
 KBUILD_CFLAGS	:= $(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS))
+# LLVM plugins may add data with absolute addresses
+KBUILD_CFLAGS	:= $(filter-out $(LLVM_PLUGINS_CFLAGS),$(KBUILD_CFLAGS))
 KASAN_SANITIZE	:= n
 KCSAN_SANITIZE	:= n
 KMSAN_SANITIZE	:= n
diff --git a/arch/x86/um/vdso/Makefile b/arch/x86/um/vdso/Makefile
index 7664cbedbe30..0e32c9ae2b5a 100644
--- a/arch/x86/um/vdso/Makefile
+++ b/arch/x86/um/vdso/Makefile
@@ -37,7 +37,7 @@ CFL := $(PROFILING) -mcmodel=small -fPIC -O2 -fasynchronous-unwind-tables -m64 \
        $(filter -g%,$(KBUILD_CFLAGS)) -fno-stack-protector \
        -fno-omit-frame-pointer -foptimize-sibling-calls
 
-$(vobjs): KBUILD_CFLAGS += $(CFL)
+$(vobjs): KBUILD_CFLAGS := $(filter-out $(LLVM_PLUGINS_CFLAGS),$(KBUILD_CFLAGS)) $(CFL)
 
 #
 # vDSO code runs in userspace and -pg doesn't help with profiling anyway.
diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile
index bbab12b8eb4b..6f69d5d67731 100644
--- a/drivers/firmware/efi/libstub/Makefile
+++ b/drivers/firmware/efi/libstub/Makefile
@@ -6,7 +6,7 @@
 # enabled, even if doing so doesn't break the build.
 #
 
-cflags-y			:= $(KBUILD_CFLAGS)
+cflags-y			:= $(filter-out $(LLVM_PLUGINS_CFLAGS),$(KBUILD_CFLAGS))
 
 # x86 links the stub into the decompressor rather than the kernel proper, so
 # drop the kernel's code model, calling convention, and stack alignment, and
diff --git a/include/linux/compiler-version.h b/include/linux/compiler-version.h
index ac1665a98a15..e6a5a9f3fbcd 100644
--- a/include/linux/compiler-version.h
+++ b/include/linux/compiler-version.h
@@ -24,6 +24,11 @@
 #include <generated/gcc-plugins.h>
 #endif
 
+/* Rebuild objects when an LLVM plugin changes. */
+#ifdef LLVM_PLUGINS
+#include <generated/llvm-plugins.h>
+#endif
+
 /*
  * If the randstruct seed itself changes (whether for GCC plugins or
  * Clang), the entire tree needs to be rebuilt since the randomization of
diff --git a/lib/vdso/Makefile.include b/lib/vdso/Makefile.include
index cedbf15f8087..848d58296ba2 100644
--- a/lib/vdso/Makefile.include
+++ b/lib/vdso/Makefile.include
@@ -1,5 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 
+# vDSO code executes outside the kernel.
+KBUILD_CFLAGS := $(filter-out $(LLVM_PLUGINS_CFLAGS),$(KBUILD_CFLAGS))
+
 GENERIC_VDSO_MK_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
 GENERIC_VDSO_DIR := $(dir $(GENERIC_VDSO_MK_PATH))
 
diff --git a/scripts/Makefile b/scripts/Makefile
index d7fbb7ca098b..bc5dacc29207 100644
--- a/scripts/Makefile
+++ b/scripts/Makefile
@@ -66,6 +66,7 @@ hostprogs += unifdef gen_packed_field_checks
 targets += module.lds
 
 subdir-$(CONFIG_GCC_PLUGINS) += gcc-plugins
+subdir-$(CONFIG_LLVM_PLUGINS) += llvm-plugins
 subdir-$(CONFIG_GENKSYMS) += genksyms
 subdir-$(CONFIG_GENDWARFKSYMS) += gendwarfksyms
 subdir-$(CONFIG_SECURITY_SELINUX) += selinux
diff --git a/scripts/Makefile.llvm-plugins b/scripts/Makefile.llvm-plugins
new file mode 100644
index 000000000000..e18f4907b015
--- /dev/null
+++ b/scripts/Makefile.llvm-plugins
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+LLVM_PLUGIN := $(llvm-plugin-y)
+LLVM_PLUGINS_CFLAGS := $(strip $(addprefix -fpass-plugin=$(objtree)/scripts/llvm-plugins/, \
+	$(LLVM_PLUGIN)) -DLLVM_PLUGINS)
+export LLVM_PLUGIN LLVM_PLUGINS_CFLAGS
+
+KBUILD_CFLAGS += $(LLVM_PLUGINS_CFLAGS)
diff --git a/scripts/llvm-plugins/Kconfig b/scripts/llvm-plugins/Kconfig
new file mode 100644
index 000000000000..37db99159b9c
--- /dev/null
+++ b/scripts/llvm-plugins/Kconfig
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+menuconfig LLVM_PLUGINS
+	bool "LLVM plugins"
+	depends on CC_IS_CLANG
+	depends on $(success,$(LLVM_CONFIG) --version)
+	# The plugins are built against the headers of llvm-config and loaded
+	# by clang, so both must come from the same LLVM release
+	depends on $(success,test "$(shell,$(LLVM_CONFIG) --version 2>/dev/null | cut -d. -f1)" = "$(shell,$(CC) -dumpversion | cut -d. -f1)")
+	# The host C++ compiler must accept the LLVM headers and flags
+	depends on $(success,echo '#include <llvm/Config/llvm-config.h>' | $(HOSTCXX) $(shell,$(LLVM_CONFIG) --cxxflags 2>/dev/null) -x c++ -fsyntax-only -)
+	default y
+	help
+	  LLVM plugins add compiler passes to kernel and module builds.
+	  Building them requires LLVM development headers and llvm-config
+	  of the same major version as clang. Each plugin has its own option
+	  below and is built only if that option is enabled.
diff --git a/scripts/llvm-plugins/Makefile b/scripts/llvm-plugins/Makefile
new file mode 100644
index 000000000000..a1f8503c8bcd
--- /dev/null
+++ b/scripts/llvm-plugins/Makefile
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+always-y += $(LLVM_PLUGIN)
+targets += $(LLVM_PLUGIN)
+clean-files += *.so
+
+# Run llvm-config only to build a plugin, not when this Makefile is parsed
+llvm_cxxflags = $(shell $(LLVM_CONFIG) --cxxflags)
+llvm_ldflags = $(shell $(LLVM_CONFIG) --ldflags)
+
+quiet_cmd_llvm_plugin = HOSTCXX $@
+      cmd_llvm_plugin = $(HOSTCXX) -Wp,-MD,$(depfile) $(KBUILD_HOSTCXXFLAGS) \
+		$(llvm_cxxflags) -fPIC -shared $(KBUILD_HOSTLDFLAGS) -o $@ $< \
+		$(llvm_ldflags)
+
+$(obj)/%.so: $(src)/%.cpp FORCE
+	$(call if_changed_dep,llvm_plugin)
+
+$(obj)/../../include/generated/llvm-plugins.h: $(addprefix $(obj)/, $(LLVM_PLUGIN)) FORCE
+	$(call if_changed,touch)
+always-y += ../../include/generated/llvm-plugins.h
-- 
2.43.0

[PATCH v2 bpf-next 08/18] bpf: Add the bpf_scan_ir LLVM plugin

[Anton Protopopov]

Add a new bpf_scan_ir.cpp LLVM plugin selected by the new
CONFIG_CLANG_PLUGIN_BPF_SCAN_IR option. The new plugin depends on
DEBUG_INFO_BTF, as the information it produces eventually ends up in
kernel/module BTF. It needs LLVM 19 or later and supports only x86_64
and arm64. For now LTO_CLANG is not supported. It is a planned
follow-up. The new pass runs after optimization over the IR of the
kernel and modules.

The purpose of this plugin is to gather information about functions,
which can be targets for BPF fmodret programs (have paths without side
effects which return errors). The records about functions are stored in
the new .BTF_fmodret_candidates section which is created by plugin and
later is processed by the resolve_btfids tool. The actual plugin logic is
added in the following commits.

Keep .BTF_fmodret_candidates and its relocations in the temporary BTF
link and discard it before the later links, so the build-only function
references do not create runtime relocations. Keep it in modules until
their BTF is generated and remove it then, also when BTF generation is
skipped. Without module BTF, discard it at the module link. Let modpost
accept the section. Let objtool skip its relocations in IBT validation:
a record has the address of its function, which is not an indirect
branch target. The vDSO, the EFI stub and the early boot code are built
without the plugins, so position-independent startup objects do not
contain the absolute function pointers of the records.

The record format and the linker script are in scripts/bpf, so a future
GCC plugin can share them.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 MAINTAINERS                          |   7 ++
 include/asm-generic/vmlinux.lds.h    |   3 +-
 scripts/Makefile.llvm-plugins        |   2 +
 scripts/Makefile.modfinal            |   1 +
 scripts/Makefile.vmlinux             |   3 +
 scripts/bpf/fmodret.lds              |   6 ++
 scripts/bpf/fmodret_metadata.h       |  29 ++++++
 scripts/gen-btf.sh                   |   4 +-
 scripts/link-vmlinux.sh              |   7 ++
 scripts/llvm-plugins/Kconfig         |  16 ++++
 scripts/llvm-plugins/bpf_scan_ir.cpp | 126 +++++++++++++++++++++++++++
 scripts/mod/modpost.c                |   1 +
 scripts/module.lds.S                 |   5 ++
 tools/objtool/check.c                |   1 +
 14 files changed, 209 insertions(+), 2 deletions(-)
 create mode 100644 scripts/bpf/fmodret.lds
 create mode 100644 scripts/bpf/fmodret_metadata.h
 create mode 100644 scripts/llvm-plugins/bpf_scan_ir.cpp

diff --git a/MAINTAINERS b/MAINTAINERS
index 02b6c8963c63..dfbb45ecc716 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -5064,6 +5064,13 @@ S:	Supported
 F:	arch/x86/net/
 X:	arch/x86/net/bpf_jit_comp32.c
 
+BPF [COMPILER PLUGINS]
+M:	Anton Protopopov <a.s.protopopov@gmail.com>
+L:	bpf@vger.kernel.org
+S:	Maintained
+F:	scripts/bpf/
+F:	scripts/llvm-plugins/bpf_*
+
 BPF [CORE]
 M:	Alexei Starovoitov <ast@kernel.org>
 M:	Daniel Borkmann <daniel@iogearbox.net>
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index f8254a88e05d..4e18758b5ffc 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -886,7 +886,8 @@
 		KLP_SYMID
 
 #define MODINFO								\
-		.modinfo (INFO) : { KEEP(*(.modinfo)) }
+		.modinfo (INFO) : { KEEP(*(.modinfo)) }	\
+		.BTF_fmodret_candidates 0 (INFO) : { KEEP(*(.BTF_fmodret_candidates)) }
 
 #ifdef CONFIG_GENERIC_BUG
 #define BUG_TABLE							\
diff --git a/scripts/Makefile.llvm-plugins b/scripts/Makefile.llvm-plugins
index e18f4907b015..9ff364f4db59 100644
--- a/scripts/Makefile.llvm-plugins
+++ b/scripts/Makefile.llvm-plugins
@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
+llvm-plugin-$(CONFIG_CLANG_PLUGIN_BPF_SCAN_IR) += bpf_scan_ir.so
+
 LLVM_PLUGIN := $(llvm-plugin-y)
 LLVM_PLUGINS_CFLAGS := $(strip $(addprefix -fpass-plugin=$(objtree)/scripts/llvm-plugins/, \
 	$(LLVM_PLUGIN)) -DLLVM_PLUGINS)
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index 01a37ec872b9..042c31e3bdfb 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -42,6 +42,7 @@ quiet_cmd_btf_ko = BTF [M] $@
       cmd_btf_ko = 							\
 	if [ ! -f $(objtree)/vmlinux ]; then				\
 		printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
+		$(OBJCOPY) --remove-section=.BTF_fmodret_candidates $@;	\
 	else	\
 		$(CONFIG_SHELL) $(srctree)/scripts/gen-btf.sh --btf_base $(objtree)/vmlinux $@; \
 	fi;
diff --git a/scripts/Makefile.vmlinux b/scripts/Makefile.vmlinux
index 95b523fe4f9e..c8651454c3d3 100644
--- a/scripts/Makefile.vmlinux
+++ b/scripts/Makefile.vmlinux
@@ -73,6 +73,9 @@ vmlinux.unstripped: scripts/link-vmlinux.sh vmlinux.o .vmlinux.export.o $(KBUILD
 ifdef CONFIG_DEBUG_INFO_BTF
 vmlinux.unstripped: $(RESOLVE_BTFIDS) $(srctree)/scripts/gen-btf.sh
 endif
+ifdef CONFIG_CLANG_PLUGIN_BPF_SCAN_IR
+vmlinux.unstripped: $(srctree)/scripts/bpf/fmodret.lds
+endif
 
 ifdef CONFIG_BUILDTIME_TABLE_SORT
 vmlinux.unstripped: scripts/sorttable
diff --git a/scripts/bpf/fmodret.lds b/scripts/bpf/fmodret.lds
new file mode 100644
index 000000000000..6bdfde9e7679
--- /dev/null
+++ b/scripts/bpf/fmodret.lds
@@ -0,0 +1,6 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+SECTIONS {
+	/DISCARD/ : {
+		*(.BTF_fmodret_candidates)
+	}
+}
diff --git a/scripts/bpf/fmodret_metadata.h b/scripts/bpf/fmodret_metadata.h
new file mode 100644
index 000000000000..209f266f89a3
--- /dev/null
+++ b/scripts/bpf/fmodret_metadata.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef FMODRET_METADATA_H
+#define FMODRET_METADATA_H
+
+/*
+ * The records which the BPF compiler plugins write to .BTF_fmodret_candidates
+ * for resolve_btfids, one for each function with values and for each weak
+ * definition. If a function has two weak definitions, the record of the one
+ * which loses the link would describe the body of the other, so both need a
+ * record to make the function ambiguous. A record is packed, target-endian and
+ * aligned to the target pointer size:
+ *
+ *	u32 magic, version, size, flags;
+ *	ptr function;
+ *	u32 nr_values, reserved;
+ *	s64 values[nr_values];
+ *
+ * The values are the errors which clean paths of the function return, sorted
+ * and unique.
+ */
+#define FMODRET_META_MAGIC 0x46524d32U
+#define FMODRET_META_VERSION 1U
+#define FMODRET_META_HEADER_SIZE(ptr_size) (24 + (ptr_size))
+#define FMODRET_META_VALUE_SIZE 8
+
+#define FMODRET_FN_WEAK (1U << 0)
+#define FMODRET_FN_RET64 (1U << 1)
+
+#endif
diff --git a/scripts/gen-btf.sh b/scripts/gen-btf.sh
index b17f1189298a..5bca864631e0 100755
--- a/scripts/gen-btf.sh
+++ b/scripts/gen-btf.sh
@@ -120,7 +120,9 @@ gen_btf_o()
 
 embed_btf_data()
 {
-	${OBJCOPY} --add-section .BTF=${ELF_FILE}.BTF ${ELF_FILE}
+	# resolve_btfids has consumed the compiler plugin records of the module
+	${OBJCOPY} --add-section .BTF=${ELF_FILE}.BTF \
+		--remove-section=.BTF_fmodret_candidates ${ELF_FILE}
 
 	# a module might not have a .BTF_ids or .BTF.base section
 	btf_base="${ELF_FILE}.BTF.base"
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index d5710e900bd4..f39e2d1f1400 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -89,6 +89,13 @@ vmlinux_link()
 		ldlibs=
 	fi
 
+	if is_enabled CONFIG_CLANG_PLUGIN_BPF_SCAN_IR; then
+		if [ "${output}" = ".tmp_vmlinux1" ]; then
+			ldflags="${ldflags} ${wl}--emit-relocs"
+		else
+			ldflags="${ldflags} ${wl}--script=${srctree}/scripts/bpf/fmodret.lds"
+		fi
+	fi
 	ldflags="${ldflags} ${wl}--script=${objtree}/${KBUILD_LDS}"
 
 	# The kallsyms linking does not need debug symbols included.
diff --git a/scripts/llvm-plugins/Kconfig b/scripts/llvm-plugins/Kconfig
index 37db99159b9c..c4d62ae5f65d 100644
--- a/scripts/llvm-plugins/Kconfig
+++ b/scripts/llvm-plugins/Kconfig
@@ -15,3 +15,19 @@ menuconfig LLVM_PLUGINS
 	  Building them requires LLVM development headers and llvm-config
 	  of the same major version as clang. Each plugin has its own option
 	  below and is built only if that option is enabled.
+
+if LLVM_PLUGINS
+
+config CLANG_PLUGIN_BPF_SCAN_IR
+	bool "Scan optimized LLVM IR for BPF"
+	depends on X86_64 || ARM64
+	depends on DEBUG_INFO_BTF
+	depends on !LTO_CLANG
+	depends on CLANG_VERSION >= 190000
+	help
+	  Find the functions which return errors on paths without side
+	  effects and describe them in .BTF_fmodret_candidates for
+	  resolve_btfids. The section is discarded from the final kernel
+	  and modules.
+
+endif
diff --git a/scripts/llvm-plugins/bpf_scan_ir.cpp b/scripts/llvm-plugins/bpf_scan_ir.cpp
new file mode 100644
index 000000000000..6700896749d4
--- /dev/null
+++ b/scripts/llvm-plugins/bpf_scan_ir.cpp
@@ -0,0 +1,126 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* bpf_scan_ir: optimizer-last LLVM IR analysis for BPF. */
+
+#include "../bpf/fmodret_metadata.h"
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/Config/llvm-config.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/GlobalVariable.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/Passes/PassBuilder.h"
+#if __has_include("llvm/Plugins/PassPlugin.h")
+#include "llvm/Plugins/PassPlugin.h"
+#else
+#include "llvm/Passes/PassPlugin.h"
+#endif
+#include "llvm/Transforms/Utils/ModuleUtils.h"
+
+using namespace llvm;
+
+namespace {
+
+/* The errors which clean paths of a function return */
+using ReturnValues = SmallVector<int64_t, 8>;
+
+struct FunctionDescription {
+	Function *Fn;
+	unsigned Flags;
+	ReturnValues Values;
+};
+
+static void emitDescriptions(Module &M, ArrayRef<FunctionDescription> Functions)
+{
+	auto *I32 = Type::getInt32Ty(M.getContext());
+	auto *I64 = Type::getInt64Ty(M.getContext());
+	unsigned PtrSize = M.getDataLayout().getPointerSize();
+	auto N = [&](unsigned V) -> Constant * { return ConstantInt::get(I32, V); };
+
+	for (const auto &F : Functions) {
+		unsigned Size = FMODRET_META_HEADER_SIZE(PtrSize) +
+				F.Values.size() * FMODRET_META_VALUE_SIZE;
+		SmallVector<Constant *, 16> Fields = {
+			N(FMODRET_META_MAGIC), N(FMODRET_META_VERSION), N(Size), N(F.Flags),
+			F.Fn, N(F.Values.size()), N(0),
+		};
+
+		for (int64_t V : F.Values)
+			Fields.push_back(ConstantInt::getSigned(I64, V));
+		Constant *Record = ConstantStruct::getAnon(M.getContext(), Fields, true);
+		auto *GV = new GlobalVariable(M, Record->getType(), true, GlobalValue::PrivateLinkage,
+					      Record, "fmodret.function." + F.Fn->getName());
+		GV->setSection(".BTF_fmodret_candidates");
+		GV->setAlignment(M.getDataLayout().getPointerABIAlignment(0));
+		appendToUsed(M, {GV});
+	}
+}
+
+#if LLVM_VERSION_MAJOR >= 24
+class BpfScanIRPass : public RequiredPassInfoMixin<BpfScanIRPass>
+#else
+class BpfScanIRPass : public PassInfoMixin<BpfScanIRPass>
+#endif
+{
+public:
+	PreservedAnalyses run(Module &M, ModuleAnalysisManager &MAM)
+	{
+		// Avoid emitting duplicate descriptions if bpf_scan_ir runs again
+		// on the same module.
+		if (M.getNamedMetadata("bpf_scan_ir.done"))
+			return PreservedAnalyses::all();
+
+		SmallVector<FunctionDescription, 32> Descriptions;
+
+		for (Function &F : M) {
+			if (F.isIntrinsic() || F.isDeclaration() || F.hasAvailableExternallyLinkage())
+				continue;
+			Type *Ret = F.getReturnType();
+			unsigned Flags = 0;
+			ReturnValues Values;
+
+			if (!Ret->isIntegerTy(32) && !Ret->isIntegerTy(64))
+				continue;
+			if (F.isWeakForLinker())
+				Flags |= FMODRET_FN_WEAK;
+			if (Ret->isIntegerTy(64))
+				Flags |= FMODRET_FN_RET64;
+			/*
+			 * Nothing finds the values yet. A weak definition always
+			 * has a record, see fmodret_metadata.h.
+			 */
+			if (!Values.empty() || (Flags & FMODRET_FN_WEAK))
+				Descriptions.push_back({&F, Flags, std::move(Values)});
+		}
+		if (Descriptions.empty())
+			return PreservedAnalyses::all();
+		emitDescriptions(M, Descriptions);
+		M.getOrInsertNamedMetadata("bpf_scan_ir.done");
+
+		PreservedAnalyses PA;
+		PA.preserve<FunctionAnalysisManagerModuleProxy>();
+		PA.preserveSet<CFGAnalyses>();
+		return PA;
+	}
+
+	static bool isRequired() { return true; }
+};
+
+} // namespace
+
+extern "C" LLVM_ATTRIBUTE_WEAK PassPluginLibraryInfo llvmGetPassPluginInfo()
+{
+	return {
+		LLVM_PLUGIN_API_VERSION,
+		"bpf_scan_ir",
+		"1.0",
+		[](PassBuilder &PB) {
+			PB.registerOptimizerLastEPCallback(
+				[](ModulePassManager &MPM, OptimizationLevel, auto...) {
+					MPM.addPass(BpfScanIRPass());
+				   }
+			);
+		}
+	};
+}
diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index 75374c64b8cc..3bae98549228 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -772,6 +772,7 @@ static const char *const section_white_list[] =
 	".llvm.call-graph-profile",	/* call graph */
 	"__llvm_covfun",
 	"__llvm_covmap",
+	".BTF_fmodret_candidates",
 	".klp.symid",			/* objtool --klp-symids */
 	NULL
 };
diff --git a/scripts/module.lds.S b/scripts/module.lds.S
index d0f200428957..e6ba499dc876 100644
--- a/scripts/module.lds.S
+++ b/scripts/module.lds.S
@@ -27,6 +27,11 @@ SECTIONS {
 	__ex_table		0 : ALIGN(4) { KEEP(*(__ex_table)) }
 
 	__patchable_function_entries 0 : { *(__patchable_function_entries) }
+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+	.BTF_fmodret_candidates	0 (INFO) : { KEEP(*(.BTF_fmodret_candidates)) }
+#else
+	/DISCARD/ : { *(.BTF_fmodret_candidates) }
+#endif
 
 	.init.klp_funcs		0 : ALIGN(8) { KEEP(*(.init.klp_funcs)) }
 	.init.klp_objects	0 : ALIGN(8) { KEEP(*(.init.klp_objects)) }
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index 931c5c412f15..e6120bee4505 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -4657,6 +4657,7 @@ static int validate_ibt(struct objtool_file *file)
 		    !strcmp(sec->name, "__jump_table")			||
 		    !strcmp(sec->name, ".init.klp_funcs")		||
 		    !strcmp(sec->name, "__mcount_loc")			||
+		    !strcmp(sec->name, ".BTF_fmodret_candidates")	||
 		    !strcmp(sec->name, ".llvm.call-graph-profile")	||
 		    !strcmp(sec->name, ".llvm_bb_addr_map")		||
 		    !strcmp(sec->name, "__tracepoints")			||
-- 
2.43.0

[PATCH v2 bpf-next 09/18] bpf: Find fmod_ret candidates in bpf_scan_ir

[Anton Protopopov]

Find the functions which can become fmod_ret targets: functions which
return a 32 or 64 bit integer and have a path from the entry without
observable side effects (a clean path) which returns an error. A
fmod_ret program which returns this error instead of calling the
function then behaves like the path. List the errors in the record of
the function, sorted and unique.

Take only the blocks which no dirty path reaches: after a dirty path
joins a clean one, a branch may depend on what the dirty path did. Only
the block of the return may also be reached by dirty paths, as it
returns the same constant on all paths.

For now an error must be a constant returned by a clean path. The
following changes find more of the errors.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 scripts/llvm-plugins/bpf_scan_ir.cpp | 117 ++++++++++++++++++++++++++-
 1 file changed, 113 insertions(+), 4 deletions(-)

diff --git a/scripts/llvm-plugins/bpf_scan_ir.cpp b/scripts/llvm-plugins/bpf_scan_ir.cpp
index 6700896749d4..6828fa73717d 100644
--- a/scripts/llvm-plugins/bpf_scan_ir.cpp
+++ b/scripts/llvm-plugins/bpf_scan_ir.cpp
@@ -3,11 +3,17 @@
 
 #include "../bpf/fmodret_metadata.h"
 
+#include <algorithm>
+#include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/Analysis.h"
+#include "llvm/IR/BasicBlock.h"
+#include "llvm/IR/CFG.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/GlobalVariable.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/Passes/PassBuilder.h"
@@ -22,9 +28,114 @@ using namespace llvm;
 
 namespace {
 
+static constexpr int MaxErrno = 4095;
+
 /* The errors which clean paths of a function return */
 using ReturnValues = SmallVector<int64_t, 8>;
 
+/* Return true and set N if V is an errno constant */
+static bool errnoConstant(const Value *V, int64_t &N)
+{
+	auto *C = dyn_cast<ConstantInt>(V);
+
+	if (!C || (C->getBitWidth() != 32 && C->getBitWidth() != 64))
+		return false;
+
+	N = C->getSExtValue();
+	return N >= -MaxErrno && N < 0;
+}
+
+static bool cleanInstruction(const Instruction &I)
+{
+	if (auto *Intrinsic = dyn_cast<IntrinsicInst>(&I)) {
+		switch (Intrinsic->getIntrinsicID()) {
+		case Intrinsic::assume:
+		case Intrinsic::lifetime_start:
+		case Intrinsic::lifetime_end:
+		case Intrinsic::invariant_start:
+		case Intrinsic::invariant_end:
+			return true;
+		default:
+			break;
+		}
+	}
+	return !I.mayHaveSideEffects();
+}
+
+static bool cleanBB(BasicBlock *BB)
+{
+	for (Instruction &I : *BB) {
+		if (!cleanInstruction(I))
+			return false;
+	}
+	return true;
+}
+
+/* Add the errors which clean paths of F are known to return to VALUES */
+static void cleanReturnValues(Function &F, ReturnValues &Values)
+{
+	SmallPtrSet<BasicBlock *, 32> CleanBBs;
+	SmallVector<BasicBlock *, 32> Worklist;
+	bool Changed;
+
+	/* The clean blocks which the entry reaches through clean blocks */
+	if (cleanBB(&F.getEntryBlock())) {
+		CleanBBs.insert(&F.getEntryBlock());
+		Worklist.push_back(&F.getEntryBlock());
+	}
+	while (!Worklist.empty()) {
+		BasicBlock *BB = Worklist.pop_back_val();
+
+		for (BasicBlock *Successor : successors(BB)) {
+			if (!CleanBBs.contains(Successor) && cleanBB(Successor)) {
+				CleanBBs.insert(Successor);
+				Worklist.push_back(Successor);
+			}
+		}
+	}
+
+	/*
+	 * Keep the blocks which only clean paths reach: after a dirty path joins
+	 * a clean one, a branch may depend on what the dirty path did
+	 */
+	do {
+		Changed = false;
+		for (BasicBlock &BB : F) {
+			if (!CleanBBs.contains(&BB))
+				continue;
+			for (BasicBlock *Pred : predecessors(&BB)) {
+				if (!CleanBBs.contains(Pred)) {
+					CleanBBs.erase(&BB);
+					Changed = true;
+					break;
+				}
+			}
+		}
+	} while (Changed);
+
+	/*
+	 * The returns of the clean blocks which only clean paths reach or which
+	 * one of them branches to. Dirty paths may also reach the latter, but
+	 * the returned constant is the same on all paths.
+	 */
+	for (BasicBlock &BB : F) {
+		auto *Return = dyn_cast<ReturnInst>(BB.getTerminator());
+		bool Entered = CleanBBs.contains(&BB);
+		int64_t N;
+
+		if (!Return || !Return->getReturnValue() || !cleanBB(&BB))
+			continue;
+		for (BasicBlock *Pred : predecessors(&BB))
+			Entered |= CleanBBs.contains(Pred);
+		if (!Entered)
+			continue;
+		if (errnoConstant(Return->getReturnValue(), N))
+			Values.push_back(N);
+	}
+	llvm::sort(Values);
+	Values.erase(std::unique(Values.begin(), Values.end()), Values.end());
+}
+
 struct FunctionDescription {
 	Function *Fn;
 	unsigned Flags;
@@ -86,10 +197,8 @@ public:
 				Flags |= FMODRET_FN_WEAK;
 			if (Ret->isIntegerTy(64))
 				Flags |= FMODRET_FN_RET64;
-			/*
-			 * Nothing finds the values yet. A weak definition always
-			 * has a record, see fmodret_metadata.h.
-			 */
+			cleanReturnValues(F, Values);
+			/* A weak definition always has a record, see fmodret_metadata.h */
 			if (!Values.empty() || (Flags & FMODRET_FN_WEAK))
 				Descriptions.push_back({&F, Flags, std::move(Values)});
 		}
-- 
2.43.0

[PATCH v2 bpf-next 10/18] bpf: Allow writes to private stack objects in bpf_scan_ir

[Anton Protopopov]

A write to a stack object of the function is not observable outside of
it as long as the object has not escaped. Let such writes and memsets
stay on a clean path. A write after the object may have been captured,
a volatile or atomic one and a write to other memory still make the
path dirty.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 scripts/llvm-plugins/bpf_scan_ir.cpp | 85 +++++++++++++++++++++-------
 1 file changed, 64 insertions(+), 21 deletions(-)

diff --git a/scripts/llvm-plugins/bpf_scan_ir.cpp b/scripts/llvm-plugins/bpf_scan_ir.cpp
index 6828fa73717d..0fc60b2130b1 100644
--- a/scripts/llvm-plugins/bpf_scan_ir.cpp
+++ b/scripts/llvm-plugins/bpf_scan_ir.cpp
@@ -4,13 +4,17 @@
 #include "../bpf/fmodret_metadata.h"
 
 #include <algorithm>
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
+#include "llvm/Analysis/CaptureTracking.h"
+#include "llvm/Analysis/ValueTracking.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/Analysis.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CFG.h"
 #include "llvm/IR/Constants.h"
+#include "llvm/IR/Dominators.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
@@ -45,41 +49,76 @@ static bool errnoConstant(const Value *V, int64_t &N)
 	return N >= -MaxErrno && N < 0;
 }
 
-static bool cleanInstruction(const Instruction &I)
-{
-	if (auto *Intrinsic = dyn_cast<IntrinsicInst>(&I)) {
-		switch (Intrinsic->getIntrinsicID()) {
-		case Intrinsic::assume:
-		case Intrinsic::lifetime_start:
-		case Intrinsic::lifetime_end:
-		case Intrinsic::invariant_start:
-		case Intrinsic::invariant_end:
-			return true;
-		default:
-			break;
+class CleanInstructions {
+	DominatorTree &DT;
+	DenseMap<const Instruction *, bool> PrivateWrites;
+
+	bool privateDestination(const Value *Pointer, const Instruction &I)
+	{
+		auto It = PrivateWrites.find(&I);
+		if (It != PrivateWrites.end())
+			return It->second;
+		auto *Allocation = dyn_cast<AllocaInst>(getUnderlyingObject(Pointer));
+		bool Private = false;
+
+		if (Allocation && Allocation->getFunction() == I.getFunction()) {
+			// A later call may receive the initialized object. Only captures
+			// that can precede this write make the write externally visible.
+#if LLVM_VERSION_MAJOR >= 21
+			Private = !PointerMayBeCapturedBefore(Allocation, true, &I, &DT, true, 256);
+#else
+			Private = !PointerMayBeCapturedBefore(Allocation, true, true, &I, &DT, true, 256);
+#endif
 		}
+		PrivateWrites[&I] = Private;
+		return Private;
 	}
-	return !I.mayHaveSideEffects();
-}
 
-static bool cleanBB(BasicBlock *BB)
+public:
+	explicit CleanInstructions(DominatorTree &DT) : DT(DT) {}
+
+	bool isClean(const Instruction &I)
+	{
+		if (auto *Store = dyn_cast<StoreInst>(&I))
+			return !Store->isVolatile() && !Store->isAtomic() &&
+			       privateDestination(Store->getPointerOperand(), I);
+		if (auto *Memory = dyn_cast<MemIntrinsic>(&I))
+			return !Memory->isVolatile() &&
+			       privateDestination(Memory->getRawDest(), I);
+		if (auto *Intrinsic = dyn_cast<IntrinsicInst>(&I)) {
+			switch (Intrinsic->getIntrinsicID()) {
+			case Intrinsic::assume:
+			case Intrinsic::lifetime_start:
+			case Intrinsic::lifetime_end:
+			case Intrinsic::invariant_start:
+			case Intrinsic::invariant_end:
+				return true;
+			default:
+				break;
+			}
+		}
+		return !I.mayHaveSideEffects();
+	}
+};
+
+static bool cleanBB(BasicBlock *BB, CleanInstructions &Clean)
 {
 	for (Instruction &I : *BB) {
-		if (!cleanInstruction(I))
+		if (!Clean.isClean(I))
 			return false;
 	}
 	return true;
 }
 
 /* Add the errors which clean paths of F are known to return to VALUES */
-static void cleanReturnValues(Function &F, ReturnValues &Values)
+static void cleanReturnValues(Function &F, CleanInstructions &Clean, ReturnValues &Values)
 {
 	SmallPtrSet<BasicBlock *, 32> CleanBBs;
 	SmallVector<BasicBlock *, 32> Worklist;
 	bool Changed;
 
 	/* The clean blocks which the entry reaches through clean blocks */
-	if (cleanBB(&F.getEntryBlock())) {
+	if (cleanBB(&F.getEntryBlock(), Clean)) {
 		CleanBBs.insert(&F.getEntryBlock());
 		Worklist.push_back(&F.getEntryBlock());
 	}
@@ -87,7 +126,7 @@ static void cleanReturnValues(Function &F, ReturnValues &Values)
 		BasicBlock *BB = Worklist.pop_back_val();
 
 		for (BasicBlock *Successor : successors(BB)) {
-			if (!CleanBBs.contains(Successor) && cleanBB(Successor)) {
+			if (!CleanBBs.contains(Successor) && cleanBB(Successor, Clean)) {
 				CleanBBs.insert(Successor);
 				Worklist.push_back(Successor);
 			}
@@ -123,7 +162,7 @@ static void cleanReturnValues(Function &F, ReturnValues &Values)
 		bool Entered = CleanBBs.contains(&BB);
 		int64_t N;
 
-		if (!Return || !Return->getReturnValue() || !cleanBB(&BB))
+		if (!Return || !Return->getReturnValue() || !cleanBB(&BB, Clean))
 			continue;
 		for (BasicBlock *Pred : predecessors(&BB))
 			Entered |= CleanBBs.contains(Pred);
@@ -183,6 +222,7 @@ public:
 			return PreservedAnalyses::all();
 
 		SmallVector<FunctionDescription, 32> Descriptions;
+		auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
 
 		for (Function &F : M) {
 			if (F.isIntrinsic() || F.isDeclaration() || F.hasAvailableExternallyLinkage())
@@ -197,7 +237,10 @@ public:
 				Flags |= FMODRET_FN_WEAK;
 			if (Ret->isIntegerTy(64))
 				Flags |= FMODRET_FN_RET64;
-			cleanReturnValues(F, Values);
+			auto &DT = FAM.getResult<DominatorTreeAnalysis>(F);
+			CleanInstructions Clean(DT);
+
+			cleanReturnValues(F, Clean, Values);
 			/* A weak definition always has a record, see fmodret_metadata.h */
 			if (!Values.empty() || (Flags & FMODRET_FN_WEAK))
 				Descriptions.push_back({&F, Flags, std::move(Values)});
-- 
2.43.0

[PATCH v2 bpf-next 11/18] bpf: Follow PHIs in bpf_scan_ir

[Anton Protopopov]

An error often reaches the return through a PHI in the block of the
return which joins it with the results of other paths, some of them
dirty. Take the incoming values of such a PHI on the edges from the
blocks which only clean paths reach. Do not follow a PHI in an earlier
block: a later branch may split its values between clean and dirty
paths.

Take also the errors which LazyValueInfo knows exactly: an incoming
value on its edge and a returned value at the return.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 scripts/llvm-plugins/bpf_scan_ir.cpp | 84 ++++++++++++++++++++++++++--
 1 file changed, 79 insertions(+), 5 deletions(-)

diff --git a/scripts/llvm-plugins/bpf_scan_ir.cpp b/scripts/llvm-plugins/bpf_scan_ir.cpp
index 0fc60b2130b1..81de8f941568 100644
--- a/scripts/llvm-plugins/bpf_scan_ir.cpp
+++ b/scripts/llvm-plugins/bpf_scan_ir.cpp
@@ -8,11 +8,13 @@
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Analysis/CaptureTracking.h"
+#include "llvm/Analysis/LazyValueInfo.h"
 #include "llvm/Analysis/ValueTracking.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/Analysis.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CFG.h"
+#include "llvm/IR/ConstantRange.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/GlobalVariable.h"
@@ -34,9 +36,40 @@ namespace {
 
 static constexpr int MaxErrno = 4095;
 
+// is Range in [-4095, 0) ?
+static bool isErrnoRange(const ConstantRange &Range)
+{
+	unsigned Width = Range.getBitWidth();
+
+	if (Width != 32 && Width != 64)
+		return false;
+
+	auto L = APInt(Width, 0) - APInt(Width, MaxErrno);
+	auto R = APInt(Width, 0);
+	return !Range.isEmptySet() && ConstantRange(L, R).contains(Range);
+}
+
 /* The errors which clean paths of a function return */
 using ReturnValues = SmallVector<int64_t, 8>;
 
+/* Return true and set N if RANGE holds a single errno */
+static bool singleErrno(const ConstantRange &Range, int64_t &N)
+{
+	const APInt *C = Range.getSingleElement();
+
+	if (!C || !isErrnoRange(Range))
+		return false;
+	N = C->getSExtValue();
+	return true;
+}
+
+/* Return true and set N if the value at use U is a single errno */
+static bool singleErrnoAtUse(const Use &U, LazyValueInfo &LVI, int64_t &N)
+{
+	return U.get()->getType()->isIntegerTy() &&
+	       singleErrno(LVI.getConstantRangeAtUse(U, false), N);
+}
+
 /* Return true and set N if V is an errno constant */
 static bool errnoConstant(const Value *V, int64_t &N)
 {
@@ -49,6 +82,42 @@ static bool errnoConstant(const Value *V, int64_t &N)
 	return N >= -MaxErrno && N < 0;
 }
 
+/*
+ * Add the errors which V, the value returned in RET, is known to be on the
+ * edges to RET from CLEANBBS to VALUES. V may be a PHI in RET. A PHI in an
+ * earlier block is not followed: a later branch may split its values between
+ * clean and dirty paths.
+ */
+static void collectErrnos(Value *V, BasicBlock *Ret, LazyValueInfo &LVI,
+			  const SmallPtrSetImpl<BasicBlock *> &CleanBBs, ReturnValues &Values)
+{
+	int64_t N;
+
+	auto *Phi = dyn_cast<PHINode>(V);
+
+	if (!Phi || Phi->getParent() != Ret)
+		return;
+	for (unsigned i = 0; i < Phi->getNumIncomingValues(); i++) {
+		Value *Incoming = Phi->getIncomingValue(i);
+		BasicBlock *Pred = Phi->getIncomingBlock(i);
+
+		if (!CleanBBs.contains(Pred))
+			continue;
+		if (errnoConstant(Incoming, N)) {
+			Values.push_back(N);
+			continue;
+		}
+		/*
+		 * LazyValueInfo knows the ranges of integers only. On an edge it
+		 * also uses the branch condition, as in "if (err == -EINVAL) goto
+		 * out", where err is the error.
+		 */
+		if (Incoming->getType()->isIntegerTy() &&
+		    singleErrno(LVI.getConstantRangeOnEdge(Incoming, Pred, Ret), N))
+			Values.push_back(N);
+	}
+}
+
 class CleanInstructions {
 	DominatorTree &DT;
 	DenseMap<const Instruction *, bool> PrivateWrites;
@@ -111,7 +180,8 @@ static bool cleanBB(BasicBlock *BB, CleanInstructions &Clean)
 }
 
 /* Add the errors which clean paths of F are known to return to VALUES */
-static void cleanReturnValues(Function &F, CleanInstructions &Clean, ReturnValues &Values)
+static void cleanReturnValues(Function &F, LazyValueInfo &LVI, CleanInstructions &Clean,
+			      ReturnValues &Values)
 {
 	SmallPtrSet<BasicBlock *, 32> CleanBBs;
 	SmallVector<BasicBlock *, 32> Worklist;
@@ -154,8 +224,7 @@ static void cleanReturnValues(Function &F, CleanInstructions &Clean, ReturnValue
 
 	/*
 	 * The returns of the clean blocks which only clean paths reach or which
-	 * one of them branches to. Dirty paths may also reach the latter, but
-	 * the returned constant is the same on all paths.
+	 * one of them branches to
 	 */
 	for (BasicBlock &BB : F) {
 		auto *Return = dyn_cast<ReturnInst>(BB.getTerminator());
@@ -168,8 +237,11 @@ static void cleanReturnValues(Function &F, CleanInstructions &Clean, ReturnValue
 			Entered |= CleanBBs.contains(Pred);
 		if (!Entered)
 			continue;
-		if (errnoConstant(Return->getReturnValue(), N))
+		if (singleErrnoAtUse(Return->getOperandUse(0), LVI, N)) {
 			Values.push_back(N);
+			continue;
+		}
+		collectErrnos(Return->getReturnValue(), &BB, LVI, CleanBBs, Values);
 	}
 	llvm::sort(Values);
 	Values.erase(std::unique(Values.begin(), Values.end()), Values.end());
@@ -237,10 +309,11 @@ public:
 				Flags |= FMODRET_FN_WEAK;
 			if (Ret->isIntegerTy(64))
 				Flags |= FMODRET_FN_RET64;
+			auto &LVI = FAM.getResult<LazyValueAnalysis>(F);
 			auto &DT = FAM.getResult<DominatorTreeAnalysis>(F);
 			CleanInstructions Clean(DT);
 
-			cleanReturnValues(F, Clean, Values);
+			cleanReturnValues(F, LVI, Clean, Values);
 			/* A weak definition always has a record, see fmodret_metadata.h */
 			if (!Values.empty() || (Flags & FMODRET_FN_WEAK))
 				Descriptions.push_back({&F, Flags, std::move(Values)});
@@ -253,6 +326,7 @@ public:
 		PreservedAnalyses PA;
 		PA.preserve<FunctionAnalysisManagerModuleProxy>();
 		PA.preserveSet<CFGAnalyses>();
+		PA.preserve<LazyValueAnalysis>();
 		return PA;
 	}
 
-- 
2.43.0

[PATCH v2 bpf-next 12/18] bpf: Follow integer casts in bpf_scan_ir

[Anton Protopopov]

Follow sign extensions and truncations between integers of at least 32
bits in the block of the return. They keep an error, for example an int
error which a function returns as long.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 scripts/llvm-plugins/bpf_scan_ir.cpp | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/scripts/llvm-plugins/bpf_scan_ir.cpp b/scripts/llvm-plugins/bpf_scan_ir.cpp
index 81de8f941568..a3c6336662fc 100644
--- a/scripts/llvm-plugins/bpf_scan_ir.cpp
+++ b/scripts/llvm-plugins/bpf_scan_ir.cpp
@@ -82,17 +82,39 @@ static bool errnoConstant(const Value *V, int64_t &N)
 	return N >= -MaxErrno && N < 0;
 }
 
+/* A sign extension or a truncation between integers of at least 32 bits */
+static bool preservesErrnoCast(const CastInst *Cast)
+{
+	Type *Src = Cast->getSrcTy(), *Dst = Cast->getDestTy();
+
+	if (Src->isIntegerTy() && Dst->isIntegerTy())
+		return (Cast->getOpcode() == Instruction::SExt ||
+			Cast->getOpcode() == Instruction::Trunc) &&
+		       Src->getIntegerBitWidth() >= 32 && Dst->getIntegerBitWidth() >= 32;
+	return false;
+}
+
 /*
  * Add the errors which V, the value returned in RET, is known to be on the
- * edges to RET from CLEANBBS to VALUES. V may be a PHI in RET. A PHI in an
- * earlier block is not followed: a later branch may split its values between
- * clean and dirty paths.
+ * edges to RET from CLEANBBS to VALUES. Follow the casts in RET which keep the
+ * error to a PHI in RET. A PHI in an earlier block is not followed: a later
+ * branch may split its values between clean and dirty paths.
  */
 static void collectErrnos(Value *V, BasicBlock *Ret, LazyValueInfo &LVI,
 			  const SmallPtrSetImpl<BasicBlock *> &CleanBBs, ReturnValues &Values)
 {
 	int64_t N;
 
+	while (auto *Cast = dyn_cast<CastInst>(V)) {
+		if (Cast->getParent() != Ret || !preservesErrnoCast(Cast))
+			return;
+		if (singleErrnoAtUse(Cast->getOperandUse(0), LVI, N)) {
+			Values.push_back(N);
+			return;
+		}
+		V = Cast->getOperand(0);
+	}
+
 	auto *Phi = dyn_cast<PHINode>(V);
 
 	if (!Phi || Phi->getParent() != Ret)
-- 
2.43.0

[PATCH v2 bpf-next 13/18] bpf: Check the return values of fmod_ret programs on found targets

[Anton Protopopov]

A later patch lets resolve_btfids make the functions found by bpf_scan_ir
fmod_ret targets. Each of them gets a "bpf_fmodret_retval:" decl tag with
the values it returns on paths without side effects, for example
"bpf_fmodret_retval:-22,-12". Returning such a value instead of calling
the function looks to the callers like a real failure.

Read the tag when an fmod_ret program is loaded and check at every exit
that the program returns 0 or one of these values.

A function which the static rules allow keeps today's behaviour. So do
the functions declared in BTF_FMODRET sets, which have no such tag.

Only a kernel function has such a tag, so do not look for it when the
target is a BPF program. Remember the values in the program, and do not
let an extension replace an fmod_ret program with restricted return
values: it could return any value. Check this at load and when an
extension is attached to another program later. Do not allow tail calls
in such a program either: the program it jumps to returns for it, and a
program array does not check the values it may return.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 include/linux/bpf.h   |   6 ++
 kernel/bpf/syscall.c  |   1 +
 kernel/bpf/verifier.c | 140 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 147 insertions(+)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 520fd72e21fd..56a5e4b9f138 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1969,6 +1969,12 @@ struct bpf_prog_aux {
 	const struct btf_type *attach_func_proto;
 	/* function name for valid attach_btf_id */
 	const char *attach_func_name;
+	/*
+	 * The values which an fmod_ret program may return besides 0, from the
+	 * "bpf_fmodret_retval:" tag of its target, or NULL
+	 */
+	s64 *fmodret_retvals;
+	u32 fmodret_retval_cnt;
 	struct bpf_prog **func;
 	struct bpf_prog_aux *main_prog_aux;
 	/* main prog: there when the prog has timed may_goto */
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 9004f2673ccf..0a78f66a1622 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2490,6 +2490,7 @@ static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred)
 	module_put(prog->aux->mod);
 	kfree(prog->aux->kfunc_tab);
 	kfree(prog->aux->ctx_arg_info);
+	kfree(prog->aux->fmodret_retvals);
 	if (prog->aux->attach_btf)
 		btf_put(prog->aux->attach_btf);
 
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 20b1b9ff574c..f3476b12c58a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13561,6 +13561,14 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 	}
 
 	if (func_id == BPF_FUNC_tail_call) {
+		/*
+		 * The program which a tail call jumps to returns for this one,
+		 * and a program array does not check the values it may return.
+		 */
+		if (env->prog->aux->fmodret_retvals) {
+			verbose(env, "tail calls are not allowed in fmod_ret programs with restricted return values\n");
+			return -EINVAL;
+		}
 		if (env->cur_state->curframe) {
 			struct bpf_verifier_state *branch;
 
@@ -20260,6 +20268,67 @@ static bool program_returns_void(struct bpf_verifier_env *env)
 	return false;
 }
 
+/*
+ * A function which the build found by analysing its code has a
+ * "bpf_fmodret_retval:" decl tag with the values it returns on paths without
+ * side effects, for example "bpf_fmodret_retval:-22,-12". See the comment at
+ * the top of tools/bpf/resolve_btfids/main.c. An fmod_ret program attached to
+ * it may return only these values, or 0 to call the function.
+ */
+#define BTF_FMODRET_RETVAL_TAG "bpf_fmodret_retval:"
+
+/* Parse the next value of the list at *LIST and move *LIST past it */
+static int fmodret_retval_next(const char **list, s64 *value)
+{
+	const char *end = strchrnul(*list, ',');
+	size_t len = end - *list;
+	char buf[24];
+
+	if (!len || len >= sizeof(buf) || (*end && !end[1]))
+		return -EINVAL;
+	memcpy(buf, *list, len);
+	buf[len] = 0;
+	*list = *end ? end + 1 : end;
+	return kstrtos64(buf, 10, value);
+}
+
+static bool fmodret_retval_allowed(const struct bpf_prog_aux *aux, u64 value)
+{
+	const struct btf_type *t;
+	u32 i;
+
+	/* "w0 = -22" zero-extends, the caller of an int function sees -22 */
+	t = btf_type_skip_modifiers(aux->attach_btf, aux->attach_func_proto->type, NULL);
+	if (t->size == 4 && !upper_32_bits(value))
+		value = (s32)value;
+	if (!value)
+		return true;
+	for (i = 0; i < aux->fmodret_retval_cnt; i++) {
+		if (aux->fmodret_retvals[i] == (s64)value)
+			return true;
+	}
+	return false;
+}
+
+/* The program must return a constant: 0 or one of the values of the tag */
+static int check_fmodret_retval(struct bpf_verifier_env *env, int regno,
+				const char *exit_ctx, const char *reg_name)
+{
+	const struct bpf_prog_aux *aux = env->prog->aux;
+	struct bpf_reg_state *reg = reg_state(env, regno);
+	u32 i;
+
+	if (reg->type == SCALAR_VALUE && tnum_is_const(reg->var_off) &&
+	    fmodret_retval_allowed(aux, reg->var_off.value))
+		return 0;
+
+	verbose(env, "%s the register %s must be a constant, it should be 0 or one of", exit_ctx, reg_name);
+	for (i = 0; i < aux->fmodret_retval_cnt; i++)
+		verbose(env, "%s%lld", i ? "," : " ", aux->fmodret_retvals[i]);
+	verbose(env, "\n");
+	return -EINVAL;
+}
+
 static int check_return_code(struct bpf_verifier_env *env, int regno, const char *reg_name)
 {
 	const char *exit_ctx = "At program exit";
@@ -20316,6 +20385,9 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	if (prog_type == BPF_PROG_TYPE_CGROUP_SKB && (env->prog->expected_attach_type == BPF_CGROUP_INET_EGRESS))
 		enforce_attach_type_range = tnum_range(2, 3);
 
+	if (env->prog->aux->fmodret_retvals)
+		return check_fmodret_retval(env, regno, exit_ctx, reg_name);
+
 	if (!return_retval_range(env, &range))
 		return 0;
 
@@ -23759,6 +23831,58 @@ static int check_attach_modify_return(unsigned long addr, const char *func_name)
 
 #endif /* CONFIG_FUNCTION_ERROR_INJECTION */
 
+/*
+ * Programs attached to a function found by the analysis may return only the
+ * values of its "bpf_fmodret_retval:" tag. A function which the static rules
+ * allow stays unrestricted.
+ */
+static int fmodret_retval_init(struct bpf_verifier_env *env, u32 btf_id,
+			       unsigned long addr, const char *tname)
+{
+	struct bpf_prog_aux *aux = env->prog->aux;
+	const struct btf *btf = aux->attach_btf;
+	const struct btf_type *t = btf_type_by_id(btf, btf_id);
+	s64 *retvals = NULL;
+	const char *values, *list;
+	u32 cnt, size = 0;
+	s64 v;
+
+	if (!check_attach_modify_return(addr, tname))
+		return 0;
+	values = btf_find_decl_tag_value(btf, t, -1, BTF_FMODRET_RETVAL_TAG);
+	if (IS_ERR(values))
+		return PTR_ERR(values) == -ENOENT ? 0 : PTR_ERR(values);
+
+	t = btf_type_by_id(btf, t->type);
+	t = t ? btf_type_skip_modifiers(btf, t->type, NULL) : NULL;
+	if (t && (btf_type_is_int(t) || btf_is_any_enum(t)))
+		size = t->size;
+	if (size != 4 && size != 8)
+		goto invalid;
+
+	for (list = values, cnt = 1; *list; list++)
+		cnt += *list == ',';
+	retvals = kmalloc_array(cnt, sizeof(*retvals), GFP_KERNEL_ACCOUNT);
+	if (!retvals)
+		return -ENOMEM;
+	for (list = values, cnt = 0; *list;) {
+		if (fmodret_retval_next(&list, &v) || v >= 0 || (size == 4 && v < S32_MIN))
+			goto invalid;
+		retvals[cnt++] = v;
+	}
+	if (!cnt)
+		goto invalid;
+
+	aux->fmodret_retvals = retvals;
+	aux->fmodret_retval_cnt = cnt;
+	return 0;
+
+invalid:
+	kfree(retvals);
+	verbose(env, "invalid %s%s tag of %s()\n", BTF_FMODRET_RETVAL_TAG, values, tname);
+	return -EINVAL;
+}
+
 static bool is_tracing_multi_id(const struct bpf_prog *prog, u32 btf_id)
 {
 	return is_tracing_multi(prog->expected_attach_type) && bpf_multi_func_btf_id[0] == btf_id;
@@ -24173,6 +24297,15 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 			bpf_log(log, "Cannot extend fentry/fexit/fsession\n");
 			return -EINVAL;
 		}
+		/*
+		 * An extension could return any value for an fmod_ret program
+		 * whose return values are restricted. This is also checked when
+		 * an extension is attached to another program than at load.
+		 */
+		if (prog_extension && tgt_prog->aux->fmodret_retvals) {
+			bpf_log(log, "Cannot extend an fmod_ret program with restricted return values\n");
+			return -EINVAL;
+		}
 	} else {
 		if (prog_extension) {
 			bpf_log(log, "Cannot replace kernel functions\n");
@@ -24552,6 +24685,13 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
 		return -EINVAL;
 	}
 
+	/* Only a kernel function has a "bpf_fmodret_retval:" tag */
+	if (prog->expected_attach_type == BPF_MODIFY_RETURN && !tgt_prog) {
+		ret = fmodret_retval_init(env, btf_id, tgt_info.tgt_addr, tgt_info.tgt_name);
+		if (ret)
+			return ret;
+	}
+
 	/*
 	 * We don't get trampoline for tracing_multi programs at this point,
 	 * it's done when tracing_multi link is created.
-- 
2.43.0

[PATCH v2 bpf-next 14/18] bpf: Require the JIT for fmod_ret programs on found 64-bit targets

[Anton Protopopov]

The interpreter returns only the low 32 bits of the return value of a
program, while the trampoline passes all 64 bits to the caller of the
function. If an fmod_ret program on a found target which returns long
is interpreted, then a valid return value like -14 reaches the caller
as 4294967282. This value is not 0, so the function is skipped, and it
is not an error either.

Require the JIT for such programs, as for other features which the
interpreter does not support. The caller of a function which returns int
reads only the low 32 bits, so nothing changes for it.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 kernel/bpf/verifier.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index f3476b12c58a..5045d9fa68b9 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -23875,6 +23875,9 @@ static int fmodret_retval_init(struct bpf_verifier_env *env, u32 btf_id,
 
 	aux->fmodret_retvals = retvals;
 	aux->fmodret_retval_cnt = cnt;
+	/* The interpreter returns only 32 bits */
+	if (size == 8)
+		env->prog->jit_required = 1;
 	return 0;
 
 invalid:
-- 
2.43.0

[PATCH v2 bpf-next 15/18] resolve_btfids: Tag fmod_ret candidates as targets

[Anton Protopopov]

Read the records which bpf_scan_ir writes to .BTF_fmodret_candidates and
choose the functions which can become fmod_ret targets. Map each record
to its function through its relocation in a module or through its
address in vmlinux. A record of a strong definition wins over the
records of the weak definitions it overrides. A strong definition
without values has no record, so use the record of a weak definition
only if the symbol of the function is still weak after the link. Every
weak definition has a record, so a function with two weak definitions is
ambiguous.

Choose a function if the record lists at least one error, the function
has an ftrace call site, is not init, exit or noinstr code, its name
belongs to exactly one ELF function and one BTF FUNC and it returns an
integer of the size the record describes.

Tag each chosen function with "bpf_fmodret" and with
"bpf_fmodret_retval:" followed by the errors it returns on paths without
side effects, for example "bpf_fmodret_retval:-22,-12". The values are
signed decimal numbers, sorted and separated by commas. The verifier
lets programs attached to such a function return only these values or 0.

A function in a BTF_FMODRET set keeps only its own tags.

Mark the range of patchable function entries in the linker script so the
ftrace call sites in vmlinux can be told from the mcount ones.
Take the ftrace call sites of a module from the section the kernel uses:
x86 also has call padding entries in __patchable_function_entries.

With -v, report what was found.

Add the new files to the BPF [COMPILER PLUGINS] entry in MAINTAINERS.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 MAINTAINERS                        |    1 +
 include/asm-generic/vmlinux.lds.h  |    2 +
 scripts/Makefile.btf               |    1 +
 tools/bpf/resolve_btfids/Build     |    1 +
 tools/bpf/resolve_btfids/fmodret.c | 1205 ++++++++++++++++++++++++++++
 tools/bpf/resolve_btfids/fmodret.h |   28 +
 tools/bpf/resolve_btfids/main.c    |   64 ++
 7 files changed, 1302 insertions(+)
 create mode 100644 tools/bpf/resolve_btfids/fmodret.c
 create mode 100644 tools/bpf/resolve_btfids/fmodret.h

diff --git a/MAINTAINERS b/MAINTAINERS
index dfbb45ecc716..29940e8d7205 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -5070,6 +5070,7 @@ L:	bpf@vger.kernel.org
 S:	Maintained
 F:	scripts/bpf/
 F:	scripts/llvm-plugins/bpf_*
+F:	tools/bpf/resolve_btfids/fmodret.*
 
 BPF [CORE]
 M:	Alexei Starovoitov <ast@kernel.org>
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 4e18758b5ffc..77e86ff2f9a4 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -198,7 +198,9 @@
 #define MCOUNT_REC()	. = ALIGN(8);				\
 			__start_mcount_loc = .;			\
 			KEEP(*(__mcount_loc))			\
+			__start_patchable_function_entries = .;	\
 			KEEP_PATCHABLE				\
+			__stop_patchable_function_entries = .;	\
 			__stop_mcount_loc = .;			\
 			FTRACE_STUB_HACK			\
 			ftrace_ops_list_func = arch_ftrace_ops_list_func;
diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
index 98012344a81e..3aa3d084cc6c 100644
--- a/scripts/Makefile.btf
+++ b/scripts/Makefile.btf
@@ -35,5 +35,6 @@ resolve-btfids-flags-y :=
 resolve-btfids-flags-$(CONFIG_WERROR) += --fatal_warnings
 resolve-btfids-flags-$(if $(KBUILD_EXTMOD),y) += --distill_base
 resolve-btfids-flags-$(if $(KBUILD_VERBOSE),y) += --verbose
+resolve-btfids-flags-$(CONFIG_FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY) += --patchable_ftrace
 
 export RESOLVE_BTFIDS_FLAGS := $(resolve-btfids-flags-y)
diff --git a/tools/bpf/resolve_btfids/Build b/tools/bpf/resolve_btfids/Build
index 602e1508bf09..af777afa451b 100644
--- a/tools/bpf/resolve_btfids/Build
+++ b/tools/bpf/resolve_btfids/Build
@@ -1,6 +1,7 @@
 hostprogs := resolve_btfids
 
 resolve_btfids-y += main.o
+resolve_btfids-y += fmodret.o
 resolve_btfids-y += btf_colors.o
 resolve_btfids-y += rbtree.o
 resolve_btfids-y += zalloc.o
diff --git a/tools/bpf/resolve_btfids/fmodret.c b/tools/bpf/resolve_btfids/fmodret.c
new file mode 100644
index 000000000000..df44b93eea73
--- /dev/null
+++ b/tools/bpf/resolve_btfids/fmodret.c
@@ -0,0 +1,1205 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Read the records which the bpf_scan_ir compiler plugin writes to
+ * .BTF_fmodret_candidates, see scripts/llvm-plugins/bpf_scan_ir.cpp, and find
+ * the functions which can become fmod_ret targets. A record lists the errors
+ * which clean paths of a function return, that is paths without side effects.
+ *
+ * A function is chosen if
+ *  - exactly one record describes its body, or one record of a strong
+ *    definition next to records of the weak ones it overrides,
+ *  - the record is of a strong definition or the symbol is still weak,
+ *  - the record lists at least one error,
+ *  - it has an ftrace call site and is not init, exit or noinstr code,
+ *  - its name belongs to exactly one ELF function and one BTF FUNC,
+ *  - it returns an integer of the size the record describes.
+ *
+ * The plugin records no errors for functions which return pointers yet.
+ */
+
+#include <errno.h>
+#include <gelf.h>
+#include <libelf.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <bpf/btf.h>
+#include <linux/kernel.h>
+
+#include "fmodret.h"
+#include "../../../scripts/bpf/fmodret_metadata.h"
+
+#define FMODRET_CANDIDATES_SECTION ".BTF_fmodret_candidates"
+#define NO_FUNCTION SIZE_MAX
+
+struct elf_function {
+	const char *name;
+	GElf_Addr value;
+	GElf_Xword size;
+	Elf64_Section shndx;
+	bool traceable;
+	/* The symbol is weak: no strong definition replaced it at the link */
+	bool weak_symbol;
+	/* The records which describe the body at this address */
+	unsigned int nr_strong, nr_weak;
+	size_t strong, weak;
+};
+
+struct elf_function_list {
+	struct elf_function *functions;
+	size_t nr;
+	size_t capacity;
+};
+
+struct fmodret_stats {
+	size_t records, unmapped, with_values;
+	size_t ambiguous_body, not_traceable, ambiguous_name;
+	size_t overridden, no_btf, return_mismatch, added;
+};
+
+static int find_section(Elf *elf, size_t shstrndx, const char *wanted,
+			Elf_Scn **result)
+{
+	Elf_Scn *scn = NULL;
+
+	while ((scn = elf_nextscn(elf, scn))) {
+		GElf_Shdr shdr;
+		const char *name;
+
+		if (!gelf_getshdr(scn, &shdr))
+			return -1;
+		name = elf_strptr(elf, shstrndx, shdr.sh_name);
+		if (!name)
+			return -1;
+		if (!strcmp(name, wanted)) {
+			*result = scn;
+			return 0;
+		}
+	}
+
+	*result = NULL;
+	return 0;
+}
+
+static int find_symbol(Elf *elf, const char *wanted, GElf_Sym *result,
+		       bool *found)
+{
+	Elf_Scn *scn = NULL;
+
+	*found = false;
+	while ((scn = elf_nextscn(elf, scn))) {
+		GElf_Shdr shdr;
+		Elf_Data *data;
+		size_t nr_symbols;
+		size_t i;
+
+		if (!gelf_getshdr(scn, &shdr))
+			return -1;
+		if (shdr.sh_type != SHT_SYMTAB)
+			continue;
+		if (!shdr.sh_entsize || shdr.sh_size % shdr.sh_entsize)
+			return -EINVAL;
+		data = elf_getdata(scn, NULL);
+		if (!data || elf_getdata(scn, data))
+			return -1;
+		nr_symbols = shdr.sh_size / shdr.sh_entsize;
+		for (i = 0; i < nr_symbols; i++) {
+			const char *name;
+			GElf_Sym sym;
+
+			if (!gelf_getsym(data, i, &sym))
+				return -1;
+			if (sym.st_shndx == SHN_UNDEF || !sym.st_name)
+				continue;
+			name = elf_strptr(elf, shdr.sh_link, sym.st_name);
+			if (!name)
+				return -1;
+			if (strcmp(name, wanted))
+				continue;
+			*result = sym;
+			*found = true;
+			return 0;
+		}
+	}
+	return 0;
+}
+
+static int section_for_address(Elf *elf, GElf_Addr address,
+			       Elf64_Section *result)
+{
+	Elf_Scn *scn = NULL;
+
+	while ((scn = elf_nextscn(elf, scn))) {
+		GElf_Shdr shdr;
+
+		if (!gelf_getshdr(scn, &shdr))
+			return -1;
+		if (!(shdr.sh_flags & SHF_EXECINSTR) || address < shdr.sh_addr ||
+		    address - shdr.sh_addr >= shdr.sh_size)
+			continue;
+		*result = elf_ndxscn(scn);
+		return 0;
+	}
+	*result = SHN_UNDEF;
+	return 0;
+}
+
+static bool relocation_offset(const GElf_Shdr *target, GElf_Addr offset,
+			      size_t width, size_t *result)
+{
+	if (target->sh_size < width)
+		return false;
+	if (offset >= target->sh_addr &&
+	    offset - target->sh_addr <= target->sh_size - width) {
+		*result = offset - target->sh_addr;
+		return true;
+	}
+	/* ET_REL relocation offsets are relative to their target section. */
+	if (offset <= target->sh_size - width) {
+		*result = offset;
+		return true;
+	}
+	return false;
+}
+
+static uint64_t read_elf_value(const unsigned char *p, size_t size,
+			       unsigned int encoding)
+{
+	uint64_t value = 0;
+	size_t i;
+
+	if (encoding == ELFDATA2LSB) {
+		for (i = 0; i < size; i++)
+			value |= (uint64_t)p[i] << (8 * i);
+	} else {
+		for (i = 0; i < size; i++)
+			value = (value << 8) | p[i];
+	}
+	return value;
+}
+
+static int copy_section_data(Elf_Scn *scn, const GElf_Shdr *shdr,
+			     unsigned char **result)
+{
+	Elf_Data *data = NULL;
+	unsigned char *buf;
+
+	buf = calloc(1, shdr->sh_size ?: 1);
+	if (!buf)
+		return -ENOMEM;
+	while ((data = elf_getdata(scn, data))) {
+		if (data->d_off > shdr->sh_size ||
+		    data->d_size > shdr->sh_size - data->d_off) {
+			free(buf);
+			return -EINVAL;
+		}
+		memcpy(buf + data->d_off, data->d_buf, data->d_size);
+	}
+	*result = buf;
+	return 0;
+}
+
+static int add_elf_function(struct elf_function_list *list, const char *name,
+			    const GElf_Sym *sym)
+{
+	struct elf_function *functions;
+	size_t capacity;
+
+	if (list->nr == list->capacity) {
+		capacity = list->capacity ? 2 * list->capacity : 1024;
+		if (capacity > SIZE_MAX / sizeof(*functions))
+			return -E2BIG;
+		functions = realloc(list->functions, capacity * sizeof(*functions));
+		if (!functions)
+			return -ENOMEM;
+		list->functions = functions;
+		list->capacity = capacity;
+	}
+
+	list->functions[list->nr++] = (struct elf_function) {
+		.name = name,
+		.value = sym->st_value,
+		.size = sym->st_size,
+		.shndx = sym->st_shndx,
+		.weak_symbol = GELF_ST_BIND(sym->st_info) == STB_WEAK,
+		.strong = NO_FUNCTION,
+		.weak = NO_FUNCTION,
+	};
+	return 0;
+}
+
+static int collect_elf_functions(Elf *elf, struct elf_function_list *list)
+{
+	Elf_Scn *scn = NULL;
+
+	while ((scn = elf_nextscn(elf, scn))) {
+		GElf_Shdr shdr;
+		Elf_Data *data;
+		size_t nr_symbols;
+		size_t i;
+
+		if (!gelf_getshdr(scn, &shdr))
+			return -1;
+		if (shdr.sh_type != SHT_SYMTAB)
+			continue;
+		if (!shdr.sh_entsize || shdr.sh_size % shdr.sh_entsize)
+			return -EINVAL;
+		data = elf_getdata(scn, NULL);
+		if (!data || elf_getdata(scn, data))
+			return -1;
+
+		nr_symbols = shdr.sh_size / shdr.sh_entsize;
+		for (i = 0; i < nr_symbols; i++) {
+			const char *name;
+			GElf_Sym sym;
+			int err;
+
+			if (!gelf_getsym(data, i, &sym))
+				return -1;
+			if (GELF_ST_TYPE(sym.st_info) != STT_FUNC ||
+			    sym.st_shndx == SHN_UNDEF || !sym.st_name)
+				continue;
+			name = elf_strptr(elf, shdr.sh_link, sym.st_name);
+			if (!name)
+				return -1;
+			err = add_elf_function(list, name, &sym);
+			if (err)
+				return err;
+		}
+	}
+	return 0;
+}
+
+static int cmp_elf_function_location(const void *va, const void *vb)
+{
+	const struct elf_function *a = va;
+	const struct elf_function *b = vb;
+
+	if (a->shndx != b->shndx)
+		return a->shndx > b->shndx ? 1 : -1;
+	if (a->value != b->value)
+		return a->value > b->value ? 1 : -1;
+	return 0;
+}
+
+static int cmp_elf_function_name(const void *va, const void *vb)
+{
+	const struct elf_function *a = va;
+	const struct elf_function *b = vb;
+
+	return strcmp(a->name, b->name) ?: cmp_elf_function_location(va, vb);
+}
+
+/* Return the first function at VALUE in SHNDX. Functions are sorted by location. */
+static size_t function_at(const struct elf_function_list *list,
+			  Elf64_Section shndx, GElf_Addr value)
+{
+	size_t lo = 0, hi = list->nr;
+
+	while (lo < hi) {
+		size_t mid = lo + (hi - lo) / 2;
+		const struct elf_function *f = &list->functions[mid];
+
+		if (f->shndx < shndx || (f->shndx == shndx && f->value < value))
+			lo = mid + 1;
+		else
+			hi = mid;
+	}
+	if (lo == list->nr || list->functions[lo].shndx != shndx ||
+	    list->functions[lo].value != value)
+		return NO_FUNCTION;
+	return lo;
+}
+
+static bool function_contains(const struct elf_function *function,
+			      GElf_Addr address)
+{
+	if (address < function->value)
+		return false;
+	if (!function->size)
+		return address == function->value;
+	return address - function->value < function->size;
+}
+
+enum ftrace_site_kind {
+	FTRACE_SITE_MCOUNT,
+	FTRACE_SITE_PATCHABLE,
+	FTRACE_SITE_UNKNOWN,
+};
+
+/*
+ * The NOPs which arm64 puts before a function with CALL_OPS. From the entry of
+ * an overridden weak function, whose body has no symbol left, a longer distance
+ * could reach the next function. Other arches may need more.
+ */
+#define MAX_PATCHABLE_FUNCTION_ENTRY_SIZE 8
+
+/* Return true if function I or another function at its address contains ADDRESS */
+static bool alias_contains(const struct elf_function_list *list, size_t i,
+			   GElf_Addr address)
+{
+	const struct elf_function *f = &list->functions[i];
+
+	for (;;) {
+		if (function_contains(&list->functions[i], address))
+			return true;
+		if (!i || list->functions[i - 1].shndx != f->shndx ||
+		    list->functions[i - 1].value != f->value)
+			return false;
+		i--;
+	}
+}
+
+/*
+ * Mark the functions at (SHNDX, VALUE) traceable, or, with CHECK, only those
+ * which contain ADDRESS.
+ */
+static void mark_aliases(struct elf_function_list *list, Elf64_Section shndx,
+			 GElf_Addr value, GElf_Addr address, bool check)
+{
+	size_t first = function_at(list, shndx, value), i;
+
+	if (first == NO_FUNCTION)
+		return;
+	for (i = first; i < list->nr && list->functions[i].shndx == shndx &&
+	     list->functions[i].value == value; i++) {
+		if (check && !function_contains(&list->functions[i], address))
+			continue;
+		list->functions[i].traceable = true;
+	}
+}
+
+/*
+ * Mark the function containing an ftrace call site. Functions are sorted by
+ * section and address. Mark all aliases at the same address: at run time BTF
+ * attachment is name based and may resolve to any of them.
+ */
+static void mark_traceable_function(struct elf_function_list *list,
+				    Elf64_Section shndx, GElf_Addr address,
+				    enum ftrace_site_kind kind)
+{
+	size_t left = 0;
+	size_t right = list->nr;
+	size_t first;
+
+	/* Find the first function after ADDRESS in SHNDX. */
+	while (left < right) {
+		size_t middle = left + (right - left) / 2;
+		struct elf_function *function = &list->functions[middle];
+
+		if (function->shndx < shndx ||
+		    (function->shndx == shndx && function->value <= address))
+			left = middle + 1;
+		else
+			right = middle;
+	}
+	/*
+	 * A patchable entry is at the start of its function, a few bytes into it
+	 * (after an arm64 BTI) or in the NOPs before it
+	 */
+	first = SIZE_MAX;
+	if (kind != FTRACE_SITE_MCOUNT) {
+		if (left && list->functions[left - 1].shndx == shndx &&
+		    address - list->functions[left - 1].value <
+					MAX_PATCHABLE_FUNCTION_ENTRY_SIZE &&
+		    alias_contains(list, left - 1, address))
+			first = left - 1;
+		else if (left < list->nr && list->functions[left].shndx == shndx &&
+			 list->functions[left].value - address <=
+					MAX_PATCHABLE_FUNCTION_ENTRY_SIZE)
+			first = left;
+	}
+	if (first != SIZE_MAX) {
+		mark_aliases(list, shndx, list->functions[first].value, address, false);
+		return;
+	}
+	if (kind == FTRACE_SITE_PATCHABLE || !left)
+		return;
+
+	/* The nearest preceding function owns the call site. */
+	first = left - 1;
+	if (list->functions[first].shndx != shndx)
+		return;
+	mark_aliases(list, shndx, list->functions[first].value, address, true);
+}
+
+static int mark_relocated_ftrace_sites(Elf *elf, size_t target_ndx,
+				       const unsigned char *target_data,
+				       const GElf_Shdr *target_shdr,
+				       size_t begin, size_t end,
+				       size_t ptr_size, unsigned int encoding,
+				       struct elf_function_list *functions,
+				       enum ftrace_site_kind kind,
+				       bool *relocated)
+{
+	Elf_Scn *scn = NULL;
+
+	*relocated = false;
+	while ((scn = elf_nextscn(elf, scn))) {
+		GElf_Shdr shdr;
+		Elf_Data *data;
+		size_t nr_relocations;
+		size_t i;
+
+		if (!gelf_getshdr(scn, &shdr))
+			return -1;
+		if ((shdr.sh_type != SHT_RELA && shdr.sh_type != SHT_REL) ||
+		    (shdr.sh_info && shdr.sh_info != target_ndx))
+			continue;
+		if (!shdr.sh_entsize || shdr.sh_size % shdr.sh_entsize)
+			return -EINVAL;
+		data = elf_getdata(scn, NULL);
+		if (!data || elf_getdata(scn, data))
+			return -1;
+
+		nr_relocations = shdr.sh_size / shdr.sh_entsize;
+		for (i = 0; i < nr_relocations; i++) {
+			Elf_Scn *symtab_scn;
+			Elf_Data *symtab_data;
+			GElf_Shdr symtab_shdr;
+			GElf_Sxword addend;
+			GElf_Addr raw_offset;
+			GElf_Addr address;
+			GElf_Sym sym;
+			size_t sym_ndx;
+			size_t offset;
+
+			if (shdr.sh_type == SHT_RELA) {
+				GElf_Rela rela;
+
+				if (!gelf_getrela(data, i, &rela))
+					return -1;
+				sym_ndx = GELF_R_SYM(rela.r_info);
+				raw_offset = rela.r_offset;
+				addend = rela.r_addend;
+			} else {
+				GElf_Rel rel;
+
+				if (!gelf_getrel(data, i, &rel))
+					return -1;
+				sym_ndx = GELF_R_SYM(rel.r_info);
+				raw_offset = rel.r_offset;
+				if (!relocation_offset(target_shdr, raw_offset,
+						       ptr_size, &offset))
+					continue;
+				addend = read_elf_value(target_data + offset,
+							ptr_size, encoding);
+			}
+			if (!relocation_offset(target_shdr, raw_offset, ptr_size,
+					       &offset))
+				continue;
+			if (offset < begin || offset >= end)
+				continue;
+			*relocated = true;
+
+			/*
+			 * PIE vmlinux uses symbol-less RELATIVE relocations. Its
+			 * addend is the link-time call-site address, represented as
+			 * a signed ELF addend.
+			 */
+			if (!sym_ndx) {
+				Elf64_Section target_shndx;
+
+				address = (GElf_Addr)addend;
+				if (section_for_address(elf, address,
+							&target_shndx))
+					return -1;
+				if (target_shndx != SHN_UNDEF)
+					mark_traceable_function(functions,
+								target_shndx,
+								address, kind);
+				continue;
+			}
+
+			symtab_scn = elf_getscn(elf, shdr.sh_link);
+			if (!symtab_scn ||
+			    !gelf_getshdr(symtab_scn, &symtab_shdr))
+				return -1;
+			symtab_data = elf_getdata(symtab_scn, NULL);
+			if (!symtab_data || elf_getdata(symtab_scn, symtab_data) ||
+			    !gelf_getsym(symtab_data, sym_ndx, &sym))
+				return -1;
+			if (sym.st_shndx == SHN_UNDEF)
+				continue;
+			if (sym.st_shndx >= SHN_LORESERVE)
+				continue;
+			if (addend < 0) {
+				GElf_Addr delta = -(addend + 1);
+
+				delta++;
+				if (delta > sym.st_value)
+					continue;
+				address = sym.st_value - delta;
+			} else {
+				if ((GElf_Addr)addend > UINT64_MAX - sym.st_value)
+					continue;
+				address = sym.st_value + addend;
+			}
+			mark_traceable_function(functions, sym.st_shndx, address,
+						kind);
+		}
+	}
+	return 0;
+}
+
+static int read_ftrace_site_range(Elf *elf, Elf_Scn *scn, size_t begin,
+				  size_t end, size_t ptr_size,
+				  unsigned int encoding,
+				  struct elf_function_list *functions,
+				  enum ftrace_site_kind kind)
+{
+	unsigned char *buf = NULL;
+	GElf_Shdr shdr;
+	bool relocated;
+	size_t offset;
+	int err;
+
+	if (!gelf_getshdr(scn, &shdr))
+		return -1;
+	if (begin > end || end > shdr.sh_size ||
+	    (end - begin) % ptr_size)
+		return -EINVAL;
+	if (begin == end)
+		return 0;
+	err = copy_section_data(scn, &shdr, &buf);
+	if (err)
+		return err;
+	err = mark_relocated_ftrace_sites(elf, elf_ndxscn(scn), buf, &shdr,
+					  begin, end, ptr_size, encoding,
+					  functions, kind, &relocated);
+	if (err || relocated)
+		goto out;
+
+	for (offset = begin; offset < end; offset += ptr_size) {
+		GElf_Addr address;
+		Elf64_Section shndx;
+
+		address = read_elf_value(buf + offset, ptr_size, encoding);
+		err = section_for_address(elf, address, &shndx);
+		if (err)
+			goto out;
+		if (shndx != SHN_UNDEF)
+			mark_traceable_function(functions, shndx, address, kind);
+	}
+out:
+	free(buf);
+	return err;
+}
+
+static bool section_prefix(const char *name, const char *prefix)
+{
+	size_t len = strlen(prefix);
+
+	return !strncmp(name, prefix, len) &&
+	       (!name[len] || name[len] == '.');
+}
+
+static bool section_cannot_be_attached(const char *name)
+{
+	return section_prefix(name, ".init.text") ||
+	       section_prefix(name, ".exit.text") ||
+	       section_prefix(name, ".noinstr.text");
+}
+
+static int remove_unattachable_functions(Elf *elf,
+					 struct elf_function_list *functions,
+					 size_t shstrndx)
+{
+	GElf_Sym noinstr_start;
+	GElf_Sym noinstr_end;
+	bool found_start;
+	bool found_end;
+	size_t i;
+	int err;
+
+	err = find_symbol(elf, "__noinstr_text_start", &noinstr_start,
+			  &found_start);
+	if (err)
+		return err;
+	err = find_symbol(elf, "__noinstr_text_end", &noinstr_end, &found_end);
+	if (err)
+		return err;
+
+	for (i = 0; i < functions->nr; i++) {
+		struct elf_function *function = &functions->functions[i];
+		Elf_Scn *scn;
+		GElf_Shdr shdr;
+		const char *name;
+
+		if (!function->traceable)
+			continue;
+		scn = elf_getscn(elf, function->shndx);
+		if (!scn || !gelf_getshdr(scn, &shdr))
+			return -1;
+		name = elf_strptr(elf, shstrndx, shdr.sh_name);
+		if (!name)
+			return -1;
+		if (section_cannot_be_attached(name))
+			function->traceable = false;
+		else if (found_start && found_end &&
+			 function->shndx == noinstr_start.st_shndx &&
+			 noinstr_start.st_shndx == noinstr_end.st_shndx &&
+			 function->value >= noinstr_start.st_value &&
+			 function->value < noinstr_end.st_value)
+			function->traceable = false;
+	}
+	return 0;
+}
+
+static int read_ftrace_symbol_range(Elf *elf, const GElf_Sym *start,
+				    const GElf_Sym *stop, size_t ptr_size,
+				    unsigned int encoding,
+				    struct elf_function_list *functions,
+				    enum ftrace_site_kind kind)
+{
+	Elf_Scn *scn;
+	GElf_Shdr shdr;
+	size_t begin;
+	size_t end;
+
+	if (start->st_shndx != stop->st_shndx)
+		return -EINVAL;
+	scn = elf_getscn(elf, start->st_shndx);
+	if (!scn || !gelf_getshdr(scn, &shdr))
+		return -1;
+	if (start->st_value < shdr.sh_addr || stop->st_value < start->st_value ||
+	    stop->st_value - shdr.sh_addr > shdr.sh_size)
+		return -EINVAL;
+	begin = start->st_value - shdr.sh_addr;
+	end = stop->st_value - shdr.sh_addr;
+	return read_ftrace_site_range(elf, scn, begin, end, ptr_size, encoding,
+				       functions, kind);
+}
+
+/*
+ * Mark the functions with an ftrace call site. A module keeps the sites in the
+ * section FTRACE_CALLSITE_SECTION: __patchable_function_entries if PATCHABLE,
+ * else __mcount_loc. x86 also has call padding entries, which are not call
+ * sites, in __patchable_function_entries. vmlinux merges the sites into init
+ * data between the __start_mcount_loc and __stop_mcount_loc symbols, see
+ * MCOUNT_REC().
+ */
+static int read_ftrace_sites(Elf *elf, size_t shstrndx,
+			     size_t ptr_size, unsigned int encoding,
+			     struct elf_function_list *functions, bool patchable)
+{
+	const char *section = patchable ? "__patchable_function_entries" : "__mcount_loc";
+	Elf_Scn *scn = NULL;
+	GElf_Sym start;
+	GElf_Sym stop;
+	GElf_Sym patchable_start;
+	GElf_Sym patchable_stop;
+	bool found_start;
+	bool found_stop;
+	bool found_patchable_start;
+	bool found_patchable_stop;
+	GElf_Shdr shdr;
+	int err;
+
+	err = find_section(elf, shstrndx, section, &scn);
+	if (err)
+		return err;
+	if (scn) {
+		if (!gelf_getshdr(scn, &shdr))
+			return -1;
+		err = read_ftrace_site_range(elf, scn, 0, shdr.sh_size,
+					     ptr_size, encoding, functions,
+					     patchable ? FTRACE_SITE_PATCHABLE :
+							 FTRACE_SITE_MCOUNT);
+		if (err)
+			return err;
+	}
+
+	err = find_symbol(elf, "__start_mcount_loc", &start, &found_start);
+	if (err)
+		return err;
+	err = find_symbol(elf, "__stop_mcount_loc", &stop, &found_stop);
+	if (err)
+		return err;
+	err = find_symbol(elf, "__start_patchable_function_entries",
+			  &patchable_start, &found_patchable_start);
+	if (err)
+		return err;
+	err = find_symbol(elf, "__stop_patchable_function_entries",
+			  &patchable_stop, &found_patchable_stop);
+	if (err)
+		return err;
+	if (found_start && found_stop) {
+		if (found_patchable_start && found_patchable_stop) {
+			err = read_ftrace_symbol_range(elf, &start,
+						       &patchable_start,
+						       ptr_size, encoding,
+						       functions,
+						       FTRACE_SITE_MCOUNT);
+			if (err)
+				return err;
+			err = read_ftrace_symbol_range(elf, &patchable_start,
+						       &patchable_stop,
+						       ptr_size, encoding,
+						       functions,
+						       FTRACE_SITE_PATCHABLE);
+		} else {
+			err = read_ftrace_symbol_range(elf, &start, &stop,
+						       ptr_size, encoding,
+						       functions,
+						       FTRACE_SITE_UNKNOWN);
+		}
+		if (err)
+			return err;
+	}
+
+	return remove_unattachable_functions(elf, functions, shstrndx);
+}
+
+/*
+ * Find the function each record points to. In a module the pointer is a
+ * relocation, in a linked image it is the address. REFS maps each pointer
+ * sized slot of the section to a function.
+ */
+static int resolve_record_relocations(Elf *elf, const struct elf_function_list *functions,
+				      size_t metadata_ndx, const GElf_Shdr *metadata_shdr,
+				      const unsigned char *metadata, size_t *refs,
+				      size_t ptr_size, unsigned int encoding)
+{
+	Elf_Scn *scn = NULL;
+
+	while ((scn = elf_nextscn(elf, scn))) {
+		Elf_Data *data, *symbols;
+		Elf_Scn *symtab;
+		GElf_Shdr shdr;
+		size_t i;
+
+		if (!gelf_getshdr(scn, &shdr))
+			return -1;
+		if ((shdr.sh_type != SHT_RELA && shdr.sh_type != SHT_REL) ||
+		    shdr.sh_info != metadata_ndx)
+			continue;
+		if (!shdr.sh_entsize || shdr.sh_size % shdr.sh_entsize)
+			return -EINVAL;
+		data = elf_getdata(scn, NULL);
+		symtab = elf_getscn(elf, shdr.sh_link);
+		symbols = symtab ? elf_getdata(symtab, NULL) : NULL;
+		if (!data || !symbols || elf_getdata(scn, data))
+			return -1;
+		for (i = 0; i < shdr.sh_size / shdr.sh_entsize; i++) {
+			GElf_Rela rela;
+			GElf_Sym sym;
+			size_t offset;
+
+			if (shdr.sh_type == SHT_RELA) {
+				if (!gelf_getrela(data, i, &rela))
+					return -1;
+			} else {
+				GElf_Rel rel;
+
+				if (!gelf_getrel(data, i, &rel))
+					return -1;
+				rela = (GElf_Rela){ .r_offset = rel.r_offset,
+						    .r_info = rel.r_info };
+				/* REL addends are stored in the pointer itself. */
+				if (!relocation_offset(metadata_shdr, rel.r_offset, ptr_size,
+						       &offset))
+					return -EINVAL;
+				rela.r_addend = read_elf_value(metadata + offset, ptr_size,
+							       encoding);
+			}
+			if (!relocation_offset(metadata_shdr, rela.r_offset, ptr_size, &offset) ||
+			    offset % ptr_size)
+				return -EINVAL;
+			if (!gelf_getsym(symbols, GELF_R_SYM(rela.r_info), &sym))
+				return -1;
+			if (sym.st_shndx == SHN_UNDEF || sym.st_shndx >= SHN_LORESERVE)
+				continue;
+			refs[offset / ptr_size] = function_at(functions, sym.st_shndx,
+							      sym.st_value + rela.r_addend);
+		}
+	}
+	return 0;
+}
+
+static size_t record_function(Elf *elf, const struct elf_function_list *functions,
+			      const unsigned char *buf, const size_t *refs,
+			      size_t offset, size_t ptr_size, unsigned int encoding)
+{
+	Elf64_Section section;
+	GElf_Ehdr ehdr;
+	GElf_Addr value;
+
+	if (refs[offset / ptr_size] != NO_FUNCTION)
+		return refs[offset / ptr_size];
+	if (!gelf_getehdr(elf, &ehdr) || ehdr.e_type == ET_REL)
+		return NO_FUNCTION;
+	value = read_elf_value(buf + offset, ptr_size, encoding);
+	if (!value || section_for_address(elf, value, &section))
+		return NO_FUNCTION;
+	return function_at(functions, section, value);
+}
+
+/* Remember the records of a body for all the functions at its address */
+static void add_record(struct elf_function_list *functions, size_t first,
+		       size_t offset, bool weak)
+{
+	struct elf_function *f = &functions->functions[first];
+	size_t i;
+
+	for (i = first; i < functions->nr &&
+	     !cmp_elf_function_location(f, &functions->functions[i]); i++) {
+		if (weak) {
+			functions->functions[i].nr_weak++;
+			functions->functions[i].weak = offset;
+		} else {
+			functions->functions[i].nr_strong++;
+			functions->functions[i].strong = offset;
+		}
+	}
+}
+
+static int parse_records(Elf *elf, struct elf_function_list *functions,
+			 const unsigned char *buf, size_t size, const size_t *refs,
+			 size_t ptr_size, unsigned int encoding,
+			 struct fmodret_stats *stats)
+{
+	size_t offset = 0;
+
+	while (offset < size) {
+		unsigned int flags, length, nr_values;
+		size_t function;
+
+		if (size - offset < FMODRET_META_HEADER_SIZE(ptr_size) ||
+		    read_elf_value(buf + offset, 4, encoding) != FMODRET_META_MAGIC ||
+		    read_elf_value(buf + offset + 4, 4, encoding) != FMODRET_META_VERSION)
+			goto invalid;
+		length = read_elf_value(buf + offset + 8, 4, encoding);
+		flags = read_elf_value(buf + offset + 12, 4, encoding);
+		nr_values = read_elf_value(buf + offset + 16 + ptr_size, 4, encoding);
+		if (length < FMODRET_META_HEADER_SIZE(ptr_size) || length > size - offset ||
+		    length % ptr_size ||
+		    nr_values != (length - FMODRET_META_HEADER_SIZE(ptr_size)) /
+				 FMODRET_META_VALUE_SIZE ||
+		    (length - FMODRET_META_HEADER_SIZE(ptr_size)) % FMODRET_META_VALUE_SIZE)
+			goto invalid;
+		stats->records++;
+		function = record_function(elf, functions, buf, refs, offset + 16,
+					   ptr_size, encoding);
+		/* A function in a discarded section has no ELF function */
+		if (function == NO_FUNCTION)
+			stats->unmapped++;
+		else
+			add_record(functions, function, offset, flags & FMODRET_FN_WEAK);
+		offset += length;
+	}
+	return 0;
+invalid:
+	fprintf(stderr,
+		"invalid or old fmodret metadata at offset %#zx; rebuild all plugin objects\n",
+		offset);
+	return -EINVAL;
+}
+
+/* Return the record which describes the body of F, or NO_FUNCTION */
+static size_t chosen_record(const struct elf_function *f)
+{
+	if (f->nr_strong == 1)
+		return f->strong;
+	if (!f->nr_strong && f->nr_weak == 1)
+		return f->weak;
+	return NO_FUNCTION;
+}
+
+struct btf_func {
+	const char *name;
+	__u32 id;
+};
+
+static int cmp_btf_func(const void *a, const void *b)
+{
+	const struct btf_func *x = a, *y = b;
+
+	return strcmp(x->name, y->name);
+}
+
+/*
+ * Return the FUNCs of BTF, without those of its base, sorted by name. The BTF
+ * is sorted only after the tags are added, and searching all of it for each
+ * candidate is slow for vmlinux.
+ */
+static struct btf_func *collect_btf_funcs(const struct btf *btf, size_t *cnt)
+{
+	const struct btf *base = btf__base_btf(btf);
+	__u32 id, nr_types = btf__type_cnt(btf);
+	struct btf_func *funcs;
+	size_t n = 0;
+
+	funcs = malloc(nr_types * sizeof(*funcs));
+	if (!funcs)
+		return NULL;
+	for (id = base ? btf__type_cnt(base) : 1; id < nr_types; id++) {
+		const struct btf_type *t = btf__type_by_id(btf, id);
+
+		if (!btf_is_func(t))
+			continue;
+		funcs[n].name = btf__name_by_offset(btf, t->name_off);
+		funcs[n++].id = id;
+	}
+	qsort(funcs, n, sizeof(*funcs), cmp_btf_func);
+	*cnt = n;
+	return funcs;
+}
+
+/* Return the BTF ID of the only FUNC named NAME, or 0 */
+static int find_btf_func(const struct btf_func *funcs, size_t cnt, const char *name)
+{
+	const struct btf_func key = { .name = name }, *f;
+
+	f = bsearch(&key, funcs, cnt, sizeof(*funcs), cmp_btf_func);
+	if (!f)
+		return 0;
+	if ((f > funcs && !strcmp(f[-1].name, name)) ||
+	    (f + 1 < funcs + cnt && !strcmp(f[1].name, name)))
+		return 0;
+	return f->id;
+}
+
+/* Return the size of the integer which FUNC returns, or 0 */
+static __u32 btf_func_int_return_size(const struct btf *btf, int func)
+{
+	const struct btf_type *t = btf__type_by_id(btf, func);
+	int ret;
+
+	t = btf__type_by_id(btf, t->type);
+	if (!t || !btf_is_func_proto(t) || !t->type)
+		return 0;
+	ret = btf__resolve_type(btf, t->type);
+	if (ret < 0)
+		return 0;
+	t = btf__type_by_id(btf, ret);
+	if (!btf_is_int(t) && !btf_is_enum(t) && !btf_is_enum64(t))
+		return 0;
+	return t->size;
+}
+
+/* Format the values of the record at OFFSET as "-22,-12" */
+static char *format_values(const unsigned char *buf, size_t offset, size_t ptr_size,
+			   unsigned int encoding)
+{
+	unsigned int nr = read_elf_value(buf + offset + 16 + ptr_size, 4, encoding);
+	const unsigned char *p = buf + offset + FMODRET_META_HEADER_SIZE(ptr_size);
+	size_t size = (size_t)nr * 21 + 1, len = 0;
+	char *s = malloc(size);
+	unsigned int i;
+
+	if (!s)
+		return NULL;
+	for (i = 0; i < nr; i++) {
+		int64_t value = read_elf_value(p + i * FMODRET_META_VALUE_SIZE,
+					       FMODRET_META_VALUE_SIZE, encoding);
+
+		len += snprintf(s + len, size - len, "%s%lld", i ? "," : "",
+				(long long)value);
+	}
+	s[len] = 0;
+	return s;
+}
+
+static int cmp_retval(const void *va, const void *vb)
+{
+	const struct fmodret_retval *a = va;
+	const struct fmodret_retval *b = vb;
+
+	return a->btf_id > b->btf_id ? 1 : a->btf_id < b->btf_id ? -1 : 0;
+}
+
+/* Choose the targets among the functions and describe them in RESULT */
+static int choose_functions(struct elf_function_list *functions, const struct btf *btf,
+			    const struct btf_func *btf_funcs, size_t nr_btf_funcs,
+			    const unsigned char *buf, size_t ptr_size,
+			    unsigned int encoding, struct fmodret_retvals *result,
+			    struct fmodret_stats *stats, int verbose)
+{
+	size_t i = 0, j;
+
+	qsort(functions->functions, functions->nr, sizeof(*functions->functions),
+	      cmp_elf_function_name);
+	for (i = 0; i < functions->nr; i = j) {
+		const struct elf_function *f = &functions->functions[i];
+		size_t record = chosen_record(f);
+		const char *reason = NULL;
+		unsigned int flags = 0;
+		struct fmodret_retval *r;
+		__u32 size;
+		int id = 0;
+
+		for (j = i + 1; j < functions->nr; j++)
+			if (strcmp(f->name, functions->functions[j].name))
+				break;
+
+		if (!f->nr_strong && !f->nr_weak)
+			continue;
+		if (record == NO_FUNCTION) {
+			stats->ambiguous_body++;
+			reason = "ambiguous-body";
+			goto rejected;
+		}
+		/*
+		 * A strong definition without values has no record. If it
+		 * replaced a weak one, only the record of the weak one points
+		 * to its body.
+		 */
+		if (!f->nr_strong && !f->weak_symbol) {
+			stats->overridden++;
+			reason = "overridden-weak";
+			goto rejected;
+		}
+		if (!read_elf_value(buf + record + 16 + ptr_size, 4, encoding))
+			continue;
+		stats->with_values++;
+		flags = read_elf_value(buf + record + 12, 4, encoding);
+		if (j - i != 1) {
+			stats->ambiguous_name++;
+			reason = "ambiguous-name";
+			goto rejected;
+		}
+		if (!f->traceable) {
+			stats->not_traceable++;
+			reason = "not-traceable";
+			goto rejected;
+		}
+		id = find_btf_func(btf_funcs, nr_btf_funcs, f->name);
+		if (id <= 0) {
+			stats->no_btf++;
+			reason = "no-unique-BTF-FUNC";
+			goto rejected;
+		}
+		size = btf_func_int_return_size(btf, id);
+		if (size != (flags & FMODRET_FN_RET64 ? 8 : 4)) {
+			stats->return_mismatch++;
+			reason = "BTF-return-type";
+		}
+rejected:
+		if (reason) {
+			if (verbose > 1)
+				fprintf(stderr, "fmodret: rejected %s: %s\n", f->name, reason);
+			continue;
+		}
+
+		if (result->cnt == result->cap) {
+			size_t cap = result->cap ? 2 * result->cap : 256;
+
+			r = realloc(result->retvals, cap * sizeof(*r));
+			if (!r)
+				return -ENOMEM;
+			result->retvals = r;
+			result->cap = cap;
+		}
+		r = &result->retvals[result->cnt];
+		r->btf_id = id;
+		r->values = format_values(buf, record, ptr_size, encoding);
+		if (!r->values)
+			return -ENOMEM;
+		result->cnt++;
+		stats->added++;
+		if (verbose > 1)
+			fprintf(stderr, "fmodret: %s: BTF ID %d, retval %s\n",
+				f->name, id, r->values);
+	}
+	if (result->cnt)
+		qsort(result->retvals, result->cnt, sizeof(*result->retvals), cmp_retval);
+	return 0;
+}
+
+int fmodret_collect(Elf *elf, const struct btf *btf, struct fmodret_retvals *result,
+		    bool patchable_ftrace, int verbose)
+{
+	struct elf_function_list functions = {};
+	struct btf_func *btf_funcs = NULL;
+	struct fmodret_stats stats = {};
+	unsigned char *buf = NULL;
+	size_t nr_btf_funcs;
+	size_t *refs = NULL;
+	size_t shstrndx, ptr_size, i;
+	unsigned int encoding;
+	Elf_Scn *scn;
+	GElf_Shdr shdr;
+	int err;
+
+	if (elf_getshdrstrndx(elf, &shstrndx))
+		return -EINVAL;
+	err = find_section(elf, shstrndx, FMODRET_CANDIDATES_SECTION, &scn);
+	if (err || !scn)
+		return err;
+	if (!gelf_getshdr(scn, &shdr))
+		return -EINVAL;
+	ptr_size = gelf_getclass(elf) == ELFCLASS64 ? 8 : 4;
+	encoding = elf_getident(elf, NULL)[EI_DATA];
+	if (shdr.sh_size % ptr_size)
+		return -EINVAL;
+
+	err = collect_elf_functions(elf, &functions);
+	if (err)
+		goto out;
+	qsort(functions.functions, functions.nr, sizeof(*functions.functions),
+	      cmp_elf_function_location);
+
+	err = copy_section_data(scn, &shdr, &buf);
+	if (err)
+		goto out;
+	refs = malloc((shdr.sh_size / ptr_size ?: 1) * sizeof(*refs));
+	if (!refs) {
+		err = -ENOMEM;
+		goto out;
+	}
+	for (i = 0; i < shdr.sh_size / ptr_size; i++)
+		refs[i] = NO_FUNCTION;
+	err = resolve_record_relocations(elf, &functions, elf_ndxscn(scn), &shdr, buf,
+					 refs, ptr_size, encoding);
+	if (err)
+		goto out;
+	err = parse_records(elf, &functions, buf, shdr.sh_size, refs, ptr_size,
+			    encoding, &stats);
+	if (err)
+		goto out;
+	err = read_ftrace_sites(elf, shstrndx, ptr_size, encoding, &functions,
+				patchable_ftrace);
+	if (err)
+		goto out;
+	btf_funcs = collect_btf_funcs(btf, &nr_btf_funcs);
+	if (!btf_funcs) {
+		err = -ENOMEM;
+		goto out;
+	}
+	err = choose_functions(&functions, btf, btf_funcs, nr_btf_funcs, buf, ptr_size,
+			       encoding, result, &stats, verbose);
+	if (err)
+		goto out;
+
+	if (verbose)
+		fprintf(stderr,
+			"fmodret: %zu records (%zu without a function), %zu functions with values, %zu added; rejected: %zu ambiguous body, %zu ambiguous name, %zu not traceable, %zu overridden weak, %zu without unique BTF FUNC, %zu BTF return type\n",
+			stats.records, stats.unmapped, stats.with_values, stats.added,
+			stats.ambiguous_body,
+			stats.ambiguous_name, stats.not_traceable, stats.overridden,
+			stats.no_btf, stats.return_mismatch);
+out:
+	if (err == -1)
+		fprintf(stderr, "failed to read %s: %s\n", FMODRET_CANDIDATES_SECTION,
+			elf_errmsg(-1));
+	free(btf_funcs);
+	free(refs);
+	free(buf);
+	free(functions.functions);
+	return err;
+}
+
+void fmodret_free(struct fmodret_retvals *retvals)
+{
+	size_t i;
+
+	for (i = 0; i < retvals->cnt; i++)
+		free(retvals->retvals[i].values);
+	free(retvals->retvals);
+	retvals->retvals = NULL;
+	retvals->cnt = retvals->cap = 0;
+}
diff --git a/tools/bpf/resolve_btfids/fmodret.h b/tools/bpf/resolve_btfids/fmodret.h
new file mode 100644
index 000000000000..c1d5db64970b
--- /dev/null
+++ b/tools/bpf/resolve_btfids/fmodret.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __RESOLVE_BTFIDS_FMODRET_H
+#define __RESOLVE_BTFIDS_FMODRET_H
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <libelf.h>
+#include <linux/types.h>
+
+struct btf;
+
+/* A function found by the analysis, and the values it may return */
+struct fmodret_retval {
+	__u32 btf_id;
+	char *values;
+};
+
+/* Sorted by BTF ID */
+struct fmodret_retvals {
+	struct fmodret_retval *retvals;
+	size_t cnt, cap;
+};
+
+int fmodret_collect(Elf *elf, const struct btf *btf, struct fmodret_retvals *result,
+		    bool patchable_ftrace, int verbose);
+void fmodret_free(struct fmodret_retvals *retvals);
+
+#endif
diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c
index e0fb7525a5ea..124c4f01c887 100644
--- a/tools/bpf/resolve_btfids/main.c
+++ b/tools/bpf/resolve_btfids/main.c
@@ -75,6 +75,12 @@
  * and also "bpf_fmodret_sleepable" when KF_SLEEPABLE is set, so the sleepable
  * tag never comes alone. The kernel builds the set of fmod_ret targets from
  * these tags when it loads BTF.
+ *
+ * Functions which the analysis of the compiler plugin records, see fmodret.c,
+ * get "bpf_fmodret" too, and "bpf_fmodret_retval:" with the values they
+ * return on paths without side effects, such as "bpf_fmodret_retval:-22,-12".
+ * The verifier lets programs attached to them return only these values or 0.
+ * A function in a BTF_FMODRET set keeps only its own tags.
  */
 
 #define  _GNU_SOURCE
@@ -96,6 +102,7 @@
 #include <bpf/btf.h>
 #include <bpf/libbpf.h>
 #include "btf_colors.h"
+#include "fmodret.h"
 #include <subcmd/parse-options.h>
 
 #define BTF_IDS_SECTION	".BTF_ids"
@@ -154,6 +161,11 @@ struct object {
 	struct btf *orig_btf;
 	bool distill_base;
 	bool extract_inline;
+	/* The ftrace call sites are in __patchable_function_entries */
+	bool patchable_ftrace;
+
+	/* Functions found by the analysis of the compiler plugin, see fmodret.c */
+	struct fmodret_retvals fmodret_retvals;
 
 	struct {
 		int		 fd;
@@ -205,6 +217,7 @@ struct btf_links {
 #define DECL_TAG_KFUNC "bpf_kfunc"
 #define DECL_TAG_FMODRET "bpf_fmodret"
 #define DECL_TAG_FMODRET_SLEEPABLE "bpf_fmodret_sleepable"
+#define DECL_TAG_FMODRET_RETVAL "bpf_fmodret_retval:"
 
 #define KF_SLEEPABLE	(1 << 5)
 #define KF_FASTCALL	(1 << 12)
@@ -1792,6 +1805,40 @@ static int add_decl_tag(struct btf2btf_context *ctx, const char *tag_name,
 	return push_decl_tag_id(ctx, new_id);
 }
 
+static bool has_set_func(const struct rb_root *root, u32 btf_id)
+{
+	const struct rb_node *node = root->rb_node;
+
+	while (node) {
+		const struct kfunc *k = rb_entry(node, struct kfunc, rb_node);
+
+		if (btf_id < k->btf_id)
+			node = node->rb_left;
+		else if (btf_id > k->btf_id)
+			node = node->rb_right;
+		else
+			return true;
+	}
+	return false;
+}
+
+static int add_fmodret_retval_tags(struct btf2btf_context *ctx,
+				   const struct fmodret_retval *retval)
+{
+	char *tag;
+	int err;
+
+	err = add_decl_tag(ctx, DECL_TAG_FMODRET, retval->btf_id, -1);
+	if (err)
+		return err;
+
+	if (asprintf(&tag, "%s%s", DECL_TAG_FMODRET_RETVAL, retval->values) < 0)
+		return -ENOMEM;
+	err = add_decl_tag(ctx, tag, retval->btf_id, -1);
+	free(tag);
+	return err;
+}
+
 static int btf2btf(struct object *obj)
 {
 	struct btf2btf_context ctx = {};
@@ -1841,6 +1888,16 @@ static int btf2btf(struct object *obj)
 		}
 	}
 
+	for (size_t i = 0; i < obj->fmodret_retvals.cnt; i++) {
+		const struct fmodret_retval *retval = &obj->fmodret_retvals.retvals[i];
+
+		if (has_set_func(&ctx.fmodrets, retval->btf_id))
+			continue;
+		err = add_fmodret_retval_tags(&ctx, retval);
+		if (err)
+			goto out;
+	}
+
 	err = 0;
 out:
 	free(ctx.decl_tags);
@@ -2194,6 +2251,8 @@ int main(int argc, const char **argv)
 			    "distill --btf_base and emit .BTF.base section data"),
 		OPT_BOOLEAN(0, "inline", &obj.extract_inline,
 			    "extract location BTF into a .BTF.inline file"),
+		OPT_BOOLEAN(0, "patchable_ftrace", &obj.patchable_ftrace,
+			    "ftrace call sites are in __patchable_function_entries"),
 		OPT_CALLBACK(0, "btf_link", &btf_links, "section:module:btf-file",
 			     "patch a BTF link with --patch_btfids", parse_btf_link),
 		OPT_STRING(0, "patch_btfids", &btfids_path, "file",
@@ -2252,6 +2311,10 @@ int main(int argc, const char **argv)
 	if (load_btf(&obj))
 		goto out;
 
+	if (fmodret_collect(obj.efile.elf, obj.btf, &obj.fmodret_retvals,
+			    obj.patchable_ftrace, verbose))
+		goto out;
+
 	if (btf2btf(&obj))
 		goto out;
 
@@ -2309,6 +2372,7 @@ int main(int argc, const char **argv)
 	btf_id__free_all(&obj.funcs);
 	btf_id__free_all(&obj.sets);
 	free(obj.addr_syms);
+	fmodret_free(&obj.fmodret_retvals);
 	free(obj.efile.func_symbols);
 	if (obj.efile.elf) {
 		elf_end(obj.efile.elf);
-- 
2.43.0

[PATCH v2 bpf-next 16/18] selftests/bpf: Test fmod_ret candidates found by bpf_scan_ir

[Anton Protopopov]

Add bpf_testmod_retval.c with functions for bpf_scan_ir to a new module
bpf_test_retval.ko. The "test_retval:" decl tag of each lists the values
which bpf_scan_ir must find. An empty list means that the function must
not become an fmod_ret target. Cover constant errors of both return
widths, assumptions, private stack writes, PHIs, sign extensions,
PTR_ERR() of an ERR_PTR() from an inlined helper, which is not
recognized yet, errors which are not known exactly, observable effects,
a notrace function, a function on the error injection list, a declared
target, a weak definition with an error which a strong one without
errors in bpf_testmod_retval_strong.c replaces and two weak definitions,
of which the one with an error in bpf_testmod_retval_strong.c loses the
link.

Build both files only with the plugin: their functions mean nothing
without it and bpf_testmod_retval.c uses clang extensions such as
__builtin_assume(). The module itself is always built so the selftests
can install it.

The functions are not in bpf_testmod.ko because kbuild first links a
module built from several objects with "ld -r" and GNU ld places
.data..percpu after .debug_info there. pahole then attributes the
per-CPU variables of bpf_testmod to .debug_info and drops them from its
BTF. The functions can be merged back into bpf_testmod.ko once pahole
looks up variables only in allocated sections.

Check that the tags in the module BTF match. Check that the verifier
lets fmod_ret programs return only 0 or the listed values on every path,
in the low 32 bits for an int function and in all 64 bits for an s64
one. Declared targets and the static rules stay unrestricted. Check that
an extension cannot replace a function of an fmod_ret program with
restricted return values, neither at load nor when it is attached later,
and that such a program cannot use tail calls.

Skip the test when bpf_test_retval has no "test_retval:" tags, that is
when it is built without the plugin.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../selftests/bpf/prog_tests/fmodret_retval.c | 211 ++++++++++++
 .../selftests/bpf/progs/fmodret_retval.c      | 196 +++++++++++
 .../selftests/bpf/progs/fmodret_retval_ext.c  |  53 +++
 .../testing/selftests/bpf/test_kmods/Makefile |  15 +-
 .../bpf/test_kmods/bpf_test_retval_main.c     |  12 +
 .../bpf/test_kmods/bpf_testmod_retval.c       | 311 ++++++++++++++++++
 .../test_kmods/bpf_testmod_retval_strong.c    |  28 ++
 8 files changed, 826 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/fmodret_retval.c
 create mode 100644 tools/testing/selftests/bpf/progs/fmodret_retval.c
 create mode 100644 tools/testing/selftests/bpf/progs/fmodret_retval_ext.c
 create mode 100644 tools/testing/selftests/bpf/test_kmods/bpf_test_retval_main.c
 create mode 100644 tools/testing/selftests/bpf/test_kmods/bpf_testmod_retval.c
 create mode 100644 tools/testing/selftests/bpf/test_kmods/bpf_testmod_retval_strong.c

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 77682bf28c6e..84554a2adfc1 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -47,7 +47,7 @@ TEST_PROGS_EXTENDED := \
 	ima_setup.sh verify_sig_setup.sh
 
 TEST_KMODS := bpf_testmod.ko bpf_test_no_cfi.ko bpf_test_modorder_x.ko \
-	bpf_test_modorder_y.ko bpf_test_rqspinlock.ko
+	bpf_test_modorder_y.ko bpf_test_rqspinlock.ko bpf_test_retval.ko
 TEST_KMOD_TARGETS = $(addprefix $(OUTPUT)/,$(TEST_KMODS))
 
 # Compile but not part of 'make run_tests'
diff --git a/tools/testing/selftests/bpf/prog_tests/fmodret_retval.c b/tools/testing/selftests/bpf/prog_tests/fmodret_retval.c
new file mode 100644
index 000000000000..7b374fe68130
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/fmodret_retval.c
@@ -0,0 +1,211 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+#include <bpf/btf.h>
+#include "fmodret_retval.skel.h"
+#include "fmodret_retval_ext.skel.h"
+
+#define TAG_FMODRET		"bpf_fmodret"
+#define TAG_RETVAL		"bpf_fmodret_retval:"
+#define TAG_TEST_RETVAL		"test_retval:"
+
+#define RETVAL_MODULE		"bpf_test_retval"
+
+/* The decl tags of a function */
+struct func_tags {
+	const char *test_retval;
+	const char *retval;
+	bool target;
+};
+
+static bool tag_value(const char *name, const char *key, const char **value)
+{
+	size_t len = strlen(key);
+
+	if (strncmp(name, key, len))
+		return false;
+	*value = name + len;
+	return true;
+}
+
+/*
+ * Return the decl tags of the functions of module BTF, indexed by BTF ID. Only
+ * the types of the module itself are read, not those of its vmlinux base.
+ */
+static struct func_tags *collect_tags(const struct btf *btf, int start_id)
+{
+	int id, nr_types = btf__type_cnt(btf);
+	struct func_tags *tags;
+
+	tags = calloc(nr_types, sizeof(*tags));
+	if (!tags)
+		return NULL;
+	for (id = start_id; id < nr_types; id++) {
+		const struct btf_type *t = btf__type_by_id(btf, id);
+		struct func_tags *f;
+		const char *name;
+
+		if (!btf_is_decl_tag(t) || btf_decl_tag(t)->component_idx != -1 ||
+		    t->type >= nr_types)
+			continue;
+		f = &tags[t->type];
+		name = btf__name_by_offset(btf, t->name_off);
+		if (!tag_value(name, TAG_TEST_RETVAL, &f->test_retval) &&
+		    !tag_value(name, TAG_RETVAL, &f->retval) && !strcmp(name, TAG_FMODRET))
+			f->target = true;
+	}
+
+	return tags;
+}
+
+/* Compare the tags of every fixture with the values it expects */
+static void test_tags(const struct btf *btf, const struct func_tags *tags, int start_id)
+{
+	int id, nr_types = btf__type_cnt(btf), nr_fixtures = 0;
+
+	for (id = start_id; id < nr_types; id++) {
+		const struct btf_type *t = btf__type_by_id(btf, id);
+		const char *name;
+
+		if (!btf_is_func(t) || !tags[id].test_retval)
+			continue;
+		name = btf__name_by_offset(btf, t->name_off);
+		ASSERT_STREQ(tags[id].retval ?: "", tags[id].test_retval, name);
+		if (*tags[id].test_retval)
+			ASSERT_TRUE(tags[id].target, name);
+		else if (strcmp(name, "bpf_testmod_retval_declared"))
+			ASSERT_FALSE(tags[id].target, name);
+		nr_fixtures++;
+	}
+	ASSERT_GT(nr_fixtures, 0, "fixtures");
+
+	id = btf__find_by_name_kind(btf, "bpf_testmod_retval_declared", BTF_KIND_FUNC);
+	if (ASSERT_GE(id, start_id, "declared"))
+		ASSERT_TRUE(tags[id].target, "declared_target");
+}
+
+/* Open the programs again with only the extension, to replace FUNC of TARGET */
+static struct fmodret_retval_ext *open_extension(struct bpf_program *target,
+						 const char *func, char *log, size_t log_size)
+{
+	struct fmodret_retval_ext *skel;
+
+	skel = fmodret_retval_ext__open();
+	if (!ASSERT_OK_PTR(skel, "fmodret_retval_ext__open"))
+		return NULL;
+	bpf_program__set_autoload(skel->progs.restricted, false);
+	bpf_program__set_autoload(skel->progs.unrestricted, false);
+	if (!ASSERT_OK(bpf_program__set_attach_target(skel->progs.extension,
+						      bpf_program__fd(target), func),
+		       "set_attach_target")) {
+		fmodret_retval_ext__destroy(skel);
+		return NULL;
+	}
+	bpf_program__set_log_buf(skel->progs.extension, log, log_size);
+	return skel;
+}
+
+/*
+ * An extension could return any value for an fmod_ret program, so it must not
+ * replace one whose return values are restricted, neither at load nor when it
+ * is attached later to another program.
+ */
+static void test_extension(void)
+{
+	struct fmodret_retval_ext *targets, *skel = NULL;
+	struct bpf_link *link;
+	char log[4096] = {};
+
+	if (!is_jit_enabled()) {
+		printf("%s:SKIP:extensions need the JIT\n", __func__);
+		test__skip();
+		return;
+	}
+
+	targets = fmodret_retval_ext__open();
+	if (!ASSERT_OK_PTR(targets, "fmodret_retval_ext__open"))
+		return;
+	bpf_program__set_autoload(targets->progs.extension, false);
+	if (!ASSERT_OK(fmodret_retval_ext__load(targets), "load_targets"))
+		goto out;
+
+	skel = open_extension(targets->progs.restricted, "restricted_value", log, sizeof(log));
+	if (!skel)
+		goto out;
+	ASSERT_ERR(fmodret_retval_ext__load(skel), "load_restricted");
+	ASSERT_HAS_SUBSTR(log, "Cannot extend an fmod_ret program with restricted return values",
+			  "log_restricted");
+	fmodret_retval_ext__destroy(skel);
+
+	skel = open_extension(targets->progs.unrestricted, "unrestricted_value", log, sizeof(log));
+	if (!skel)
+		goto out;
+	if (!ASSERT_OK(fmodret_retval_ext__load(skel), "load_unrestricted"))
+		goto out;
+	link = bpf_program__attach_freplace(skel->progs.extension,
+					    bpf_program__fd(targets->progs.restricted),
+					    "restricted_value");
+	if (!ASSERT_ERR_PTR(link, "attach_restricted"))
+		bpf_link__destroy(link);
+	link = bpf_program__attach_freplace(skel->progs.extension,
+					    bpf_program__fd(targets->progs.unrestricted),
+					    "unrestricted_value");
+	if (ASSERT_OK_PTR(link, "attach_unrestricted"))
+		bpf_link__destroy(link);
+out:
+	fmodret_retval_ext__destroy(skel);
+	fmodret_retval_ext__destroy(targets);
+}
+
+void test_fmodret_retval(void)
+{
+	struct btf *vmlinux_btf, *btf = NULL;
+	struct func_tags *tags = NULL;
+	int id, start_id, nr_types;
+	bool has_fixtures = false;
+
+	if (!env.has_testmod) {
+		test__skip();
+		return;
+	}
+
+	if (!ASSERT_OK(load_module(RETVAL_MODULE ".ko", env_verbosity > VERBOSE_NONE),
+		       "load_module"))
+		return;
+	vmlinux_btf = btf__load_vmlinux_btf();
+	if (!ASSERT_OK_PTR(vmlinux_btf, "btf__load_vmlinux_btf"))
+		goto out;
+	btf = btf__load_module_btf(RETVAL_MODULE, vmlinux_btf);
+	if (!btf && errno == ENOENT) {
+		printf("%s:SKIP:%s has no BTF\n", __func__, RETVAL_MODULE);
+		test__skip();
+		goto out;
+	}
+	if (!ASSERT_OK_PTR(btf, "btf__load_module_btf"))
+		goto out;
+	start_id = btf__type_cnt(vmlinux_btf);
+	nr_types = btf__type_cnt(btf);
+	tags = collect_tags(btf, start_id);
+	if (!ASSERT_OK_PTR(tags, "collect_tags"))
+		goto out;
+
+	/* The fixtures are built only with the compiler plugin */
+	for (id = start_id; id < nr_types && !has_fixtures; id++)
+		has_fixtures = tags[id].test_retval;
+	if (!has_fixtures) {
+		printf("%s:SKIP:%s has no %s tags\n", __func__, RETVAL_MODULE, TAG_TEST_RETVAL);
+		test__skip();
+		goto out;
+	}
+
+	if (test__start_subtest("tags"))
+		test_tags(btf, tags, start_id);
+	if (test__start_subtest("extension"))
+		test_extension();
+	RUN_TESTS(fmodret_retval);
+out:
+	free(tags);
+	btf__free(btf);
+	btf__free(vmlinux_btf);
+	unload_module(RETVAL_MODULE, env_verbosity > VERBOSE_NONE);
+}
diff --git a/tools/testing/selftests/bpf/progs/fmodret_retval.c b/tools/testing/selftests/bpf/progs/fmodret_retval.c
new file mode 100644
index 000000000000..bf5c6f53a999
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/fmodret_retval.c
@@ -0,0 +1,196 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * fmod_ret programs on the targets which the LLVM IR scanner finds in
+ * bpf_test_retval, see test_kmods/bpf_testmod_retval.c. They may return only
+ * 0 or the values of the "bpf_fmodret_retval:" tag of their target.
+ */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+/* bpf_testmod_retval_constant() returns -EINVAL */
+SEC("fmod_ret/bpf_testmod_retval_constant")
+__success
+int BPF_PROG(constant_einval, int ret)
+{
+	return -22;
+}
+
+SEC("fmod_ret/bpf_testmod_retval_constant")
+__success
+int BPF_PROG(constant_zero, int ret)
+{
+	return 0;
+}
+
+SEC("fmod_ret/bpf_testmod_retval_constant")
+__failure __msg("it should be 0 or one of -22")
+int BPF_PROG(constant_other, int ret)
+{
+	return -1;
+}
+
+SEC("fmod_ret/bpf_testmod_retval_constant")
+__failure __msg("it should be 0 or one of -22")
+__naked void constant_unknown(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/* The function returns int, so only the low 32 bits count */
+SEC("fmod_ret/bpf_testmod_retval_constant")
+__success
+__naked void constant_32bit(void)
+{
+	asm volatile (
+	"w0 = -22;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* Not 0, so the function is not called, but the caller would see 0 */
+SEC("fmod_ret/bpf_testmod_retval_constant")
+__failure __msg("it should be 0 or one of -22")
+__naked void constant_high_bits(void)
+{
+	asm volatile (
+	"r0 = 1;"
+	"r0 <<= 32;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* bpf_testmod_retval_join() returns -EINVAL or -ERANGE */
+SEC("fmod_ret/bpf_testmod_retval_join")
+__success
+int BPF_PROG(join_einval, int *result, int value, int ret)
+{
+	return -22;
+}
+
+SEC("fmod_ret/bpf_testmod_retval_join")
+__success
+int BPF_PROG(join_erange, int *result, int value, int ret)
+{
+	return -34;
+}
+
+SEC("fmod_ret/bpf_testmod_retval_join")
+__failure __msg("it should be 0 or one of -34,-22")
+int BPF_PROG(join_other, int *result, int value, int ret)
+{
+	return -1;
+}
+
+/* Each path returns an allowed value */
+SEC("fmod_ret/bpf_testmod_retval_join")
+__success
+__naked void join_both_paths(void)
+{
+	asm volatile (
+	"r2 = *(u64 *)(r1 + 8);"
+	"if r2 s< 0 goto 1f;"
+	"r0 = -34;"
+	"exit;"
+"1:"
+	"r0 = -22;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/*
+ * One of two paths to the same exit returns a forbidden value. The verifier
+ * checks the allowed one first and must not prune the other.
+ */
+SEC("fmod_ret/bpf_testmod_retval_join")
+__failure __msg("it should be 0 or one of -34,-22")
+__naked void join_one_forbidden(void)
+{
+	asm volatile (
+	"r2 = *(u64 *)(r1 + 8);"
+	"if r2 s< 0 goto 1f;"
+	"r0 = -34;"
+	"goto 2f;"
+"1:"
+	"r0 = -1;"
+"2:"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* bpf_testmod_retval_wide() returns s64, so all 64 bits count */
+SEC("fmod_ret/bpf_testmod_retval_wide")
+__success
+__load_if_JITed()
+__naked void wide_64bit(void)
+{
+	asm volatile (
+	"r0 = -14;"
+	"exit;"
+	::: __clobber_all);
+}
+
+SEC("fmod_ret/bpf_testmod_retval_wide")
+__failure __msg("it should be 0 or one of -14")
+__naked void wide_32bit(void)
+{
+	asm volatile (
+	"w0 = -14;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* Declared targets are not restricted, in module and vmlinux BTF */
+SEC("fmod_ret/bpf_testmod_retval_declared")
+__success
+int BPF_PROG(declared_module_any, int ret)
+{
+	return -1;
+}
+
+SEC("fmod_ret/bpf_modify_return_test")
+__success
+int BPF_PROG(declared_vmlinux_any, int a, int *b, int ret)
+{
+	return -1;
+}
+
+/* The error injection list allows any value although the target has a tag */
+SEC("fmod_ret/bpf_testmod_retval_injectable")
+__success
+int BPF_PROG(static_rule_tagged, int value, int ret)
+{
+	return -1;
+}
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+/* A program in a program array may return any value */
+SEC("fmod_ret/bpf_testmod_retval_constant")
+__failure __msg("tail calls are not allowed in fmod_ret programs with restricted return values")
+int BPF_PROG(constant_tail_call, int ret)
+{
+	bpf_tail_call(ctx, &jmp_table, 0);
+	return 0;
+}
+
+SEC("fmod_ret/bpf_testmod_retval_declared")
+__success
+int BPF_PROG(declared_tail_call, int ret)
+{
+	bpf_tail_call(ctx, &jmp_table, 0);
+	return 0;
+}
diff --git a/tools/testing/selftests/bpf/progs/fmodret_retval_ext.c b/tools/testing/selftests/bpf/progs/fmodret_retval_ext.c
new file mode 100644
index 000000000000..dd105eea05ee
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/fmodret_retval_ext.c
@@ -0,0 +1,53 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * An extension of fmod_ret programs, see prog_tests/fmodret_retval.c. The
+ * return values of the program on bpf_testmod_retval_constant() are restricted
+ * by its "bpf_fmodret_retval:" tag, those of the program on the declared target
+ * are not. An extension cannot replace the main function of a tracing program,
+ * so each of them calls a global function which the extension replaces.
+ */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+/* Keeps clang from folding the calls */
+int value_bias;
+
+__noinline int restricted_value(int value)
+{
+	return value + value_bias;
+}
+
+__noinline int unrestricted_value(int value)
+{
+	return value + value_bias;
+}
+
+SEC("fmod_ret/bpf_testmod_retval_constant")
+int BPF_PROG(restricted, int ret)
+{
+	int zero = 0;
+
+	if (restricted_value(ret))
+		return -22;
+	/*
+	 * fmod_ret BPF progs must return constants (0 or -22 in this case).
+	 * The barrier here forces GCC to return a constant
+	 */
+	barrier_var(zero);
+	return zero;
+}
+
+SEC("fmod_ret/bpf_testmod_retval_declared")
+int BPF_PROG(unrestricted, int ret)
+{
+	return unrestricted_value(ret);
+}
+
+SEC("freplace")
+int extension(int value)
+{
+	return -1;
+}
diff --git a/tools/testing/selftests/bpf/test_kmods/Makefile b/tools/testing/selftests/bpf/test_kmods/Makefile
index 031c7454ce65..f4af3c3ab54c 100644
--- a/tools/testing/selftests/bpf/test_kmods/Makefile
+++ b/tools/testing/selftests/bpf/test_kmods/Makefile
@@ -19,10 +19,23 @@ Q = @
 endif
 
 MODULES = bpf_testmod.ko bpf_test_no_cfi.ko bpf_test_modorder_x.ko \
-	bpf_test_modorder_y.ko bpf_test_rqspinlock.ko
+	bpf_test_modorder_y.ko bpf_test_rqspinlock.ko bpf_test_retval.ko
 
 $(foreach m,$(MODULES),$(eval obj-m += $(m:.ko=.o)))
 
+bpf_test_retval-y := bpf_test_retval_main.o
+# The functions for bpf_scan_ir use clang extensions and need the plugin
+bpf_test_retval-$(CONFIG_CLANG_PLUGIN_BPF_SCAN_IR) += bpf_testmod_retval.o \
+						      bpf_testmod_retval_strong.o
+# Instrumentation and branch profiling make every path of the fixtures dirty
+GCOV_PROFILE_bpf_testmod_retval.o := n
+KCOV_INSTRUMENT_bpf_testmod_retval.o := n
+KMSAN_SANITIZE_bpf_testmod_retval.o := n
+KASAN_SANITIZE_bpf_testmod_retval.o := n
+KCSAN_SANITIZE_bpf_testmod_retval.o := n
+UBSAN_SANITIZE_bpf_testmod_retval.o := n
+CFLAGS_bpf_testmod_retval.o += -DDISABLE_BRANCH_PROFILING
+
 CFLAGS_bpf_testmod.o = -I$(src)
 
 # When BPF_STRICT_BUILD != 0, a missing KDIR is fatal (the default).
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_test_retval_main.c b/tools/testing/selftests/bpf/test_kmods/bpf_test_retval_main.c
new file mode 100644
index 000000000000..c3a9093fa331
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_test_retval_main.c
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A module for the functions of bpf_testmod_retval.c and
+ * bpf_testmod_retval_strong.c, see prog_tests/fmodret_retval.c. They are not
+ * in bpf_testmod: pahole drops the per-CPU variables of a module which GNU ld
+ * links from several objects.
+ */
+#include <linux/module.h>
+
+MODULE_AUTHOR("Anton Protopopov");
+MODULE_DESCRIPTION("BPF selftest module for fmod_ret targets found by bpf_scan_ir");
+MODULE_LICENSE("GPL");
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_retval.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_retval.c
new file mode 100644
index 000000000000..fd63c6317cb9
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_retval.c
@@ -0,0 +1,311 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Functions for the fmod_ret targets which the LLVM IR scanner finds, see
+ * prog_tests/fmodret_retval.c. The "test_retval:" decl tag of each lists the
+ * values the scanner must find on paths without side effects. An empty list
+ * means that the function must not become such a target.
+ */
+#include <linux/atomic.h>
+#include <linux/btf_ids.h>
+#include <linux/compiler.h>
+#include <linux/err.h>
+#include <linux/errno.h>
+#include <linux/error-injection.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+#define RETVAL(values)	__attribute__((btf_decl_tag("test_retval:" values)))
+
+static __used int bpf_testmod_retval_state;
+static __used int *bpf_testmod_retval_captured;
+
+static noinline void bpf_testmod_retval_effect(void)
+{
+	WRITE_ONCE(bpf_testmod_retval_state, 1);
+}
+
+static noinline void bpf_testmod_retval_capture(int *ptr)
+{
+	WRITE_ONCE(bpf_testmod_retval_captured, ptr);
+}
+
+static noinline int bpf_testmod_retval_store(int value)
+{
+	WRITE_ONCE(bpf_testmod_retval_state, value);
+	return value;
+}
+
+static noinline int bpf_testmod_retval_clear(int value)
+{
+	WRITE_ONCE(bpf_testmod_retval_state, 0);
+	return value - 1;
+}
+
+/* Constant errors and both return widths */
+static __used noinline RETVAL("-22") int bpf_testmod_retval_constant(void)
+{
+	return -EINVAL;
+}
+
+static __used noinline RETVAL("-14") s64 bpf_testmod_retval_wide(void)
+{
+	return -EFAULT;
+}
+
+static __used noinline RETVAL("-1") int bpf_testmod_retval_first_errno(void)
+{
+	return -1;
+}
+
+static __used noinline RETVAL("-4095") int bpf_testmod_retval_last_errno(void)
+{
+	return -MAX_ERRNO;
+}
+
+/*
+ * PTR_ERR() of the ERR_PTR() of an inlined helper. Not recognized yet: the
+ * error reaches the return through a PHI in an earlier block.
+ */
+static __always_inline int *bpf_testmod_retval_slot(int *buf, int idx)
+{
+	if (idx < 0)
+		return ERR_PTR(-EINVAL);
+	if (idx > 8)
+		return ERR_PTR(-EFAULT);
+	return &buf[idx];
+}
+
+static __used noinline RETVAL("") int bpf_testmod_retval_err_ptr(int *buf, int idx)
+{
+	int *slot = bpf_testmod_retval_slot(buf, idx);
+
+	if (IS_ERR(slot))
+		return PTR_ERR(slot);
+	*slot = 1;
+	return 0;
+}
+
+/* A clean error path may join a dirty success path */
+static __used noinline RETVAL("-22") int bpf_testmod_retval_before_store(int *result, int value)
+{
+	if (value < 0)
+		return -EINVAL;
+	*result = value;
+	return 0;
+}
+
+static __used noinline RETVAL("-34,-22") int bpf_testmod_retval_join(int *result, int value)
+{
+	int ret;
+
+	if (value < 0)
+		ret = -EINVAL;
+	else if (value > 32)
+		ret = -ERANGE;
+	else {
+		*result = value;
+		ret = 0;
+	}
+	return ret;
+}
+
+/*
+ * An int error which a long function returns: the PHI of the two calls keeps
+ * the sign extension in the block of the return
+ */
+static __used noinline RETVAL("-22") long bpf_testmod_retval_sign_extend(int value)
+{
+	int ret;
+
+	if (value < 0)
+		ret = -EINVAL;
+	else if (value > 32)
+		ret = bpf_testmod_retval_store(value);
+	else
+		ret = bpf_testmod_retval_clear(value);
+	return ret;
+}
+
+/* An assumption does not make a path dirty */
+static __used noinline RETVAL("-22") int bpf_testmod_retval_assume(int value)
+{
+	__builtin_assume(value > 0);
+	return -EINVAL;
+}
+
+/* Writes to private stack objects are clean, optnone keeps them in the IR */
+static __used noinline RETVAL("-22") __attribute__((optnone))
+int bpf_testmod_retval_private_store(int value)
+{
+	int scratch[2];
+
+	scratch[0] = value;
+	scratch[1] = 0;
+	return -EINVAL;
+}
+
+static __used noinline RETVAL("-22") __attribute__((optnone))
+int bpf_testmod_retval_private_memset(void)
+{
+	char scratch[32];
+
+	__builtin_memset(scratch, 0, sizeof(scratch));
+	return -EINVAL;
+}
+
+/* An error which is not known exactly is not recorded */
+static __used noinline RETVAL("") int bpf_testmod_retval_masked(unsigned int value)
+{
+	return -1 - (value & 31);
+}
+
+static __used noinline RETVAL("") long bpf_testmod_retval_ptr_err(void *ptr, int *state)
+{
+	if (IS_ERR(ptr))
+		return PTR_ERR(ptr);
+	*state = 1;
+	return 0;
+}
+
+/* Pointer functions are not targets yet */
+static __used noinline RETVAL("") void *bpf_testmod_retval_error_pointer(void)
+{
+	return ERR_PTR(-ENOMEM);
+}
+
+/* Every error path has an observable effect */
+static __used noinline RETVAL("") int bpf_testmod_retval_argument_store(int *state)
+{
+	*state = 1;
+	return -EINVAL;
+}
+
+static __used noinline RETVAL("") int bpf_testmod_retval_global_store(void)
+{
+	bpf_testmod_retval_state = 1;
+	return -EINVAL;
+}
+
+static __used noinline RETVAL("") int bpf_testmod_retval_volatile_read(int *state)
+{
+	return READ_ONCE(*state) ? -EINVAL : -ERANGE;
+}
+
+static __used noinline RETVAL("") int bpf_testmod_retval_atomic(atomic_t *state)
+{
+	atomic_inc(state);
+	return -EINVAL;
+}
+
+static __used noinline RETVAL("") int bpf_testmod_retval_call(void)
+{
+	bpf_testmod_retval_effect();
+	return -EINVAL;
+}
+
+static __used noinline RETVAL("") int bpf_testmod_retval_barrier(void)
+{
+	barrier();
+	return -EINVAL;
+}
+
+static __used noinline RETVAL("")
+int bpf_testmod_retval_only_error_is_dirty(int *state, bool reject)
+{
+	if (reject) {
+		*state = 1;
+		return -EINVAL;
+	}
+	return 0;
+}
+
+static __used noinline RETVAL("")
+int bpf_testmod_retval_lock_held(spinlock_t *lock) __acquires(lock)
+{
+	spin_lock(lock);
+	return -EINVAL;
+}
+
+static __used noinline RETVAL("") int bpf_testmod_retval_stack_escape(void)
+{
+	int scratch;
+
+	bpf_testmod_retval_capture(&scratch);
+	scratch = 1;
+	return -EINVAL;
+}
+
+static __used noinline RETVAL("") int bpf_testmod_retval_external_memset(void *dest)
+{
+	__builtin_memset(dest, 0, 32);
+	return -EINVAL;
+}
+
+/* Not recognized yet, although they have no side effects */
+static __used noinline RETVAL("") int bpf_testmod_retval_balanced_spinlock(spinlock_t *lock)
+{
+	spin_lock(lock);
+	spin_unlock(lock);
+	return -EINVAL;
+}
+
+static __used noinline RETVAL("") int bpf_testmod_retval_input_errno(int error)
+{
+	if (error >= -MAX_ERRNO && error < 0)
+		return error;
+	return 0;
+}
+
+/* No ftrace call site, but call padding in __patchable_function_entries on x86 */
+static __used noinline RETVAL("") notrace int bpf_testmod_retval_notrace(void)
+{
+	return -EINVAL;
+}
+
+/*
+ * A weak definition with an error, replaced by a strong one without errors in
+ * bpf_testmod_retval_strong.c
+ */
+int bpf_testmod_retval_overridden(void);
+
+__weak noinline __attribute__((btf_decl_tag("test_retval:")))
+int bpf_testmod_retval_overridden(void)
+{
+	return -EINVAL;
+}
+
+/*
+ * Two weak definitions: this one with a side effect is linked first and wins
+ * over the one with an error in bpf_testmod_retval_strong.c
+ */
+int bpf_testmod_retval_two_weak(void);
+
+__weak noinline __attribute__((btf_decl_tag("test_retval:")))
+int bpf_testmod_retval_two_weak(void)
+{
+	bpf_testmod_retval_effect();
+	return 0;
+}
+
+/*
+ * A function on the error injection list gets its tag, but the static rules let
+ * programs return any value
+ */
+static __used noinline RETVAL("-22") int bpf_testmod_retval_injectable(int value)
+{
+	if (value < 0)
+		return -EINVAL;
+	bpf_testmod_retval_effect();
+	return 0;
+}
+ALLOW_ERROR_INJECTION(bpf_testmod_retval_injectable, ERRNO);
+
+/* A declared target keeps only its own tags */
+static __used noinline RETVAL("") int bpf_testmod_retval_declared(void)
+{
+	return -EINVAL;
+}
+
+BTF_FMODRET_START(bpf_testmod_retval_declared_ids)
+BTF_ID_FLAGS(func, bpf_testmod_retval_declared)
+BTF_FMODRET_END(bpf_testmod_retval_declared_ids)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_retval_strong.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_retval_strong.c
new file mode 100644
index 000000000000..a246d7f619bc
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_retval_strong.c
@@ -0,0 +1,28 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Replaces the weak bpf_testmod_retval_overridden() of bpf_testmod_retval.c.
+ * This definition returns no errors, so the function must not become an
+ * fmod_ret target, see prog_tests/fmodret_retval.c.
+ *
+ * The weak bpf_testmod_retval_two_weak() here loses the link to the one of
+ * bpf_testmod_retval.c, so its error must not make the function a target
+ * either.
+ */
+#include <linux/compiler.h>
+#include <linux/errno.h>
+
+int bpf_testmod_retval_overridden(void);
+
+noinline __attribute__((btf_decl_tag("test_retval:")))
+int bpf_testmod_retval_overridden(void)
+{
+	return 0;
+}
+
+int bpf_testmod_retval_two_weak(void);
+
+__weak noinline __attribute__((btf_decl_tag("test_retval:")))
+int bpf_testmod_retval_two_weak(void)
+{
+	return -EINVAL;
+}
-- 
2.43.0

[PATCH v2 bpf-next 17/18] selftests/bpf: Enable bpf_scan_ir in the config

[Anton Protopopov]

The bpf_scan_ir plugin is off by default. Without it bpf_test_retval has
no fixtures for the plugin and fmodret_retval skips. Enable the plugin in
the selftests config, so the CI jobs which build the kernel with clang
build and test it. Kconfig enables the plugin only on x86 and arm64 with
clang 19 or newer and a matching llvm-config, so elsewhere
bpf_test_retval is built without the fixtures and fmodret_retval skips.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 tools/testing/selftests/bpf/config | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index 49dfe939467c..5461c32cd3cd 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -11,6 +11,7 @@ CONFIG_BPF_STREAM_PARSER=y
 CONFIG_BPF_SYSCALL=y
 # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set
 CONFIG_CGROUP_BPF=y
+CONFIG_CLANG_PLUGIN_BPF_SCAN_IR=y
 CONFIG_CRYPTO_HMAC=y
 CONFIG_CRYPTO_MLDSA=y
 CONFIG_CRYPTO_SHA256=y
@@ -56,6 +57,7 @@ CONFIG_IPV6_TUNNEL=y
 CONFIG_KEYS=y
 CONFIG_LIRC=y
 CONFIG_LIVEPATCH=y
+CONFIG_LLVM_PLUGINS=y
 CONFIG_LWTUNNEL=y
 CONFIG_LWTUNNEL_BPF=y
 CONFIG_MEMCG=y
-- 
2.43.0

[PATCH v2 bpf-next 18/18] bpftool: List and show fmod_ret targets

[Anton Protopopov]

Most fmod_ret targets are now found by the kernel build and only exist
as BTF decl tags. Add a bpftool fmodret command to see them.

The `bpftool fmodret list` command prints the functions tagged with
"bpf_fmodret" in vmlinux and in the loaded modules, and marks the ones
which sleepable programs may attach to.

The `bpftool fmodret show` command prints a function in detail: its
prototype, if it was declared as a target or found by the build, and the
values which fmod_ret programs may return. A function is given by name
or BTF ID.

Both commands read BTF from /sys/kernel/btf or from a file and support
JSON output.

Functions which the verifier allows by its static rules, such as the
security hooks, may also have the tags and be listed. Even if they are
listed, the restrictions of the tags do not apply to them. This may
change in the future by either not listing them, or listing them without
restrictions.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 .../bpftool/Documentation/bpftool-fmodret.rst | 111 +++
 tools/bpf/bpftool/Documentation/bpftool.rst   |   2 +-
 tools/bpf/bpftool/bash-completion/bpftool     |  31 +
 tools/bpf/bpftool/fmodret.c                   | 896 ++++++++++++++++++
 tools/bpf/bpftool/main.c                      |   3 +-
 tools/bpf/bpftool/main.h                      |   1 +
 6 files changed, 1042 insertions(+), 2 deletions(-)
 create mode 100644 tools/bpf/bpftool/Documentation/bpftool-fmodret.rst
 create mode 100644 tools/bpf/bpftool/fmodret.c

diff --git a/tools/bpf/bpftool/Documentation/bpftool-fmodret.rst b/tools/bpf/bpftool/Documentation/bpftool-fmodret.rst
new file mode 100644
index 000000000000..1580fd614010
--- /dev/null
+++ b/tools/bpf/bpftool/Documentation/bpftool-fmodret.rst
@@ -0,0 +1,111 @@
+.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+
+===============
+bpftool-fmodret
+===============
+-------------------------------------------------------------------------------
+tool to inspect BPF fmod_ret targets
+-------------------------------------------------------------------------------
+
+:Manual section: 8
+
+.. include:: substitutions.rst
+
+SYNOPSIS
+========
+
+**bpftool** [*OPTIONS*] **fmodret** *COMMAND*
+
+*OPTIONS* := { |COMMON_OPTIONS| | { **-B** | **--base-btf** } *FILE* }
+
+*COMMANDS* := { **list** | **show** | **help** }
+
+FMODRET COMMANDS
+================
+
+| **bpftool** **fmodret list** [*OBJECT*]
+| **bpftool** **fmodret show** *FUNC* [*OBJECT*]
+| **bpftool** **fmodret help**
+|
+| *FUNC* := { **name** *NAME* | **id** *BTF_ID* }
+| *OBJECT* := { **file** *FILE* | **vmlinux** | **module** *NAME* }
+
+DESCRIPTION
+===========
+
+bpftool fmodret list [*OBJECT*]
+    List the fmod_ret targets, the functions which the kernel build tags with
+    the **bpf_fmodret** BTF decl tag, in vmlinux and in loaded modules. Each
+    entry contains the BTF ID of the function, its name and the name of the
+    object it belongs to. Targets which sleepable programs may attach to are
+    marked **sleepable**.
+
+    With **file**, read the BTF from *FILE* (for example a vmlinux image or a
+    **BTF** file) instead of the running kernel. With **vmlinux** or
+    **module**, list only the targets of vmlinux or of the module *NAME*.
+
+    With **--json**, each entry contains **btf_id**, **name**, **owner** and
+    **sleepable** fields.
+
+    The verifier also accepts functions by its static rules, such as the error
+    injection list or **security_** hooks. Such functions are listed only if
+    the kernel build tagged them. Even if they are listed, the restrictions of
+    their **bpf_fmodret_retval:** tags do not apply to them: programs may
+    return any value, and sleepable programs may attach to some of them without
+    the **sleepable** mark. This may change in the future by either not listing
+    them, or listing them without restrictions. Normal verifier and tracing
+    checks apply to all of them.
+
+bpftool fmodret show *FUNC* [*OBJECT*]
+    Show the function named *NAME* or with BTF ID *BTF_ID* in detail. It does
+    not have to be a target. A name may belong to several functions, for
+    example to functions of different modules. A BTF ID is looked up in
+    vmlinux, unless **module** or **file** is given. *OBJECT* is as for
+    **list**.
+
+    Besides the BTF ID, the name and the object, show the prototype of the
+    function and if it is a target: **found by the kernel build**, **declared**
+    or **not tagged**. For a target show if sleepable programs may attach to it
+    and the values which programs may return: 0 and the values of its
+    **bpf_fmodret_retval:** tag with the names of the errors on the host, or
+    **any** for a declared target. A tag which the verifier would reject,
+    malformed, not the only one or with values which the return type cannot
+    hold, is shown as invalid. The values do not apply to a function which the
+    static rules allow, see **list**. For the running kernel, also show the
+    type of the entry of the function in the error injection list, if the list
+    is available in debugfs.
+
+    With **--json**, each function contains **btf_id**, **name**, **owner**,
+    **prototype**, **target** (**found**, **declared** or **none**) and
+    **sleepable** fields, a **retval** array for a target found by the build
+    with the values programs may return, 0 included, or a **retval_error**
+    string if its tag is invalid, and an **error_injection** field if the
+    function is in the error injection list. A BTF ID gives one function, a
+    name an array of them.
+
+bpftool fmodret help
+    Print short help message.
+
+OPTIONS
+========
+
+.. include:: common_options.rst
+
+-B, --base-btf *FILE*
+    Pass the base BTF of a split BTF *FILE*, usually the BTF of vmlinux. Raw
+    BTF of a module needs it. A module image may carry the base it was built
+    against in its **.BTF.base** section instead, but its BTF IDs then differ
+    from those of the running kernel.
+
+EXAMPLES
+========
+
+**# bpftool fmodret show name bpf_testmod_retval_join module bpf_test_retval**
+
+::
+
+  123456  bpf_testmod_retval_join (bpf_test_retval)
+          int bpf_testmod_retval_join(int *result, int value)
+          target: found by the kernel build
+          sleepable: no
+          retval: 0, -34 (ERANGE), -22 (EINVAL)
diff --git a/tools/bpf/bpftool/Documentation/bpftool.rst b/tools/bpf/bpftool/Documentation/bpftool.rst
index f38ae5c40439..2c64c22a5026 100644
--- a/tools/bpf/bpftool/Documentation/bpftool.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool.rst
@@ -21,7 +21,7 @@ SYNOPSIS
 **bpftool** **version**
 
 *OBJECT* := { **map** | **prog** | **link** | **cgroup** | **perf** | **net** | **feature** |
-**btf** | **gen** | **struct_ops** | **iter** }
+**btf** | **gen** | **struct_ops** | **iter** | **fmodret** }
 
 *OPTIONS* := { { **-V** | **--version** } | |COMMON_OPTIONS| }
 
diff --git a/tools/bpf/bpftool/bash-completion/bpftool b/tools/bpf/bpftool/bash-completion/bpftool
index a28af31937ad..83288c22b3d3 100644
--- a/tools/bpf/bpftool/bash-completion/bpftool
+++ b/tools/bpf/bpftool/bash-completion/bpftool
@@ -675,6 +675,37 @@ _bpftool()
                     ;;
             esac
             ;;
+        fmodret)
+            case $command in
+                list|show)
+                    case $prev in
+                        file)
+                            _filedir
+                            ;;
+                        module)
+                            COMPREPLY=( $( compgen -W "$( command ls \
+                                /sys/kernel/btf 2>/dev/null | \
+                                command grep -vx -e vmlinux -e '.*\.inline' )" \
+                                -- "$cur" ) )
+                            ;;
+                        name|id)
+                            ;;
+                        *)
+                            if [[ $prev == show ]]; then
+                                COMPREPLY=( $( compgen -W 'name id' -- "$cur" ) )
+                            else
+                                _bpftool_one_of_list 'file vmlinux module'
+                            fi
+                            ;;
+                    esac
+                    return 0
+                    ;;
+                *)
+                    [[ $prev == $object ]] && \
+                        COMPREPLY=( $( compgen -W 'help list show' -- "$cur" ) )
+                    ;;
+            esac
+            ;;
         map)
             case $command in
                 show|list|dump|peek|pop|dequeue|freeze)
diff --git a/tools/bpf/bpftool/fmodret.c b/tools/bpf/bpftool/fmodret.c
new file mode 100644
index 000000000000..941166da53cc
--- /dev/null
+++ b/tools/bpf/bpftool/fmodret.c
@@ -0,0 +1,896 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+
+#ifndef _GNU_SOURCE
+#define _GNU_SOURCE
+#endif
+#include <dirent.h>
+#include <errno.h>
+#include <limits.h>
+#include <stdarg.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <bpf/btf.h>
+#include <bpf/libbpf.h>
+
+#include "main.h"
+
+/*
+ * Decl tags added by resolve_btfids to functions in BTF_FMODRET sets. A target
+ * which sleepable programs may attach to has both tags. A target which the
+ * build found by analysing its code has a "bpf_fmodret_retval:" tag with the
+ * values programs may return, such as "bpf_fmodret_retval:-22,-12".
+ */
+#define TAG_FMODRET		"bpf_fmodret"
+#define TAG_FMODRET_SLEEPABLE	"bpf_fmodret_sleepable"
+#define TAG_FMODRET_RETVAL	"bpf_fmodret_retval:"
+
+#define SYS_BTF_DIR		"/sys/kernel/btf"
+#define ERROR_INJECTION_LIST	"/sys/kernel/debug/error_injection/list"
+
+struct fmodret_entry {
+	__u32 btf_id;
+	bool target;
+	bool sleepable;
+	const char *retval;
+	/* More than one "bpf_fmodret_retval:" tag, which the verifier rejects */
+	bool retval_dup;
+};
+
+struct fmodret_filter {
+	/* Only vmlinux, or only the module with this name */
+	bool vmlinux;
+	const char *module;
+	/* The function to show, by name or by BTF ID */
+	const char *name;
+	__u32 btf_id;
+	unsigned int found;
+	/* The BTF is the one of the running kernel */
+	bool live;
+};
+
+typedef int (*btf_cb)(const char *owner, const struct btf *btf,
+		      struct fmodret_filter *filter);
+
+#define ERRNO_NAME(e) { e, #e }
+
+/* The errors of the uapi and the kernel internal ones, which libc lacks */
+static const struct {
+	int value;
+	const char *name;
+} errno_names[] = {
+	ERRNO_NAME(EPERM),
+	ERRNO_NAME(ENOENT),
+	ERRNO_NAME(ESRCH),
+	ERRNO_NAME(EINTR),
+	ERRNO_NAME(EIO),
+	ERRNO_NAME(ENXIO),
+	ERRNO_NAME(E2BIG),
+	ERRNO_NAME(ENOEXEC),
+	ERRNO_NAME(EBADF),
+	ERRNO_NAME(ECHILD),
+	ERRNO_NAME(EAGAIN),
+	ERRNO_NAME(ENOMEM),
+	ERRNO_NAME(EACCES),
+	ERRNO_NAME(EFAULT),
+	ERRNO_NAME(ENOTBLK),
+	ERRNO_NAME(EBUSY),
+	ERRNO_NAME(EEXIST),
+	ERRNO_NAME(EXDEV),
+	ERRNO_NAME(ENODEV),
+	ERRNO_NAME(ENOTDIR),
+	ERRNO_NAME(EISDIR),
+	ERRNO_NAME(EINVAL),
+	ERRNO_NAME(ENFILE),
+	ERRNO_NAME(EMFILE),
+	ERRNO_NAME(ENOTTY),
+	ERRNO_NAME(ETXTBSY),
+	ERRNO_NAME(EFBIG),
+	ERRNO_NAME(ENOSPC),
+	ERRNO_NAME(ESPIPE),
+	ERRNO_NAME(EROFS),
+	ERRNO_NAME(EMLINK),
+	ERRNO_NAME(EPIPE),
+	ERRNO_NAME(EDOM),
+	ERRNO_NAME(ERANGE),
+	ERRNO_NAME(EDEADLK),
+#if defined(EDEADLOCK) && EDEADLOCK != EDEADLK
+	ERRNO_NAME(EDEADLOCK),
+#endif
+	ERRNO_NAME(ENAMETOOLONG),
+	ERRNO_NAME(ENOLCK),
+	ERRNO_NAME(ENOSYS),
+	ERRNO_NAME(ENOTEMPTY),
+	ERRNO_NAME(ELOOP),
+	ERRNO_NAME(ENOMSG),
+	ERRNO_NAME(EIDRM),
+	ERRNO_NAME(ECHRNG),
+	ERRNO_NAME(EL2NSYNC),
+	ERRNO_NAME(EL3HLT),
+	ERRNO_NAME(EL3RST),
+	ERRNO_NAME(ELNRNG),
+	ERRNO_NAME(EUNATCH),
+	ERRNO_NAME(ENOCSI),
+	ERRNO_NAME(EL2HLT),
+	ERRNO_NAME(EBADE),
+	ERRNO_NAME(EBADR),
+	ERRNO_NAME(EXFULL),
+	ERRNO_NAME(ENOANO),
+	ERRNO_NAME(EBADRQC),
+	ERRNO_NAME(EBADSLT),
+	ERRNO_NAME(EBFONT),
+	ERRNO_NAME(ENOSTR),
+	ERRNO_NAME(ENODATA),
+	ERRNO_NAME(ETIME),
+	ERRNO_NAME(ENOSR),
+	ERRNO_NAME(ENONET),
+	ERRNO_NAME(ENOPKG),
+	ERRNO_NAME(EREMOTE),
+	ERRNO_NAME(ENOLINK),
+	ERRNO_NAME(EADV),
+	ERRNO_NAME(ESRMNT),
+	ERRNO_NAME(ECOMM),
+	ERRNO_NAME(EPROTO),
+	ERRNO_NAME(EMULTIHOP),
+	ERRNO_NAME(EDOTDOT),
+	ERRNO_NAME(EBADMSG),
+	ERRNO_NAME(EOVERFLOW),
+	ERRNO_NAME(ENOTUNIQ),
+	ERRNO_NAME(EBADFD),
+	ERRNO_NAME(EREMCHG),
+	ERRNO_NAME(ELIBACC),
+	ERRNO_NAME(ELIBBAD),
+	ERRNO_NAME(ELIBSCN),
+	ERRNO_NAME(ELIBMAX),
+	ERRNO_NAME(ELIBEXEC),
+	ERRNO_NAME(EILSEQ),
+	ERRNO_NAME(ERESTART),
+	ERRNO_NAME(ESTRPIPE),
+	ERRNO_NAME(EUSERS),
+	ERRNO_NAME(ENOTSOCK),
+	ERRNO_NAME(EDESTADDRREQ),
+	ERRNO_NAME(EMSGSIZE),
+	ERRNO_NAME(EPROTOTYPE),
+	ERRNO_NAME(ENOPROTOOPT),
+	ERRNO_NAME(EPROTONOSUPPORT),
+	ERRNO_NAME(ESOCKTNOSUPPORT),
+	ERRNO_NAME(EOPNOTSUPP),
+	ERRNO_NAME(EPFNOSUPPORT),
+	ERRNO_NAME(EAFNOSUPPORT),
+	ERRNO_NAME(EADDRINUSE),
+	ERRNO_NAME(EADDRNOTAVAIL),
+	ERRNO_NAME(ENETDOWN),
+	ERRNO_NAME(ENETUNREACH),
+	ERRNO_NAME(ENETRESET),
+	ERRNO_NAME(ECONNABORTED),
+	ERRNO_NAME(ECONNRESET),
+	ERRNO_NAME(ENOBUFS),
+	ERRNO_NAME(EISCONN),
+	ERRNO_NAME(ENOTCONN),
+	ERRNO_NAME(ESHUTDOWN),
+	ERRNO_NAME(ETOOMANYREFS),
+	ERRNO_NAME(ETIMEDOUT),
+	ERRNO_NAME(ECONNREFUSED),
+	ERRNO_NAME(EHOSTDOWN),
+	ERRNO_NAME(EHOSTUNREACH),
+	ERRNO_NAME(EALREADY),
+	ERRNO_NAME(EINPROGRESS),
+	ERRNO_NAME(ESTALE),
+	ERRNO_NAME(EUCLEAN),
+	ERRNO_NAME(ENOTNAM),
+	ERRNO_NAME(ENAVAIL),
+	ERRNO_NAME(EISNAM),
+	ERRNO_NAME(EREMOTEIO),
+	ERRNO_NAME(EDQUOT),
+	ERRNO_NAME(ENOMEDIUM),
+	ERRNO_NAME(EMEDIUMTYPE),
+	ERRNO_NAME(ECANCELED),
+	ERRNO_NAME(ENOKEY),
+	ERRNO_NAME(EKEYEXPIRED),
+	ERRNO_NAME(EKEYREVOKED),
+	ERRNO_NAME(EKEYREJECTED),
+	ERRNO_NAME(EOWNERDEAD),
+	ERRNO_NAME(ENOTRECOVERABLE),
+	ERRNO_NAME(ERFKILL),
+	ERRNO_NAME(EHWPOISON),
+#ifdef EFTYPE
+	ERRNO_NAME(EFTYPE),
+#endif
+	{ 512, "ERESTARTSYS" },
+	{ 513, "ERESTARTNOINTR" },
+	{ 514, "ERESTARTNOHAND" },
+	{ 515, "ENOIOCTLCMD" },
+	{ 516, "ERESTART_RESTARTBLOCK" },
+	{ 517, "EPROBE_DEFER" },
+	{ 518, "EOPENSTALE" },
+	{ 519, "ENOPARAM" },
+	{ 521, "EBADHANDLE" },
+	{ 522, "ENOTSYNC" },
+	{ 523, "EBADCOOKIE" },
+	{ 524, "ENOTSUPP" },
+	{ 525, "ETOOSMALL" },
+	{ 526, "ESERVERFAULT" },
+	{ 527, "EBADTYPE" },
+	{ 528, "EJUKEBOX" },
+	{ 529, "EIOCBQUEUED" },
+	{ 530, "ERECALLCONFLICT" },
+	{ 531, "ENOGRACE" },
+};
+
+static const char *errno_name(long long value)
+{
+	size_t i;
+
+	for (i = 0; i < ARRAY_SIZE(errno_names); i++)
+		if (-(long long)errno_names[i].value == value)
+			return errno_names[i].name;
+	return NULL;
+}
+
+static int cmp_entry(const void *a, const void *b)
+{
+	const struct fmodret_entry *x = a, *y = b;
+
+	return (x->btf_id > y->btf_id) - (x->btf_id < y->btf_id);
+}
+
+/* The first type ID which belongs to BTF and not to its base */
+static __u32 first_own_type(const struct btf *btf)
+{
+	const struct btf *base = btf__base_btf(btf);
+
+	return base ? btf__type_cnt(base) : 1;
+}
+
+/*
+ * Return the function which T tags with decl tag NAME, or 0. A NAME which ends
+ * with ':' matches the tags which start with it.
+ */
+static __u32 tag_func(const struct btf *btf, const struct btf_type *t, const char *name)
+{
+	size_t len = strlen(name);
+	const char *tag;
+
+	if (!btf_is_decl_tag(t) || btf_decl_tag(t)->component_idx != -1)
+		return 0;
+	tag = btf__name_by_offset(btf, t->name_off);
+	if (strncmp(tag, name, len) || (name[len - 1] != ':' && tag[len]))
+		return 0;
+	if (!btf_is_func(btf__type_by_id(btf, t->type)))
+		return 0;
+	return t->type;
+}
+
+/* Return the fmod_ret targets of BTF, sorted by BTF ID */
+static int collect_targets(const struct btf *btf, struct fmodret_entry **targets,
+			   size_t *target_cnt)
+{
+	__u32 id, func_id, start_id = first_own_type(btf);
+	__u32 nr_types = btf__type_cnt(btf);
+	struct fmodret_entry *entries = NULL, *entry, *tmp;
+	size_t cnt = 0, capacity = 0;
+
+	for (id = start_id; id < nr_types; id++) {
+		func_id = tag_func(btf, btf__type_by_id(btf, id), TAG_FMODRET);
+		if (!func_id)
+			continue;
+
+		if (cnt == capacity) {
+			capacity = capacity ? 2 * capacity : 64;
+			tmp = realloc(entries, capacity * sizeof(*entries));
+			if (!tmp) {
+				free(entries);
+				return -ENOMEM;
+			}
+			entries = tmp;
+		}
+		entries[cnt].btf_id = func_id;
+		entries[cnt].sleepable = false;
+		cnt++;
+	}
+	if (cnt)
+		qsort(entries, cnt, sizeof(*entries), cmp_entry);
+
+	/* The other tags mark functions which are already targets */
+	for (id = start_id; cnt && id < nr_types; id++) {
+		const struct btf_type *t = btf__type_by_id(btf, id);
+		struct fmodret_entry key = {};
+
+		key.btf_id = tag_func(btf, t, TAG_FMODRET_SLEEPABLE);
+		if (key.btf_id) {
+			entry = bsearch(&key, entries, cnt, sizeof(*entries), cmp_entry);
+			if (entry)
+				entry->sleepable = true;
+		}
+	}
+
+	*targets = entries;
+	*target_cnt = cnt;
+	return 0;
+}
+
+/* Fill ENTRY from the decl tags of function FUNC_ID */
+static void func_entry(const struct btf *btf, __u32 func_id, struct fmodret_entry *entry)
+{
+	__u32 id, nr_types = btf__type_cnt(btf);
+
+	memset(entry, 0, sizeof(*entry));
+	entry->btf_id = func_id;
+	for (id = first_own_type(btf); id < nr_types; id++) {
+		const struct btf_type *t = btf__type_by_id(btf, id);
+
+		if (tag_func(btf, t, TAG_FMODRET) == func_id)
+			entry->target = true;
+		else if (tag_func(btf, t, TAG_FMODRET_SLEEPABLE) == func_id)
+			entry->sleepable = true;
+		else if (tag_func(btf, t, TAG_FMODRET_RETVAL) == func_id) {
+			entry->retval_dup = entry->retval;
+			entry->retval = btf__name_by_offset(btf, t->name_off) +
+					strlen(TAG_FMODRET_RETVAL);
+		}
+	}
+}
+
+static void list_target(const char *owner, const struct btf *btf,
+			const struct fmodret_entry *entry)
+{
+	const struct btf_type *t = btf__type_by_id(btf, entry->btf_id);
+	const char *name = btf__name_by_offset(btf, t->name_off);
+
+	if (json_output) {
+		jsonw_start_object(json_wtr);
+		jsonw_uint_field(json_wtr, "btf_id", entry->btf_id);
+		jsonw_string_field(json_wtr, "name", name);
+		jsonw_string_field(json_wtr, "owner", owner);
+		jsonw_bool_field(json_wtr, "sleepable", entry->sleepable);
+		jsonw_end_object(json_wtr);
+	} else {
+		printf("%u  %s (%s)%s\n", entry->btf_id, name, owner,
+		       entry->sleepable ? " sleepable" : "");
+	}
+}
+
+static int list_btf(const char *owner, const struct btf *btf,
+		    struct fmodret_filter *filter)
+{
+	struct fmodret_entry *targets = NULL;
+	size_t cnt = 0, i;
+	int err;
+
+	err = collect_targets(btf, &targets, &cnt);
+	if (err) {
+		p_err("failed to collect fmod_ret targets of %s: %s", owner,
+		      strerror(-err));
+		return err;
+	}
+	for (i = 0; i < cnt; i++)
+		list_target(owner, btf, &targets[i]);
+	free(targets);
+	return 0;
+}
+
+struct error_injection {
+	char *name;
+	char *module;
+	char *type;
+};
+
+static struct error_injection *error_injections;
+static size_t error_injection_cnt;
+
+/*
+ * Read the error injection list of the running kernel. Its lines are
+ * "FUNC\tTYPE" and "FUNC [MODULE]\tTYPE". Without debugfs the list is empty.
+ */
+static void read_error_injection_list(void)
+{
+	struct error_injection *tmp;
+	char *line = NULL;
+	size_t len = 0;
+	FILE *f;
+
+	f = fopen(ERROR_INJECTION_LIST, "r");
+	if (!f)
+		return;
+	while (getline(&line, &len, f) > 0) {
+		char *name, *module = NULL, *type, *save;
+
+		/* "NAME [MODULE]\tTYPE" or "NAME\tTYPE" */
+		name = strtok_r(line, " \t\n", &save);
+		type = strtok_r(NULL, " \t\n", &save);
+		if (type && type[0] == '[') {
+			module = type + 1;
+			module[strcspn(module, "]")] = '\0';
+			type = strtok_r(NULL, " \t\n", &save);
+		}
+		if (!name || !type)
+			continue;
+		tmp = realloc(error_injections,
+			      (error_injection_cnt + 1) * sizeof(*error_injections));
+		if (!tmp)
+			break;
+		error_injections = tmp;
+		tmp = &error_injections[error_injection_cnt];
+		tmp->name = strdup(name);
+		tmp->module = strdup(module ?: "");
+		tmp->type = strdup(type);
+		if (!tmp->name || !tmp->module || !tmp->type) {
+			free(tmp->name);
+			free(tmp->module);
+			free(tmp->type);
+			break;
+		}
+		error_injection_cnt++;
+	}
+	free(line);
+	fclose(f);
+}
+
+static void free_error_injection_list(void)
+{
+	size_t i;
+
+	for (i = 0; i < error_injection_cnt; i++) {
+		free(error_injections[i].name);
+		free(error_injections[i].module);
+		free(error_injections[i].type);
+	}
+	free(error_injections);
+	error_injections = NULL;
+	error_injection_cnt = 0;
+}
+
+/* Return the error injection type of function NAME of OWNER, or NULL */
+static const char *error_injection_type(const char *name, const char *owner)
+{
+	const char *module = strcmp(owner, "vmlinux") ? owner : "";
+	size_t i;
+
+	for (i = 0; i < error_injection_cnt; i++)
+		if (!strcmp(error_injections[i].name, name) &&
+		    !strcmp(error_injections[i].module, module))
+			return error_injections[i].type;
+	return NULL;
+}
+
+static void __printf(2, 0) prototype_printf(void *ctx, const char *fmt, va_list args)
+{
+	vfprintf(ctx, fmt, args);
+}
+
+/* Return the declaration of function FUNC_ID, or NULL */
+static char *func_prototype(const struct btf *btf, __u32 func_id)
+{
+	const struct btf_type *t = btf__type_by_id(btf, func_id);
+	LIBBPF_OPTS(btf_dump_emit_type_decl_opts, opts,
+		.field_name = btf__name_by_offset(btf, t->name_off),
+	);
+	struct btf_dump *d;
+	char *buf = NULL;
+	size_t size = 0;
+	FILE *f;
+	int err;
+
+	f = open_memstream(&buf, &size);
+	if (!f)
+		return NULL;
+	d = btf_dump__new(btf, prototype_printf, f, NULL);
+	err = libbpf_get_error(d);
+	if (!err) {
+		err = btf_dump__emit_type_decl(d, t->type, &opts);
+		btf_dump__free(d);
+	}
+	fclose(f);
+	if (err) {
+		free(buf);
+		return NULL;
+	}
+	return buf;
+}
+
+/* Show 0 and the values of a "bpf_fmodret_retval:" tag as a JSON array */
+static void show_retval_json(const char *retval)
+{
+	const char *p = retval;
+	char *end;
+
+	jsonw_name(json_wtr, "retval");
+	jsonw_start_array(json_wtr);
+	jsonw_int(json_wtr, 0);
+	while (*p) {
+		long long value;
+
+		errno = 0;
+		value = strtoll(p, &end, 10);
+		if (end == p || errno)
+			break;
+		jsonw_int(json_wtr, value);
+		p = *end == ',' ? end + 1 : end;
+	}
+	jsonw_end_array(json_wtr);
+}
+
+/*
+ * Check the "bpf_fmodret_retval:" tag of FUNC_ID as the verifier does: negative
+ * decimal numbers separated by commas, which the integer or enum of 4 or 8 bytes
+ * that the function returns can hold. Return why it is invalid, or NULL.
+ */
+static const char *retval_invalid(const struct btf *btf, __u32 func_id,
+				  const char *retval)
+{
+	const struct btf_type *t = btf__type_by_id(btf, func_id);
+	const char *p = retval;
+	int ret_id;
+	char *end;
+
+	t = btf__type_by_id(btf, t->type);
+	ret_id = t ? btf__resolve_type(btf, t->type) : -EINVAL;
+	t = ret_id > 0 ? btf__type_by_id(btf, ret_id) : NULL;
+	if (!t || !(btf_is_int(t) || btf_is_any_enum(t)) || (t->size != 4 && t->size != 8))
+		return "the function does not return an integer of 4 or 8 bytes";
+	for (;;) {
+		long long value;
+
+		if (*p != '-')
+			return "malformed tag";
+		errno = 0;
+		value = strtoll(p, &end, 10);
+		if (end == p || errno || value >= 0)
+			return "malformed tag";
+		if (t->size == 4 && value < INT_MIN)
+			return "a value does not fit the return type";
+		if (!*end)
+			return NULL;
+		if (*end != ',')
+			return "malformed tag";
+		p = end + 1;
+	}
+}
+
+/* Show the values of a "bpf_fmodret_retval:" tag with the names of errors */
+static void show_retval_plain(const char *retval)
+{
+	const char *p = retval, *name;
+	char *end;
+
+	printf("\tretval: 0");
+	while (*p) {
+		long long value;
+
+		errno = 0;
+		value = strtoll(p, &end, 10);
+		if (end == p || errno)
+			break;
+		printf(", %lld", value);
+		name = errno_name(value);
+		if (name)
+			printf(" (%s)", name);
+		p = *end == ',' ? end + 1 : end;
+	}
+	printf("\n");
+}
+
+static int show_function(const char *owner, const struct btf *btf,
+			 const struct fmodret_entry *entry,
+			 struct fmodret_filter *filter)
+{
+	const struct btf_type *t = btf__type_by_id(btf, entry->btf_id);
+	const char *name = btf__name_by_offset(btf, t->name_off);
+	const char *injection = NULL, *invalid = NULL;
+	char *prototype;
+
+	/* The verifier rejects programs on a function with an invalid tag */
+	if (entry->target && entry->retval_dup)
+		invalid = "more than one tag";
+	else if (entry->target && entry->retval)
+		invalid = retval_invalid(btf, entry->btf_id, entry->retval);
+	prototype = func_prototype(btf, entry->btf_id);
+	if (!prototype) {
+		p_err("failed to print the prototype of %s", name);
+		return -EINVAL;
+	}
+	if (filter->live)
+		injection = error_injection_type(name, owner);
+
+	if (json_output) {
+		jsonw_start_object(json_wtr);
+		jsonw_uint_field(json_wtr, "btf_id", entry->btf_id);
+		jsonw_string_field(json_wtr, "name", name);
+		jsonw_string_field(json_wtr, "owner", owner);
+		jsonw_string_field(json_wtr, "prototype", prototype);
+		jsonw_string_field(json_wtr, "target", !entry->target ? "none" :
+				   entry->retval ? "found" : "declared");
+		jsonw_bool_field(json_wtr, "sleepable", entry->target && entry->sleepable);
+		if (invalid)
+			jsonw_string_field(json_wtr, "retval_error", invalid);
+		else if (entry->target && entry->retval)
+			show_retval_json(entry->retval);
+		if (injection)
+			jsonw_string_field(json_wtr, "error_injection", injection);
+		jsonw_end_object(json_wtr);
+	} else {
+		printf("%u  %s (%s)\n", entry->btf_id, name, owner);
+		printf("\t%s\n", prototype);
+		if (entry->target) {
+			printf("\ttarget: %s\n", entry->retval ? "found by the kernel build" :
+			       "declared");
+			printf("\tsleepable: %s\n", entry->sleepable ? "yes" : "no");
+			if (invalid)
+				printf("\tretval: invalid, %s\n", invalid);
+			else if (entry->retval)
+				show_retval_plain(entry->retval);
+			else
+				printf("\tretval: any\n");
+		} else {
+			printf("\ttarget: not tagged, the verifier may accept it by other rules\n");
+		}
+		if (injection)
+			printf("\terror injection: %s\n", injection);
+	}
+	free(prototype);
+	filter->found++;
+	return 0;
+}
+
+static int show_btf(const char *owner, const struct btf *btf,
+		    struct fmodret_filter *filter)
+{
+	__u32 id, nr_types = btf__type_cnt(btf);
+	struct fmodret_entry entry;
+	int err;
+
+	if (filter->btf_id) {
+		if (filter->btf_id < first_own_type(btf) || filter->btf_id >= nr_types ||
+		    !btf_is_func(btf__type_by_id(btf, filter->btf_id)))
+			return 0;
+		func_entry(btf, filter->btf_id, &entry);
+		return show_function(owner, btf, &entry, filter);
+	}
+
+	for (id = first_own_type(btf); id < nr_types; id++) {
+		const struct btf_type *t = btf__type_by_id(btf, id);
+
+		if (!btf_is_func(t) ||
+		    strcmp(btf__name_by_offset(btf, t->name_off), filter->name))
+			continue;
+		func_entry(btf, id, &entry);
+		err = show_function(owner, btf, &entry, filter);
+		if (err)
+			return err;
+	}
+	return 0;
+}
+
+static int filter_module(const struct dirent *entry)
+{
+	size_t len = strlen(entry->d_name);
+
+	/* Skip ".", "..", vmlinux and the ".inline" companions of modules */
+	return entry->d_name[0] != '.' && strcmp(entry->d_name, "vmlinux") &&
+	       !(len > 7 && !strcmp(entry->d_name + len - 7, ".inline"));
+}
+
+static int walk_modules(struct btf *vmlinux, struct fmodret_filter *filter, btf_cb cb)
+{
+	struct dirent **modules = NULL;
+	int i, nr_modules, err = 0;
+
+	if (filter->module) {
+		struct btf *btf;
+
+		btf = btf__load_module_btf(filter->module, vmlinux);
+		err = libbpf_get_error(btf);
+		if (err) {
+			p_err("failed to load BTF of module %s: %s", filter->module,
+			      strerror(-err));
+			return err;
+		}
+		err = cb(filter->module, btf, filter);
+		btf__free(btf);
+		return err;
+	}
+
+	nr_modules = scandir(SYS_BTF_DIR, &modules, filter_module, alphasort);
+	if (nr_modules < 0) {
+		/* There is no module directory without module BTF, which is fine */
+		if (errno == ENOENT)
+			return 0;
+		err = -errno;
+		p_err("failed to read %s: %s", SYS_BTF_DIR, strerror(errno));
+		return err;
+	}
+	for (i = 0; i < nr_modules; i++) {
+		const char *name = modules[i]->d_name;
+		struct btf *btf;
+
+		btf = btf__load_module_btf(name, vmlinux);
+		err = libbpf_get_error(btf);
+		if (err == -ENOENT) {
+			/* The module was unloaded meanwhile */
+			err = 0;
+			continue;
+		}
+		if (err) {
+			p_err("failed to load BTF of module %s: %s", name, strerror(-err));
+			break;
+		}
+		err = cb(name, btf, filter);
+		btf__free(btf);
+		if (err)
+			break;
+	}
+	for (i = 0; i < nr_modules; i++)
+		free(modules[i]);
+	free(modules);
+	return err;
+}
+
+static int walk_live_btf(struct fmodret_filter *filter, btf_cb cb)
+{
+	struct btf *vmlinux;
+	int err = 0;
+
+	vmlinux = btf__load_vmlinux_btf();
+	err = libbpf_get_error(vmlinux);
+	if (err) {
+		p_err("failed to load vmlinux BTF: %s", strerror(-err));
+		return err;
+	}
+	if (!filter->module)
+		err = cb("vmlinux", vmlinux, filter);
+	if (!err && !filter->vmlinux)
+		err = walk_modules(vmlinux, filter, cb);
+	btf__free(vmlinux);
+	return err;
+}
+
+static int walk_file_btf(const char *path, struct fmodret_filter *filter, btf_cb cb)
+{
+	struct btf *btf;
+	int err;
+
+	btf = base_btf ? btf__parse_split(path, base_btf) : btf__parse(path, NULL);
+	err = libbpf_get_error(btf);
+	if (err) {
+		p_err("failed to parse BTF from %s: %s", path, strerror(-err));
+		return err;
+	}
+	err = cb(path, btf, filter);
+	btf__free(btf);
+	return err;
+}
+
+/* Parse the optional OBJECT := { file FILE | vmlinux | module NAME } */
+static int parse_object(int argc, char **argv, const char **path,
+			struct fmodret_filter *filter)
+{
+	if (!argc)
+		return 0;
+	if (is_prefix(*argv, "vmlinux")) {
+		filter->vmlinux = true;
+		NEXT_ARG();
+	} else if (is_prefix(*argv, "file")) {
+		NEXT_ARG();
+		if (!REQ_ARGS(1))
+			return -1;
+		*path = GET_ARG();
+	} else if (is_prefix(*argv, "module")) {
+		NEXT_ARG();
+		if (!REQ_ARGS(1))
+			return -1;
+		filter->module = GET_ARG();
+	}
+	if (argc)
+		return BAD_ARG();
+	return 0;
+}
+
+static int walk_btf(const char *path, struct fmodret_filter *filter, btf_cb cb)
+{
+	return path ? walk_file_btf(path, filter, cb) : walk_live_btf(filter, cb);
+}
+
+static int do_list(int argc, char **argv)
+{
+	struct fmodret_filter filter = {};
+	const char *path = NULL;
+	int err;
+
+	if (parse_object(argc, argv, &path, &filter))
+		return -1;
+
+	if (json_output)
+		jsonw_start_array(json_wtr);
+	err = walk_btf(path, &filter, list_btf);
+	if (json_output)
+		jsonw_end_array(json_wtr);
+	return err;
+}
+
+static int do_show(int argc, char **argv)
+{
+	struct fmodret_filter filter = {};
+	const char *path = NULL;
+	unsigned long id;
+	char *endptr;
+	int err;
+
+	if (!REQ_ARGS(2))
+		return -1;
+	if (is_prefix(*argv, "name")) {
+		NEXT_ARG();
+		filter.name = GET_ARG();
+	} else if (is_prefix(*argv, "id")) {
+		NEXT_ARG();
+		errno = 0;
+		id = strtoul(*argv, &endptr, 0);
+		if (errno || *endptr || !id || id > UINT_MAX || **argv == '-') {
+			p_err("can't parse %s as BTF ID", *argv);
+			return -1;
+		}
+		filter.btf_id = id;
+		NEXT_ARG();
+	} else {
+		p_err("expected 'name' or 'id', got: '%s'", *argv);
+		return -1;
+	}
+	if (parse_object(argc, argv, &path, &filter))
+		return -1;
+	/* The BTF IDs of different modules overlap */
+	if (filter.btf_id && !path && !filter.module)
+		filter.vmlinux = true;
+
+	filter.live = !path;
+	if (filter.live)
+		read_error_injection_list();
+	/* A BTF ID gives one function, a name an array of them */
+	if (json_output && !filter.btf_id)
+		jsonw_start_array(json_wtr);
+	err = walk_btf(path, &filter, show_btf);
+	if (!err && !filter.found) {
+		if (filter.name)
+			p_err("no function named %s", filter.name);
+		else
+			p_err("no function with BTF ID %u", filter.btf_id);
+		err = -ENOENT;
+	}
+	if (json_output && !filter.btf_id)
+		jsonw_end_array(json_wtr);
+	free_error_injection_list();
+	return err;
+}
+
+static int do_help(int argc, char **argv)
+{
+	if (json_output) {
+		jsonw_null(json_wtr);
+		return 0;
+	}
+
+	fprintf(stderr,
+		"Usage: %1$s %2$s list [OBJECT]\n"
+		"       %1$s %2$s show FUNC [OBJECT]\n"
+		"       %1$s %2$s help\n"
+		"\n"
+		"       FUNC := { name NAME | id BTF_ID }\n"
+		"       OBJECT := { file FILE | vmlinux | module NAME }\n"
+		"       " HELP_SPEC_OPTIONS " |\n"
+		"                    {-B|--base-btf} FILE }\n"
+		"\n",
+		bin_name, "fmodret");
+	return 0;
+}
+
+static const struct cmd cmds[] = {
+	{ "list",	do_list },
+	{ "show",	do_show },
+	{ "help",	do_help },
+	{ 0 }
+};
+
+int do_fmodret(int argc, char **argv)
+{
+	return cmd_select(cmds, argc, argv, do_help);
+}
diff --git a/tools/bpf/bpftool/main.c b/tools/bpf/bpftool/main.c
index d15fec71b64e..e2a7571b097c 100644
--- a/tools/bpf/bpftool/main.c
+++ b/tools/bpf/bpftool/main.c
@@ -65,7 +65,7 @@ static int do_help(int argc, char **argv)
 		"       %s batch file FILE\n"
 		"       %s version\n"
 		"\n"
-		"       OBJECT := { prog | map | link | cgroup | perf | net | feature | btf | gen | struct_ops | iter | token }\n"
+		"       OBJECT := { prog | map | link | cgroup | perf | net | feature | btf | gen | struct_ops | iter | fmodret | token }\n"
 		"       " HELP_SPEC_OPTIONS " |\n"
 		"                    {-V|--version} }\n"
 		"",
@@ -91,6 +91,7 @@ static const struct cmd commands[] = {
 	{ "gen",	do_gen },
 	{ "struct_ops",	do_struct_ops },
 	{ "iter",	do_iter },
+	{ "fmodret",	do_fmodret },
 	{ "token",	do_token },
 	{ "version",	do_version },
 	{ 0 }
diff --git a/tools/bpf/bpftool/main.h b/tools/bpf/bpftool/main.h
index b1eb4b5d991b..4d877b803b47 100644
--- a/tools/bpf/bpftool/main.h
+++ b/tools/bpf/bpftool/main.h
@@ -178,6 +178,7 @@ int do_tracelog(int argc, char **arg) __weak;
 int do_feature(int argc, char **argv) __weak;
 int do_struct_ops(int argc, char **argv) __weak;
 int do_iter(int argc, char **argv) __weak;
+int do_fmodret(int argc, char **argv) __weak;
 int do_token(int argc, char **argv) __weak;
 
 int parse_u32_arg(int *argc, char ***argv, __u32 *val, const char *what);
-- 
2.43.0

Re: [PATCH v2 bpf-next 11/18] bpf: Follow PHIs in bpf_scan_ir

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1924 bytes --]

> bpf: Follow PHIs in bpf_scan_ir

TL;DR: Nits only: in bpf_scan_ir.cpp, isErrnoRange() duplicates
errnoConstant()'s errno check, so the range is defined twice; also a comment
style nit.

> diff --git a/scripts/llvm-plugins/bpf_scan_ir.cpp b/scripts/llvm-plugins/bpf_scan_ir.cpp
> --- a/scripts/llvm-plugins/bpf_scan_ir.cpp
> +++ b/scripts/llvm-plugins/bpf_scan_ir.cpp

[ ... ]

> @@ -34,9 +36,40 @@ namespace {
>  
>  static constexpr int MaxErrno = 4095;
>  
> +// is Range in [-4095, 0) ?
> +static bool isErrnoRange(const ConstantRange &Range)
> +{
> +	unsigned Width = Range.getBitWidth();
> +
> +	if (Width != 32 && Width != 64)
> +		return false;
> +
> +	auto L = APInt(Width, 0) - APInt(Width, MaxErrno);
> +	auto R = APInt(Width, 0);
> +	return !Range.isEmptySet() && ConstantRange(L, R).contains(Range);
> +}
> +

this isn't a bug, but could singleErrno() and errnoConstant() share one
check?

isErrnoRange() has a single caller, singleErrno(), which only passes it a
range that getSingleElement() already showed to hold exactly one value.
For that input the width check, the isEmptySet() test and the wrapped
ConstantRange containment do the same thing as errnoConstant() below
(width of 32 or 64, then N >= -MaxErrno && N < 0), so the errno range is
now defined in two places, and the !Range.isEmptySet() test can never
fail.

For example, errnoConstant() could call
singleErrno(ConstantRange(C->getValue()), N), or singleErrno() could test
the sign-extended element against MaxErrno directly.

Also, the "// is Range in [-4095, 0) ?" comment is the only function-level
comment in the file using // rather than /* */, and it hardcodes -4095
where the code uses MaxErrno.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42662799554

Re: [PATCH v2 bpf-next 18/18] bpftool: List and show fmod_ret targets

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1359 bytes --]

> bpftool: List and show fmod_ret targets

TL;DR: Nit only: bpftool.rst adds fmodret to the OBJECT list without a
matching *FMODRET-COMMANDS* line like the other objects have; suggests adding
one.

> diff --git a/tools/bpf/bpftool/Documentation/bpftool.rst b/tools/bpf/bpftool/Documentation/bpftool.rst
> index f38ae5c40439..2c64c22a5026 100644
> --- a/tools/bpf/bpftool/Documentation/bpftool.rst
> +++ b/tools/bpf/bpftool/Documentation/bpftool.rst
> @@ -21,7 +21,7 @@ SYNOPSIS
>  **bpftool** **version**
>  
>  *OBJECT* := { **map** | **prog** | **link** | **cgroup** | **perf** | **net** | **feature** |
> -**btf** | **gen** | **struct_ops** | **iter** }
> +**btf** | **gen** | **struct_ops** | **iter** | **fmodret** }

This isn't a bug, but fmodret is added to the OBJECT list without a
matching *FMODRET-COMMANDS* line.  Every other object in that list has
one (*MAP-COMMANDS*, *PROG-COMMANDS*, ... *ITER-COMMANDS*), as set up by
commit 6585abea98ae ("bpftool: man: Add missing top level docs").

Should a line like this be added after *ITER-COMMANDS*?

  *FMODRET-COMMANDS* := { **list** | **show** | **help** }


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42662799554

Re: [PATCH v2 bpf-next 01/18] bpf: Add BTF ID sets for fmod_ret targets

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2233 bytes --]

> bpf: Add BTF ID sets for fmod_ret targets

TL;DR: Nits only in resolve_btfids: a comment may misleadingly suggest entries
with a missing function are rejected (they only warn), and a new outer "off"
is shadowed by the inner one.

> diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c
> --- a/tools/bpf/resolve_btfids/main.c
> +++ b/tools/bpf/resolve_btfids/main.c

[ ... ]

> @@ -1344,7 +1355,16 @@ static bool is_arena_param(const struct btf *btf, const struct btf_param *param)
>  	       param_name_has_suffix(name, PARAM_SUFFIX_ARENA_NULLABLE);
>  }
>  
> -static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx)
> +typedef int (*set_func_fn)(struct btf2btf_context *ctx, const char *name,
> +			   u32 btf_id, u32 flags);
> +
> +/*
> + * Call FN for each function in the BTF ID sets which have SET_FLAG set. Reject
> + * an entry with flags other than ALLOWED_FLAGS, also if its function is missing.
> + */

This isn't a bug, but could this read "even if its function is missing"?
As written it can suggest that entries with a missing function are
rejected, while collect_set_funcs() only prints a WARN, increments
warnings and continues to the next entry.

[ ... ]

> @@ -1356,24 +1376,39 @@ static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx)
>  	for (next = rb_first(&obj->sets); next; next = rb_next(next)) {
>  		struct btf_id_set8 *set8;
>  		struct btf_id *set_id;
> -		u64 set_addr;
> +		u64 set_addr, off;

[ ... ]

>  		for (u32 i = 0; i < set_id->cnt; i++) {
>  			size_t off = (char *)&set8->pairs[i] - (char *)set8;
>  			const char *name = find_name_by_addr(obj, set_addr + off);

This isn't a bug, but would a distinct name for the new outer offset, for
example set_off, make it clearer which offset "set_addr + off" uses here?

The inner "size_t off" (offset of a pair within the set) now shadows the
outer "off" (offset of the set within .BTF_ids), and the inner declaration
would trigger -Wshadow.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42662799554

Re: [PATCH v2 bpf-next 13/18] bpf: Check the return values of fmod_ret programs on found targets

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3043 bytes --]

> bpf: Check the return values of fmod_ret programs on found targets

TL;DR: The fmod_ret return check may wrongly reject valid programs: when
states returning two allowed constants merge at a shared exit, R0 becomes a
range, not a constant, and the load fails.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> +/* The program must return a constant: 0 or one of the values of the tag */
> +static int check_fmodret_retval(struct bpf_verifier_env *env, int regno,
> +				const char *exit_ctx, const char *reg_name)
> +{
> +	const struct bpf_prog_aux *aux = env->prog->aux;
> +	struct bpf_reg_state *reg = reg_state(env, regno);
> +	u32 i;
> +
> +	if (reg->type == SCALAR_VALUE && tnum_is_const(reg->var_off) &&
> +	    fmodret_retval_allowed(aux, reg->var_off.value))
> +		return 0;

Can this reject valid programs when two allowed constants are joined at a
shared exit?

The main pass is the fixpoint in kernel/bpf/flow.c, and flow_merge() joins
states that reach the same basic-block head.  For SCALAR_VALUE, join_reg()
-> join_range() produces the union range.  flow_join_makes_up() only
refuses the join when flow_gap() finds values in the union that neither
state has, and only while flow_has_room().  The FLOW_NEED_ARG table has no
entry for R0 at BPF_EXIT, so flow_needed_consts_differ() does not keep the
constants apart.

For a target tagged "bpf_fmodret_retval:-12,-11":

  r2 = *(u64 *)(r1 + 0)
  r0 = -11
  if r2 == 0 goto +1
  r0 = -12
  exit

The states at exit (R0=-11 and R0=-12) are disjoint but leave no gap, so
they become R0=[-12,-11] and tnum_is_const() fails.  The load then fails
with:

  At program exit the register R0 must be a constant, it should be 0 or
  one of -12,-11

even though every path returns an allowed value.

The same can happen with the common pattern "r0 = 0 ... r0 = -1; exit" when
the tag contains -1, since the 64-bit constants 0 and 0xffffffffffffffff
touch on the circle and are joined.  Once a program uses half of the state
limits, flow_has_room() is false and even gapped values such as -34 and -22
are joined.

The selftest join_both_paths avoids this only by using two separate exit
instructions, and whether exits are merged is the compiler's choice.

Every other exit check so far was a range check via retval_range_within(),
but this one needs set membership.  Should R0 at the main-prog exit (and R1
at bpf_throw without a callback) be added to the FLOW_NEED_ARG table for
programs with fmodret_retvals, or should a small bounded range be accepted
when every value in it is 0 or in the list?

The feature becomes reachable once 1c163e52401e ("resolve_btfids: Tag
fmod_ret candidates as targets") emits the tags, and no later commit in the
series changes this.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42662799554

Re: [PATCH v2 bpf-next 15/18] resolve_btfids: Tag fmod_ret candidates as targets

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3005 bytes --]

> resolve_btfids: Tag fmod_ret candidates as targets

TL;DR: resolve_btfids may tag KF_IMPLICIT_ARGS kfuncs (e.g.
bpf_wq_set_callback) as fmod_ret targets; the trampoline may then clobber the
implicit argument, and the kfunc could dereference a garbage pointer.

> diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c
> --- a/tools/bpf/resolve_btfids/main.c
> +++ b/tools/bpf/resolve_btfids/main.c

[ ... ]

> @@ -1841,6 +1888,16 @@ static int btf2btf(struct object *obj)
>  		}
>  	}
>  
> +	for (size_t i = 0; i < obj->fmodret_retvals.cnt; i++) {
> +		const struct fmodret_retval *retval = &obj->fmodret_retvals.retvals[i];
> +
> +		if (has_set_func(&ctx.fmodrets, retval->btf_id))
> +			continue;
> +		err = add_fmodret_retval_tags(&ctx, retval);
> +		if (err)
> +			goto out;
> +	}
> +

Can this loop tag a kfunc with KF_IMPLICIT_ARGS as an fmod_ret target?

The kfunc loop earlier in btf2btf() has already rewritten the BTF
prototype of such a kfunc through process_kfunc_with_implicit_args():

	t = (struct btf_type *)btf__type_by_id(btf, kfunc->btf_id);
	t->type = new_proto_id;

so the BTF FUNC no longer lists the implicit arguments.  fmodret_collect()
ran before btf2btf() and compared only the return type
(btf_func_int_return_size()), never the argument count.  The only check
here is has_set_func(&ctx.fmodrets, ...), which does not consult
ctx.kfuncs, so the same btf_id still gets "bpf_fmodret" and
"bpf_fmodret_retval:...".

bpf_wq_set_callback() looks like a concrete case.  It is built with the
plugin, has an ftrace call site, and starts with a side-effect free
error return:

	if (flags)
		return -EINVAL;
	return __bpf_async_set_callback(async, callback_fn, aux->prog);

so it would get "bpf_fmodret_retval:-22".  The kernel then adds it to
BTF_KFUNC_HOOK_FMODRET, and bpf_check_attach_target() accepts
BPF_MODIFY_RETURN through btf_kfunc_is_modify_return().

The trampoline model is built from the 3-argument prototype.  On x86,
__arch_prepare_bpf_trampoline() runs the fmod_ret programs and then calls:

	restore_regs(&prog, pos_of_slot, nr_regs, regs_off)

with nr_regs = 3 before calling the original function.  rcx (aux) is
caller-saved, so __bpf_prog_enter(), the program and __bpf_prog_exit()
may have overwritten it.  If the program returns 0, which the verifier
allows, bpf_wq_set_callback() dereferences a garbage aux->prog.

The same seems to apply to the other KF_IMPLICIT_ARGS kfuncs that return
int (bpf_task_work_schedule_*, bpf_list_push_*, scx_bpf_*) whenever the
analysis finds an error path for them.

Would it make sense to skip retvals whose btf_id is a kfunc with
KF_IMPLICIT_ARGS in ctx.kfuncs, or any kfunc whose prototype was
rewritten?  No later commit in the series changes resolve_btfids.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42662799554