[PATCH bpf-next v3 0/9] bpf: make the vmlinux BTF an on-demand loadable module (CONFIG_DEBUG_INFO_BTF=m) to save ~5.4 MB memory

[PATCH bpf-next v3 0/9] bpf: make the vmlinux BTF an on-demand loadable module (CONFIG_DEBUG_INFO_BTF=m) to save ~5.4 MB memory

[Jay Wang]

Based on and tested against bpf-next commit 6ac134ec9306 ("Merge branch
'file-descriptor-interface-for-bpf-streams'").

This series makes CONFIG_DEBUG_INFO_BTF a tristate, so that it can be
set to =m.  With =m the vmlinux BTF is carried by a module, btf_vmlinux,
that the kernel loads the first time anything needs the BTF.  On a
system where nothing does, that saves ~5.4 MB of RAM with a distribution
config; on a system that uses BTF, nothing differs from =y.  =y itself
is untouched.

Problem
-------

The vmlinux BTF that CONFIG_DEBUG_INFO_BTF=y builds into the kernel
image takes ~5.4 MB of memory, resident from boot whether anything uses
it or not.  On small instances that is not negligible.

A distribution cannot simply turn it off for the users who do not need
it: it ships one kernel build for all its users, and BTF is not debug
info anymore.  CO-RE, fentry/fexit, kfuncs, struct_ops, sched_ext and
bpf-lsm all depend on it, so =n takes those away from everyone who does
use them. 

Hence this series adds CONFIG_DEBUG_INFO_BTF=m: the BTF becomes an
on-demand module.
Users who never use BTF get the memory back; for users who do, the first
use loads it automatically and everything works as with =y.

Approach
--------

Do not remove anything, defer it.  The BTF is generated exactly as
before, but with =m it is not part of the kernel image: it is packed
into a module, btf_vmlinux.ko, which the kernel loads itself the first
time anything needs the BTF.  Once loaded, the BTF stays, and nothing
differs from =y.

Making that work ran into five problems.  The first three are existing
components that rely on the vmlinux BTF being present from boot, which
a loadable module cannot provide; the last two follow from the BTF no
longer being part of the image.

1. Verifier.  Problem: bpf_check() fetched the vmlinux BTF for every
   program, so the first socket filter at boot would have loaded the
   module on every system.  Solution: fetch the BTF only where kernel
   types enter a program (attach_btf, kfunc calls, ksyms, map pointer
   access, helpers that take or return kernel pointers).  A program
   using none of these never touches it.

2. Initcall registrations.  Problem: kfunc, dtor kfunc and struct_ops
   registrations run from initcalls and need the parsed BTF, which
   would again pull it in at boot.  Solution: queue them and apply the
   queue when the BTF is parsed, before it is published, so no program
   can ever see a vmlinux BTF that lacks its kfuncs or struct_ops.

3. Module BTF.  Problem: module BTF is split BTF against the vmlinux
   BTF and was parsed at module load.  A module loaded before the
   vmlinux BTF cannot be parsed yet, and the module notifier cannot
   load btf_vmlinux (that would nest a module load inside a module
   load).  Solution: keep the module's BTF aside (the same copy
   btf_parse_module() makes with =y, so module BTF costs the same in
   both; the saving is the vmlinux BTF only), expose it in
   /sys/kernel/btf right away, and parse and register it, together with
   the module's own kfuncs and struct_ops, once the vmlinux BTF arrives.
   Module BTF thus works regardless of load order, including out-of-tree
   modules with a .BTF.base, whose sysfs reader waits for the relocation.

4. Trust.  Problem: the verifier treats the BTF as the description of
   the kernel's types, so a carrier from a different build must be
   refused even if vermagic lets it load.  Solution: link the size and
   SHA-256 of the BTF into the image and check them when the module
   loads.  The linked-in size also lets /sys/kernel/btf/vmlinux report
   its final size before the load, which tooling expects.

5. Tooling.  Problem: pahole --btf_base for module BTF, bpftool and
   external module builds read .BTF from the vmlinux ELF.  Solution:
   keep .BTF in the ELF as a non-loadable section, so it is in the file
   but not in the image.

CONFIG_BPF_PRELOAD is made unavailable with =m: its preloaded programs
attach through the vmlinux BTF, so every bpffs mount, which systemd does
at boot, would load it and defeat the point.

Relation to the inline BTF series
---------------------------------

Alan's inline BTF series [2], now in bpf-next, adds inline function
information to BTF, which is even larger than the BTF itself, and its
cover letter leaves
delivering that information on demand, through a module and sysfs, to a
follow-up.  When Alexei suggested taking the same route for the vmlinux
BTF, Alan pointed out where the difficulty would lie [3].

However, that approach cannot be used directly for the vmlinux BTF,
because of what depends on the data:

1. Boot-time consumers.  Nothing needs inline information at boot, so
   loading it late only concerns the sysfs file.  The core vmlinux BTF
   is needed at boot by the verifier, by kfunc and struct_ops
   registrations, and by every module's BTF.  Therefore we need to make
   each of those wait for the BTF instead of expecting it at boot: the
   verifier fetches it only when a program brings kernel types in, and
   the registrations are queued and replayed once it is parsed.  That
   is most of this series.

2. Module BTF.  Module BTF is split against the vmlinux BTF, so a module
   loaded before it has nothing to be parsed against, and the module
   notifier cannot load btf_vmlinux itself.  Therefore we need to keep
   the module's BTF aside, create its sysfs file right away, and parse
   and register it once the vmlinux BTF arrives; the file of a module
   built against a distilled base (.BTF.base) serves it once relocated,
   the others serve the raw bytes as they are.  Alan named this as the
   hard part; it works here regardless of whether the module loads
   before or after the vmlinux BTF.

The btf_vmlinux module added here could carry the inline information as
well when that follow-up comes, so both would share one mechanism.

Patches Structure
-----------------

Patch 1 (refactor, no functional change): btf_parse_module() takes the
vmlinux BTF as an argument and can adopt an existing copy of the module
BTF data instead of duplicating it.  Needed so that module BTF kept
aside at load time can be parsed later without moving the buffer the
sysfs file points at.

Patch 2 (refactor, no functional change): splits the kfunc, dtor kfunc
and struct_ops registration functions into "find the BTF for the owner"
and "add the registration to this BTF", so the second half can be
replayed on a queued registration.

Patch 3 (verifier): stops fetching the vmlinux BTF up front in
bpf_check() and fetches it where kernel types enter a program instead.
Adds bpf_peek_btf_vmlinux() for the helpers that run in program
context and cannot load anything.  With =y the BTF is parsed at boot
anyway, so this is invisible there.

Patch 4 (carrier): the runtime side of taking the vmlinux BTF from the
btf_vmlinux module: request it on first use, copy it out of the module
in the BTF module notifier after checking size and SHA-256 against
.BTF.meta, and serve /sys/kernel/btf/vmlinux from the copy with its
size known from boot.  All under IS_MODULE(CONFIG_DEBUG_INFO_BTF), so
unreachable until patch 9.

Patch 5 (vmlinux registrations): queues kfunc, dtor kfunc and struct_ops
registrations for vmlinux made from initcalls and applies them when the
BTF is parsed, before it is published.

Patch 6 (module BTF): keeps the BTF of modules loaded before the vmlinux
BTF, with their own queued registrations, and parses, registers and
publishes it when the vmlinux BTF arrives.

Patch 7 (.BTF.base sysfs): gives such a module with a .BTF.base its
/sys/kernel/btf file from load, with a reader that waits for the
relocation.  Patches 5-7 are also unreachable until patch 9.

Patch 8 (preparation, no functional change): the #ifdef, Makefile and
Kconfig checks of CONFIG_DEBUG_INFO_BTF that must hold for both =y and
=m use IS_ENABLED(), $(subst m,y,...) and DEBUG_INFO_BTF=n, in bpf,
tracing, netfilter, xfrm, Rust and modules.

Patch 9 (kbuild and Kconfig): makes CONFIG_DEBUG_INFO_BTF a tristate;
with =m links .BTF into vmlinux as a non-loadable section, emits
.BTF.meta, builds btf_vmlinux.ko with the vmlinux .BTF as its payload,
strips .BTF from vmlinux (module BTF is generated against
vmlinux.unstripped), excludes CONFIG_BPF_PRELOAD, and documents the
option.

Patches 1-3 and 8 are independently useful or neutral; 4-7 are dead code
until 9 flips the switch, which keeps each bisect step building and
behaving as before.

Testing
-------

Tested with 1 GiB of memory, same tree, =y against =m, both with
CONFIG_DEBUG_INFO_BTF_MODULES=y.  The on-demand behaviour is easy to
see by hand on an =m kernel:

  # lsmod | grep btf_vmlinux
      -> nothing: the BTF is not loaded at boot.
  # ls -la /sys/kernel/btf/vmlinux
      -> the file exists with its final size (from .BTF.meta), although
         the BTF behind it is not loaded yet.
  # modprobe ext4 nf_conntrack
  # ls /sys/kernel/btf/
      -> ext4, nf_conntrack, ... appear immediately, although their
         BTF is only kept aside, not parsed: there is no vmlinux BTF to
         parse it against yet.
  # lsmod | grep btf_vmlinux
      -> still nothing: loading modules does not load the vmlinux BTF.
  # cat /sys/kernel/btf/ext4 > /dev/null
  # lsmod | grep btf_vmlinux
      -> still nothing: a module's BTF file is served from the raw copy,
         reading it does not need the vmlinux BTF.
  # grep VmallocUsed /proc/meminfo
      -> baseline.
  # cat /sys/kernel/btf/vmlinux > /dev/null
  # lsmod | grep btf_vmlinux
      -> btf_vmlinux ... [permanent]: the first use loaded it, and it
         cannot be unloaded.
  # grep VmallocUsed /proc/meminfo
      -> up by ~5.5 MB: the BTF copy, allocated only now.
  # bpftool btf list
      -> vmlinux and every loaded module now have BTF ids; the modules
         loaded before were parsed and registered on the way.

The same with an out-of-tree module (built with M=, so its BTF is split
against a distilled base, .BTF.base), on a fresh boot:

  # insmod btf_extmod.ko
  # ls -la /sys/kernel/btf/btf_extmod
      -> the file exists with its final size, although the BTF behind
         it is only valid once relocated against the vmlinux BTF.
  # lsmod | grep btf_vmlinux
      -> nothing: loading the module does not load the vmlinux BTF.
  # cat /sys/kernel/btf/btf_extmod > /dev/null
  # lsmod | grep btf_vmlinux
      -> btf_vmlinux ... [permanent]: reading this file loaded the
         vmlinux BTF, relocated the module's BTF and then returned it.
  # bpftool btf dump file /sys/kernel/btf/btf_extmod
      -> the module's own types, resolved against the vmlinux BTF.
  # rmmod btf_extmod
      -> unloads normally; its file goes away with it.

Any BPF program that uses kernel types (kprobe with
bpf_get_current_task_btf(), a kfunc call, fentry, a global subprogram
taking the context, bpf_snprintf_btf(), CO-RE, ...) triggers the same
load; a plain socket filter does not.

Results:

 - MemTotal is ~5.4 MB higher with =m while the BTF is unused, which is
   the size of the .BTF section.  Once the BTF is in use, MemFree is the
   same within run-to-run noise.
 - Modules loaded before the trigger (ext4, nf_conntrack, which
   registers kfuncs from its init, xfrm_interface) get BTF ids once the
   BTF is loaded; a struct_ops map for tcp_congestion_ops and a syscall
   program calling bpf_task_from_pid() work; nf_nat loaded afterwards
   takes the usual path.
 - stat() of /sys/kernel/btf/vmlinux reports the final size before the
   load; fstat/read/mmap agree afterwards.
 - A carrier with one byte of .BTF changed is refused with -EINVAL.
 - With the in-tree bpftool and clang-built programs as the first user
   on a fresh boot: bpftool btf dump file /sys/kernel/btf/vmlinux,
   bpftool btf list, a socket filter with a global subprogram taking
   struct __sk_buff *, a raw_tp program calling bpf_snprintf_btf(), a
   CO-RE field read, and a program calling a kfunc of an out-of-tree
   module (distilled .BTF.base, kfunc registered from init, loaded
   before the vmlinux BTF), and a read of that module's own sysfs file,
   each load the BTF and work; rmmod of that module afterwards is clean.
 - lockdep and kmemleak kernels are clean in all of the above.
 - =y and =n build and behave as before; =m without module BTF works;
   every patch builds on its own.
 - The boot image on disk shrinks by the compressed BTF with =m
   (15.0 MB to 13.2 MB here).

Changes since v2 [5]:

 - Rebased onto current bpf-next; v2 no longer applied there.
 - Patch 8: the RUST and GENDWARFKSYMS pahole restrictions, written as
   "depends on !DEBUG_INFO_BTF", now say DEBUG_INFO_BTF=n, so they
   still hold with =m (found while checking the Sashiko question on
   bool options depending on DEBUG_INFO_BTF, which Kconfig handles:
   a bool whose dependency is m can still be y).

Changes since v1 [4]:

 - Split for review: v1 patch 5 is now patches 5-7 (vmlinux
   registrations, module BTF, .BTF.base sysfs), v1 patch 6 is now
   patches 8-9 (preparation of the existing checks, the switch).
 - Fetch sites added for the program context type table
   (bpf_ctx_convert: global subprograms taking the context, ctx access
   of tracing/EXT programs), for bpf_snprintf_btf()/bpf_seq_printf_btf()
   and for CO-RE candidate lookup, which now fetches before taking
   cand_cache_mutex (Sashiko, bpf-ci, Jiri).  Without
   CONFIG_DEBUG_INFO_BTF the new helper check is skipped, so nothing
   changes there.
 - Module BTF is published only after its deferred registrations are
   applied; only modules past MODULE_STATE_LIVE are replayed, with the
   module pinned; a module still in init has its queue applied at LIVE.
   Fixes the concurrent registration and COMING-module lifetime issues
   (Sashiko, bpf-ci).
 - The sysfs reader of a module with .BTF.base waits for the module's
   BTF to be relocated instead of serving the raw data (bpf-ci); sysfs
   files are no longer removed from the deferred parse path, and
   MODULE_STATE_GOING removes them outside btf_module_mutex.
 - A module whose deferred BTF fails to parse or get an id keeps the
   buffer its sysfs file serves; nothing is freed under a reader
   (Sashiko).
 - The vmlinux registration queue has its own mutex; no lock is taken
   under btf_vmlinux_lock that leads back to it (bpf-ci).
 - A failed parse is not cached with =m (Sashiko).
 - Only the carrier depends on vmlinux in Makefile.modfinal; POSIX dd
   instead of head -c (Sashiko).
 - .BTF stripped from vmlinux with =m, so no boot image carries it,
   also where the image is an ELF copy of vmlinux; module BTF is
   generated against vmlinux.unstripped (Alan).
 - Kconfig help: initramfs note, module BTF accounting (Alan).
 - btf_struct_ops_add() renamed btf_struct_ops_register() (bpf-ci);
   its stub only defined where used.
 - Tests with bpftool/libbpf userspace and an out-of-tree .BTF.base
   module added (Alan).

[1] https://lore.kernel.org/all/20260917000201.25581-2-wanjay@amazon.com/
[2] https://lore.kernel.org/bpf/20260916074118.1007116-1-alan.maguire@oracle.com/
[3] https://lore.kernel.org/all/33592fca-88a8-44aa-8d94-40e1e604554e@oracle.com/
[4] https://lore.kernel.org/bpf/20260923053948.30617-1-wanjay@amazon.com/
[5] https://lore.kernel.org/bpf/20260925211314.5118-1-wanjay@amazon.com/

Jay Wang (9):
  bpf: pass the vmlinux BTF to btf_parse_module() and let it adopt the
    data
  bpf: split the kfunc, dtor kfunc and struct_ops registration bodies
  bpf: fetch the vmlinux BTF where kernel types enter a program
  bpf: take the vmlinux BTF from the btf_vmlinux module
  bpf: defer vmlinux kfunc and struct_ops registrations
  bpf: keep module BTF until the vmlinux BTF is available
  bpf: expose deferred .BTF.base module BTF in sysfs from module load
  bpf, trace, net: prepare CONFIG_DEBUG_INFO_BTF checks for a tristate
  kbuild, bpf: allow building the vmlinux BTF as a module

 Documentation/bpf/btf.rst         |  35 ++
 Makefile                          |   8 +-
 include/asm-generic/vmlinux.lds.h |  31 +-
 include/linux/bpf.h               |   1 +
 include/linux/btf.h               |   6 +
 include/linux/btf_ids.h           |   2 +-
 include/linux/compiler_types.h    |   2 +-
 include/linux/module.h            |   2 +-
 include/trace/trace_events.h      |   2 +-
 init/Kconfig                      |   2 +-
 kernel/bpf/Makefile               |   6 +-
 kernel/bpf/bpf_struct_ops.c       |   3 +-
 kernel/bpf/btf.c                  | 922 ++++++++++++++++++++++++++----
 kernel/bpf/btf_vmlinux.c          |  23 +
 kernel/bpf/preload/Kconfig        |   4 +
 kernel/bpf/syscall.c              |   6 +
 kernel/bpf/sysfs_btf.c            |  85 ++-
 kernel/bpf/verifier.c             | 122 +++-
 kernel/module/Kconfig             |   2 +-
 kernel/module/main.c              |   4 +-
 kernel/trace/bpf_trace.c          |   3 +-
 kernel/trace/trace_syscalls.c     |   6 +-
 lib/Kconfig.debug                 |  22 +-
 net/netfilter/Makefile            |   6 +-
 net/xfrm/Makefile                 |   4 +-
 scripts/Makefile.modfinal         |  26 +-
 scripts/Makefile.vmlinux          |   5 +
 scripts/gen-btf.sh                |  95 ++-
 scripts/link-vmlinux.sh           |  25 +-
 29 files changed, 1302 insertions(+), 158 deletions(-)
 create mode 100644 kernel/bpf/btf_vmlinux.c


base-commit: 6ac134ec930642d5b574b63a72a0e999c2470c64
-- 
2.47.3

[PATCH bpf-next v3 1/9] bpf: pass the vmlinux BTF to btf_parse_module() and let it adopt the data

[Jay Wang]

Make btf_parse_module() take the vmlinux BTF as an argument instead of
fetching it with bpf_get_btf_vmlinux(), and add a data_owned flag: when
set, the passed .BTF data is an already kvmalloc()ed copy that the new
btf takes ownership of on success (on failure the caller keeps it).

Factor the sysfs file creation out of the module notifier into
btf_module_sysfs_add() and the teardown into btf_module_free(), and set
btf_mod->module right after the allocation rather than under the mutex.

No functional change.  This prepares for CONFIG_DEBUG_INFO_BTF=m, where a
module can be loaded before the vmlinux BTF is available: its .BTF is
then copied and exposed in sysfs first and parsed later, at which point
the parser must take the copy as is so that the sysfs file keeps
pointing at valid data.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 kernel/bpf/btf.c | 105 +++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 41 deletions(-)

diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 9bcfefdfb734..49c4ea7f75c5 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -6868,16 +6868,20 @@ __u32 btf_relocate_id(const struct btf *btf, __u32 id)
 
 #ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 
-static struct btf *btf_parse_module(const char *module_name, const void *data,
-				    unsigned int data_size, void *base_data,
-				    unsigned int base_data_size)
+/*
+ * Parse split module BTF against @vmlinux_btf.  @data is the module's .BTF
+ * section; if @data_owned, it is an already kvmalloc()ed copy that the new
+ * btf takes ownership of on success (on failure the caller keeps it).
+ */
+static struct btf *btf_parse_module(const char *module_name, struct btf *vmlinux_btf,
+				    void *data, unsigned int data_size, bool data_owned,
+				    void *base_data, unsigned int base_data_size)
 {
-	struct btf *btf = NULL, *vmlinux_btf, *base_btf = NULL;
+	struct btf *btf = NULL, *base_btf = NULL;
 	struct btf_verifier_env *env = NULL;
 	struct bpf_verifier_log *log;
 	int err = 0;
 
-	vmlinux_btf = bpf_get_btf_vmlinux();
 	if (IS_ERR(vmlinux_btf))
 		return vmlinux_btf;
 	if (!vmlinux_btf)
@@ -6914,7 +6918,10 @@ static struct btf *btf_parse_module(const char *module_name, const void *data,
 	btf->named_start_id = 0;
 	strscpy(btf->name, module_name);
 
-	btf->data = kvmemdup(data, data_size, GFP_KERNEL | __GFP_NOWARN);
+	if (data_owned)
+		btf->data = data;
+	else
+		btf->data = kvmemdup(data, data_size, GFP_KERNEL | __GFP_NOWARN);
 	if (!btf->data) {
 		err = -ENOMEM;
 		goto errout;
@@ -6957,7 +6964,8 @@ static struct btf *btf_parse_module(const char *module_name, const void *data,
 	if (!IS_ERR(base_btf) && base_btf != vmlinux_btf)
 		btf_free(base_btf);
 	if (btf) {
-		kvfree(btf->data);
+		if (!data_owned)
+			kvfree(btf->data);
 		kvfree(btf->types);
 		kfree(btf);
 	}
@@ -8962,6 +8970,48 @@ static DEFINE_MUTEX(btf_module_mutex);
 
 static void purge_cand_cache(struct btf *btf);
 
+static int btf_module_sysfs_add(struct btf_module *btf_mod, const char *name,
+				void *data, size_t data_size)
+{
+	struct bin_attribute *attr;
+	int err;
+
+	if (!IS_ENABLED(CONFIG_SYSFS))
+		return 0;
+
+	attr = kzalloc_obj(*attr);
+	if (!attr)
+		return -ENOMEM;
+
+	sysfs_bin_attr_init(attr);
+	attr->attr.name = name;
+	attr->attr.mode = 0444;
+	attr->size = data_size;
+	attr->private = data;
+	attr->read = sysfs_bin_attr_simple_read;
+
+	err = sysfs_create_bin_file(btf_kobj, attr);
+	if (err) {
+		pr_warn("failed to register module [%s] BTF in sysfs: %d\n",
+			name, err);
+		kfree(attr);
+		return err;
+	}
+
+	btf_mod->sysfs_attr = attr;
+	return 0;
+}
+
+static void btf_module_free(struct btf_module *btf_mod)
+{
+	if (btf_mod->sysfs_attr)
+		sysfs_remove_bin_file(btf_kobj, btf_mod->sysfs_attr);
+	purge_cand_cache(btf_mod->btf);
+	btf_put(btf_mod->btf);
+	kfree(btf_mod->sysfs_attr);
+	kfree(btf_mod);
+}
+
 static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 			     void *module)
 {
@@ -8982,7 +9032,10 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 			err = -ENOMEM;
 			goto out;
 		}
-		btf = btf_parse_module(mod->name, mod->btf_data, mod->btf_data_size,
+		btf_mod->module = module;
+
+		btf = btf_parse_module(mod->name, bpf_get_btf_vmlinux(),
+				       mod->btf_data, mod->btf_data_size, false,
 				       mod->btf_base_data, mod->btf_base_data_size);
 		if (IS_ERR(btf)) {
 			kfree(btf_mod);
@@ -9004,37 +9057,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 
 		purge_cand_cache(NULL);
 		mutex_lock(&btf_module_mutex);
-		btf_mod->module = module;
 		btf_mod->btf = btf;
 		list_add(&btf_mod->list, &btf_modules);
 		mutex_unlock(&btf_module_mutex);
 
-		if (IS_ENABLED(CONFIG_SYSFS)) {
-			struct bin_attribute *attr;
-
-			attr = kzalloc_obj(*attr);
-			if (!attr)
-				goto out;
-
-			sysfs_bin_attr_init(attr);
-			attr->attr.name = btf->name;
-			attr->attr.mode = 0444;
-			attr->size = btf->data_size;
-			attr->private = btf->data;
-			attr->read = sysfs_bin_attr_simple_read;
-
-			err = sysfs_create_bin_file(btf_kobj, attr);
-			if (err) {
-				pr_warn("failed to register module [%s] BTF in sysfs: %d\n",
-					mod->name, err);
-				kfree(attr);
-				err = 0;
-				goto out;
-			}
-
-			btf_mod->sysfs_attr = attr;
-		}
-
+		/* not fatal, the module BTF is usable without the sysfs file */
+		btf_module_sysfs_add(btf_mod, btf->name, btf->data, btf->data_size);
 		break;
 	case MODULE_STATE_LIVE:
 		mutex_lock(&btf_module_mutex);
@@ -9061,12 +9089,7 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 			 */
 			btf_free_id(btf_mod->btf);
 			list_del(&btf_mod->list);
-			if (btf_mod->sysfs_attr)
-				sysfs_remove_bin_file(btf_kobj, btf_mod->sysfs_attr);
-			purge_cand_cache(btf_mod->btf);
-			btf_put(btf_mod->btf);
-			kfree(btf_mod->sysfs_attr);
-			kfree(btf_mod);
+			btf_module_free(btf_mod);
 			break;
 		}
 		mutex_unlock(&btf_module_mutex);
-- 
2.47.3

[PATCH bpf-next v3 2/9] bpf: split the kfunc, dtor kfunc and struct_ops registration bodies

[Jay Wang]

Split __register_btf_kfunc_id_set(), register_btf_id_dtor_kfuncs() and
__register_bpf_struct_ops() into the part that looks up the BTF for the
owner and the part that adds the registration to a given BTF:
btf_kfunc_id_set_add(), btf_dtor_kfuncs_add() and
btf_struct_ops_register().

In is_valid_value_type(), look up bpf_struct_ops_common_value in the btf
the function was given rather than in the btf_vmlinux global.  The id is
a vmlinux id and a module BTF resolves it through its base, so the result
is the same; the function already uses the passed btf for every other
lookup.

No functional change.  With CONFIG_DEBUG_INFO_BTF=m, registrations made
from initcalls before the vmlinux BTF is available are queued and applied
later by the BTF parsing code, which needs the add-to-this-btf half on
its own; the struct_ops ones are applied before the parsed vmlinux BTF is
published, i.e. while btf_vmlinux is still NULL.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 kernel/bpf/bpf_struct_ops.c |  3 +-
 kernel/bpf/btf.c            | 91 ++++++++++++++++++++++---------------
 2 files changed, 57 insertions(+), 37 deletions(-)

diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c
index 1178acd72296..bf3004908d15 100644
--- a/kernel/bpf/bpf_struct_ops.c
+++ b/kernel/bpf/bpf_struct_ops.c
@@ -103,7 +103,8 @@ static bool is_valid_value_type(struct btf *btf, s32 value_id,
 	}
 	member = btf_type_member(vt);
 	mt = btf_type_by_id(btf, member->type);
-	common_value_type = btf_type_by_id(btf_vmlinux,
+	/* a vmlinux id resolves through the base BTF of a module BTF too */
+	common_value_type = btf_type_by_id(btf,
 					   st_ops_ids[IDX_ST_OPS_COMMON_VALUE_ID]);
 	if (mt != common_value_type) {
 		pr_warn("The first member of %s should be bpf_struct_ops_common_value\n",
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 49c4ea7f75c5..93b5a509baef 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -9665,11 +9665,26 @@ u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id,
 	return btf_kfunc_id_set_contains(btf, BTF_KFUNC_HOOK_FMODRET, kfunc_btf_id);
 }
 
+static int btf_kfunc_id_set_add(struct btf *btf, enum btf_kfunc_hook hook,
+				const struct btf_kfunc_id_set *kset)
+{
+	int ret, i;
+
+	for (i = 0; i < kset->set->cnt; i++) {
+		ret = btf_check_kfunc_protos(btf, btf_relocate_id(btf, kset->set->pairs[i].id),
+					     kset->set->pairs[i].flags);
+		if (ret)
+			return ret;
+	}
+
+	return btf_populate_kfunc_set(btf, hook, kset);
+}
+
 static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook,
 				       const struct btf_kfunc_id_set *kset)
 {
 	struct btf *btf;
-	int ret, i;
+	int ret;
 
 	btf = btf_get_module_btf(kset->owner);
 	if (!btf)
@@ -9677,16 +9692,7 @@ static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook,
 	if (IS_ERR(btf))
 		return PTR_ERR(btf);
 
-	for (i = 0; i < kset->set->cnt; i++) {
-		ret = btf_check_kfunc_protos(btf, btf_relocate_id(btf, kset->set->pairs[i].id),
-					     kset->set->pairs[i].flags);
-		if (ret)
-			goto err_out;
-	}
-
-	ret = btf_populate_kfunc_set(btf, hook, kset);
-
-err_out:
+	ret = btf_kfunc_id_set_add(btf, hook, kset);
 	btf_put(btf);
 	return ret;
 }
@@ -9778,21 +9784,13 @@ static int btf_check_dtor_kfuncs(struct btf *btf, const struct btf_id_dtor_kfunc
 	return 0;
 }
 
-/* This function must be invoked only from initcalls/module init functions */
-int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,
-				struct module *owner)
+static int btf_dtor_kfuncs_add(struct btf *btf, const struct btf_id_dtor_kfunc *dtors,
+			       u32 add_cnt)
 {
 	struct btf_id_dtor_kfunc_tab *tab;
-	struct btf *btf;
 	u32 tab_cnt, i;
 	int ret;
 
-	btf = btf_get_module_btf(owner);
-	if (!btf)
-		return check_btf_kconfigs(owner, "dtor kfuncs");
-	if (IS_ERR(btf))
-		return PTR_ERR(btf);
-
 	if (add_cnt >= BTF_DTOR_KFUNC_MAX_CNT) {
 		pr_err("cannot register more than %d kfunc destructors\n", BTF_DTOR_KFUNC_MAX_CNT);
 		ret = -E2BIG;
@@ -9849,6 +9847,23 @@ int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_c
 end:
 	if (ret)
 		btf_free_dtor_kfunc_tab(btf);
+	return ret;
+}
+
+/* This function must be invoked only from initcalls/module init functions */
+int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,
+				struct module *owner)
+{
+	struct btf *btf;
+	int ret;
+
+	btf = btf_get_module_btf(owner);
+	if (!btf)
+		return check_btf_kconfigs(owner, "dtor kfuncs");
+	if (IS_ERR(btf))
+		return PTR_ERR(btf);
+
+	ret = btf_dtor_kfuncs_add(btf, dtors, add_cnt);
 	btf_put(btf);
 	return ret;
 }
@@ -10487,32 +10502,36 @@ bpf_struct_ops_find(struct btf *btf, u32 type_id)
 	return NULL;
 }
 
-int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)
+static int btf_struct_ops_register(struct btf *btf, struct bpf_struct_ops *st_ops)
 {
 	struct bpf_verifier_log *log;
-	struct btf *btf;
-	int err = 0;
-
-	btf = btf_get_module_btf(st_ops->owner);
-	if (!btf)
-		return check_btf_kconfigs(st_ops->owner, "struct_ops");
-	if (IS_ERR(btf))
-		return PTR_ERR(btf);
+	int err;
 
 	log = kzalloc_obj(*log, GFP_KERNEL | __GFP_NOWARN);
-	if (!log) {
-		err = -ENOMEM;
-		goto errout;
-	}
+	if (!log)
+		return -ENOMEM;
 
 	log->level = BPF_LOG_KERNEL;
 
 	err = btf_add_struct_ops(btf, st_ops, log);
 
-errout:
 	kfree(log);
-	btf_put(btf);
+	return err;
+}
 
+int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)
+{
+	struct btf *btf;
+	int err;
+
+	btf = btf_get_module_btf(st_ops->owner);
+	if (!btf)
+		return check_btf_kconfigs(st_ops->owner, "struct_ops");
+	if (IS_ERR(btf))
+		return PTR_ERR(btf);
+
+	err = btf_struct_ops_register(btf, st_ops);
+	btf_put(btf);
 	return err;
 }
 EXPORT_SYMBOL_GPL(__register_bpf_struct_ops);
-- 
2.47.3

[PATCH bpf-next v3 3/9] bpf: fetch the vmlinux BTF where kernel types enter a program

[Jay Wang]

bpf_check() fetches the vmlinux BTF up front for every program, whether
the program uses kernel types or not.  With the upcoming
CONFIG_DEBUG_INFO_BTF=m that fetch loads a module and parses 5 MiB of
BTF, and since systemd loads socket filters at boot, it would happen on
every system, whether anything uses BTF or not.

Stop fetching up front and fetch at the points where kernel types enter
the verifier state instead:

 - bpf_add_kfunc_call(), for the first kfunc call of a program;
 - check_pseudo_btf_id(), for ldimm64 of a kernel variable;
 - check_ptr_to_map_access(), for accessing a map pointer's fields;
 - check_helper_call(), when the helper's prototype takes or returns a
   PTR_TO_BTF_ID, or is bpf_snprintf_btf()/bpf_seq_printf_btf(), which
   take the kernel type id inside a struct btf_ptr instead
   (helper_uses_vmlinux_btf());
 - the program context type table, bpf_ctx_convert, which
   btf_parse_vmlinux() fills in: global subprograms taking the context
   (btf_prepare_func_args()) and context access of tracing and EXT
   programs (btf_translate_to_vmlinux()) go through it, so its readers
   fetch the BTF (bpf_ctx_convert_type());
 - CO-RE candidate lookup (bpf_core_apply(), btf_get_ptr_to_btf_id()),
   which fetches before taking cand_cache_mutex, so that loading the
   BTF never happens under that mutex; bpf_core_find_cands() itself
   only peeks.

Together with the existing fetch in bpf_prog_load() for attach_btf and
the struct_ops map creation, every way kernel types enter a program
goes through one of these sites.  A program that uses none of them,
such as a socket filter, no longer touches the vmlinux BTF.

The two callers that used the result of bpf_get_btf_vmlinux() without
checking it, btf_prepare_func_args() and btf_check_kfunc_name(), now do.

bpf_snprintf_btf() and bpf_seq_printf_btf() run in program context and
cannot afford a fetch that may sleep.  Add bpf_peek_btf_vmlinux(), which
returns the parsed vmlinux BTF or NULL without parsing anything, and use
it there; the helpers fail with -EINVAL if the BTF is not parsed, as they
do on a kernel without BTF.

With CONFIG_DEBUG_INFO_BTF=y the vmlinux BTF is parsed at boot by the
first kfunc registration, and without BTF the new helper check is
skipped, so nothing changes for either.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 include/linux/bpf.h      |  1 +
 kernel/bpf/btf.c         | 56 ++++++++++++++++++++++++++++++-----
 kernel/bpf/verifier.c    | 64 +++++++++++++++++++++++++++++++++++-----
 kernel/trace/bpf_trace.c |  3 +-
 4 files changed, 108 insertions(+), 16 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 4bae3796c42f..e46a14809dd4 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -3183,6 +3183,7 @@ static inline s32 bpf_call_args_imm(s16 idx)
 #endif
 
 struct btf *bpf_get_btf_vmlinux(void);
+struct btf *bpf_peek_btf_vmlinux(void);
 
 /* Map specifics */
 struct xdp_frame;
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 93b5a509baef..514f9832058a 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -6478,12 +6478,25 @@ static u8 bpf_ctx_convert_map[] = {
 #undef BPF_MAP_TYPE
 #undef BPF_LINK_TYPE
 
+/*
+ * bpf_ctx_convert.t is filled in by btf_parse_vmlinux().  With
+ * CONFIG_DEBUG_INFO_BTF=m that may not have run yet: the program context
+ * types are kernel types too, so this is one of the places that loads the
+ * vmlinux BTF.  Only called from the verifier, which may sleep.
+ */
+static const struct btf_type *bpf_ctx_convert_type(void)
+{
+	if (IS_ERR_OR_NULL(bpf_get_btf_vmlinux()))
+		return NULL;
+	return bpf_ctx_convert.t;
+}
+
 static const struct btf_type *find_canonical_prog_ctx_type(enum bpf_prog_type prog_type)
 {
 	const struct btf_type *conv_struct;
 	const struct btf_member *ctx_type;
 
-	conv_struct = bpf_ctx_convert.t;
+	conv_struct = bpf_ctx_convert_type();
 	if (!conv_struct)
 		return NULL;
 	/* prog_type is valid bpf program type. No need for bounds check. */
@@ -6499,7 +6512,7 @@ static int find_kern_ctx_type_id(enum bpf_prog_type prog_type)
 	const struct btf_type *conv_struct;
 	const struct btf_member *ctx_type;
 
-	conv_struct = bpf_ctx_convert.t;
+	conv_struct = bpf_ctx_convert_type();
 	if (!conv_struct)
 		return -EFAULT;
 	/* prog_type is valid bpf program type. No need for bounds check. */
@@ -6758,7 +6771,11 @@ int get_kern_ctx_btf_id(struct bpf_verifier_log *log, enum bpf_prog_type prog_ty
 	const struct btf_type *kctx_type;
 	u32 kctx_type_id;
 
-	conv_struct = bpf_ctx_convert.t;
+	conv_struct = bpf_ctx_convert_type();
+	if (!conv_struct) {
+		bpf_log(log, "btf_vmlinux is malformed\n");
+		return -EINVAL;
+	}
 	/* get member for kernel ctx type */
 	kctx_member = btf_type_member(conv_struct) + bpf_ctx_convert_map[prog_type] * 2 + 1;
 	kctx_type_id = kctx_member->type;
@@ -8236,6 +8253,13 @@ static int btf_get_ptr_to_btf_id(struct bpf_verifier_log *log, int arg_idx,
 		t = btf_type_by_id(btf, t->type);
 	}
 
+	/* candidates are kernel types: load the vmlinux BTF, outside the mutex */
+	if (IS_ERR_OR_NULL(bpf_get_btf_vmlinux())) {
+		bpf_log(log, "arg#%d reference type('%s %s') needs the vmlinux BTF\n",
+			arg_idx, btf_type_str(t), __btf_name_by_offset(btf, t->name_off));
+		return -EINVAL;
+	}
+
 	mutex_lock(&cand_cache_mutex);
 	cc = bpf_core_find_cands(&ctx, type_id);
 	if (IS_ERR(cc)) {
@@ -8587,7 +8611,10 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			if (kern_type_id < 0)
 				return kern_type_id;
 
+			/* present: btf_get_ptr_to_btf_id() found the candidate in it */
 			vmlinux_btf = bpf_get_btf_vmlinux();
+			if (IS_ERR_OR_NULL(vmlinux_btf))
+				return -EINVAL;
 			ref_t = btf_type_by_id(vmlinux_btf, kern_type_id);
 			if (!btf_type_is_struct(ref_t)) {
 				tname = __btf_name_by_offset(vmlinux_btf, t->name_off);
@@ -9325,12 +9352,18 @@ static int btf_check_kfunc_name(struct btf *btf, const char *func_name, u32 kind
 #ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	struct btf_module *btf_mod, *tmp;
 #endif
+	struct btf *vmlinux_btf;
 	s32 id;
 
 	if (!btf_is_module(btf))
 		return 0;
 
-	id = btf_find_by_name_kind(bpf_get_btf_vmlinux(), func_name, kind);
+	/* a module BTF only exists once the vmlinux BTF is parsed */
+	vmlinux_btf = bpf_get_btf_vmlinux();
+	if (IS_ERR_OR_NULL(vmlinux_btf))
+		return -EINVAL;
+
+	id = btf_find_by_name_kind(vmlinux_btf, func_name, kind);
 	if (id >= 0) {
 		pr_err("kfunc %s (id: %d) is already present in vmlinux.\n",
 		       func_name, id);
@@ -10130,9 +10163,11 @@ bpf_core_find_cands(struct bpf_core_ctx *ctx, u32 local_type_id)
 	const char *name;
 	int id;
 
-	main_btf = bpf_get_btf_vmlinux();
-	if (IS_ERR(main_btf))
-		return ERR_CAST(main_btf);
+	/*
+	 * Callers fetch the vmlinux BTF before taking cand_cache_mutex, so
+	 * that loading it (CONFIG_DEBUG_INFO_BTF=m) happens outside the lock.
+	 */
+	main_btf = bpf_peek_btf_vmlinux();
 	if (!main_btf)
 		return ERR_PTR(-EINVAL);
 
@@ -10235,6 +10270,13 @@ int bpf_core_apply(struct bpf_core_ctx *ctx, const struct bpf_core_relo *relo,
 		struct bpf_cand_cache *cc;
 		int i;
 
+		/* candidates are kernel types: load the vmlinux BTF, outside the mutex */
+		if (IS_ERR_OR_NULL(bpf_get_btf_vmlinux())) {
+			bpf_log(ctx->log, "relo #%u: needs the vmlinux BTF\n", relo_idx);
+			kfree(specs);
+			return -EINVAL;
+		}
+
 		mutex_lock(&cand_cache_mutex);
 		cc = bpf_core_find_cands(ctx, relo->type_id);
 		if (IS_ERR(cc)) {
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 03dbc0e00398..da86162ba6d9 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2882,7 +2882,8 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	tab = prog_aux->kfunc_tab;
 	btf_tab = prog_aux->kfunc_btf_tab;
 	if (!tab) {
-		if (!btf_vmlinux) {
+		/* with CONFIG_DEBUG_INFO_BTF=m this is where the vmlinux BTF gets loaded */
+		if (IS_ERR_OR_NULL(bpf_get_btf_vmlinux())) {
 			verbose(env, "calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF\n");
 			return -ENOTSUPP;
 		}
@@ -6512,7 +6513,8 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env,
 	u32 btf_id;
 	int ret;
 
-	if (!btf_vmlinux) {
+	/* with CONFIG_DEBUG_INFO_BTF=m this is where the vmlinux BTF gets loaded */
+	if (IS_ERR_OR_NULL(bpf_get_btf_vmlinux())) {
 		verbose(env, "map_ptr access not supported without CONFIG_DEBUG_INFO_BTF\n");
 		return -ENOTSUPP;
 	}
@@ -12101,6 +12103,24 @@ static int release_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 	return err;
 }
 
+/* Does calling helper @func_id bring kernel BTF types into the program? */
+static bool helper_uses_vmlinux_btf(enum bpf_func_id func_id,
+				    const struct bpf_func_proto *fn)
+{
+	int i;
+
+	/* these take the kernel type id in a struct btf_ptr, not in a register */
+	if (func_id == BPF_FUNC_snprintf_btf || func_id == BPF_FUNC_seq_printf_btf)
+		return true;
+	if (base_type(fn->ret_type) == RET_PTR_TO_BTF_ID)
+		return true;
+	for (i = 0; i < MAX_BPF_FUNC_ARGS; i++) {
+		if (base_type(fn->arg_type[i]) == ARG_PTR_TO_BTF_ID)
+			return true;
+	}
+	return false;
+}
+
 static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			     int *insn_idx_p)
 {
@@ -12170,6 +12190,18 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 		return err;
 	}
 
+	/*
+	 * Helpers that take or return kernel BTF pointers need the vmlinux
+	 * BTF; with CONFIG_DEBUG_INFO_BTF=m this is where it gets loaded.
+	 * Without CONFIG_DEBUG_INFO_BTF they keep failing as they always did.
+	 */
+	if (IS_ENABLED(CONFIG_DEBUG_INFO_BTF) && helper_uses_vmlinux_btf(func_id, fn) &&
+	    IS_ERR_OR_NULL(bpf_get_btf_vmlinux())) {
+		verbose(env, "helper %s#%d is not supported without vmlinux BTF\n",
+			func_id_name(func_id), func_id);
+		return -ENOTSUPP;
+	}
+
 	if (fn->might_sleep && !in_sleepable_context(env)) {
 		const char *suggestion;
 
@@ -19853,12 +19885,13 @@ static int check_pseudo_btf_id(struct bpf_verifier_env *env,
 			return -EINVAL;
 		}
 	} else {
-		if (!btf_vmlinux) {
+		/* with CONFIG_DEBUG_INFO_BTF=m this is where the vmlinux BTF gets loaded */
+		btf = bpf_get_btf_vmlinux();
+		if (IS_ERR_OR_NULL(btf)) {
 			verbose(env, "kernel is missing BTF, make sure CONFIG_DEBUG_INFO_BTF=y is specified in Kconfig.\n");
 			return -EINVAL;
 		}
-		btf_get(btf_vmlinux);
-		btf = btf_vmlinux;
+		btf_get(btf);
 	}
 
 	err = __check_pseudo_btf_id(env, insn, aux, btf);
@@ -21912,6 +21945,17 @@ struct btf *bpf_get_btf_vmlinux(void)
 	return btf;
 }
 
+/*
+ * The vmlinux BTF if it has been parsed already, else NULL.  Unlike
+ * bpf_get_btf_vmlinux() this never loads or parses anything, so it is safe
+ * to call from a running BPF program.
+ */
+struct btf *bpf_peek_btf_vmlinux(void)
+{
+	/* Pairs with the smp_store_release() in bpf_get_btf_vmlinux() */
+	return smp_load_acquire(&btf_vmlinux);
+}
+
 /*
  * The add_fd_from_fd_array() is executed only if fd_array_cnt is non-zero. In
  * this case expect that every file descriptor in the array is either a map or
@@ -22511,7 +22555,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret)
 		goto err_prep;
 
-	bpf_get_btf_vmlinux();
+	/*
+	 * The vmlinux BTF is not fetched up front: with CONFIG_DEBUG_INFO_BTF=m
+	 * it is loaded on demand, at the points where kernel types enter the
+	 * program (attach_btf, kfuncs, ksyms, map pointers, BTF-typed helpers).
+	 */
 
 	/* Serialize verification of unprivileged programs. */
 	if (!is_priv)
@@ -22532,10 +22580,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 
 	mark_verifier_state_clean(env);
 
-	if (IS_ERR(btf_vmlinux)) {
+	if (IS_ERR(bpf_peek_btf_vmlinux())) {
 		/* Either gcc or pahole or kernel are broken. */
 		verbose(env, "in-kernel BTF is malformed\n");
-		ret = PTR_ERR(btf_vmlinux);
+		ret = PTR_ERR(bpf_peek_btf_vmlinux());
 		goto skip_full_check;
 	}
 
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 195f78db9bda..c022b2877f0b 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1015,7 +1015,8 @@ static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
 	if (btf_ptr_size != sizeof(struct btf_ptr))
 		return -EINVAL;
 
-	*btf = bpf_get_btf_vmlinux();
+	/* Called from a running program: only use the BTF if it is parsed. */
+	*btf = bpf_peek_btf_vmlinux();
 
 	if (IS_ERR_OR_NULL(*btf))
 		return IS_ERR(*btf) ? PTR_ERR(*btf) : -EINVAL;
-- 
2.47.3

[PATCH bpf-next v3 4/9] bpf: take the vmlinux BTF from the btf_vmlinux module

[Jay Wang]

Add the runtime side of delivering the vmlinux BTF as a module: with
CONFIG_DEBUG_INFO_BTF=m the BTF is carried by a module named btf_vmlinux
and installed by the BTF module notifier when it loads, and whoever needs
the BTF first loads the module.  Nothing in this patch is reachable yet:
CONFIG_DEBUG_INFO_BTF is still a bool and every new path is under
IS_MODULE(CONFIG_DEBUG_INFO_BTF); the kbuild side and the Kconfig change
follow.

With CONFIG_DEBUG_INFO_BTF=y the vmlinux BTF, 5.4 MiB on x86-64 with a
distribution config, is part of the kernel image and resident from boot
whether anything uses it or not.  Most systems never do.  Carrying it in
a module that is loaded on first use makes the memory a cost of using
BTF rather than of having a kernel that supports it.

btf_vmlinux_data() hands out the raw vmlinux BTF: from __start_BTF with
=y, or with =m from a vmalloc_user() copy that the notifier makes when
the btf_vmlinux module loads.  If the copy is not there and the caller
asked to load, it calls request_module("btf_vmlinux"); the notifier
installs the copy before init_module() returns, so the data is either
present afterwards or the module is not available (yet).  The result is
not cached, a later call retries.  btf_parse_vmlinux() and
bpf_get_btf_vmlinux() use it; the latter loads outside btf_vmlinux_lock
so that the notifier is never blocked by the caller, and returns NULL
like a kernel without BTF when the module cannot be loaded.  A failed
parse is not remembered with =m: the payload was checked against the
kernel when the module loaded, so a failure is a resource problem, and
the next caller retries.

The verifier trusts the BTF as the description of this kernel's types, so
the notifier only accepts a payload whose size and SHA-256 match the
values linked into the kernel as .BTF.meta (struct btf_vmlinux_meta,
filled in by scripts/gen-btf.sh in a later patch).  A carrier from
another build is refused with -EINVAL even if vermagic lets it load, and
a second carrier is ignored.  The copy is never freed: as with =y, the
BTF stays for the lifetime of the kernel, and the carrier has no exit.

The module notifier, btf_parse_module() and the btf_data fields in struct
module are compiled for CONFIG_DEBUG_INFO_BTF_MODULES or =m; with =m and
no module BTF, the notifier only recognizes the carrier.

/sys/kernel/btf/vmlinux exists from boot with its final size, which is
known from .BTF.meta before the BTF is loaded; the first read() or mmap()
loads it.  mmap() uses remap_vmalloc_range() on the copy.  This keeps
stat() working before the load, which is what the btf_sysfs selftest
does.

BPF_BTF_GET_NEXT_ID loads the BTF too: kernel BTFs get their ids when
the vmlinux BTF is parsed, and whoever enumerates BTF ids wants them.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 include/linux/btf.h    |   1 +
 include/linux/module.h |   2 +-
 kernel/bpf/btf.c       | 156 ++++++++++++++++++++++++++++++++++++++---
 kernel/bpf/syscall.c   |   6 ++
 kernel/bpf/sysfs_btf.c |  85 +++++++++++++++++++++-
 kernel/bpf/verifier.c  |  51 ++++++++++----
 kernel/module/main.c   |   4 +-
 7 files changed, 278 insertions(+), 27 deletions(-)

diff --git a/include/linux/btf.h b/include/linux/btf.h
index 4b63bb91550a..81e6c65fe5f6 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -602,6 +602,7 @@ __u32 *btf_field_iter_next(struct btf_field_iter *it);
 const char *btf_name_by_offset(const struct btf *btf, u32 offset);
 const char *btf_str_by_offset(const struct btf *btf, u32 offset);
 struct btf *btf_parse_vmlinux(void);
+void *btf_vmlinux_data(u32 *size, bool load);
 struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog);
 u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog);
 int btf_kfunc_check_flag(const struct btf *btf, u32 kfunc_btf_id, u32 flag);
diff --git a/include/linux/module.h b/include/linux/module.h
index 96cc98568eea..82734996a862 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -497,7 +497,7 @@ struct module {
 	unsigned int num_bpf_raw_events;
 	struct bpf_raw_event_map *bpf_raw_events;
 #endif
-#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+#if IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES) || IS_MODULE(CONFIG_DEBUG_INFO_BTF)
 	unsigned int btf_data_size;
 	unsigned int btf_base_data_size;
 	void *btf_data;
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 514f9832058a..1dd7f9650ae8 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -29,6 +29,7 @@
 #include <linux/string.h>
 #include <linux/sysfs.h>
 #include <linux/overflow.h>
+#include <crypto/sha2.h>
 #include <linux/bitops.h>
 
 #include <net/netfilter/nf_bpf_link.h>
@@ -6444,10 +6445,73 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr,
 	return ERR_PTR(err);
 }
 
+#if IS_BUILTIN(CONFIG_DEBUG_INFO_BTF)
 extern char __start_BTF[];
 extern char __stop_BTF[];
+#endif
 extern struct btf *btf_vmlinux;
 
+#if IS_MODULE(CONFIG_DEBUG_INFO_BTF)
+/*
+ * With CONFIG_DEBUG_INFO_BTF=m the vmlinux BTF is not part of the kernel
+ * image.  The btf_vmlinux module carries it in its .BTF section; when the
+ * module loads, btf_module_notify() copies the section here.  The copy is
+ * made with vmalloc_user() so that /sys/kernel/btf/vmlinux can be mmap()ed
+ * as with the built-in BTF.  Set once, never cleared: like the built-in
+ * BTF, once present it stays for the lifetime of the kernel.
+ *
+ * The size and SHA-256 of the BTF are linked into the kernel as .BTF.meta
+ * by scripts/gen-btf.sh: the size makes /sys/kernel/btf/vmlinux report its
+ * size before the BTF is loaded, the hash makes sure only the BTF this
+ * kernel was built with is accepted.
+ */
+struct btf_vmlinux_meta {
+	u32 size;
+	u8 sha256[SHA256_DIGEST_SIZE];
+} __packed;
+
+extern const struct btf_vmlinux_meta __start_BTF_meta[];
+#define btf_vmlinux_meta (__start_BTF_meta[0])
+
+static void *btf_vmlinux_raw;
+#endif
+
+/**
+ * btf_vmlinux_data - get the raw vmlinux BTF
+ * @size: where to store the size of the BTF, also when it is not loaded yet
+ * @load: with CONFIG_DEBUG_INFO_BTF=m, load the btf_vmlinux module if the
+ *	  BTF is not present yet; may sleep
+ *
+ * Return: the raw BTF, or NULL if it is not available.
+ */
+void *btf_vmlinux_data(u32 *size, bool load)
+{
+#if IS_BUILTIN(CONFIG_DEBUG_INFO_BTF)
+	*size = __stop_BTF - __start_BTF;
+	return __start_BTF;
+#elif IS_MODULE(CONFIG_DEBUG_INFO_BTF)
+	/* Pairs with the smp_store_release() in btf_vmlinux_module_coming() */
+	void *data = smp_load_acquire(&btf_vmlinux_raw);
+
+	if (!data && load) {
+		/*
+		 * The module notifier installs the BTF before init_module()
+		 * returns, so it is either there after this or the module is
+		 * not available (yet).  Not cached: a later call retries,
+		 * e.g. once the module becomes reachable on the root fs.
+		 */
+		request_module("btf_vmlinux");
+		/* Same pairing as above */
+		data = smp_load_acquire(&btf_vmlinux_raw);
+	}
+	*size = btf_vmlinux_meta.size;
+	return data;
+#else
+	*size = 0;
+	return NULL;
+#endif
+}
+
 #define BPF_MAP_TYPE(_id, _ops)
 #define BPF_LINK_TYPE(_id, _name)
 static union {
@@ -6848,15 +6912,22 @@ struct btf *btf_parse_vmlinux(void)
 	struct btf_verifier_env *env = NULL;
 	struct bpf_verifier_log *log;
 	struct btf *btf;
+	void *data;
+	u32 size;
 	int err;
 
+	/* The caller made sure the BTF is present, see bpf_get_btf_vmlinux() */
+	data = btf_vmlinux_data(&size, false);
+	if (!data)
+		return ERR_PTR(-ENOENT);
+
 	env = kzalloc_obj(*env, GFP_KERNEL | __GFP_NOWARN);
 	if (!env)
 		return ERR_PTR(-ENOMEM);
 
 	log = &env->log;
 	log->level = BPF_LOG_KERNEL;
-	btf = btf_parse_base(env, "vmlinux", __start_BTF, __stop_BTF - __start_BTF);
+	btf = btf_parse_base(env, "vmlinux", data, size);
 	if (IS_ERR(btf))
 		goto err_out;
 
@@ -6864,6 +6935,7 @@ struct btf *btf_parse_vmlinux(void)
 	bpf_ctx_convert.t = btf_type_by_id(btf, bpf_ctx_convert_btf_id[0]);
 	err = btf_alloc_id(btf);
 	if (err) {
+		bpf_ctx_convert.t = NULL;
 		btf_free(btf);
 		btf = ERR_PTR(err);
 	}
@@ -6883,7 +6955,7 @@ __u32 btf_relocate_id(const struct btf *btf, __u32 id)
 	return btf->base_id_map[id];
 }
 
-#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+#if IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES) || IS_MODULE(CONFIG_DEBUG_INFO_BTF)
 
 /*
  * Parse split module BTF against @vmlinux_btf.  @data is the module's .BTF
@@ -6989,7 +7061,7 @@ static struct btf *btf_parse_module(const char *module_name, struct btf *vmlinux
 	return ERR_PTR(err);
 }
 
-#endif /* CONFIG_DEBUG_INFO_BTF_MODULES */
+#endif /* CONFIG_DEBUG_INFO_BTF_MODULES || CONFIG_DEBUG_INFO_BTF=m */
 
 struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog)
 {
@@ -8983,7 +9055,16 @@ enum {
 	BTF_MODULE_F_LIVE = (1 << 0),
 };
 
-#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+/*
+ * The module notifier registers module BTF (CONFIG_DEBUG_INFO_BTF_MODULES)
+ * and picks up the vmlinux BTF from the btf_vmlinux module
+ * (CONFIG_DEBUG_INFO_BTF=m).
+ */
+#if IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES) || IS_MODULE(CONFIG_DEBUG_INFO_BTF)
+#define BTF_MODULE_NOTIFIER 1
+#endif
+
+#ifdef BTF_MODULE_NOTIFIER
 struct btf_module {
 	struct list_head list;
 	struct module *module;
@@ -9039,6 +9120,52 @@ static void btf_module_free(struct btf_module *btf_mod)
 	kfree(btf_mod);
 }
 
+#if IS_MODULE(CONFIG_DEBUG_INFO_BTF)
+/*
+ * The btf_vmlinux module carries the vmlinux BTF in its .BTF section
+ * (scripts/gen-btf.sh).  Keep a copy; the module is only the carrier and
+ * has no BTF of its own.
+ */
+static int btf_vmlinux_module_coming(struct module *mod)
+{
+	u8 sha256sum[SHA256_DIGEST_SIZE];
+	void *data;
+
+	if (btf_vmlinux_raw)
+		return 0;
+
+	/*
+	 * The verifier trusts the BTF as the description of this kernel's
+	 * types, so a BTF from a different build must not get in even if
+	 * the module otherwise loads (same release string, same vermagic).
+	 */
+	if (mod->btf_data_size != btf_vmlinux_meta.size) {
+		pr_err("module [%s]: BTF size %u does not match this kernel (%u)\n",
+		       mod->name, mod->btf_data_size, btf_vmlinux_meta.size);
+		return -EINVAL;
+	}
+	sha256(mod->btf_data, mod->btf_data_size, sha256sum);
+	if (memcmp(sha256sum, btf_vmlinux_meta.sha256, sizeof(sha256sum))) {
+		pr_err("module [%s]: BTF does not match this kernel\n", mod->name);
+		return -EINVAL;
+	}
+
+	data = vmalloc_user(mod->btf_data_size);
+	if (!data)
+		return -ENOMEM;
+	memcpy(data, mod->btf_data, mod->btf_data_size);
+
+	/* Pairs with the smp_load_acquire() in btf_vmlinux_data() */
+	smp_store_release(&btf_vmlinux_raw, data);
+	return 0;
+}
+#else
+static int btf_vmlinux_module_coming(struct module *mod)
+{
+	return 0;
+}
+#endif
+
 static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 			     void *module)
 {
@@ -9047,9 +9174,17 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 	struct btf *btf;
 	int err = 0;
 
-	if (mod->btf_data_size == 0 ||
-	    (op != MODULE_STATE_COMING && op != MODULE_STATE_LIVE &&
-	     op != MODULE_STATE_GOING))
+	if (op != MODULE_STATE_COMING && op != MODULE_STATE_LIVE &&
+	    op != MODULE_STATE_GOING)
+		goto out;
+
+	if (IS_MODULE(CONFIG_DEBUG_INFO_BTF) && !strcmp(mod->name, "btf_vmlinux")) {
+		if (op == MODULE_STATE_COMING)
+			err = btf_vmlinux_module_coming(mod);
+		goto out;
+	}
+
+	if (!IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES) || mod->btf_data_size == 0)
 		goto out;
 
 	switch (op) {
@@ -9137,7 +9272,7 @@ static int __init btf_module_init(void)
 }
 
 fs_initcall(btf_module_init);
-#endif /* CONFIG_DEBUG_INFO_BTF_MODULES */
+#endif /* BTF_MODULE_NOTIFIER */
 
 struct module *btf_try_get_module(const struct btf *btf)
 {
@@ -10100,6 +10235,11 @@ static void purge_cand_cache(struct btf *btf)
 	__purge_cand_cache(btf, module_cand_cache, MODULE_CAND_CACHE_SIZE);
 	mutex_unlock(&cand_cache_mutex);
 }
+#elif defined(BTF_MODULE_NOTIFIER)
+/* CONFIG_DEBUG_INFO_BTF=m without module BTF: nothing is ever cached */
+static void purge_cand_cache(struct btf *btf)
+{
+}
 #endif
 
 static struct bpf_cand_cache *
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index ac52f4ae414c..7905e3bcbfc7 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -6435,6 +6435,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size,
 					  &map_idr, &map_idr_lock);
 		break;
 	case BPF_BTF_GET_NEXT_ID:
+		/*
+		 * With CONFIG_DEBUG_INFO_BTF=m the kernel BTFs get ids when the
+		 * vmlinux BTF is loaded; whoever enumerates them wants them.
+		 */
+		if (IS_MODULE(CONFIG_DEBUG_INFO_BTF))
+			bpf_get_btf_vmlinux();
 		err = bpf_obj_get_next_id(&attr, uattr.user,
 					  &btf_idr, &btf_idr_lock);
 		break;
diff --git a/kernel/bpf/sysfs_btf.c b/kernel/bpf/sysfs_btf.c
index 9cbe15ce3540..b2408878412a 100644
--- a/kernel/bpf/sysfs_btf.c
+++ b/kernel/bpf/sysfs_btf.c
@@ -9,8 +9,13 @@
 #include <linux/sysfs.h>
 #include <linux/mm.h>
 #include <linux/io.h>
+#include <linux/bpf.h>
 #include <linux/btf.h>
+#include <linux/vmalloc.h>
 
+struct kobject *btf_kobj;
+
+#if IS_BUILTIN(CONFIG_DEBUG_INFO_BTF)
 /* See scripts/link-vmlinux.sh, gen_btf() func for details */
 extern char __start_BTF[];
 extern char __stop_BTF[];
@@ -49,12 +54,86 @@ static struct bin_attribute bin_attr_btf_vmlinux __ro_after_init = {
 	.mmap = btf_sysfs_vmlinux_mmap,
 };
 
-struct kobject *btf_kobj;
-
-static int __init btf_vmlinux_init(void)
+static void __init btf_sysfs_vmlinux_init(void)
 {
 	bin_attr_btf_vmlinux.private = __start_BTF;
 	bin_attr_btf_vmlinux.size = __stop_BTF - __start_BTF;
+}
+
+#else /* CONFIG_DEBUG_INFO_BTF=m */
+
+/*
+ * The BTF is carried by the btf_vmlinux module and only loaded when
+ * something needs it.  Its size is known from the start, so the file has
+ * its final size from boot; the first read() or mmap() loads the BTF.
+ */
+static void *btf_sysfs_vmlinux_load(u32 *size)
+{
+	/* Loads the module, parses the BTF and registers module BTFs. */
+	if (IS_ERR_OR_NULL(bpf_get_btf_vmlinux()))
+		return NULL;
+	return btf_vmlinux_data(size, false);
+}
+
+static ssize_t btf_sysfs_vmlinux_read(struct file *filp, struct kobject *kobj,
+				      const struct bin_attribute *attr,
+				      char *buf, loff_t off, size_t count)
+{
+	u32 size;
+	void *data = btf_sysfs_vmlinux_load(&size);
+
+	if (!data)
+		return -ENODEV;
+
+	/* sysfs clamps @off and @count to attr->size, which is @size */
+	memcpy(buf, data + off, count);
+	return count;
+}
+
+static int btf_sysfs_vmlinux_mmap(struct file *filp, struct kobject *kobj,
+				  const struct bin_attribute *attr,
+				  struct vm_area_struct *vma)
+{
+	size_t vm_size = vma->vm_end - vma->vm_start;
+	u32 size;
+	void *data = btf_sysfs_vmlinux_load(&size);
+
+	if (!data)
+		return -ENODEV;
+
+	if (vma->vm_pgoff)
+		return -EINVAL;
+
+	if (vma->vm_flags & (VM_WRITE | VM_EXEC | VM_MAYSHARE))
+		return -EACCES;
+
+	if (vm_size > PAGE_ALIGN(size))
+		return -EINVAL;
+
+	vm_flags_mod(vma, VM_DONTDUMP, VM_MAYEXEC | VM_MAYWRITE);
+	/* the copy was made with vmalloc_user() for this purpose */
+	return remap_vmalloc_range(vma, data, 0);
+}
+
+static struct bin_attribute bin_attr_btf_vmlinux __ro_after_init = {
+	.attr = { .name = "vmlinux", .mode = 0444, },
+	.read = btf_sysfs_vmlinux_read,
+	.mmap = btf_sysfs_vmlinux_mmap,
+};
+
+static void __init btf_sysfs_vmlinux_init(void)
+{
+	u32 size;
+
+	/* known before the BTF is loaded, see .BTF.meta */
+	btf_vmlinux_data(&size, false);
+	bin_attr_btf_vmlinux.size = size;
+}
+#endif
+
+static int __init btf_vmlinux_init(void)
+{
+	btf_sysfs_vmlinux_init();
 
 	if (bin_attr_btf_vmlinux.size == 0)
 		return 0;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index da86162ba6d9..84968244a10f 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21922,26 +21922,51 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt
 	return 0;
 }
 
+/*
+ * Returns the parsed vmlinux BTF, NULL if the kernel has none, or an ERR_PTR
+ * if it is malformed.  With CONFIG_DEBUG_INFO_BTF=m the BTF lives in the
+ * btf_vmlinux module; the first caller loads it and parses it.  May sleep.
+ */
 struct btf *bpf_get_btf_vmlinux(void)
 {
 	/* Pairs with the smp_store_release() on the parse path below. */
 	struct btf *btf = smp_load_acquire(&btf_vmlinux);
+	u32 size;
 
-	if (!btf && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
-		mutex_lock(&btf_vmlinux_lock);
-		btf = btf_vmlinux;
-		if (!btf) {
-			btf = btf_parse_vmlinux();
-			/*
-			 * Order the parsed BTF contents and the globals the
-			 * parse populated (e.g. bpf_ctx_convert.t) before
-			 * the pointer publication. Pairs with the acquire
-			 * on the lockless fast path above.
-			 */
-			smp_store_release(&btf_vmlinux, btf);
+	if (btf || !IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
+		return btf;
+
+	/*
+	 * Loading the module may take a while and its notifier must not be
+	 * blocked by us, so do it outside btf_vmlinux_lock.  Not available:
+	 * behave like a kernel without BTF, and retry next time.
+	 */
+	if (!btf_vmlinux_data(&size, true))
+		return NULL;
+
+	mutex_lock(&btf_vmlinux_lock);
+	btf = btf_vmlinux;
+	if (!btf) {
+		btf = btf_parse_vmlinux();
+		/*
+		 * With =m the BTF was checked against the kernel when the
+		 * module loaded, so a failure here is a resource problem
+		 * (-ENOMEM) rather than a broken BTF: do not remember it,
+		 * the next caller retries.
+		 */
+		if (IS_MODULE(CONFIG_DEBUG_INFO_BTF) && IS_ERR(btf)) {
+			mutex_unlock(&btf_vmlinux_lock);
+			return btf;
 		}
-		mutex_unlock(&btf_vmlinux_lock);
+		/*
+		 * Order the parsed BTF contents and the globals the
+		 * parse populated (e.g. bpf_ctx_convert.t) before
+		 * the pointer publication. Pairs with the acquire
+		 * on the lockless fast path above.
+		 */
+		smp_store_release(&btf_vmlinux, btf);
 	}
+	mutex_unlock(&btf_vmlinux_lock);
 	return btf;
 }
 
diff --git a/kernel/module/main.c b/kernel/module/main.c
index d0e1e0bd2ad0..694c4bc7e679 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -2718,7 +2718,7 @@ static int find_module_sections(struct module *mod, struct load_info *info)
 					   sizeof(*mod->bpf_raw_events),
 					   &mod->num_bpf_raw_events);
 #endif
-#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+#if IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES) || IS_MODULE(CONFIG_DEBUG_INFO_BTF)
 	mod->btf_data = any_section_objs(info, ".BTF", 1, &mod->btf_data_size);
 	mod->btf_base_data = any_section_objs(info, ".BTF.base", 1,
 					      &mod->btf_base_data_size);
@@ -3172,7 +3172,7 @@ static noinline int do_init_module(struct module *mod)
 		mod->mem[type].size = 0;
 	}
 
-#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+#if IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES) || IS_MODULE(CONFIG_DEBUG_INFO_BTF)
 	/* .BTF is not SHF_ALLOC and will get removed, so sanitize pointers */
 	mod->btf_data = NULL;
 	mod->btf_base_data = NULL;
-- 
2.47.3

[PATCH bpf-next v3 5/9] bpf: defer vmlinux kfunc and struct_ops registrations

[Jay Wang]

With CONFIG_DEBUG_INFO_BTF=m the vmlinux BTF is loaded on first use.  For
that to save anything, nothing may pull it in at boot.  The verifier no
longer does since the previous patches, but register_btf_kfunc_id_set(),
register_btf_id_dtor_kfuncs() and register_bpf_struct_ops() for vmlinux
run from initcalls and need the parsed BTF.

Queue them instead (btf_defer_reg()) and apply them in
btf_parse_vmlinux(), before the BTF is published, so that no program can
see a vmlinux BTF without its kfuncs and struct_ops.  Applying a
struct_ops runs its ->init(), which registers the kfunc sets of its hook;
those land back on the queue, so it is drained in a loop until a pass
adds nothing, and only then are new registrations applied directly.  The
dtor arrays are copied: every caller in the tree builds them on the stack
of its initcall.

The queue has its own lock, btf_vmlinux_regs_mutex: it is drained under
btf_vmlinux_lock, and btf_module_mutex must not nest inside that, since
purge_cand_cache() takes cand_cache_mutex under btf_module_mutex and
CO-RE takes btf_vmlinux_lock under cand_cache_mutex.

Module registrations are not queued yet; the next patch does that
together with deferring the module BTF itself.  With =y the BTF is
present from boot and nothing is queued.  Nothing here is reachable
until the Kconfig symbol becomes a tristate.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 kernel/bpf/btf.c | 207 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 207 insertions(+)

diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 1dd7f9650ae8..207954b5754a 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -6907,6 +6907,8 @@ static struct btf *btf_parse_base(struct btf_verifier_env *env, const char *name
 	return ERR_PTR(err);
 }
 
+static void btf_apply_deferred_vmlinux_regs(struct btf *btf);
+
 struct btf *btf_parse_vmlinux(void)
 {
 	struct btf_verifier_env *env = NULL;
@@ -6938,7 +6940,15 @@ struct btf *btf_parse_vmlinux(void)
 		bpf_ctx_convert.t = NULL;
 		btf_free(btf);
 		btf = ERR_PTR(err);
+		goto err_out;
 	}
+
+	/*
+	 * With CONFIG_DEBUG_INFO_BTF=m, kfunc, dtor kfunc and struct_ops
+	 * registrations for vmlinux made before the BTF was available were
+	 * queued; apply them now, before the BTF becomes visible to anyone.
+	 */
+	btf_apply_deferred_vmlinux_regs(btf);
 err_out:
 	btf_verifier_env_free(env);
 	return btf;
@@ -9064,6 +9074,33 @@ enum {
 #define BTF_MODULE_NOTIFIER 1
 #endif
 
+/*
+ * CONFIG_DEBUG_INFO_BTF=m: a kfunc, dtor kfunc or struct_ops registration
+ * made while the BTF it applies to is not available yet.  Kept until the BTF
+ * arrives, see btf_defer_reg().
+ */
+enum btf_deferred_reg_kind {
+	BTF_DEFERRED_KFUNC_SET,
+	BTF_DEFERRED_DTOR_KFUNCS,
+	BTF_DEFERRED_STRUCT_OPS,
+};
+
+struct btf_deferred_reg {
+	struct list_head list;
+	enum btf_deferred_reg_kind kind;
+	union {
+		struct {
+			enum btf_kfunc_hook hook;
+			const struct btf_kfunc_id_set *kset;
+		} kfunc;
+		struct {
+			const struct btf_id_dtor_kfunc *dtors;
+			u32 cnt;
+		} dtor;
+		struct bpf_struct_ops *st_ops;
+	};
+};
+
 #ifdef BTF_MODULE_NOTIFIER
 struct btf_module {
 	struct list_head list;
@@ -9848,12 +9885,22 @@ static int btf_kfunc_id_set_add(struct btf *btf, enum btf_kfunc_hook hook,
 	return btf_populate_kfunc_set(btf, hook, kset);
 }
 
+static int btf_defer_reg(struct module *owner, const struct btf_deferred_reg *tmpl);
+
 static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook,
 				       const struct btf_kfunc_id_set *kset)
 {
+	struct btf_deferred_reg tmpl = {
+		.kind = BTF_DEFERRED_KFUNC_SET,
+		.kfunc = { .hook = hook, .kset = kset },
+	};
 	struct btf *btf;
 	int ret;
 
+	ret = btf_defer_reg(kset->owner, &tmpl);
+	if (ret)
+		return ret > 0 ? 0 : ret;
+
 	btf = btf_get_module_btf(kset->owner);
 	if (!btf)
 		return check_btf_kconfigs(kset->owner, "kfunc");
@@ -10022,9 +10069,17 @@ static int btf_dtor_kfuncs_add(struct btf *btf, const struct btf_id_dtor_kfunc *
 int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,
 				struct module *owner)
 {
+	struct btf_deferred_reg tmpl = {
+		.kind = BTF_DEFERRED_DTOR_KFUNCS,
+		.dtor = { .dtors = dtors, .cnt = add_cnt },
+	};
 	struct btf *btf;
 	int ret;
 
+	ret = btf_defer_reg(owner, &tmpl);
+	if (ret)
+		return ret > 0 ? 0 : ret;
+
 	btf = btf_get_module_btf(owner);
 	if (!btf)
 		return check_btf_kconfigs(owner, "dtor kfuncs");
@@ -10703,9 +10758,17 @@ static int btf_struct_ops_register(struct btf *btf, struct bpf_struct_ops *st_op
 
 int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)
 {
+	struct btf_deferred_reg tmpl = {
+		.kind = BTF_DEFERRED_STRUCT_OPS,
+		.st_ops = st_ops,
+	};
 	struct btf *btf;
 	int err;
 
+	err = btf_defer_reg(st_ops->owner, &tmpl);
+	if (err)
+		return err > 0 ? 0 : err;
+
 	btf = btf_get_module_btf(st_ops->owner);
 	if (!btf)
 		return check_btf_kconfigs(st_ops->owner, "struct_ops");
@@ -10717,8 +10780,152 @@ int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)
 	return err;
 }
 EXPORT_SYMBOL_GPL(__register_bpf_struct_ops);
+#elif defined(BTF_MODULE_NOTIFIER)
+static int btf_struct_ops_register(struct btf *btf, struct bpf_struct_ops *st_ops)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
+/*
+ * CONFIG_DEBUG_INFO_BTF=m: registrations for vmlinux made before its BTF is
+ * available wait in btf_vmlinux_deferred_regs until btf_parse_vmlinux()
+ * applies them.
+ */
+#ifdef BTF_MODULE_NOTIFIER
+/*
+ * The queue has its own lock: it is drained under btf_vmlinux_lock, and
+ * btf_module_mutex must not nest inside that (purge_cand_cache() takes
+ * cand_cache_mutex under btf_module_mutex, and CO-RE fetches the vmlinux
+ * BTF under cand_cache_mutex).
+ */
+static DEFINE_MUTEX(btf_vmlinux_regs_mutex);
+static LIST_HEAD(btf_vmlinux_deferred_regs);
+/* Set when the vmlinux BTF is parsed; new registrations apply directly */
+static bool btf_vmlinux_regs_closed;
+
+/*
+ * Queue @tmpl if the BTF for @owner is not available yet.  Returns 1 if the
+ * registration was queued and is to be considered done, 0 if the caller has
+ * to apply it, or -ENOMEM.  Only vmlinux registrations are queued so far.
+ */
+static int btf_defer_reg(struct module *owner, const struct btf_deferred_reg *tmpl)
+{
+	struct list_head *head = NULL;
+	struct btf_deferred_reg *reg;
+
+	if (!IS_MODULE(CONFIG_DEBUG_INFO_BTF) || owner)
+		return 0;
+
+	guard(mutex)(&btf_vmlinux_regs_mutex);
+	if (!btf_vmlinux_regs_closed)
+		head = &btf_vmlinux_deferred_regs;
+	if (!head)
+		return 0;
+
+	reg = kmemdup(tmpl, sizeof(*reg), GFP_KERNEL);
+	if (!reg)
+		return -ENOMEM;
+	/*
+	 * kfunc id sets and struct_ops are static data of their owner, but
+	 * the dtor arrays are commonly built on the stack of the initcall.
+	 */
+	if (reg->kind == BTF_DEFERRED_DTOR_KFUNCS) {
+		reg->dtor.dtors = kmemdup_array(tmpl->dtor.dtors, tmpl->dtor.cnt,
+						sizeof(*tmpl->dtor.dtors), GFP_KERNEL);
+		if (!reg->dtor.dtors) {
+			kfree(reg);
+			return -ENOMEM;
+		}
+	}
+	list_add_tail(&reg->list, head);
+	return 1;
+}
+
+static void btf_free_deferred_reg(struct btf_deferred_reg *reg)
+{
+	if (reg->kind == BTF_DEFERRED_DTOR_KFUNCS)
+		kfree(reg->dtor.dtors);
+	kfree(reg);
+}
+
+static const char *btf_deferred_reg_name(const struct btf_deferred_reg *reg)
+{
+	switch (reg->kind) {
+	case BTF_DEFERRED_KFUNC_SET:	return "kfunc set";
+	case BTF_DEFERRED_DTOR_KFUNCS:	return "dtor kfuncs";
+	case BTF_DEFERRED_STRUCT_OPS:	return "struct_ops";
+	}
+	return "?";
+}
+
+static int btf_apply_deferred_reg(struct btf *btf, const struct btf_deferred_reg *reg)
+{
+	switch (reg->kind) {
+	case BTF_DEFERRED_KFUNC_SET:
+		return btf_kfunc_id_set_add(btf, reg->kfunc.hook, reg->kfunc.kset);
+	case BTF_DEFERRED_DTOR_KFUNCS:
+		return btf_dtor_kfuncs_add(btf, reg->dtor.dtors, reg->dtor.cnt);
+	case BTF_DEFERRED_STRUCT_OPS:
+		return btf_struct_ops_register(btf, reg->st_ops);
+	}
+	return -EINVAL;
+}
+
+/* Apply and free the registrations in @regs to @btf. */
+static void btf_apply_deferred_regs(struct btf *btf, struct list_head *regs)
+{
+	struct btf_deferred_reg *reg, *tmp;
+	int err;
+
+	list_for_each_entry_safe(reg, tmp, regs, list) {
+		err = btf_apply_deferred_reg(btf, reg);
+		if (err)
+			pr_warn("failed to register deferred %s for [%s] BTF: %d\n",
+				btf_deferred_reg_name(reg), btf->name, err);
+		list_del(&reg->list);
+		btf_free_deferred_reg(reg);
+	}
+}
+
+/*
+ * The vmlinux BTF has just been parsed; apply the registrations that waited
+ * for it.  Runs under btf_vmlinux_lock, before @btf is published, so nothing
+ * can observe a vmlinux BTF without its kfuncs and struct_ops.
+ *
+ * Applying a registration can queue further ones: a struct_ops ->init()
+ * registers the kfuncs of its hook.  Those must not go through
+ * bpf_get_btf_vmlinux() (we hold its lock), so the queue stays open until
+ * a pass applies nothing new, and only then are registrations applied directly.
+ */
+static void btf_apply_deferred_vmlinux_regs(struct btf *btf)
+{
+	LIST_HEAD(regs);
+
+	if (!IS_MODULE(CONFIG_DEBUG_INFO_BTF))
+		return;
+
+	mutex_lock(&btf_vmlinux_regs_mutex);
+	while (!list_empty(&btf_vmlinux_deferred_regs)) {
+		list_splice_init(&btf_vmlinux_deferred_regs, &regs);
+		mutex_unlock(&btf_vmlinux_regs_mutex);
+		btf_apply_deferred_regs(btf, &regs);
+		mutex_lock(&btf_vmlinux_regs_mutex);
+	}
+	btf_vmlinux_regs_closed = true;
+	mutex_unlock(&btf_vmlinux_regs_mutex);
+}
+#else
+static int btf_defer_reg(struct module *owner, const struct btf_deferred_reg *tmpl)
+{
+	return 0;
+}
+
+static void btf_apply_deferred_vmlinux_regs(struct btf *btf)
+{
+}
+#endif /* BTF_MODULE_NOTIFIER */
+
 bool btf_param_match_suffix(const struct btf *btf,
 			    const struct btf_param *arg,
 			    const char *suffix)
-- 
2.47.3

[PATCH bpf-next v3 6/9] bpf: keep module BTF until the vmlinux BTF is available

[Jay Wang]

Module BTF is split BTF against the vmlinux BTF and is parsed in the
module notifier.  With CONFIG_DEBUG_INFO_BTF=m the vmlinux BTF may not be
loaded yet when a module loads, and the notifier cannot load btf_vmlinux
(that would nest a module load in a module load).

So a module loaded before the vmlinux BTF keeps a copy of its .BTF and
.BTF.base and gets a list entry with btf == NULL; its kfunc, dtor kfunc
and struct_ops registrations wait on that entry.  Without a .BTF.base
the data is final and is exposed in /sys/kernel/btf right away (the raw
bytes need no parsing); with one, parsing relocates the data in place,
so its file is created once parsed, as with =y.  The next patch creates
that file earlier.

When the vmlinux BTF arrives, btf_parse_deferred_modules() parses the
kept copies (the copy is the one btf_parse_module() makes anyway, so an
existing sysfs file keeps pointing at valid data), applies the waiting
registrations and only then publishes the BTF with an id, so nobody sees
a module BTF without its kfuncs.  Applying walks the module list
(btf_check_kfunc_name()) and so happens with btf_module_mutex dropped
and the module pinned.  A module that is still initializing when the
vmlinux BTF arrives is only published; its init is still queueing
registrations, and MODULE_STATE_LIVE applies them once init is done,
which also keeps a module whose init fails from being touched after it
is freed.

A module whose BTF turns out to mismatch at that point is already
running and keeps running without BTF, with a warning; its entry stays,
dead, until the module goes, and keeps the raw data a sysfs file may
serve.  With =y such a module would have been refused at load time
unless CONFIG_MODULE_ALLOW_BTF_MISMATCH; that check only applies to
modules loaded after the vmlinux BTF.

Walkers of the module BTF list skip entries whose BTF is not parsed yet.
With =y the BTF is present from boot and the notifier takes the existing
path.  Still nothing is reachable until the Kconfig symbol becomes a
tristate.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 include/linux/btf.h   |   5 +
 kernel/bpf/btf.c      | 267 +++++++++++++++++++++++++++++++++++++++---
 kernel/bpf/verifier.c |   9 +-
 3 files changed, 262 insertions(+), 19 deletions(-)

diff --git a/include/linux/btf.h b/include/linux/btf.h
index 81e6c65fe5f6..fea60e36edba 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -603,6 +603,11 @@ const char *btf_name_by_offset(const struct btf *btf, u32 offset);
 const char *btf_str_by_offset(const struct btf *btf, u32 offset);
 struct btf *btf_parse_vmlinux(void);
 void *btf_vmlinux_data(u32 *size, bool load);
+#if IS_MODULE(CONFIG_DEBUG_INFO_BTF)
+void btf_parse_deferred_modules(void);
+#else
+static inline void btf_parse_deferred_modules(void) {}
+#endif
 struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog);
 u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog);
 int btf_kfunc_check_flag(const struct btf *btf, u32 kfunc_btf_id, u32 flag);
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 207954b5754a..1dae1c9b53fe 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -9077,7 +9077,7 @@ enum {
 /*
  * CONFIG_DEBUG_INFO_BTF=m: a kfunc, dtor kfunc or struct_ops registration
  * made while the BTF it applies to is not available yet.  Kept until the BTF
- * arrives, see btf_defer_reg().
+ * arrives, see btf_defer_reg() and btf_apply_deferred_regs().
  */
 enum btf_deferred_reg_kind {
 	BTF_DEFERRED_KFUNC_SET,
@@ -9102,12 +9102,30 @@ struct btf_deferred_reg {
 };
 
 #ifdef BTF_MODULE_NOTIFIER
+static void btf_free_deferred_regs(struct list_head *regs);
+static void btf_apply_deferred_regs(struct btf *btf, struct list_head *regs);
+
 struct btf_module {
 	struct list_head list;
 	struct module *module;
 	struct btf *btf;
 	struct bin_attribute *sysfs_attr;
 	int flags;
+	/*
+	 * CONFIG_DEBUG_INFO_BTF=m: a module loaded before the vmlinux BTF is
+	 * available cannot have its BTF parsed yet.  Its .BTF and .BTF.base
+	 * sections are copied here and parsed once the vmlinux BTF arrives
+	 * (btf_parse_deferred_modules()); @btf is NULL until then.
+	 * Registrations of the module's kfuncs, dtor kfuncs and struct_ops
+	 * wait in @deferred_regs.
+	 */
+	void *data;
+	void *base_data;
+	u32 data_size;
+	u32 base_data_size;
+	struct list_head deferred_regs;
+	/* the kept BTF turned out unusable; the entry stays until the module goes */
+	bool gone;
 };
 
 static LIST_HEAD(btf_modules);
@@ -9151,8 +9169,14 @@ static void btf_module_free(struct btf_module *btf_mod)
 {
 	if (btf_mod->sysfs_attr)
 		sysfs_remove_bin_file(btf_kobj, btf_mod->sysfs_attr);
-	purge_cand_cache(btf_mod->btf);
-	btf_put(btf_mod->btf);
+	if (btf_mod->btf) {
+		purge_cand_cache(btf_mod->btf);
+		btf_put(btf_mod->btf);
+	} else {
+		kvfree(btf_mod->data);
+		kvfree(btf_mod->base_data);
+	}
+	btf_free_deferred_regs(&btf_mod->deferred_regs);
 	kfree(btf_mod->sysfs_attr);
 	kfree(btf_mod);
 }
@@ -9196,11 +9220,53 @@ static int btf_vmlinux_module_coming(struct module *mod)
 	smp_store_release(&btf_vmlinux_raw, data);
 	return 0;
 }
+
+/*
+ * The vmlinux BTF is not available yet and must not be loaded from the
+ * module notifier (that would nest a module load into a module load).  Keep
+ * the module's BTF for btf_parse_deferred_modules().
+ *
+ * Without a .BTF.base section the .BTF data is final and can be exposed in
+ * sysfs right away, it needs no parsing.  With one, parsing relocates the
+ * data in place against the vmlinux BTF, so the file is created afterwards,
+ * as with =y where it also only appears once the BTF is parsed.
+ */
+static int btf_module_defer(struct btf_module *btf_mod, struct module *mod)
+{
+	btf_mod->data = kvmemdup(mod->btf_data, mod->btf_data_size,
+				 GFP_KERNEL | __GFP_NOWARN);
+	if (!btf_mod->data)
+		return -ENOMEM;
+	btf_mod->data_size = mod->btf_data_size;
+
+	if (mod->btf_base_data) {
+		btf_mod->base_data = kvmemdup(mod->btf_base_data,
+					      mod->btf_base_data_size,
+					      GFP_KERNEL | __GFP_NOWARN);
+		if (!btf_mod->base_data) {
+			kvfree(btf_mod->data);
+			return -ENOMEM;
+		}
+		btf_mod->base_data_size = mod->btf_base_data_size;
+	} else {
+		/* not fatal, the module BTF is usable without the sysfs file */
+		btf_module_sysfs_add(btf_mod, mod->name, btf_mod->data,
+				     btf_mod->data_size);
+	}
+
+	list_add(&btf_mod->list, &btf_modules);
+	return 0;
+}
 #else
 static int btf_vmlinux_module_coming(struct module *mod)
 {
 	return 0;
 }
+
+static int btf_module_defer(struct btf_module *btf_mod, struct module *mod)
+{
+	return 0;
+}
 #endif
 
 static int btf_module_notify(struct notifier_block *nb, unsigned long op,
@@ -9232,6 +9298,24 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 			goto out;
 		}
 		btf_mod->module = module;
+		INIT_LIST_HEAD(&btf_mod->deferred_regs);
+
+		if (IS_MODULE(CONFIG_DEBUG_INFO_BTF)) {
+			mutex_lock(&btf_module_mutex);
+			/* Pairs with the publication in bpf_get_btf_vmlinux() */
+			if (!smp_load_acquire(&btf_vmlinux)) {
+				err = btf_module_defer(btf_mod, mod);
+				mutex_unlock(&btf_module_mutex);
+				if (err) {
+					pr_warn("failed to keep module [%s] BTF: %d\n",
+						mod->name, err);
+					kfree(btf_mod);
+					err = 0;
+				}
+				goto out;
+			}
+			mutex_unlock(&btf_module_mutex);
+		}
 
 		btf = btf_parse_module(mod->name, bpf_get_btf_vmlinux(),
 				       mod->btf_data, mod->btf_data_size, false,
@@ -9270,6 +9354,26 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 				continue;
 
 			btf_mod->flags |= BTF_MODULE_F_LIVE;
+			if (IS_MODULE(CONFIG_DEBUG_INFO_BTF) && btf_mod->btf &&
+			    !list_empty(&btf_mod->deferred_regs)) {
+				/*
+				 * The vmlinux BTF arrived while this module was
+				 * initializing: btf_parse_deferred_modules()
+				 * parsed its BTF but left the registrations its
+				 * init queued to us, now that init is done and
+				 * the module is not going anywhere.  Applying
+				 * them walks btf_modules, so drop the mutex.
+				 */
+				LIST_HEAD(regs);
+
+				btf = btf_mod->btf;
+				btf_get(btf);
+				list_splice_init(&btf_mod->deferred_regs, &regs);
+				mutex_unlock(&btf_module_mutex);
+				btf_apply_deferred_regs(btf, &regs);
+				btf_put(btf);
+				goto out;
+			}
 			break;
 		}
 		mutex_unlock(&btf_module_mutex);
@@ -9286,7 +9390,8 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 			 * btf_try_get_module() on such BTFs will fail. This may
 			 * be called again on btf_put(), but it's ok to do so.
 			 */
-			btf_free_id(btf_mod->btf);
+			if (btf_mod->btf)
+				btf_free_id(btf_mod->btf);
 			list_del(&btf_mod->list);
 			btf_module_free(btf_mod);
 			break;
@@ -9309,6 +9414,101 @@ static int __init btf_module_init(void)
 }
 
 fs_initcall(btf_module_init);
+
+#if IS_MODULE(CONFIG_DEBUG_INFO_BTF)
+/*
+ * A kept module whose BTF cannot be used after all.  The module is loaded
+ * and stays, so there is no way to reject it: the entry stays on the list,
+ * dead, until the module goes.  A sysfs file it has keeps serving the raw
+ * data, which is kept for that.
+ */
+static void btf_module_dead(struct btf_module *btf_mod, const char *what, int err)
+{
+	pr_warn("failed to %s module [%s] BTF: %d\n", what, btf_mod->module->name, err);
+	kvfree(btf_mod->base_data);
+	btf_mod->base_data = NULL;
+	btf_free_deferred_regs(&btf_mod->deferred_regs);
+	btf_mod->gone = true;
+}
+
+/*
+ * CONFIG_DEBUG_INFO_BTF=m: the vmlinux BTF has just become available.  Parse
+ * the BTF of the modules that were loaded before it, and apply the
+ * registrations that waited for them.  Called from bpf_get_btf_vmlinux()
+ * once btf_vmlinux is published, with no locks held.
+ *
+ * A module's BTF is published (btf_mod->btf set, id allocated) only after
+ * its queued registrations are applied, so nobody sees a module BTF without
+ * its kfuncs and struct_ops, as with the vmlinux BTF.  Applying walks
+ * btf_modules (btf_check_kfunc_name()) and so needs the mutex dropped; the
+ * module is pinned for that, and the scan restarts afterwards.  A module
+ * that is still initializing is only published: its init is still queueing
+ * registrations, and MODULE_STATE_LIVE applies them once it is done.
+ */
+void btf_parse_deferred_modules(void)
+{
+	/* Pairs with the publication in bpf_get_btf_vmlinux() */
+	struct btf *vmlinux_btf = smp_load_acquire(&btf_vmlinux);
+	struct btf_module *btf_mod;
+	bool parsed = false;
+	LIST_HEAD(regs);
+	struct btf *btf;
+	int err;
+
+	if (IS_ERR_OR_NULL(vmlinux_btf))
+		return;
+
+	mutex_lock(&btf_module_mutex);
+restart:
+	list_for_each_entry(btf_mod, &btf_modules, list) {
+		if (btf_mod->btf || btf_mod->gone)
+			continue;
+
+		btf = btf_parse_module(btf_mod->module->name, vmlinux_btf,
+				       btf_mod->data, btf_mod->data_size, true,
+				       btf_mod->base_data, btf_mod->base_data_size);
+		if (IS_ERR(btf)) {
+			/* on failure the caller keeps the data */
+			btf_module_dead(btf_mod, "validate", PTR_ERR(btf));
+			continue;
+		}
+		/* btf->data is btf_mod->data now, the sysfs file keeps pointing at valid data */
+		kvfree(btf_mod->base_data);
+		btf_mod->base_data = NULL;
+
+		if ((btf_mod->flags & BTF_MODULE_F_LIVE) &&
+		    try_module_get(btf_mod->module)) {
+			list_splice_init(&btf_mod->deferred_regs, &regs);
+			mutex_unlock(&btf_module_mutex);
+			btf_apply_deferred_regs(btf, &regs);
+			mutex_lock(&btf_module_mutex);
+			module_put(btf_mod->module);
+		}
+
+		err = btf_alloc_id(btf);
+		if (err) {
+			/* give the data back to the entry, the sysfs file may serve it */
+			btf->data = NULL;
+			btf_free(btf);
+			btf_module_dead(btf_mod, "register", err);
+			goto restart;
+		}
+
+		/* modules with .BTF.base get their sysfs file now, the data is relocated */
+		if (!btf_mod->sysfs_attr)
+			btf_module_sysfs_add(btf_mod, btf->name, btf->data, btf->data_size);
+		btf_mod->data = NULL;
+		btf_mod->btf = btf;
+		parsed = true;
+		/* the list may have changed while the mutex was dropped */
+		goto restart;
+	}
+	mutex_unlock(&btf_module_mutex);
+
+	if (parsed)
+		purge_cand_cache(NULL);
+}
+#endif /* IS_MODULE(CONFIG_DEBUG_INFO_BTF) */
 #endif /* BTF_MODULE_NOTIFIER */
 
 struct module *btf_try_get_module(const struct btf *btf)
@@ -9361,8 +9561,11 @@ struct btf *btf_get_module_btf(const struct module *module)
 		if (btf_mod->module != module)
 			continue;
 
-		btf_get(btf_mod->btf);
-		btf = btf_mod->btf;
+		/* NULL while waiting for the vmlinux BTF (CONFIG_DEBUG_INFO_BTF=m) */
+		if (btf_mod->btf) {
+			btf_get(btf_mod->btf);
+			btf = btf_mod->btf;
+		}
 		break;
 	}
 	mutex_unlock(&btf_module_mutex);
@@ -9545,7 +9748,8 @@ static int btf_check_kfunc_name(struct btf *btf, const char *func_name, u32 kind
 #ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	guard(mutex)(&btf_module_mutex);
 	list_for_each_entry_safe(btf_mod, tmp, &btf_modules, list) {
-		if (btf_mod->btf == btf)
+		/* skip ourselves and, with CONFIG_DEBUG_INFO_BTF=m, unparsed BTF */
+		if (btf_mod->btf == btf || !btf_mod->btf)
 			continue;
 		id = btf_find_by_name_kind(btf_mod->btf, func_name, kind);
 		if (id >= 0) {
@@ -10788,14 +10992,16 @@ static int btf_struct_ops_register(struct btf *btf, struct bpf_struct_ops *st_op
 #endif
 
 /*
- * CONFIG_DEBUG_INFO_BTF=m: registrations for vmlinux made before its BTF is
- * available wait in btf_vmlinux_deferred_regs until btf_parse_vmlinux()
- * applies them.
+ * CONFIG_DEBUG_INFO_BTF=m: registrations made before the BTF they apply to
+ * is available.  Registrations for vmlinux wait in btf_vmlinux_deferred_regs
+ * until btf_parse_vmlinux() applies them; registrations for a module wait in
+ * its struct btf_module, under btf_module_mutex, until
+ * btf_parse_deferred_modules() does.
  */
 #ifdef BTF_MODULE_NOTIFIER
 /*
- * The queue has its own lock: it is drained under btf_vmlinux_lock, and
- * btf_module_mutex must not nest inside that (purge_cand_cache() takes
+ * The vmlinux queue has its own lock: it is drained under btf_vmlinux_lock,
+ * and btf_module_mutex must not nest inside that (purge_cand_cache() takes
  * cand_cache_mutex under btf_module_mutex, and CO-RE fetches the vmlinux
  * BTF under cand_cache_mutex).
  */
@@ -10807,19 +11013,31 @@ static bool btf_vmlinux_regs_closed;
 /*
  * Queue @tmpl if the BTF for @owner is not available yet.  Returns 1 if the
  * registration was queued and is to be considered done, 0 if the caller has
- * to apply it, or -ENOMEM.  Only vmlinux registrations are queued so far.
+ * to apply it, or -ENOMEM.
  */
 static int btf_defer_reg(struct module *owner, const struct btf_deferred_reg *tmpl)
 {
 	struct list_head *head = NULL;
 	struct btf_deferred_reg *reg;
+	struct btf_module *btf_mod;
 
-	if (!IS_MODULE(CONFIG_DEBUG_INFO_BTF) || owner)
+	if (!IS_MODULE(CONFIG_DEBUG_INFO_BTF))
 		return 0;
 
-	guard(mutex)(&btf_vmlinux_regs_mutex);
-	if (!btf_vmlinux_regs_closed)
-		head = &btf_vmlinux_deferred_regs;
+	guard(mutex)(owner ? &btf_module_mutex : &btf_vmlinux_regs_mutex);
+	if (!owner) {
+		if (!btf_vmlinux_regs_closed)
+			head = &btf_vmlinux_deferred_regs;
+	} else {
+		list_for_each_entry(btf_mod, &btf_modules, list) {
+			if (btf_mod->module != owner)
+				continue;
+			/* a dead entry has no BTF to register with, as with =y */
+			if (!btf_mod->btf && !btf_mod->gone)
+				head = &btf_mod->deferred_regs;
+			break;
+		}
+	}
 	if (!head)
 		return 0;
 
@@ -10872,7 +11090,10 @@ static int btf_apply_deferred_reg(struct btf *btf, const struct btf_deferred_reg
 	return -EINVAL;
 }
 
-/* Apply and free the registrations in @regs to @btf. */
+/*
+ * Apply and free the registrations in @regs to @btf.  For a module BTF the
+ * caller holds a reference on @btf and makes sure the owning module stays.
+ */
 static void btf_apply_deferred_regs(struct btf *btf, struct list_head *regs)
 {
 	struct btf_deferred_reg *reg, *tmp;
@@ -10888,6 +11109,16 @@ static void btf_apply_deferred_regs(struct btf *btf, struct list_head *regs)
 	}
 }
 
+static void btf_free_deferred_regs(struct list_head *regs)
+{
+	struct btf_deferred_reg *reg, *tmp;
+
+	list_for_each_entry_safe(reg, tmp, regs, list) {
+		list_del(&reg->list);
+		btf_free_deferred_reg(reg);
+	}
+}
+
 /*
  * The vmlinux BTF has just been parsed; apply the registrations that waited
  * for it.  Runs under btf_vmlinux_lock, before @btf is published, so nothing
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 84968244a10f..7577c6215535 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21925,12 +21925,14 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt
 /*
  * Returns the parsed vmlinux BTF, NULL if the kernel has none, or an ERR_PTR
  * if it is malformed.  With CONFIG_DEBUG_INFO_BTF=m the BTF lives in the
- * btf_vmlinux module; the first caller loads it and parses it.  May sleep.
+ * btf_vmlinux module; the first caller loads it, parses it and then registers
+ * the BTF of the modules that were loaded before it.  May sleep.
  */
 struct btf *bpf_get_btf_vmlinux(void)
 {
 	/* Pairs with the smp_store_release() on the parse path below. */
 	struct btf *btf = smp_load_acquire(&btf_vmlinux);
+	bool parsed = false;
 	u32 size;
 
 	if (btf || !IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
@@ -21965,8 +21967,13 @@ struct btf *bpf_get_btf_vmlinux(void)
 		 * on the lockless fast path above.
 		 */
 		smp_store_release(&btf_vmlinux, btf);
+		parsed = true;
 	}
 	mutex_unlock(&btf_vmlinux_lock);
+
+	if (parsed && IS_MODULE(CONFIG_DEBUG_INFO_BTF))
+		btf_parse_deferred_modules();
+
 	return btf;
 }
 
-- 
2.47.3

[PATCH bpf-next v3 7/9] bpf: expose deferred .BTF.base module BTF in sysfs from module load

[Jay Wang]

A module with a .BTF.base section (built out of tree) that is loaded
before the vmlinux BTF only gets its /sys/kernel/btf file once the
vmlinux BTF has been loaded and its BTF relocated, because the raw .BTF
is only valid against the distilled base and relocation rewrites it in
place.  Until then the module is missing from /sys/kernel/btf, unlike
with =y, and reading its file cannot trigger the load.

Create the file at module load instead, with its final size: relocation
only rewrites type ids and string offsets, never the length.  Its reader,
btf_module_sysfs_read_deferred(), loads the vmlinux BTF, which parses and
relocates the kept modules, then waits until this module's BTF is
published (btf_mod->ready) before serving it, so no unrelocated or
half-relocated data is ever visible.  If the module goes away or its BTF
turns out unusable first (btf_mod->gone), the read fails with -ENODEV.

Because that reader may itself be the thread running
btf_parse_deferred_modules(), and removing a sysfs file waits for its
readers, nothing removes a sysfs file from that path (a failed entry
stays dead until the module goes, which the previous patch already
arranged), and MODULE_STATE_GOING removes the file after dropping
btf_module_mutex, which the reader may need to get there.

Modules without .BTF.base are unchanged: their data is final and is
served as is.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 kernel/bpf/btf.c | 108 ++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 88 insertions(+), 20 deletions(-)

diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 1dae1c9b53fe..64583fb1f380 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -9124,17 +9124,27 @@ struct btf_module {
 	u32 data_size;
 	u32 base_data_size;
 	struct list_head deferred_regs;
-	/* the kept BTF turned out unusable; the entry stays until the module goes */
+	/*
+	 * For the sysfs reader of a module whose data is only final once its
+	 * BTF is relocated: @ready once @btf is published, @gone once the
+	 * entry is dead (parse failed or module going).  Both only ever go
+	 * from false to true; waiters sleep on btf_module_wq.
+	 */
+	bool ready;
 	bool gone;
 };
 
 static LIST_HEAD(btf_modules);
 static DEFINE_MUTEX(btf_module_mutex);
+static DECLARE_WAIT_QUEUE_HEAD(btf_module_wq);
 
 static void purge_cand_cache(struct btf *btf);
 
 static int btf_module_sysfs_add(struct btf_module *btf_mod, const char *name,
-				void *data, size_t data_size)
+				void *private, size_t size,
+				ssize_t (*read)(struct file *, struct kobject *,
+						const struct bin_attribute *,
+						char *, loff_t, size_t))
 {
 	struct bin_attribute *attr;
 	int err;
@@ -9149,9 +9159,9 @@ static int btf_module_sysfs_add(struct btf_module *btf_mod, const char *name,
 	sysfs_bin_attr_init(attr);
 	attr->attr.name = name;
 	attr->attr.mode = 0444;
-	attr->size = data_size;
-	attr->private = data;
-	attr->read = sysfs_bin_attr_simple_read;
+	attr->size = size;
+	attr->private = private;
+	attr->read = read;
 
 	err = sysfs_create_bin_file(btf_kobj, attr);
 	if (err) {
@@ -9165,8 +9175,14 @@ static int btf_module_sysfs_add(struct btf_module *btf_mod, const char *name,
 	return 0;
 }
 
+/*
+ * Called with btf_module_mutex NOT held: removing the sysfs file waits for
+ * readers to leave, and a deferred reader may need the mutex to get there.
+ */
 static void btf_module_free(struct btf_module *btf_mod)
 {
+	WRITE_ONCE(btf_mod->gone, true);
+	wake_up_all(&btf_module_wq);
 	if (btf_mod->sysfs_attr)
 		sysfs_remove_bin_file(btf_kobj, btf_mod->sysfs_attr);
 	if (btf_mod->btf) {
@@ -9221,15 +9237,57 @@ static int btf_vmlinux_module_coming(struct module *mod)
 	return 0;
 }
 
+/*
+ * sysfs reader for a module kept aside with a .BTF.base section: its .BTF is
+ * split against the distilled base and only becomes valid split BTF against
+ * the vmlinux BTF once relocated, which rewrites the buffer in place.  So
+ * first make sure the vmlinux BTF is loaded (which parses and relocates the
+ * kept modules), then wait until this module's BTF is published.  The size
+ * does not change: relocation only rewrites ids and string offsets.
+ */
+static bool btf_module_published(struct btf_module *btf_mod)
+{
+	/* Pairs with the smp_store_release() of @ready after btf_mod->btf is set */
+	return smp_load_acquire(&btf_mod->ready);
+}
+
+static ssize_t btf_module_sysfs_read_deferred(struct file *filp, struct kobject *kobj,
+					      const struct bin_attribute *attr,
+					      char *buf, loff_t off, size_t count)
+{
+	struct btf_module *btf_mod = attr->private;
+	int err;
+
+	if (IS_ERR_OR_NULL(bpf_get_btf_vmlinux()))
+		return -ENODEV;
+
+	/*
+	 * Another thread may still be relocating and publishing it; if the
+	 * module goes away or its BTF turns out unusable, btf_module_free()
+	 * or btf_parse_deferred_modules() set @gone and wake us.
+	 */
+	err = wait_event_interruptible(btf_module_wq,
+				       btf_module_published(btf_mod) ||
+				       READ_ONCE(btf_mod->gone));
+	if (err)
+		return err;
+	if (!btf_module_published(btf_mod))
+		return -ENODEV;
+
+	/* sysfs clamps @off and @count to attr->size == btf->data_size */
+	memcpy(buf, btf_mod->btf->data + off, count);
+	return count;
+}
+
 /*
  * The vmlinux BTF is not available yet and must not be loaded from the
  * module notifier (that would nest a module load into a module load).  Keep
  * the module's BTF for btf_parse_deferred_modules().
  *
- * Without a .BTF.base section the .BTF data is final and can be exposed in
- * sysfs right away, it needs no parsing.  With one, parsing relocates the
- * data in place against the vmlinux BTF, so the file is created afterwards,
- * as with =y where it also only appears once the BTF is parsed.
+ * The sysfs file is created right away with its final size, as with =y.
+ * Without a .BTF.base section the .BTF data is final and is served as is;
+ * with one, it is only valid once relocated, so its reader waits for that
+ * (btf_module_sysfs_read_deferred()).
  */
 static int btf_module_defer(struct btf_module *btf_mod, struct module *mod)
 {
@@ -9248,10 +9306,12 @@ static int btf_module_defer(struct btf_module *btf_mod, struct module *mod)
 			return -ENOMEM;
 		}
 		btf_mod->base_data_size = mod->btf_base_data_size;
-	} else {
 		/* not fatal, the module BTF is usable without the sysfs file */
+		btf_module_sysfs_add(btf_mod, mod->name, btf_mod, btf_mod->data_size,
+				     btf_module_sysfs_read_deferred);
+	} else {
 		btf_module_sysfs_add(btf_mod, mod->name, btf_mod->data,
-				     btf_mod->data_size);
+				     btf_mod->data_size, sysfs_bin_attr_simple_read);
 	}
 
 	list_add(&btf_mod->list, &btf_modules);
@@ -9345,7 +9405,8 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 		mutex_unlock(&btf_module_mutex);
 
 		/* not fatal, the module BTF is usable without the sysfs file */
-		btf_module_sysfs_add(btf_mod, btf->name, btf->data, btf->data_size);
+		btf_module_sysfs_add(btf_mod, btf->name, btf->data, btf->data_size,
+				     sysfs_bin_attr_simple_read);
 		break;
 	case MODULE_STATE_LIVE:
 		mutex_lock(&btf_module_mutex);
@@ -9393,8 +9454,10 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
 			if (btf_mod->btf)
 				btf_free_id(btf_mod->btf);
 			list_del(&btf_mod->list);
+			mutex_unlock(&btf_module_mutex);
+			/* off the list, nobody else can find it now */
 			btf_module_free(btf_mod);
-			break;
+			goto out;
 		}
 		mutex_unlock(&btf_module_mutex);
 		break;
@@ -9419,8 +9482,10 @@ fs_initcall(btf_module_init);
 /*
  * A kept module whose BTF cannot be used after all.  The module is loaded
  * and stays, so there is no way to reject it: the entry stays on the list,
- * dead, until the module goes.  A sysfs file it has keeps serving the raw
- * data, which is kept for that.
+ * dead, until the module goes.  Its sysfs file stays too, its reader, or
+ * the caller of this function, may be inside it right now: a .BTF.base
+ * reader wakes up and fails, a plain one keeps serving the raw data, which
+ * is kept for that.
  */
 static void btf_module_dead(struct btf_module *btf_mod, const char *what, int err)
 {
@@ -9428,14 +9493,17 @@ static void btf_module_dead(struct btf_module *btf_mod, const char *what, int er
 	kvfree(btf_mod->base_data);
 	btf_mod->base_data = NULL;
 	btf_free_deferred_regs(&btf_mod->deferred_regs);
-	btf_mod->gone = true;
+	WRITE_ONCE(btf_mod->gone, true);
+	wake_up_all(&btf_module_wq);
 }
 
 /*
  * CONFIG_DEBUG_INFO_BTF=m: the vmlinux BTF has just become available.  Parse
  * the BTF of the modules that were loaded before it, and apply the
  * registrations that waited for them.  Called from bpf_get_btf_vmlinux()
- * once btf_vmlinux is published, with no locks held.
+ * once btf_vmlinux is published, with no locks held -- possibly from the
+ * sysfs reader of one of these modules, which is why no sysfs file is
+ * removed here.
  *
  * A module's BTF is published (btf_mod->btf set, id allocated) only after
  * its queued registrations are applied, so nobody sees a module BTF without
@@ -9494,11 +9562,11 @@ void btf_parse_deferred_modules(void)
 			goto restart;
 		}
 
-		/* modules with .BTF.base get their sysfs file now, the data is relocated */
-		if (!btf_mod->sysfs_attr)
-			btf_module_sysfs_add(btf_mod, btf->name, btf->data, btf->data_size);
 		btf_mod->data = NULL;
 		btf_mod->btf = btf;
+		/* Pairs with the smp_load_acquire() in btf_module_sysfs_read_deferred() */
+		smp_store_release(&btf_mod->ready, true);
+		wake_up_all(&btf_module_wq);
 		parsed = true;
 		/* the list may have changed while the mutex was dropped */
 		goto restart;
-- 
2.47.3

[PATCH bpf-next v3 8/9] bpf, trace, net: prepare CONFIG_DEBUG_INFO_BTF checks for a tristate

[Jay Wang]

The next patch makes CONFIG_DEBUG_INFO_BTF a tristate.  With =m, Kconfig
defines CONFIG_DEBUG_INFO_BTF_MODULE instead of CONFIG_DEBUG_INFO_BTF,
so every check that must hold for both =y and =m has to be written for
it:

 - #ifdef CONFIG_DEBUG_INFO_BTF becomes #if IS_ENABLED(...) where the
   generated BTF and its id tables must be the same for =y and =m: the
   .BTF_ids tables (btf_ids.h), the BTF type tags (compiler_types.h), and
   the tracepoint and syscall BTF ids (trace_events.h, trace_syscalls.c).
   Leaving them would silently produce empty id sets with =m.

 - obj-$(CONFIG_DEBUG_INFO_BTF) and include-$(CONFIG_DEBUG_INFO_BTF)
   become $(subst m,y,...) where the object is built into the kernel
   regardless: sysfs_btf.o, the netfilter and xfrm kfunc objects, and
   scripts/Makefile.btf.  Otherwise =m would try to build them as
   modules (xfrm_state_bpf.o fails modpost for lack of MODULE_LICENSE)
   or skip the BTF generation flags.

 - "depends on !DEBUG_INFO_BTF" becomes "depends on DEBUG_INFO_BTF=n"
   for RUST and GENDWARFKSYMS: with =m the BTF is generated as with =y,
   so the pahole restrictions they express still apply, but !m is m,
   which a bool option takes as y.

No functional change: CONFIG_DEBUG_INFO_BTF is still a bool, for which
IS_ENABLED() and #ifdef agree, $(subst m,y,y) is y and "=n" is "!".

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 Makefile                       | 3 ++-
 include/linux/btf_ids.h        | 2 +-
 include/linux/compiler_types.h | 2 +-
 include/trace/trace_events.h   | 2 +-
 init/Kconfig                   | 2 +-
 kernel/bpf/Makefile            | 2 +-
 kernel/module/Kconfig          | 2 +-
 kernel/trace/trace_syscalls.c  | 6 +++---
 net/netfilter/Makefile         | 6 +++---
 net/xfrm/Makefile              | 4 ++--
 10 files changed, 16 insertions(+), 15 deletions(-)

diff --git a/Makefile b/Makefile
index 751a08643bf8..f561516e1735 100644
--- a/Makefile
+++ b/Makefile
@@ -1208,7 +1208,8 @@ endif
 # include additional Makefiles when needed
 include-y			:= scripts/Makefile.warn
 include-$(CONFIG_DEBUG_INFO)	+= scripts/Makefile.debug
-include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf
+# CONFIG_DEBUG_INFO_BTF is a tristate; BTF is generated for both y and m
+include-$(subst m,y,$(CONFIG_DEBUG_INFO_BTF)) += scripts/Makefile.btf
 include-$(CONFIG_KASAN)		+= scripts/Makefile.kasan
 include-$(CONFIG_KCSAN)		+= scripts/Makefile.kcsan
 include-$(CONFIG_KMSAN)		+= scripts/Makefile.kmsan
diff --git a/include/linux/btf_ids.h b/include/linux/btf_ids.h
index 8b5a9ee92513..c665afff100e 100644
--- a/include/linux/btf_ids.h
+++ b/include/linux/btf_ids.h
@@ -22,7 +22,7 @@ struct btf_id_set8 {
 	} pairs[];
 };
 
-#ifdef CONFIG_DEBUG_INFO_BTF
+#if IS_ENABLED(CONFIG_DEBUG_INFO_BTF)
 
 #include <linux/compiler.h> /* for __PASTE */
 #include <linux/compiler_attributes.h> /* for __maybe_unused */
diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
index c5921f139007..a90a99849cee 100644
--- a/include/linux/compiler_types.h
+++ b/include/linux/compiler_types.h
@@ -34,7 +34,7 @@
  * Skipped when running bindgen due to a libclang issue;
  * see https://github.com/rust-lang/rust-bindgen/issues/2244.
  */
-#if defined(CONFIG_DEBUG_INFO_BTF) && defined(CONFIG_PAHOLE_HAS_BTF_TAG) && \
+#if IS_ENABLED(CONFIG_DEBUG_INFO_BTF) && defined(CONFIG_PAHOLE_HAS_BTF_TAG) && \
 	__has_attribute(btf_type_tag) && !defined(__BINDGEN__)
 # define BTF_TYPE_TAG(value) __attribute__((btf_type_tag(#value)))
 #else
diff --git a/include/trace/trace_events.h b/include/trace/trace_events.h
index 93011f800d0f..2a0098929771 100644
--- a/include/trace/trace_events.h
+++ b/include/trace/trace_events.h
@@ -398,7 +398,7 @@ static inline notrace int trace_event_get_offsets_##call(		\
 #define _TRACE_PERF_INIT(call)
 #endif /* CONFIG_PERF_EVENTS */
 
-#if defined(CONFIG_BPF_EVENTS) && defined(CONFIG_DEBUG_INFO_BTF)
+#if defined(CONFIG_BPF_EVENTS) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)
 /*
  * Per-template BTF id list, populated at link time by resolve_btfids:
  *   [0] FUNC   __bpf_trace_<call>     (the BPF dispatcher)
diff --git a/init/Kconfig b/init/Kconfig
index 8583d9f06c52..9f9562f813c5 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2257,7 +2257,7 @@ config RUST
 	depends on !MODVERSIONS || GENDWARFKSYMS
 	depends on !GCC_PLUGIN_RANDSTRUCT
 	depends on !RANDSTRUCT
-	depends on !DEBUG_INFO_BTF || (PAHOLE_HAS_LANG_EXCLUDE && !LTO)
+	depends on DEBUG_INFO_BTF=n || (PAHOLE_HAS_LANG_EXCLUDE && !LTO)
 	depends on !CFI || HAVE_CFI_ICALL_NORMALIZE_INTEGERS_RUSTC
 	select CFI_ICALL_NORMALIZE_INTEGERS if CFI
 	depends on !KASAN || CC_IS_CLANG
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index c1f9b0d3468d..0b7db88f1bed 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -41,7 +41,7 @@ ifeq ($(CONFIG_INET),y)
 obj-$(CONFIG_BPF_SYSCALL) += reuseport_array.o
 endif
 ifeq ($(CONFIG_SYSFS),y)
-obj-$(CONFIG_DEBUG_INFO_BTF) += sysfs_btf.o
+obj-$(subst m,y,$(CONFIG_DEBUG_INFO_BTF)) += sysfs_btf.o
 endif
 ifeq ($(CONFIG_BPF_JIT),y)
 obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o
diff --git a/kernel/module/Kconfig b/kernel/module/Kconfig
index 43b1bb01fd27..da49cb984b0d 100644
--- a/kernel/module/Kconfig
+++ b/kernel/module/Kconfig
@@ -197,7 +197,7 @@ config GENDWARFKSYMS
 	# X86, requires pahole before commit 47dcb534e253 ("btf_encoder: Stop
 	# indexing symbols for VARs") or after commit 9810758003ce ("btf_encoder:
 	# Verify 0 address DWARF variables are in ELF section").
-	depends on !X86 || !DEBUG_INFO_BTF || PAHOLE_VERSION < 128 || PAHOLE_VERSION > 129
+	depends on !X86 || DEBUG_INFO_BTF=n || PAHOLE_VERSION < 128 || PAHOLE_VERSION > 129
 	help
 	  Calculate symbol versions from DWARF debugging information using
 	  gendwarfksyms. Requires DEBUG_INFO to be enabled.
diff --git a/kernel/trace/trace_syscalls.c b/kernel/trace/trace_syscalls.c
index e35744049e3f..7a0d59c308c2 100644
--- a/kernel/trace/trace_syscalls.c
+++ b/kernel/trace/trace_syscalls.c
@@ -1304,7 +1304,7 @@ struct trace_event_functions exit_syscall_print_funcs = {
 	.trace		= print_syscall_exit,
 };
 
-#if defined(CONFIG_BPF_EVENTS) && defined(CONFIG_DEBUG_INFO_BTF)
+#if defined(CONFIG_BPF_EVENTS) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)
 /* BTF id lists for the shared sys_enter/sys_exit dispatcher tracepoints. */
 BTF_ID_LIST(syscall_enter_btf_ids)
 BTF_ID(func,   __bpf_trace_sys_enter)
@@ -1321,7 +1321,7 @@ struct trace_event_class __refdata event_class_syscall_enter = {
 	.fields_array	= syscall_enter_fields_array,
 	.get_fields	= syscall_get_enter_fields,
 	.raw_init	= init_syscall_trace,
-#if defined(CONFIG_BPF_EVENTS) && defined(CONFIG_DEBUG_INFO_BTF)
+#if defined(CONFIG_BPF_EVENTS) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)
 	.btf_ids	= syscall_enter_btf_ids,
 #endif
 };
@@ -1336,7 +1336,7 @@ struct trace_event_class __refdata event_class_syscall_exit = {
 	},
 	.fields		= LIST_HEAD_INIT(event_class_syscall_exit.fields),
 	.raw_init	= init_syscall_trace,
-#if defined(CONFIG_BPF_EVENTS) && defined(CONFIG_DEBUG_INFO_BTF)
+#if defined(CONFIG_BPF_EVENTS) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)
 	.btf_ids	= syscall_exit_btf_ids,
 #endif
 };
diff --git a/net/netfilter/Makefile b/net/netfilter/Makefile
index 6bf74d488a29..a2c7f00794d2 100644
--- a/net/netfilter/Makefile
+++ b/net/netfilter/Makefile
@@ -18,7 +18,7 @@ nf_conntrack-$(CONFIG_NF_CT_PROTO_GRE) += nf_conntrack_proto_gre.o
 ifeq ($(CONFIG_NF_CONNTRACK),m)
 nf_conntrack-$(CONFIG_DEBUG_INFO_BTF_MODULES) += nf_conntrack_bpf.o
 else ifeq ($(CONFIG_NF_CONNTRACK),y)
-nf_conntrack-$(CONFIG_DEBUG_INFO_BTF) += nf_conntrack_bpf.o
+nf_conntrack-$(subst m,y,$(CONFIG_DEBUG_INFO_BTF)) += nf_conntrack_bpf.o
 endif
 
 obj-$(CONFIG_NETFILTER) = netfilter.o
@@ -65,7 +65,7 @@ nf_nat-$(CONFIG_NF_NAT_OVS) += nf_nat_ovs.o
 ifeq ($(CONFIG_NF_NAT),m)
 nf_nat-$(CONFIG_DEBUG_INFO_BTF_MODULES) += nf_nat_bpf.o
 else ifeq ($(CONFIG_NF_NAT),y)
-nf_nat-$(CONFIG_DEBUG_INFO_BTF) += nf_nat_bpf.o
+nf_nat-$(subst m,y,$(CONFIG_DEBUG_INFO_BTF)) += nf_nat_bpf.o
 endif
 
 # NAT helpers
@@ -147,7 +147,7 @@ nf_flow_table-$(CONFIG_NF_FLOW_TABLE_PROCFS) += nf_flow_table_procfs.o
 ifeq ($(CONFIG_NF_FLOW_TABLE),m)
 nf_flow_table-$(CONFIG_DEBUG_INFO_BTF_MODULES) += nf_flow_table_bpf.o
 else ifeq ($(CONFIG_NF_FLOW_TABLE),y)
-nf_flow_table-$(CONFIG_DEBUG_INFO_BTF) += nf_flow_table_bpf.o
+nf_flow_table-$(subst m,y,$(CONFIG_DEBUG_INFO_BTF)) += nf_flow_table_bpf.o
 endif
 
 obj-$(CONFIG_NF_FLOW_TABLE_INET) += nf_flow_table_inet.o
diff --git a/net/xfrm/Makefile b/net/xfrm/Makefile
index 5a1787587cb3..b7f6e5046a0e 100644
--- a/net/xfrm/Makefile
+++ b/net/xfrm/Makefile
@@ -8,7 +8,7 @@ xfrm_interface-$(CONFIG_XFRM_INTERFACE) += xfrm_interface_core.o
 ifeq ($(CONFIG_XFRM_INTERFACE),m)
 xfrm_interface-$(CONFIG_DEBUG_INFO_BTF_MODULES) += xfrm_interface_bpf.o
 else ifeq ($(CONFIG_XFRM_INTERFACE),y)
-xfrm_interface-$(CONFIG_DEBUG_INFO_BTF) += xfrm_interface_bpf.o
+xfrm_interface-$(subst m,y,$(CONFIG_DEBUG_INFO_BTF)) += xfrm_interface_bpf.o
 endif
 
 obj-$(CONFIG_XFRM) := xfrm_policy.o xfrm_state.o xfrm_hash.o \
@@ -23,4 +23,4 @@ obj-$(CONFIG_XFRM_IPCOMP) += xfrm_ipcomp.o
 obj-$(CONFIG_XFRM_INTERFACE) += xfrm_interface.o
 obj-$(CONFIG_XFRM_IPTFS) += xfrm_iptfs.o
 obj-$(CONFIG_XFRM_ESPINTCP) += espintcp.o
-obj-$(CONFIG_DEBUG_INFO_BTF) += xfrm_state_bpf.o
+obj-$(subst m,y,$(CONFIG_DEBUG_INFO_BTF)) += xfrm_state_bpf.o
-- 
2.47.3

[PATCH bpf-next v3 9/9] kbuild, bpf: allow building the vmlinux BTF as a module

[Jay Wang]

Make CONFIG_DEBUG_INFO_BTF a tristate.  With =m the vmlinux BTF is not
part of the kernel image: it is carried by a new module, btf_vmlinux, and
loaded the first time something needs it.  Nothing that works with =y
stops working; the 5.4 MiB of read-only data (distribution config) is
simply not there on systems where nothing uses it.

The only way to save that memory today is CONFIG_DEBUG_INFO_BTF=n, which
a distribution cannot ship: one binary goes to every user, and off takes
BTF away from the users of CO-RE, fentry/fexit, kfuncs, struct_ops,
sched_ext or bpf-lsm.  Whether BTF is used is a property of the
workload, not of the build, so let the first user decide.

The BTF is generated as before, but with =m the .BTF section is linked
as a non-loadable section (like .comment), so the kernel image does not
load it, and the final step that makes vmlinux from vmlinux.unstripped
strips it: every boot image made from vmlinux, whether a raw binary or
an ELF copy, is without it.  Module BTF is generated against
vmlinux.unstripped, which keeps it.  .BTF_ids stays loadable, the
verifier needs it once the BTF is loaded.  The object that carries .BTF
also carries .BTF.meta, the size and SHA-256 of the BTF (struct
btf_vmlinux_meta, checked by the module notifier); the first link, which
the BTF is generated from, gets a zeroed .BTF.meta of the same size.
kernel/bpf/btf_vmlinux.c is an empty carrier module; scripts/gen-btf.sh
gives it the vmlinux .BTF as its own .BTF section instead of generating
split BTF for it, so that one module depends on vmlinux with =m; with
CONFIG_DEBUG_INFO_BTF_MODULES all of them do, as before.

CONFIG_BPF_PRELOAD is not selectable with =m: its iterator programs
attach through the vmlinux BTF, so every bpffs mount (systemd does one
at boot) would load it and defeat the point.  Module BTF is still kept
when a module loads, as with =y; the saving is the vmlinux BTF only.
Programs that need kernel types before the root file system is mounted
need btf_vmlinux.ko in the initramfs; the Kconfig help says so.

The module has no exit: once loaded the BTF stays, as with =y.  The
runtime side -- loading the module on first use, checking it against
.BTF.meta, deferring kfunc and struct_ops registrations and module BTF
until it arrives -- and the IS_ENABLED()/$(subst m,y,...) preparation of
the existing checks are in the preceding patches; this one makes it
selectable.

Tested with 1 GiB of memory, same tree, =y vs =m, both with
CONFIG_DEBUG_INFO_BTF_MODULES=y:

 - MemTotal is ~5.4 MB higher with =m while the BTF is unused: the size
   of the .BTF section.
 - stat() of /sys/kernel/btf/vmlinux reports the BTF size before it is
   loaded, as the btf_sysfs selftest expects.
 - With BTF in use, MemFree is the same within run-to-run noise.
 - Modules loaded before the trigger (ext4, nf_conntrack and its kfuncs,
   xfrm_interface) appear in /sys/kernel/btf immediately and get BTF ids
   once the BTF is loaded; a socket filter loads without loading the
   module; a kprobe program calling bpf_get_current_task_btf(), opening
   /sys/kernel/btf/vmlinux or BPF_BTF_GET_NEXT_ID each load it.
 - A carrier module with one byte of its .BTF changed is refused with
   "BTF does not match this kernel" and leaves no state behind.
 - After the load: fstat/read/mmap of /sys/kernel/btf/vmlinux, a
   struct_ops map for tcp_congestion_ops, a syscall program calling the
   bpf_task_from_pid()/bpf_task_release() kfuncs, and modules loaded
   afterwards (nf_nat) all work as with =y.
 - =m without DEBUG_INFO_BTF_MODULES, and =y, build and pass the same
   tests.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 Documentation/bpf/btf.rst         | 35 ++++++++++++
 Makefile                          |  5 +-
 include/asm-generic/vmlinux.lds.h | 31 +++++++++-
 kernel/bpf/Makefile               |  4 ++
 kernel/bpf/btf_vmlinux.c          | 23 ++++++++
 kernel/bpf/preload/Kconfig        |  4 ++
 lib/Kconfig.debug                 | 22 ++++++-
 scripts/Makefile.modfinal         | 26 +++++++--
 scripts/Makefile.vmlinux          |  5 ++
 scripts/gen-btf.sh                | 95 +++++++++++++++++++++++++++++--
 scripts/link-vmlinux.sh           | 25 ++++++--
 11 files changed, 257 insertions(+), 18 deletions(-)
 create mode 100644 kernel/bpf/btf_vmlinux.c

diff --git a/Documentation/bpf/btf.rst b/Documentation/bpf/btf.rst
index 29de1222c3e7..1bd0d35cdb05 100644
--- a/Documentation/bpf/btf.rst
+++ b/Documentation/bpf/btf.rst
@@ -1276,6 +1276,41 @@ format.::
             .long   58
             .long   8206                    # Line 8 Col 14
 
+6.1 Kernel BTF
+--------------
+
+With CONFIG_DEBUG_INFO_BTF=y the BTF of the kernel is generated at link time
+from its DWARF and placed in the .BTF section of vmlinux, which is read-only
+data of the kernel image. It is available as /sys/kernel/btf/vmlinux and, if
+CONFIG_DEBUG_INFO_BTF_MODULES is set, module BTF is generated as split BTF
+against it and available as /sys/kernel/btf/<module>.
+
+With CONFIG_DEBUG_INFO_BTF=m the same BTF is generated, but it is not part of
+the kernel image or of the vmlinux ELF file (vmlinux.unstripped in the build
+tree keeps it, for module BTF generation). It is delivered by the
+btf_vmlinux module, which the kernel loads on demand the first time the BTF is
+needed: when /sys/kernel/btf/vmlinux is read or mmap()ed, when kernel BTF
+objects are enumerated (BPF_BTF_GET_NEXT_ID), or when a BPF program needs
+kernel type information (an attach_btf_id, a kfunc call, a ksym, a map pointer
+or a helper that takes or returns a kernel BTF pointer). Until then no memory
+is used for it, and afterwards nothing differs from =y. In particular:
+
+  * /sys/kernel/btf/vmlinux exists from boot with its final size.
+  * Modules loaded before the vmlinux BTF are exposed in /sys/kernel/btf right
+    away, their BTF is parsed and gets a BTF id once the vmlinux BTF is
+    loaded, together with their kfunc and struct_ops registrations.
+  * kfunc, dtor kfunc and struct_ops registrations of the kernel itself are
+    applied before the BTF becomes visible.
+  * The kernel only accepts the BTF it was built with: the size and SHA-256 of
+    the BTF are linked into the kernel and checked against the module.
+  * Once loaded the BTF stays; the module cannot be unloaded.
+
+If the module is not available (not installed, or the root file system is not
+mounted yet), the kernel behaves as one built without BTF and retries next
+time. CONFIG_BPF_PRELOAD is not available with =m: its iterators attach through
+the vmlinux BTF, so mounting bpffs would load it. bpf_snprintf_btf() and bpf_seq_printf_btf() only use the BTF if it has
+already been parsed, as they run in program context.
+
 7. Testing
 ==========
 
diff --git a/Makefile b/Makefile
index f561516e1735..7ce5d478abd3 100644
--- a/Makefile
+++ b/Makefile
@@ -1745,8 +1745,9 @@ endif
 #
 
 # *.ko are usually independent of vmlinux, but CONFIG_DEBUG_INFO_BTF_MODULES
-# is an exception.
-ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+# is an exception, and so is the btf_vmlinux module with CONFIG_DEBUG_INFO_BTF=m,
+# which carries the vmlinux BTF.
+ifneq ($(CONFIG_DEBUG_INFO_BTF_MODULES)$(filter m,$(CONFIG_DEBUG_INFO_BTF)),)
 KBUILD_BUILTIN := y
 modules: vmlinux
 endif
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index b2988aa12f66..204a9ee171c6 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -674,8 +674,18 @@
 
 /*
  * .BTF
+ *
+ * With CONFIG_DEBUG_INFO_BTF=y the vmlinux BTF is loaded as read-only data and
+ * bounded by __start_BTF/__stop_BTF.  With CONFIG_DEBUG_INFO_BTF=m it is
+ * linked as a non-loadable section (see BTF_NOLOAD in ELF_DETAILS), so that
+ * module BTF generation can read it from vmlinux.unstripped; it is stripped
+ * from vmlinux (scripts/Makefile.vmlinux), and the btf_vmlinux module carries
+ * a copy and provides it on demand at runtime.
+ * What is loaded instead is .BTF.meta, the size and hash of that BTF (see
+ * scripts/gen-btf.sh), empty in the first link that the BTF is generated
+ * from.  .BTF_ids is needed by the kernel in both cases.
  */
-#ifdef CONFIG_DEBUG_INFO_BTF
+#if IS_BUILTIN(CONFIG_DEBUG_INFO_BTF)
 #define BTF								\
 	. = ALIGN(PAGE_SIZE);						\
 	.BTF : AT(ADDR(.BTF) - LOAD_OFFSET) {				\
@@ -685,10 +695,28 @@
 	.BTF_ids : AT(ADDR(.BTF_ids) - LOAD_OFFSET) {			\
 		*(.BTF_ids)						\
 	}
+#elif IS_MODULE(CONFIG_DEBUG_INFO_BTF)
+#define BTF								\
+	. = ALIGN(8);							\
+	.BTF.meta : AT(ADDR(.BTF.meta) - LOAD_OFFSET) {			\
+		BOUNDED_SECTION_BY(.BTF.meta, _BTF_meta)		\
+	}								\
+	. = ALIGN(PAGE_SIZE);						\
+	.BTF_ids : AT(ADDR(.BTF_ids) - LOAD_OFFSET) {			\
+		*(.BTF_ids)						\
+	}
 #else
 #define BTF
 #endif
 
+#if IS_MODULE(CONFIG_DEBUG_INFO_BTF)
+/* quoted: BTF is a macro, an unquoted .BTF here would expand it */
+#define BTF_NOLOAD							\
+		".BTF" 0 : { *(".BTF") }
+#else
+#define BTF_NOLOAD
+#endif
+
 /*
  * Init task
  */
@@ -849,6 +877,7 @@
 /* Required sections not related to debugging. */
 #define ELF_DETAILS							\
 		.comment 0 : { *(.comment) }				\
+		BTF_NOLOAD						\
 		.symtab 0 : { *(.symtab) }				\
 		.strtab 0 : { *(.strtab) }				\
 		.shstrtab 0 : { *(.shstrtab) }				\
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 0b7db88f1bed..8a7ad4303c0d 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -43,6 +43,10 @@ endif
 ifeq ($(CONFIG_SYSFS),y)
 obj-$(subst m,y,$(CONFIG_DEBUG_INFO_BTF)) += sysfs_btf.o
 endif
+# With CONFIG_DEBUG_INFO_BTF=m the vmlinux BTF is carried by this module
+ifeq ($(CONFIG_DEBUG_INFO_BTF),m)
+obj-m += btf_vmlinux.o
+endif
 ifeq ($(CONFIG_BPF_JIT),y)
 obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o
 obj-$(CONFIG_BPF_SYSCALL) += cpumask.o
diff --git a/kernel/bpf/btf_vmlinux.c b/kernel/bpf/btf_vmlinux.c
new file mode 100644
index 000000000000..8d89b4bb3c43
--- /dev/null
+++ b/kernel/bpf/btf_vmlinux.c
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Carrier module for the vmlinux BTF when CONFIG_DEBUG_INFO_BTF=m.
+ *
+ * This module has no code of its own.  Its .BTF section is a copy of the
+ * vmlinux BTF (see scripts/gen-btf.sh), which the BTF module notifier in
+ * kernel/bpf/btf.c recognizes by module name and installs as the vmlinux BTF.
+ * The kernel loads it on demand, the first time the vmlinux BTF is needed.
+ *
+ * There is deliberately no module_exit(): once the BTF is in use it cannot
+ * be taken away again, exactly as with CONFIG_DEBUG_INFO_BTF=y.
+ */
+#include <linux/init.h>
+#include <linux/module.h>
+
+static int __init btf_vmlinux_init(void)
+{
+	return 0;
+}
+module_init(btf_vmlinux_init);
+
+MODULE_DESCRIPTION("BTF type information for vmlinux");
+MODULE_LICENSE("GPL");
diff --git a/kernel/bpf/preload/Kconfig b/kernel/bpf/preload/Kconfig
index aef7b0bc96d6..b1600bdce7a0 100644
--- a/kernel/bpf/preload/Kconfig
+++ b/kernel/bpf/preload/Kconfig
@@ -6,6 +6,10 @@ menuconfig BPF_PRELOAD
 	# The dependency on !COMPILE_TEST prevents it from being enabled
 	# in allmodconfig or allyesconfig configurations
 	depends on !COMPILE_TEST
+	# The preloaded iterators attach through the vmlinux BTF, so with
+	# CONFIG_DEBUG_INFO_BTF=m every bpffs mount would load the BTF, which
+	# defeats the point of =m on any system that mounts bpffs at boot.
+	depends on DEBUG_INFO_BTF!=m
 	help
 	  This builds kernel module with several embedded BPF programs that are
 	  pinned into BPF FS mount point as human readable files that are
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 134b15a44625..418025c5e657 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -396,7 +396,7 @@ config DEBUG_INFO_SPLIT
 	  Incompatible with older versions of ccache.
 
 config DEBUG_INFO_BTF
-	bool "Generate BTF type information"
+	tristate "Generate BTF type information"
 	depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
 	depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
 	depends on BPF_SYSCALL
@@ -408,6 +408,26 @@ config DEBUG_INFO_BTF
 	  Turning this on requires pahole v1.22 or later, which will convert
 	  DWARF type info into equivalent deduplicated BTF type info.
 
+	  If built as a module (=m), the vmlinux BTF is not part of the
+	  kernel image.  It is carried by the btf_vmlinux module, which is
+	  loaded on demand the first time the BTF is needed: when a BPF
+	  program requires kernel type information, or when
+	  /sys/kernel/btf/vmlinux is opened.  Until then, no memory is
+	  spent on it.  The vmlinux ELF file does not carry the BTF
+	  either; module BTF is generated against vmlinux.unstripped, and
+	  tools that read the BTF from a file can use that or
+	  /sys/kernel/btf/vmlinux.
+
+	  Module BTF (DEBUG_INFO_BTF_MODULES) is kept when a module loads,
+	  as with =y, and registered once the vmlinux BTF is available; the
+	  saving is the vmlinux BTF only.
+
+	  If BPF programs that use kernel types run before the root file
+	  system is mounted, put btf_vmlinux.ko into the initramfs: until
+	  the module can be loaded, such programs fail as on a kernel
+	  without BTF.  Not compatible with BPF_PRELOAD, whose iterators
+	  would load the BTF at every bpffs mount.
+
 config PAHOLE_HAS_BTF_TAG
 	def_bool PAHOLE_VERSION >= 123
 	depends on CC_IS_CLANG
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index 01a37ec872b9..6a0958cf0e6f 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -38,20 +38,34 @@ quiet_cmd_ld_ko_o = LD [M]  $@
 		$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE)		\
 		-T $(objtree)/scripts/module.lds -o $@ $(filter %.o, $^)
 
+# The ELF file with the vmlinux BTF: with CONFIG_DEBUG_INFO_BTF=m the BTF is
+# stripped from vmlinux (scripts/Makefile.vmlinux), vmlinux.unstripped keeps it.
+btf-vmlinux := $(objtree)/vmlinux$(if $(filter m,$(CONFIG_DEBUG_INFO_BTF)),.unstripped)
+
 quiet_cmd_btf_ko = BTF [M] $@
       cmd_btf_ko = 							\
-	if [ ! -f $(objtree)/vmlinux ]; then				\
+	if [ ! -f $(btf-vmlinux) ]; then				\
 		printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
 	else	\
-		$(CONFIG_SHELL) $(srctree)/scripts/gen-btf.sh --btf_base $(objtree)/vmlinux $@; \
+		$(CONFIG_SHELL) $(srctree)/scripts/gen-btf.sh --btf_base $(btf-vmlinux) $@; \
 	fi;
 
-# Re-generate module BTFs if either module's .ko or vmlinux changed
-%.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(objtree)/vmlinux) FORCE
-	+$(call if_changed,ld_ko_o)
+# Modules that get a .BTF section: all of them with CONFIG_DEBUG_INFO_BTF_MODULES,
+# otherwise only the vmlinux BTF carrier module with CONFIG_DEBUG_INFO_BTF=m.
 ifdef CONFIG_DEBUG_INFO_BTF_MODULES
-	+$(if $(newer-prereqs),$(call cmd,btf_ko))
+btf-modules := $(modules:%.o=%.ko)
+else ifeq ($(CONFIG_DEBUG_INFO_BTF),m)
+btf-modules := $(filter %/btf_vmlinux.ko,$(modules:%.o=%.ko))
+# Only the carrier depends on vmlinux, not every module
+ifdef KBUILD_BUILTIN
+$(btf-modules): $(btf-vmlinux)
+endif
 endif
+
+# Re-generate module BTFs if either module's .ko or vmlinux changed
+%.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(btf-vmlinux)) FORCE
+	+$(call if_changed,ld_ko_o)
+	+$(if $(and $(filter $@,$(btf-modules)),$(newer-prereqs)),$(call cmd,btf_ko))
 	+$(call cmd,check_tracepoint)
 
 targets += $(modules:%.o=%.ko) $(modules:%.o=%.mod.o) .module-common.o
diff --git a/scripts/Makefile.vmlinux b/scripts/Makefile.vmlinux
index fcae1e432d9a..557db1ee1f3b 100644
--- a/scripts/Makefile.vmlinux
+++ b/scripts/Makefile.vmlinux
@@ -86,6 +86,11 @@ remove-section-$(CONFIG_ARCH_VMLINUX_NEEDS_RELOCS) += '.rel*' '!.rel*.dyn'
 # for compatibility with binutils < 2.32
 # https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=c12d9fa2afe7abcbe407a00e15719e1a1350c2a7
 remove-section-$(CONFIG_ARCH_VMLINUX_NEEDS_RELOCS) += '.rel.*'
+# With CONFIG_DEBUG_INFO_BTF=m the btf_vmlinux module carries the vmlinux BTF;
+# only vmlinux.unstripped keeps it, for module BTF generation.
+ifeq ($(CONFIG_DEBUG_INFO_BTF),m)
+remove-section-y += .BTF
+endif
 
 remove-symbols := -w --strip-unneeded-symbol='__mod_device_table__*'
 
diff --git a/scripts/gen-btf.sh b/scripts/gen-btf.sh
index 8ca96eb10a69..d808d0006d32 100755
--- a/scripts/gen-btf.sh
+++ b/scripts/gen-btf.sh
@@ -22,16 +22,27 @@
 #   - ${1}.btf.o ready for linking into vmlinux
 #   - ${1}.BTF_ids with .BTF_ids data blob
 # This output is consumed by scripts/link-vmlinux.sh
+#
+# With CONFIG_DEBUG_INFO_BTF=m the .BTF section in ${1}.btf.o is not
+# allocatable, so the kernel image does not carry the BTF; vmlinux.unstripped
+# does, for module BTF generation, and scripts/Makefile.vmlinux strips it from
+# vmlinux.  ${1}.btf.o then also carries .BTF.meta, the size and SHA-256 of
+# the BTF for the kernel (struct btf_vmlinux_meta); "--placeholder ${1}"
+# produces a ${1}.btf.o with a zeroed .BTF.meta and no .BTF for the first
+# vmlinux link, which the BTF is generated from.  The btf_vmlinux module gets
+# no BTF of its own; its .BTF section is a copy of the vmlinux BTF, extracted
+# from --btf_base.
 
 set -e
 
 usage()
 {
-	echo "Usage: $0 [--btf_base <file>] <target ELF file>"
+	echo "Usage: $0 [--btf_base <file>] [--placeholder] <target ELF file>"
 	exit 1
 }
 
 BTF_BASE=""
+PLACEHOLDER=""
 
 while [ $# -gt 0 ]; do
 	case "$1" in
@@ -39,6 +50,10 @@ while [ $# -gt 0 ]; do
 		BTF_BASE="$2"
 		shift 2
 		;;
+	--placeholder)
+		PLACEHOLDER=1
+		shift
+		;;
 	-*)
 		echo "Unknown option: $1" >&2
 		usage
@@ -60,6 +75,10 @@ is_enabled() {
 	grep -q "^$1=y" ${objtree}/include/config/auto.conf
 }
 
+is_module() {
+	grep -q "^$1=m" ${objtree}/include/config/auto.conf
+}
+
 case "${KBUILD_VERBOSE}" in
 *1*)
 	set -x
@@ -79,6 +98,30 @@ gen_btf_data()
 		--btf ${btf1} "${ELF_FILE}"
 }
 
+# Write one byte with value $1 (0..255)
+put_byte()
+{
+	printf "\\$(printf '%03o' "$1")"
+}
+
+# CONFIG_DEBUG_INFO_BTF=m: write struct btf_vmlinux_meta { u32 size; u8
+# sha256[32]; } for the BTF in $1 to $2, in the target's byte order.
+gen_btf_meta()
+{
+	size=$(${CONFIG_SHELL} "${srctree}/scripts/file-size.sh" "$1")
+	sha256=$(sha256sum < "$1" | cut -d' ' -f1)
+	{
+		if is_enabled CONFIG_CPU_BIG_ENDIAN; then
+			for shift in 24 16 8 0; do put_byte $(( (size >> shift) & 255 )); done
+		else
+			for shift in 0 8 16 24; do put_byte $(( (size >> shift) & 255 )); done
+		fi
+		for byte in $(echo "${sha256}" | sed 's/../& /g'); do
+			put_byte $(( 0x${byte} ))
+		done
+	} > "$2"
+}
+
 gen_btf_o()
 {
 	btf_data=${ELF_FILE}.btf.o
@@ -88,9 +131,23 @@ gen_btf_o()
 	# deletes all symbols including __start_BTF and __stop_BTF, which will
 	# be redefined in the linker script.
 	echo "" | ${CC} ${CLANG_FLAGS} ${KBUILD_CPPFLAGS} ${KBUILD_CFLAGS} -fno-lto -c -x c -o ${btf_data} -
-	${OBJCOPY} --add-section .BTF=${ELF_FILE}.BTF \
-		--set-section-flags .BTF=alloc,readonly ${btf_data}
-	${OBJCOPY} --only-section=.BTF --strip-all ${btf_data}
+	if is_module CONFIG_DEBUG_INFO_BTF; then
+		# CONFIG_DEBUG_INFO_BTF=m: .BTF stays non-allocatable, kept in
+		# vmlinux.unstripped for module BTF but not loaded; the btf_vmlinux
+		# module provides it at runtime.  What is loaded is .BTF.meta,
+		# its size and hash, so that /sys/kernel/btf/vmlinux has the right
+		# size from boot and only the matching BTF is accepted.
+		gen_btf_meta ${ELF_FILE}.BTF ${ELF_FILE}.BTF.meta
+		${OBJCOPY} --add-section .BTF=${ELF_FILE}.BTF \
+			--set-section-flags .BTF=readonly \
+			--add-section .BTF.meta=${ELF_FILE}.BTF.meta \
+			--set-section-flags .BTF.meta=alloc,readonly ${btf_data}
+		${OBJCOPY} --only-section=.BTF --only-section=.BTF.meta --strip-all ${btf_data}
+	else
+		${OBJCOPY} --add-section .BTF=${ELF_FILE}.BTF \
+			--set-section-flags .BTF=alloc,readonly ${btf_data}
+		${OBJCOPY} --only-section=.BTF --strip-all ${btf_data}
+	fi
 
 	# Change e_type to ET_REL so that it can be used to link final vmlinux.
 	# GNU ld 2.35+ and lld do not allow an ET_EXEC input.
@@ -121,6 +178,7 @@ cleanup()
 {
 	rm -f "${ELF_FILE}.BTF.1"
 	rm -f "${ELF_FILE}.BTF"
+	rm -f "${ELF_FILE}.BTF.meta"
 	if [ "${BTFGEN_MODE}" = "module" ]; then
 		rm -f "${ELF_FILE}.BTF.base"
 		rm -f "${ELF_FILE}.BTF_ids"
@@ -133,6 +191,35 @@ if [ -n "${BTF_BASE}" ]; then
 	BTFGEN_MODE="module"
 fi
 
+if [ -n "${PLACEHOLDER}" ]; then
+	btf_data=${ELF_FILE}.btf.o
+	echo "" | ${CC} ${CLANG_FLAGS} ${KBUILD_CPPFLAGS} ${KBUILD_CFLAGS} -fno-lto -c -x c -o ${btf_data} -
+	dd if=/dev/zero of=${ELF_FILE}.BTF.meta bs=36 count=1 2>/dev/null
+	${OBJCOPY} --add-section .BTF.meta=${ELF_FILE}.BTF.meta \
+		--set-section-flags .BTF.meta=alloc,readonly ${btf_data}
+	${OBJCOPY} --only-section=.BTF.meta --strip-all ${btf_data}
+	exit 0
+fi
+
+# CONFIG_DEBUG_INFO_BTF=m: the btf_vmlinux module carries the vmlinux BTF
+# itself.  Its own types are of no interest, so instead of generating split
+# BTF for it, copy the (non-loadable) .BTF section of --btf_base
+# (vmlinux.unstripped) into the module.
+# The kernel recognizes the module by name and treats its .BTF as base BTF.
+case "${BTFGEN_MODE}:${ELF_FILE}" in
+module:*/btf_vmlinux.ko)
+	if is_module CONFIG_DEBUG_INFO_BTF; then
+		# -O binary only emits allocatable sections; make .BTF one for
+		# the extraction.  ${BTF_BASE} itself is not modified.
+		${OBJCOPY} -O binary --only-section=.BTF			\
+			--set-section-flags .BTF=alloc,load,readonly	\
+			"${BTF_BASE}" "${ELF_FILE}.BTF"
+		${OBJCOPY} --add-section .BTF="${ELF_FILE}.BTF" "${ELF_FILE}"
+		exit 0
+	fi
+	;;
+esac
+
 gen_btf_data
 
 case "${BTFGEN_MODE}" in
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index ab0b8125c8cb..a9a066267ef6 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -37,6 +37,15 @@ is_enabled() {
 	grep -q "^$1=y" include/config/auto.conf
 }
 
+is_module() {
+	grep -q "^$1=m" include/config/auto.conf
+}
+
+# =y or =m
+is_set() {
+	grep -q "^$1=[ym]" include/config/auto.conf
+}
+
 # Nice output in kbuild format
 # Will be suppressed by "make -s"
 info()
@@ -211,17 +220,25 @@ if is_enabled CONFIG_KALLSYMS; then
 	kallsyms .tmp_vmlinux0.syms .tmp_vmlinux0.kallsyms
 fi
 
-if is_enabled CONFIG_KALLSYMS || is_enabled CONFIG_DEBUG_INFO_BTF; then
+if is_module CONFIG_DEBUG_INFO_BTF; then
+	# The kernel refers to the size and hash of its BTF, which only the
+	# BTF generated from the first link can provide; link a placeholder
+	# of the same layout until then.
+	${CONFIG_SHELL} ${srctree}/scripts/gen-btf.sh --placeholder .tmp_vmlinux0
+	btf_vmlinux_bin_o=.tmp_vmlinux0.btf.o
+fi
+
+if is_enabled CONFIG_KALLSYMS || is_set CONFIG_DEBUG_INFO_BTF; then
 
 	# The kallsyms linking does not need debug symbols, but the BTF does.
-	if ! is_enabled CONFIG_DEBUG_INFO_BTF; then
+	if ! is_set CONFIG_DEBUG_INFO_BTF; then
 		strip_debug=1
 	fi
 
 	vmlinux_link .tmp_vmlinux1
 fi
 
-if is_enabled CONFIG_DEBUG_INFO_BTF; then
+if is_set CONFIG_DEBUG_INFO_BTF; then
 	info BTF .tmp_vmlinux1
 	if ! ${CONFIG_SHELL} ${srctree}/scripts/gen-btf.sh .tmp_vmlinux1; then
 		echo >&2 "Failed to generate BTF for vmlinux"
@@ -287,7 +304,7 @@ fi
 
 vmlinux_link "${VMLINUX}"
 
-if is_enabled CONFIG_DEBUG_INFO_BTF; then
+if is_set CONFIG_DEBUG_INFO_BTF; then
 	info BTFIDS ${VMLINUX}
 	${RESOLVE_BTFIDS} --patch_btfids ${btfids_vmlinux} ${VMLINUX}
 fi
-- 
2.47.3

Re: [PATCH bpf-next v3 8/9] bpf, trace, net: prepare CONFIG_DEBUG_INFO_BTF checks for a tristate

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3127 bytes --]

> diff --git a/Makefile b/Makefile
> index 751a08643bf8..f561516e1735 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1208,7 +1208,8 @@ endif
>  # include additional Makefiles when needed
>  include-y			:= scripts/Makefile.warn
>  include-$(CONFIG_DEBUG_INFO)	+= scripts/Makefile.debug
> -include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf
> +# CONFIG_DEBUG_INFO_BTF is a tristate; BTF is generated for both y and m
> +include-$(subst m,y,$(CONFIG_DEBUG_INFO_BTF)) += scripts/Makefile.btf
>  include-$(CONFIG_KASAN)		+= scripts/Makefile.kasan
>  include-$(CONFIG_KCSAN)		+= scripts/Makefile.kcsan
>  include-$(CONFIG_KMSAN)		+= scripts/Makefile.kmsan

[ ... ]

> diff --git a/include/trace/trace_events.h b/include/trace/trace_events.h
> index 93011f800d0f..2a0098929771 100644
> --- a/include/trace/trace_events.h
> +++ b/include/trace/trace_events.h
> @@ -398,7 +398,7 @@ static inline notrace int trace_event_get_offsets_##call(		\
>  #define _TRACE_PERF_INIT(call)
>  #endif /* CONFIG_PERF_EVENTS */
>
> -#if defined(CONFIG_BPF_EVENTS) && defined(CONFIG_DEBUG_INFO_BTF)
> +#if defined(CONFIG_BPF_EVENTS) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)

Can this change introduce a deadlock when DEBUG_INFO_BTF=m and the
btf_vmlinux module is not yet loaded?

Before this change, with =m the preprocessor guard was false, so
_TRACE_BTF_IDS_INIT() expanded to nothing and trace_event_class.btf_ids
stayed NULL. With IS_ENABLED() the guard is now true for both =y and =m,
so built-in tracepoint classes initialize .btf_ids and create
events/<sys>/<event>/btf_ids tracefs files.

Reading that file for a built-in event goes through event_btf_ids_read()
in kernel/trace/trace_events.c:

    scoped_guard(mutex, &event_mutex) {
        ...
        ids = call->class->btf_ids;
        ...
        btf = btf_get_module_btf(mod);   /* mod == NULL for built-in */

btf_get_module_btf(NULL) calls bpf_get_btf_vmlinux(), which in the
IS_MODULE(CONFIG_DEBUG_INFO_BTF) case calls request_module("btf_vmlinux")
when the vmlinux BTF has not been loaded yet.

request_module() waits for modprobe to exit (UMH_WAIT_PROC) while
event_mutex is still held. Meanwhile, modprobe does finit_module() ->
load_module() -> prepare_coming_module(), which calls
trace_module_notify():

    mutex_lock(&event_mutex);
    mutex_lock(&trace_types_lock);

so the btf_vmlinux module load blocks on event_mutex held by the reader,
and the reader blocks waiting for modprobe.

The reader hangs until SIGKILLed, and every other module load and
event_mutex user (enabling/disabling events, kprobe/uprobe creation, etc.)
is stuck. Trigger: with DEBUG_INFO_BTF=m and btf_vmlinux not yet loaded,
`cat /sys/kernel/tracing/events/sched/sched_switch/btf_ids` (or any
syscall event's btf_ids).

Should the BTF be resolved before taking event_mutex, or should this path
return -ENOENT when the vmlinux BTF is not yet present?

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36198628965

Re: [PATCH bpf-next v3 4/9] bpf: take the vmlinux BTF from the btf_vmlinux module

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 5071 bytes --]

> diff --git a/include/linux/btf.h b/include/linux/btf.h
> index 4b63bb91550a..81e6c65fe5f6 100644
> --- a/include/linux/btf.h
> +++ b/include/linux/btf.h
> @@ -602,6 +602,7 @@ __u32 *btf_field_iter_next(struct btf_field_iter *it);
>  const char *btf_name_by_offset(const struct btf *btf, u32 offset);
>  const char *btf_str_by_offset(const struct btf *btf, u32 offset);
>  struct btf *btf_parse_vmlinux(void);
> +void *btf_vmlinux_data(u32 *size, bool load);
>  struct bpf_prog_get_target_btf(const struct bpf_prog *prog);
>  u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog);
>  int btf_kfunc_check_flag(const struct btf *btf, u32 kfunc_btf_id, u32 flag);

[ ... ]

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index da86162ba6d9..84968244a10f 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -21922,26 +21922,51 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt
>  	return 0;
>  }
>
> +/*
> + * Returns the parsed vmlinux BTF, NULL if the kernel has none, or an ERR_PTR
> + * if it is malformed.  With CONFIG_DEBUG_INFO_BTF=m the BTF lives in the
> + * btf_vmlinux module; the first caller loads it and parses it.  May sleep.
> + */
>  struct btf *bpf_get_btf_vmlinux(void)
>  {
>  	/* Pairs with the smp_store_release() on the parse path below. */
>  	struct btf *btf = smp_load_acquire(&btf_vmlinux);
> +	u32 size;
>
> -	if (!btf && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
> -		mutex_lock(&btf_vmlinux_lock);
> -		btf = btf_vmlinux;
> -		if (!btf) {
> -			btf = btf_parse_vmlinux();
> -			/*
> -			 * Order the parsed BTF contents and the globals the
> -			 * parse populated (e.g. bpf_ctx_convert.t) before
> -			 * the pointer publication. Pairs with the acquire
> -			 * on the lockless fast path above.
> -			 */
> -			smp_store_release(&btf_vmlinux, btf);
> +	if (btf || !IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
> +		return btf;
> +
> +	/*
> +	 * Loading the module may take a while and its notifier must not be
> +	 * blocked by us, so do it outside btf_vmlinux_lock.  Not available:
> +	 * behave like a kernel without BTF, and retry next time.
> +	 */
> +	if (!btf_vmlinux_data(&size, true))
> +		return NULL;
> +
> +	mutex_lock(&btf_vmlinux_lock);
> +	btf = btf_vmlinux;
> +	if (!btf) {
> +		btf = btf_parse_vmlinux();
> +		/*
> +		 * With =m the BTF was checked against the kernel when the
> +		 * module loaded, so a failure here is a resource problem
> +		 * (-ENOMEM) rather than a broken BTF: do not remember it,
> +		 * the next caller retries.
> +		 */
> +		if (IS_MODULE(CONFIG_DEBUG_INFO_BTF) && IS_ERR(btf)) {
> +			mutex_unlock(&btf_vmlinux_lock);
> +			return btf;

Does this error path account for deterministic parse failures?

The comment states that with CONFIG_DEBUG_INFO_BTF=m a parse failure
here is a resource problem (-ENOMEM) because the payload was checked
when the module loaded. btf_vmlinux_module_coming() does verify the
size and SHA-256 match the .BTF.meta values this kernel was built with.

However, btf_parse_vmlinux() calls btf_parse_base(), which can still
fail with errors that depend only on the content:

kernel/bpf/btf.c:btf_parse_base()
    btf_parse_hdr()           -> -E2BIG, -ENOENT, -EINVAL
    btf_parse_str_sec()       -> -EINVAL
    btf_check_all_metas()     -> -EINVAL for unknown kind or bad name
    btf_check_modifier_chain_length() -> -ELOOP

These checks are deterministic. A broken BTF would fail on every
attempt, yet the code never caches the error. With CONFIG_DEBUG_INFO_BTF=y
the same condition gives 'in-kernel BTF is malformed' and the ERR_PTR
is cached once. With =m, every later caller repeats the full parse of
the ~5.4 MiB BTF while holding btf_vmlinux_lock: kvmalloc of the types
array, a walk over all metas, then the free.

Callers include bpf_prog_load(), struct_ops map creation,
BPF_BTF_GET_NEXT_ID (which runs before the CAP_SYS_ADMIN check in
bpf_obj_get_next_id()), and read()/mmap() of the 0444
/sys/kernel/btf/vmlinux. An unprivileged user can therefore make the
kernel re-parse repeatedly and serialize other BTF users behind the
mutex.

Also, /sys/kernel/btf/vmlinux returns -ENODEV even though the raw bytes
are present, because btf_sysfs_vmlinux_load() requires a successful
parse.

Would it be better to cache only on content errors (e.g. all errors
except -ENOMEM) and retry resource failures, or to skip caching only
on specific errors?

>  		}
> -		mutex_unlock(&btf_vmlinux_lock);
> +		/*
> +		 * Order the parsed BTF contents and the globals the
> +		 * parse populated (e.g. bpf_ctx_convert.t) before
> +		 * the pointer publication. Pairs with the acquire
> +		 * on the lockless fast path above.
> +		 */
> +		smp_store_release(&btf_vmlinux, btf);
>  	}
> +	mutex_unlock(&btf_vmlinux_lock);
>  	return btf;
>  }

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36198628965

Re: [PATCH bpf-next v3 7/9] bpf: expose deferred .BTF.base module BTF in sysfs from module load

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 5908 bytes --]

> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index 1dae1c9b53fe..64583fb1f380 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c

[ ... ]

> @@ -9165,8 +9175,14 @@ static int btf_module_sysfs_add(struct btf_module *btf_mod, const char *name,
>  	return 0;
>  }
>
> +/*
> + * Called with btf_module_mutex NOT held: removing the sysfs file waits for
> + * readers to leave, and a deferred reader may need the mutex to get there.
> + */
>  static void btf_module_free(struct btf_module *btf_mod)
>  {
> +	WRITE_ONCE(btf_mod->gone, true);
> +	wake_up_all(&btf_module_wq);
>  	if (btf_mod->sysfs_attr)
>  		sysfs_remove_bin_file(btf_kobj, btf_mod->sysfs_attr);

[ ... ]

> @@ -9221,15 +9237,57 @@ static int btf_vmlinux_module_coming(struct module *mod)
>  	return 0;
>  }
>
> +/*
> + * sysfs reader for a module kept aside with a .BTF.base section: its .BTF is
> + * split against the distilled base and only becomes valid split BTF against
> + * the vmlinux BTF once relocated, which rewrites the buffer in place.  So
> + * first make sure the vmlinux BTF is loaded (which parses and relocates the
> + * kept modules), then wait until this module's BTF is published.  The size
> + * does not change: relocation only rewrites ids and string offsets.
> + */
> +static bool btf_module_published(struct btf_module *btf_mod)
> +{
> +	/* Pairs with the smp_store_release() of @ready after btf_mod->btf is set */
> +	return smp_load_acquire(&btf_mod->ready);
> +}
> +
> +static ssize_t btf_module_sysfs_read_deferred(struct file *filp, struct kobject *kobj,
> +					      const struct bin_attribute *attr,
> +					      char *buf, loff_t off, size_t count)
> +{
> +	struct btf_module *btf_mod = attr->private;
> +	int err;
> +
> +	if (IS_ERR_OR_NULL(bpf_get_btf_vmlinux()))
> +		return -ENODEV;

[ ... ]

> @@ -9393,8 +9454,10 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
>  			if (btf_mod->btf)
>  				btf_free_id(btf_mod->btf);
>  			list_del(&btf_mod->list);
> +			mutex_unlock(&btf_module_mutex);
> +			/* off the list, nobody else can find it now */
>  			btf_module_free(btf_mod);
> -			break;
> +			goto out;
>  		}
>  		mutex_unlock(&btf_module_mutex);
>  		break;

Can btf_module_notify() deadlock here during MODULE_STATE_GOING?

The MODULE_STATE_GOING notifier runs from
blocking_notifier_call_chain(&module_notify_list, ...), which holds
module_notify_list.rwsem for read. This patch moves btf_module_free()
outside btf_module_mutex, and btf_module_free() calls
sysfs_remove_bin_file(), which waits uninterruptibly in kernfs_drain()
until every active reader has returned.

This patch made that reader btf_module_sysfs_read_deferred(), which
starts with:

    if (IS_ERR_OR_NULL(bpf_get_btf_vmlinux()))
            return -ENODEV;

If the vmlinux BTF has not been loaded yet, bpf_get_btf_vmlinux() calls
btf_vmlinux_data(&size, true), which calls request_module("btf_vmlinux").
That waits (UMH_WAIT_PROC | UMH_KILLABLE) for modprobe.

modprobe's load_module() calls prepare_coming_module(), which calls
blocking_notifier_call_chain_robust(&module_notify_list,
MODULE_STATE_COMING, ...) and needs down_read() on the same rwsem.

If a writer is queued on that rwsem, new readers block once the handoff
bit is set. Writers come from register_module_notifier() or
unregister_module_notifier(), called for example from kunit_init() when
kunit is modular, or from a module exit that unregisters its notifier.

The resulting cycle is:

  rmmod X (GOING, holds rwsem for read)
    -> kernfs_drain waits for the reader of /sys/kernel/btf/X

  reader of X
    -> request_module("btf_vmlinux") waits for modprobe

  modprobe btf_vmlinux
    -> down_read(module_notify_list.rwsem) waits for the queued writer

  writer
    -> down_write waits for rmmod X to release its read side

Everything stays in D state until the reader gets SIGKILL. Meanwhile
every other module load or unload also stalls on the notifier chain.

Before this patch, a deferred .BTF.base module had no sysfs file until
btf_parse_deferred_modules() had relocated it. No reader could block, so
the drain in GOING could not wait on a module load.

The new code conflicts with the rationale stated in btf_module_defer():
"The vmlinux BTF ... must not be loaded from the module notifier (that
would nest a module load into a module load)". The GOING notifier now
waits on exactly that nested load, through the reader.

Possible fixes: do not trigger the vmlinux BTF load from this reader
(serve -ENODEV/-EAGAIN until the BTF is published, or only wait when
btf_vmlinux is already set); or remove the sysfs file in a way that does
not wait under the notifier chain (for example,
kernfs_break_active_protection() in the reader around the
request_module()/wait, with the btf_mod lifetime guarded separately).

Also, this patch changes the lock ordering that affects a comment
elsewhere in the file. The comment for btf_vmlinux_regs_mutex justifies
the separate mutex by saying that purge_cand_cache() takes
cand_cache_mutex under btf_module_mutex. That was only true through
btf_module_free() on the GOING path, which this patch moves outside
btf_module_mutex (as shown above).

The remaining purge_cand_cache() callers, in btf_module_notify() COMING
and at the end of btf_parse_deferred_modules(), already run without
btf_module_mutex. No btf_module_mutex -> cand_cache_mutex chain is left,
so the stated reason for the ordering rule is gone.

The ordering that matters now is kernfs active reference ->
btf_vmlinux_lock/btf_module_mutex, which comes from the new deferred
sysfs reader. Should that comment be updated to reflect the current lock
ordering rationale?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36198628965

Re: [PATCH bpf-next v3 5/9] bpf: defer vmlinux kfunc and struct_ops registrations

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1733 bytes --]

> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index 1dd7f9650ae8..207954b5754a 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c

[ ... ]

> +#ifdef BTF_MODULE_NOTIFIER
> +/*
> + * The queue has its own lock: it is drained under btf_vmlinux_lock, and
> + * btf_module_mutex must not nest inside that (purge_cand_cache() takes
> + * cand_cache_mutex under btf_module_mutex, and CO-RE fetches the vmlinux
> + * BTF under cand_cache_mutex).
> + */
> +static DEFINE_MUTEX(btf_vmlinux_regs_mutex);

Does the lock ordering explanation match the actual code?  Looking at
the earlier patch 806240f10744 ("bpf: fetch the vmlinux BTF where
kernel types enter a program"), both CO-RE callers now call
bpf_get_btf_vmlinux() before taking cand_cache_mutex:

kernel/bpf/btf.c:bpf_core_apply() {
	/* candidates are kernel types: load the vmlinux BTF, outside the mutex */
	if (IS_ERR_OR_NULL(bpf_get_btf_vmlinux())) {
	...
	mutex_lock(&cand_cache_mutex);

and bpf_core_find_cands() uses only bpf_peek_btf_vmlinux() under the
mutex, which takes no lock.  Its comment reads "Callers fetch the
vmlinux BTF before taking cand_cache_mutex."

So the btf_vmlinux_lock -> btf_module_mutex -> cand_cache_mutex ->
btf_vmlinux_lock cycle described here does not exist in this tree.

The separate lock is fine. Could the comment and commit message explain
what actually prevents btf_module_mutex from nesting under
btf_vmlinux_lock, or point to what the next patch adds that would
create that constraint?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36198628965

Re: [PATCH bpf-next v3 9/9] kbuild, bpf: allow building the vmlinux BTF as a module

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 22382 bytes --]

> Make CONFIG_DEBUG_INFO_BTF a tristate.  With =m the vmlinux BTF is not
> part of the kernel image: it is carried by a new module, btf_vmlinux, and
> loaded the first time something needs it.  Nothing that works with =y
> stops working; the 5.4 MiB of read-only data (distribution config) is
> simply not there on systems where nothing uses it.

The statement "Nothing that works with =y stops working" appears to have
exceptions. The patch itself adds this to kernel/bpf/preload/Kconfig:

>     depends on DEBUG_INFO_BTF!=m

This makes BPF_PRELOAD no longer selectable with =m, and a later paragraph
acknowledges this ("CONFIG_BPF_PRELOAD is not selectable with =m").

Additionally, BPF programs that need kernel types before the root file
system is mounted work with =y but fail with =m unless btf_vmlinux.ko is
in the initramfs, which the changelog mentions later.

Could the opening sentence be reworded to acknowledge these exceptions,
for example: "Apart from BPF_PRELOAD and early-boot users without
btf_vmlinux.ko in the initramfs, nothing that works with =y stops
working"?

> diff --git a/Documentation/bpf/btf.rst b/Documentation/bpf/btf.rst
> index 29de1222c3e7..1bd0d35cdb05 100644
> --- a/Documentation/bpf/btf.rst
> +++ b/Documentation/bpf/btf.rst
> @@ -1276,6 +1276,41 @@ format.::
>              .long   58
>              .long   8206                    # Line 8 Col 14
>
> +6.1 Kernel BTF
> +--------------
> +
> +With CONFIG_DEBUG_INFO_BTF=y the BTF of the kernel is generated at link time
> +from its DWARF and placed in the .BTF section of vmlinux, which is read-only
> +data of the kernel image. It is available as /sys/kernel/btf/vmlinux and, if
> +CONFIG_DEBUG_INFO_BTF_MODULES is set, module BTF is generated as split BTF
> +against it and available as /sys/kernel/btf/<module>.
> +
> +With CONFIG_DEBUG_INFO_BTF=m the same BTF is generated, but it is not part of
> +the kernel image or of the vmlinux ELF file (vmlinux.unstripped in the build
> +tree keeps it, for module BTF generation). It is delivered by the
> +btf_vmlinux module, which the kernel loads on demand the first time the BTF is
> +needed: when /sys/kernel/btf/vmlinux is read or mmap()ed, when kernel BTF
> +objects are enumerated (BPF_BTF_GET_NEXT_ID), or when a BPF program needs
> +kernel type information (an attach_btf_id, a kfunc call, a ksym, a map pointer
> +or a helper that takes or returns a kernel BTF pointer). Until then no memory
> +is used for it, and afterwards nothing differs from =y. In particular:

The new section lists when the btf_vmlinux module is loaded, and mentions
that bpf_snprintf_btf() and bpf_seq_printf_btf() handle the missing-BTF
case because they run in program context. The ftrace argument printer is
another caller that can reach the loader, and it can run with interrupts
disabled.

With CONFIG_DEBUG_INFO_BTF=m, CONFIG_FUNCTION_TRACE_ARGS is still enabled
because it depends on PROBE_EVENTS_BTF_ARGS, which is a bool that treats
the m dependency as y. The func-args and funcgraph-args trace options
print arguments through this call chain:

kernel/trace/trace_output.c:print_function_args():
  t = btf_find_func_proto(name, &btf);
kernel/trace/trace_btf.c:btf_find_func_proto():
  id = bpf_find_btf_id(func_name, BTF_KIND_FUNC, btf_p);
kernel/bpf/btf.c:bpf_find_btf_id():
  btf = bpf_get_btf_vmlinux();
kernel/bpf/verifier.c:bpf_get_btf_vmlinux():
  if (!btf_vmlinux_data(&size, true))
kernel/bpf/btf.c:btf_vmlinux_data():
  request_module("btf_vmlinux");

This is also reached from ftrace_dump() in the oops/panic notifier and
from sysrq-z:

kernel/trace/trace.c:ftrace_dump_one():
  local_irq_save(flags);
  ...
  ret = print_trace_line(&iter);  -> ... -> print_function_args()

request_module() sleeps: it calls down_timeout() and
call_modprobe(..., UMH_WAIT_PROC), which waits for a usermode helper.
If the BTF has not been parsed yet, bpf_get_btf_vmlinux() takes
mutex_lock(&btf_vmlinux_lock) with GFP_KERNEL allocations. Either way,
this path sleeps with interrupts disabled. In panic, the other CPUs have
been stopped, so the modprobe wait may hang and block the dump or any
reboot after it.

With =y, late_initcall kfunc registrations parse the vmlinux BTF during
boot, so this path only does the lockless acquire load. With =m,
btf_defer_reg() queues those registrations and does not load the BTF,
so it can stay NULL until the trace printer runs.

Can this be triggered with CONFIG_DEBUG_INFO_BTF=m, func-args or
funcgraph-args set, ftrace_dump_on_oops (or sysrq-z), and no BPF user
of the vmlinux BTF since boot?

One option is to make print_function_args() use bpf_peek_btf_vmlinux(),
as bpf_btf_printf_prepare() already does. The doc trigger list also
leaves out other non-program loaders: reading the trace file with
func-args, kprobe/fprobe events with BTF arguments, struct_ops map
creation, bpffs delegate_* options and netfilter ctx access.

> +  * /sys/kernel/btf/vmlinux exists from boot with its final size.
> +  * Modules loaded before the vmlinux BTF are exposed in /sys/kernel/btf right
> +    away, their BTF is parsed and gets a BTF id once the vmlinux BTF is
> +    loaded, together with their kfunc and struct_ops registrations.
> +  * kfunc, dtor kfunc and struct_ops registrations of the kernel itself are
> +    applied before the BTF becomes visible.
> +  * The kernel only accepts the BTF it was built with: the size and SHA-256 of
> +    the BTF are linked into the kernel and checked against the module.
> +  * Once loaded the BTF stays; the module cannot be unloaded.
> +
> +If the module is not available (not installed, or the root file system is not
> +mounted yet), the kernel behaves as one built without BTF and retries next
> +time. CONFIG_BPF_PRELOAD is not available with =m: its iterators attach through
> +the vmlinux BTF, so mounting bpffs would load it. bpf_snprintf_btf() and bpf_seq_printf_btf() only use the BTF if it has
> +already been parsed, as they run in program context.

[ ... ]

> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> index 134b15a44625..418025c5e657 100644
> --- a/lib/Kconfig.debug
> +++ b/lib/Kconfig.debug
> @@ -396,7 +396,7 @@ config DEBUG_INFO_SPLIT
>  	  Incompatible with older versions of ccache.
>
>  config DEBUG_INFO_BTF
> -	bool "Generate BTF type information"
> +	tristate "Generate BTF type information"
>  	depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
>  	depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
>  	depends on BPF_SYSCALL

Does making DEBUG_INFO_BTF selectable as =m keep PROBE_EVENTS_BTF_ARGS
and FUNCTION_TRACE_ARGS enabled? Since bool dependents treat m as y, the
tracing output path would call the vmlinux BTF lookup, and under =m that
lookup can sleep.

The chain from the function_graph printers goes:
print_fn_trace() -> print_function_args() -> btf_find_func_proto() ->
bpf_find_btf_id() -> bpf_get_btf_vmlinux() ->
btf_vmlinux_data(&size, true) -> request_module("btf_vmlinux").

request_module() calls down_timeout(), kmalloc(GFP_KERNEL) in
call_modprobe(), and waits for modprobe with UMH_WAIT_PROC. These
printers run from ftrace_dump_one(), which does local_irq_save(flags)
before looping on print_trace_line(). ftrace_dump_one() is reached from
ftrace_dump_on_oops/panic, sysrq-z, <func>:dump function probes, kdb
ftdump and RCU stall dumps.

If the func-args or funcgraph-args option is on and nothing has loaded
the BTF yet, the dump hits 'sleeping function called from invalid
context' and schedules with IRQs disabled. In the panic/oops path the
other CPUs are stopped and usermode helpers cannot run, so the dump can
hang.

With =y the BTF is parsed at boot and bpf_get_btf_vmlinux() returns from
its fast path, so this path could not sleep before this change. Earlier
in the series, a commit added bpf_peek_btf_vmlinux() for
bpf_snprintf_btf()/bpf_seq_printf_btf() because they 'cannot afford a
fetch that may sleep', but the tracing output path was not converted.

Should btf_find_func_proto() use a non-loading lookup
(bpf_peek_btf_vmlinux()) or skip the lookup when !preemptible(), or
should PROBE_EVENTS_BTF_ARGS/FUNCTION_TRACE_ARGS depend on
DEBUG_INFO_BTF=y?

> @@ -408,6 +408,26 @@ config DEBUG_INFO_BTF
>  	  Turning this on requires pahole v1.22 or later, which will convert
>  	  DWARF type info into equivalent deduplicated BTF type info.
>
> +	  If built as a module (=m), the vmlinux BTF is not part of the
> +	  kernel image.  It is carried by the btf_vmlinux module, which is
> +	  loaded on demand the first time the BTF is needed: when a BPF
> +	  program requires kernel type information, or when
> +	  /sys/kernel/btf/vmlinux is opened.  Until then, no memory is
> +	  spent on it.  The vmlinux ELF file does not carry the BTF
> +	  either; module BTF is generated against vmlinux.unstripped, and
> +	  tools that read the BTF from a file can use that or
> +	  /sys/kernel/btf/vmlinux.

The help text says the btf_vmlinux module is loaded when
/sys/kernel/btf/vmlinux is opened. Does opening the file actually load
the module?

Looking at kernel/bpf/sysfs_btf.c, the file is a bin_attribute, and
struct bin_attribute has no open callback. The BTF is loaded only from
btf_sysfs_vmlinux_read() and btf_sysfs_vmlinux_mmap(), and the comment
there says 'the first read() or mmap() loads the BTF'. stat() and open()
without a read leave the module unloaded, which the commit message's
btf_sysfs selftest note about stat() relies on.

Should this say 'first read or mapped' instead of 'opened'? The commit
message has the same wording ('opening /sys/kernel/btf/vmlinux ... each
load it').

> +	  Module BTF (DEBUG_INFO_BTF_MODULES) is kept when a module loads,
> +	  as with =y, and registered once the vmlinux BTF is available; the
> +	  saving is the vmlinux BTF only.
> +
> +	  If BPF programs that use kernel types run before the root file
> +	  system is mounted, put btf_vmlinux.ko into the initramfs: until
> +	  the module can be loaded, such programs fail as on a kernel
> +	  without BTF.  Not compatible with BPF_PRELOAD, whose iterators
> +	  would load the BTF at every bpffs mount.

Once DEBUG_INFO_BTF can be =m, does 'make localmodconfig' silently turn
it off?

scripts/kconfig/streamline_config.pl maps loaded modules to configs only
through obj-$(CONFIG_X) lines. Any =m config it cannot tie to a loaded
module is written out as '# CONFIG_X is not set'. The only rule for the
carrier module is kernel/bpf/Makefile 'ifeq ($(CONFIG_DEBUG_INFO_BTF),m)'
/ 'obj-m += btf_vmlinux.o', which the script cannot map, and btf_vmlinux
is usually not loaded when lsmod is captured anyway.

So a distribution config with CONFIG_DEBUG_INFO_BTF=m loses
DEBUG_INFO_BTF under localmodconfig. olddefconfig then also drops
DEBUG_INFO_BTF_MODULES, SCHED_CLASS_EXT, NET_SCH_BPF, IO_URING_BPF_OPS,
BINFMT_MISC_BPF and PROBE_EVENTS_BTF_ARGS. With =y these options were
kept, because the script leaves built-in options alone. The only
workaround is LMC_KEEP.

Could the Makefile use an obj-$(CONFIG_DEBUG_INFO_BTF)-style rule that
streamline_config can map, or could streamline_config preserve this
symbol?

[ ... ]

> diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
> index 01a37ec872b9..6a0958cf0e6f 100644
> --- a/scripts/Makefile.modfinal
> +++ b/scripts/Makefile.modfinal
> @@ -38,20 +38,34 @@ quiet_cmd_ld_ko_o = LD [M]  $@
>  		$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE)		\
>  		-T $(objtree)/scripts/module.lds -o $@ $(filter %.o, $^)
>
> +# The ELF file with the vmlinux BTF: with CONFIG_DEBUG_INFO_BTF=m the BTF is
> +# stripped from vmlinux (scripts/Makefile.vmlinux), vmlinux.unstripped keeps it.
> +btf-vmlinux := $(objtree)/vmlinux$(if $(filter m,$(CONFIG_DEBUG_INFO_BTF)),.unstripped)
> +
>  quiet_cmd_btf_ko = BTF [M] $@
>        cmd_btf_ko = 							\
> -	if [ ! -f $(objtree)/vmlinux ]; then				\
> +	if [ ! -f $(btf-vmlinux) ]; then				\
>  		printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
>  	else	\
> -		$(CONFIG_SHELL) $(srctree)/scripts/gen-btf.sh --btf_base $(objtree)/vmlinux $@; \
> +		$(CONFIG_SHELL) $(srctree)/scripts/gen-btf.sh --btf_base $(btf-vmlinux) $@; \
>  	fi;

With CONFIG_DEBUG_INFO_BTF=m, external modules (M=) now only get BTF if
vmlinux.unstripped exists. The in-tree packaging still ships only vmlinux
for this purpose.

scripts/package/PKGBUILD's _package-debug does 'install -Dt
"${debugdir}" -m644 vmlinux' and 'ln -sr "${debugdir}/vmlinux"
"${builddir}/vmlinux"' so that modules built against
/usr/lib/modules/$KERNELRELEASE/build get split BTF. With =y that works.
With =m, Makefile.vmlinux strips .BTF from that vmlinux, and cmd_btf_ko
tests for build/vmlinux.unstripped, which nothing installs.

So every DKMS/out-of-tree module built against a pacman-pkg kernel with
=m (and DEBUG_INFO_BTF_MODULES, which defaults to y) is built without
.BTF, and the build still succeeds. The only sign is the stderr line
"Skipping BTF generation for foo.ko due to unavailability of vmlinux",
which is misleading because build/vmlinux is present; the missing file is
vmlinux.unstripped.

At runtime those modules have no BTF: fentry/fexit and BTF-typed tracing
of their functions are unavailable, and register_btf_kfunc_id_set() from
such a module falls into check_btf_kconfigs() ("missing module BTF,
cannot register kfunc"). This contradicts the changelog's "Nothing that
works with =y stops working" for the external-module flow the tree itself
packages.

Should scripts/package/PKGBUILD (and any other shipped build-tree layout)
provide vmlinux.unstripped when CONFIG_DEBUG_INFO_BTF=m, or should
cmd_btf_ko use another base source? At minimum, the error message should
name $(btf-vmlinux) instead of saying "vmlinux".

>
> -# Re-generate module BTFs if either module's .ko or vmlinux changed
> -%.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(objtree)/vmlinux) FORCE
> -	+$(call if_changed,ld_ko_o)
> +# Modules that get a .BTF section: all of them with CONFIG_DEBUG_INFO_BTF_MODULES,
> +# otherwise only the vmlinux BTF carrier module with CONFIG_DEBUG_INFO_BTF=m.
>  ifdef CONFIG_DEBUG_INFO_BTF_MODULES
> -	+$(if $(newer-prereqs),$(call cmd,btf_ko))
> +btf-modules := $(modules:%.o=%.ko)
> +else ifeq ($(CONFIG_DEBUG_INFO_BTF),m)
> +btf-modules := $(filter %/btf_vmlinux.ko,$(modules:%.o=%.ko))
> +# Only the carrier depends on vmlinux, not every module
> +ifdef KBUILD_BUILTIN
> +$(btf-modules): $(btf-vmlinux)
> +endif
>  endif
> +
> +# Re-generate module BTFs if either module's .ko or vmlinux changed
> +%.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(btf-vmlinux)) FORCE
> +	+$(call if_changed,ld_ko_o)
> +	+$(if $(and $(filter $@,$(btf-modules)),$(newer-prereqs)),$(call cmd,btf_ko))
>  	+$(call cmd,check_tracepoint)
>
>  targets += $(modules:%.o=%.ko) $(modules:%.o=%.mod.o) .module-common.o

[ ... ]

> diff --git a/scripts/Makefile.vmlinux b/scripts/Makefile.vmlinux
> index fcae1e432d9a..557db1ee1f3b 100644
> --- a/scripts/Makefile.vmlinux
> +++ b/scripts/Makefile.vmlinux
> @@ -86,6 +86,11 @@ remove-section-$(CONFIG_ARCH_VMLINUX_NEEDS_RELOCS) += '.rel*' '!.rel*.dyn'
>  # for compatibility with binutils < 2.32
>  # https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=c12d9fa2afe7abcbe407a00e15719e1a1350c2a7
>  remove-section-$(CONFIG_ARCH_VMLINUX_NEEDS_RELOCS) += '.rel.*'
> +# With CONFIG_DEBUG_INFO_BTF=m the btf_vmlinux module carries the vmlinux BTF;
> +# only vmlinux.unstripped keeps it, for module BTF generation.
> +ifeq ($(CONFIG_DEBUG_INFO_BTF),m)
> +remove-section-y += .BTF
> +endif

With CONFIG_DEBUG_INFO_BTF=m this strips .BTF from the vmlinux ELF file,
but the in-tree consumers that read the vmlinux BTF from that file were
not updated. Only Makefile.modfinal was switched to vmlinux.unstripped.

The tools pick their BTF source like this (tools/bpf/bpftool/Makefile):

  VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux)                   \
                       $(if $(KBUILD_OUTPUT),$(KBUILD_OUTPUT)/vmlinux) \
                       ../../../vmlinux                          \
                       /sys/kernel/btf/vmlinux                   \
                       /boot/vmlinux-$(shell uname -r)
  VMLINUX_BTF ?= $(abspath $(firstword $(wildcard $(VMLINUX_BTF_PATHS))))
  ...
  $(OUTPUT)vmlinux.h: $(VMLINUX_BTF) $(BPFTOOL_BOOTSTRAP)
      $(QUIET_GEN)$(BPFTOOL_BOOTSTRAP) btf dump file $< format c > $@

$(wildcard) only checks that a file exists. In a tree where an =m kernel
was built, it selects the build-tree vmlinux, which now has no .BTF
section, and never reaches /sys/kernel/btf/vmlinux. btf_parse_elf() in
tools/lib/bpf/btf.c then fails:

  if (!secs.btf_data) {
      pr_warn("failed to find '%s' ELF section in %s\n", BTF_ELF_SEC, path);
      err = -ENODATA;

do_dump() in bpftool reports "failed to load BTF from ...: No data
available" and the vmlinux.h rule fails. The same
VMLINUX_BTF_PATHS/firstword/wildcard logic followed by 'bpftool btf dump
file $(VMLINUX_BTF) format c' appears in
tools/testing/selftests/bpf/Makefile, tools/sched_ext/Makefile,
tools/testing/selftests/sched_ext/Makefile, samples/bpf/Makefile,
samples/hid/Makefile, drivers/hid/bpf/progs/Makefile and
tools/testing/selftests/hid/Makefile.

So after an =m kernel build, the default builds of bpftool (with
skeletons), the BPF selftests and the sched_ext tools fail unless
VMLINUX_BTF is overridden. With =y all of these work. This contradicts
the changelog's "Nothing that works with =y stops working". The Kconfig
help says tools "can use that [vmlinux.unstripped] or
/sys/kernel/btf/vmlinux", but none of the in-tree tools do so by default.
No commit in the series touches tools/.

In-tree packaging has a related gap. scripts/package/PKGBUILD
_package-debug installs vmlinux and links it as build/vmlinux. With =y,
that is what Makefile.modfinal needs to generate BTF for external
modules. With =m, Makefile.modfinal looks for vmlinux.unstripped, which
is not packaged, and the packaged vmlinux no longer has .BTF. External
module BTF generation against those packages is therefore skipped. The
builddeb and kernel.spec debug vmlinux copies also no longer carry BTF.

Should VMLINUX_BTF_PATHS list vmlinux.unstripped before vmlinux, or
should the vmlinux ELF file keep the non-loadable .BTF?

>
>  remove-symbols := -w --strip-unneeded-symbol='__mod_device_table__*'
>
> @@ -93,7 +98,7 @@ quiet_cmd_strip_relocs = STRIPREL
>        cmd_strip_relocs = $(OBJCOPY) $(foreach x,$(remove-section-y),--remove-section=$(x))	\
>  			      --set-section-flags .BTF=noload		\
>  			      $(remove-symbols)				\
> -			      -o $@ $<
> +			      -o $@ $< $(if $(filter .BTF,$(remove-section-y))&& [ -n "$(remove-section-y)" ])
>
>  vmlinux: vmlinux.unstripped $(and $(CONFIG_DEBUG_INFO_BTF),$(RESOLVE_BTFIDS)) FORCE
>  	$(call if_changed,strip_relocs)

[ ... ]

> diff --git a/scripts/gen-btf.sh b/scripts/gen-btf.sh
> index 8ca96eb10a69..d808d0006d32 100755
> --- a/scripts/gen-btf.sh
> +++ b/scripts/gen-btf.sh

[ ... ]

> @@ -191,6 +251,35 @@ if [ -n "${BTF_BASE}" ]; then
>  	BTFGEN_MODE="module"
>  fi
>
> +if [ -n "${PLACEHOLDER}" ]; then
> +	btf_data=${ELF_FILE}.btf.o
> +	echo "" | ${CC} ${CLANG_FLAGS} ${KBUILD_CPPFLAGS} ${KBUILD_CFLAGS} -fno-lto -c -x c -o ${btf_data} -
> +	dd if=/dev/zero of=${ELF_FILE}.BTF.meta bs=36 count=1 2>/dev/null
> +	${OBJCOPY} --add-section .BTF.meta=${ELF_FILE}.BTF.meta \
> +		--set-section-flags .BTF.meta=alloc,readonly ${btf_data}
> +	${OBJCOPY} --only-section=.BTF.meta --strip-all ${btf_data}
> +	exit 0
> +fi
> +
> +# CONFIG_DEBUG_INFO_BTF=m: the btf_vmlinux module carries the vmlinux BTF
> +# itself.  Its own types are of no interest, so instead of generating split
> +# BTF for it, copy the (non-loadable) .BTF section of --btf_base
> +# (vmlinux.unstripped) into the module.
> +# The kernel recognizes the module by name and treats its .BTF as base BTF.
> +case "${BTFGEN_MODE}:${ELF_FILE}" in
> +module:*/btf_vmlinux.ko)
> +	if is_module CONFIG_DEBUG_INFO_BTF; then
> +		# -O binary only emits allocatable sections; make .BTF one for
> +		# the extraction.  ${BTF_BASE} itself is not modified.
> +		${OBJCOPY} -O binary --only-section=.BTF			\
> +			--set-section-flags .BTF=alloc,load,readonly	\
> +			"${BTF_BASE}" "${ELF_FILE}.BTF"
> +		${OBJCOPY} --add-section .BTF="${ELF_FILE}.BTF" "${ELF_FILE}"
> +		exit 0
> +	fi
> +	;;
> +esac

The carrier module is recognized only when ELF_FILE contains a '/' before
btf_vmlinux.ko. That holds for normal and O= builds (ELF_FILE is
kernel/bpf/btf_vmlinux.ko). Does it hold for 'make M=kernel/bpf'?

Since the M= working-directory change, the top Makefile changes into the
module directory. scripts/Makefile.build then skips the $(obj)/ prefix
when obj is '.', so modules.order lists 'btf_vmlinux.o' and
Makefile.modfinal passes $@ = 'btf_vmlinux.ko'. 'module:btf_vmlinux.ko'
does not match 'module:*/btf_vmlinux.ko', so gen-btf.sh goes on to
gen_btf_data/embed_btf_data.

With CONFIG_DEBUG_INFO_BTF_MODULES=y the carrier gets its own split BTF
(plus a distilled .BTF.base, because KBUILD_EXTMOD adds --distill_base)
instead of a copy of the vmlinux BTF. Without BTF_MODULES,
Makefile.modfinal's matching '$(filter %/btf_vmlinux.ko,...)' also
misses, so the carrier gets no .BTF at all.

In both cases btf_vmlinux_module_coming() fails its check 'if
(mod->btf_data_size != btf_vmlinux_meta.size) { pr_err(...); return
-EINVAL; }', btf_module_notify() returns
notifier_from_errno(-EINVAL), and prepare_coming_module() makes the
module load fail.

After 'make M=kernel/bpf modules_install' the broken carrier goes to
$(MODLIB)/updates (Makefile.modinst: 'INSTALL_MOD_DIR ?= updates'), which
depmod searches ahead of kernel/ by default. request_module("btf_vmlinux")
from btf_vmlinux_data() then keeps loading the broken copy, and the
vmlinux BTF stays unavailable until that file is removed.

Could the match here (and the same in the Makefile.modfinal filter) cover
the bare name as well, e.g. 'module:btf_vmlinux.ko|module:*/btf_vmlinux.ko)'?

>
>  gen_btf_data
>
>  case "${BTFGEN_MODE}" in

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36198628965