Re: [PATCH bpf-next v4 00/12] bpf: make the vmlinux BTF an on-demand loadable module (CONFIG_DEBUG_INFO_BTF=m) to save ~5.4 MB memory

[Jay Wang <wanjay@amazon.com>]

>    - If I know I don't need BPF on the workload, and these 5.4 MB
>      bite me, I can just turn off BPF

They cannot, even when they know they do not need it.  As the cover
letter says, the cloud provider usually ships one kernel build to all of
its customers.  Customers install their own packages and workloads around
that kernel, but they do not build or customize the kernel.  Turning
BPF off is not a switch they have.

> What I find strange is for a user of this scale *to not know* whether
> BPF will be used in the workload or not, which is what lazy load may
> solve.

They do know.  They run predictable workloads on a large number of
small instances, and BPF is not part of them.  They spread the load by
resource usage, so at that scale every byte matters: more free memory
on each instance means fewer instances in total.

>    * Compress it.
[...]
>      Keep zstd-compressed blob in the kernel image and decompress and
>      parse it synchronously on first use.

Thanks for putting this forward.  This compression approach is simpler 
than loading a module.  I expect the overall code complexity to stay similar
to v3, since both need to find the first user and defer the module BTF parsing
until then, but it should be easier to merge into mainline because it avoids
the request_module() deadlocks.

Would you like to prepare it for a formal submission, or would you
like me to do so?  Either works for me.  Ideally it lands in time for
the next LTS kernel, so that we can adopt it there.

From what I can see, most of the work left is in that deferral.
Whoever needs the BTF first also parses the waiting module BTFs and
replays the queued registrations, struct_ops ->init() included, under
whatever locks it holds at that point, e.g. event_mutex or
bpf_verifier_lock, or inside the module notifier.  So the
cand_cache_mutex ABBA you found may not be the only deadlock.  Besides
that, only x86_64 was touched so far, so it needs extending to the
other architectures as well.