Re: [PATCH bpf-next v4 00/12] bpf: make the vmlinux BTF an on-demand loadable module (CONFIG_DEBUG_INFO_BTF=m) to save ~5.4 MB memory

[Alan Maguire <alan.maguire@oracle.com>]

>> A question I had: is it really worth 2k of complicated kernel code to
>> *maybe sometimes* save <10Mb of memory?
> 
> There are many reasons this is worth it.  The main ones:
> 
> 1. This did not start from a number we picked, but from a real use
>    case: users who run large fleets of small instances, 1 GiB of
>    memory or less, with their workloads sized to fit.  We are not able
>    to disclose more detail, but for them 5.4 MB on every instance is
>    real money, and it can be exactly what pushes a workload over its
>    memory budget and onto the next instance size.  That is why we took
>    this on, even though we knew it would not be a small change.
> 
> 2. Loading code and data only when a system needs them is what kernel
>    modules exist for.  And most of them take well under ~1 MB once
>    loaded (nf_conntrack, overlay, vfat); even big ones like ext4, kvm
>    and btrfs stay around 1-2.5 MB.  The vmlinux BTF is 5.4 MB.
> 
> 3. We are not alone in trying to keep BTF out of memory until it is
>    needed.  The inline BTF work [1] plans to deliver its data, which
>    is even larger, through a module too.  The .BTF.link record and the
>    resolve_btfids option that serve both are already part of this
>    series (patch 10), so this is not machinery for the vmlinux BTF
>    alone.

The other consideration that embedded folks have raised is disk footprint;
apparently by moving vmlinux .BTF to a module that is a win for them due
to the way embedded systems are partitioned (vmlinux image on a different
partition from modules). And for them a lower memory footprint is a win too.
So from my perspective it is worth doing, especially if the only other option is 
to switch BTF off and lose most BPF functionality. Between small VMs
and embedded systems that could amount to a lot of systems.

However it may make sense to approach via a multi-phase delivery. The inline 
stuff is pretty close, so if I can land that with the resolve_btfids support
for --btf_link that reduces the footprint of this work somewhat. Then perhaps
it might make sense to split into a prerequisite series that does the prep work
for module-delivered BTF (roughly patches 1-5), avoiding issues around module
load by reorganizing when we access vmlinux BTF. Then the remainder - stuff
which is tied more closely to module vmlinux BTF delivery - would be a smaller 
series (I'm planning on sending out the inline kbuild stuff later today all
going well). Others may have different suggestions, but that may make the work
easier to land; the multi-series approach definitely helped with the inline work
FWIW.

Alan