Re: [PATCH] kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG

[Alexander Lobakin <alobakin@pm.me>]

From: Sami Tolvanen <samitolvanen@google.com>
Date: Fri, 16 Jul 2021 16:18:42 -0700

> Hi Al,
>
> On Fri, Jul 16, 2021 at 4:00 PM Alexander Lobakin <alobakin@pm.me> wrote:
> >
> > From:   Sami Tolvanen <samitolvanen@google.com>
> > Date:   Fri, 16 Jul 2021 13:45:45 -0700
> >
> > > With CONFIG_LTO_CLANG, we currently link modules into native
> > > code just before modpost, which means with TRIM_UNUSED_KSYMS
> > > enabled, we still look at the LLVM bitcode in the .o files when
> > > generating the list of used symbols. As the bitcode doesn't
> > > yet have calls to compiler intrinsics and llvm-nm doesn't see
> > > function references that only exist in function-level inline
> > > assembly, we currently need a whitelist for TRIM_UNUSED_KSYMS to
> > > work with LTO.
> > >
> > > This change moves module LTO linking to happen earlier, and
> > > thus avoids the issue with LLVM bitcode and TRIM_UNUSED_KSYMS
> > > entirely, allowing us to also drop the whitelist from
> > > gen_autoksyms.sh.
> > >
> > > Link: https://github.com/ClangBuiltLinux/linux/issues/1369
> > > Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
> > > ---
> > >  scripts/Makefile.build    | 25 ++++++++++++++++++++++++-
> > >  scripts/Makefile.lib      |  7 +++++++
> > >  scripts/Makefile.modfinal | 21 ++-------------------
> > >  scripts/Makefile.modpost  | 22 +++-------------------
> > >  scripts/gen_autoksyms.sh  | 12 ------------
> > >  5 files changed, 36 insertions(+), 51 deletions(-)
> > >
> > > diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> > > index 10b2f2380d6f..80e0fa810870 100644
> > > --- a/scripts/Makefile.build
> > > +++ b/scripts/Makefile.build
> > > @@ -202,6 +202,7 @@ sub_cmd_record_mcount =                                   \
> > >       if [ $(@) != "scripts/mod/empty.o" ]; then      \
> > >               $(objtree)/scripts/recordmcount $(RECORDMCOUNT_FLAGS) "$(@)";   \
> > >       fi;
> > > +/
> >
> > Seems like a leftover or a random typo here.
>
> Oops, indeed.
>
> > >  recordmcount_source := $(srctree)/scripts/recordmcount.c \
> > >                   $(srctree)/scripts/recordmcount.h
> > >  else
> > > @@ -271,12 +272,34 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) $$(objtool_dep) FORCE
> > >       $(call if_changed_rule,cc_o_c)
> > >       $(call cmd,force_checksrc)
> > >
> > > +ifdef CONFIG_LTO_CLANG
> > > +# Module .o files may contain LLVM bitcode, compile them into native code
> > > +# before ELF processing
> > > +quiet_cmd_cc_lto_link_modules = LTO [M] $@
> > > +cmd_cc_lto_link_modules =                                            \
> > > +     $(LD) $(ld_flags) -r -o $@                                      \
> > > +             $(shell [ -s $(@:.lto.o=.o.symversions) ] &&            \
> > > +                     echo -T $(@:.lto.o=.o.symversions))             \
> > > +             --whole-archive $^
> > > +
> > > +ifdef CONFIG_STACK_VALIDATION
> > > +# objtool was skipped for LLVM bitcode, run it now that we have compiled
> > > +# modules into native code
> > > +cmd_cc_lto_link_modules += ;                                         \
> > > +     $(objtree)/tools/objtool/objtool $(objtool_args)                \
> >
> > Now $(part-of-module) inside $(objtool_args) doesn't get expanded
> > properly, because previously it was being called on x.ko, and now
> > it's being called on x.lto.o. $(basename $@) returns "x.lto" instead
> > of "x", and Make doesn't find "x.lto.o" in $(real-objs-m).

To be more precise:

Previously, objtool was being called from Makefile.modfinal, where
part-of-module is hardcoded to 'y'. Now it's being called from
Makefile.build, and part-of-module is being calculated the same
way as for non-LTO build (when objtool is being called on each
object file rather than final composite object).
So, part-of-module and objtool invocation is now correct for modules
with single source file, but wrong for multi-object modules.

The simplest fix is to append '--module' to objtool args
unconditionally when we're trying to process .lto.o file.

> > An example of objtool args dump:
> >
> >   LTO [M] fs/btrfs/btrfs.lto.o
> > Call: ./tools/objtool/objtool orc generate --no-fp --no-unreachable --retpoline --uaccess fs/btrfs/btrfs.lto.o
> > fs/btrfs/btrfs.lto.o: warning: objtool: static_call: can't find static_call_key symbol: __SCK__might_resched
>
> Curiously I didn't see objtool warnings when building allmodconfig,
> but you're obviously correct here. I'll fix this in v2.
>
> > As can be seen, objtools command line no longer contains "--module".
> > And this warning about "can't find static_call_key" can appear only
> > in case of !module -> no -m|--module param was given.
> >
> > As a result, modules get broken and the kernel panics after loading
> > initramfs.
>
> Thanks for taking a look!
>
> Sami

Thanks for working on ClangLTO/CFI!
Al