[PATCH] kbuild: rust: pass AutoFDO flags to Clang with inline helpers

[PATCH] kbuild: rust: pass AutoFDO flags to Clang with inline helpers

[Jenhao Chen]

When CONFIG_RUST_INLINE_HELPERS=y, cmd_rustc_o_rs and cmd_rustc_library
compile Rust sources to LLVM bitcode (.bc), link rust/helpers/helpers.bc
via llvm-link, and then invoke $(CC) to compile the combined bitcode
into the final ELF object (.o).

When commit a960c2cdb19e ("kbuild: rust: add AutoFDO support") was
tested with CONFIG_RUST_INLINE_HELPERS=y and CONFIG_LTO_CLANG_THIN=y,
cmd_rustc_o_rs skipped the post-llvm-link $(CC) step under
CONFIG_LTO_CLANG and handed the linked bitcode directly to ld.lld, where
the ThinLTO backend applied --lto-sample-profile,
-plugin-opt=-split-machine-functions, and LLD's default function
sections. Non-LTO builds (!CONFIG_LTO_CLANG), however, ran the
post-llvm-link $(CC) step without $(CFLAGS_AUTOFDO_CLANG) ever since
commit 3a2486cc1da5 ("kbuild: rust: provide an option to inline C
helpers into Rust").

Subsequently, commit 5febf432df1c ("kbuild: rust: keep Rust objects out
of Clang LTO with inline helpers") updated cmd_rustc_o_rs and
rust/Makefile to always invoke $(CC) on the linked bitcode even when
CONFIG_LTO_CLANG=y. Because that $(CC) invocation only passes
$(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS)) and does not include
$(CFLAGS_AUTOFDO_CLANG) (which is normally appended to _c_flags in
scripts/Makefile.lib), the final code generation step for Rust objects
is compiled without -ffunction-sections, -fsplit-machine-functions,
-fprofile-sample-use, or flow-sensitive discriminator flags whenever
CONFIG_RUST_INLINE_HELPERS=y.

As a result, Rust crates built with both CONFIG_RUST_INLINE_HELPERS=y
and CONFIG_AUTOFDO_CLANG=y emit all functions into a single monolithic
.text section (0 .text.<func> sections instead of 417 in
rust_binder_main.o and 424 in rust/kernel.o on arm64 with
CLANG_AUTOFDO_PROFILE set), preventing the linker from performing
AutoFDO hot/cold function clustering and discarding SamplePGO weights on
the inlined C helpers during final code generation.

Factor out _autofdo_c_flags in scripts/Makefile.lib (preserving per-file
and per-directory AUTOFDO_PROFILE opt-outs) and pass it to the
post-llvm-link $(CC) invocation in scripts/Makefile.build and
rust/Makefile.

Fixes: 3a2486cc1da5 ("kbuild: rust: provide an option to inline C helpers into Rust")
Fixes: 5febf432df1c ("kbuild: rust: keep Rust objects out of Clang LTO with inline helpers")
Signed-off-by: Jenhao Chen <jenhaochen@google.com>
---
 rust/Makefile          | 2 +-
 scripts/Makefile.build | 2 +-
 scripts/Makefile.lib   | 3 ++-
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/rust/Makefile b/rust/Makefile
index 0ac9ab28cc2b..f2d9b4411720 100644
--- a/rust/Makefile
+++ b/rust/Makefile
@@ -658,7 +658,7 @@ quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L
 	$(if $(link_helper),;$(LLVM_LINK) --internalize --suppress-warnings $(patsubst %.o,%.bc,$@) \
 		$(obj)/helpers/helpers$(if $(part-of-module),_module).bc -o $(patsubst %.o,%.m.bc,$@); \
 		$(CC) $(CLANG_FLAGS) $(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS)) \
-		$(CC_FLAGS_RUST_INLINE_HELPERS) -Wno-override-module -c $(patsubst %.o,%.m.bc,$@) -o $@ \
+		$(_autofdo_c_flags) $(CC_FLAGS_RUST_INLINE_HELPERS) -Wno-override-module -c $(patsubst %.o,%.m.bc,$@) -o $@ \
 		$(cmd_ld_single)) \
 	$(if $(rustc_objcopy),;$(OBJCOPY) $(rustc_objcopy) $@) \
 	$(cmd_objtool)
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 6127feb970c6..bd7bd05e9fe9 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -369,7 +369,7 @@ quiet_cmd_rustc_o_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
 	$(if $(CONFIG_RUST_INLINE_HELPERS),;$(LLVM_LINK) --internalize --suppress-warnings $(patsubst %.o,%.bc,$@) \
 		$(objtree)/rust/helpers/helpers$(if $(part-of-module),_module).bc -o $(patsubst %.o,%.m.bc,$@); \
 		$(CC) $(CLANG_FLAGS) $(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS)) \
-		$(CC_FLAGS_RUST_INLINE_HELPERS) -Wno-override-module -c $(patsubst %.o,%.m.bc,$@) -o $@ \
+		$(_autofdo_c_flags) $(CC_FLAGS_RUST_INLINE_HELPERS) -Wno-override-module -c $(patsubst %.o,%.m.bc,$@) -o $@ \
 		$(cmd_ld_single)) \
 	$(cmd_objtool)
 
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 19e96b30a122..e72efa5c6c05 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -120,9 +120,10 @@ endif
 # enable (depends on variables AUTOFDO_PROFILE_obj.o and AUTOFDO_PROFILE).
 #
 ifeq ($(CONFIG_AUTOFDO_CLANG),y)
-_c_flags += $(if $(patsubst n%,, \
+_autofdo_c_flags = $(if $(patsubst n%,, \
 	$(AUTOFDO_PROFILE_$(target-stem).o)$(AUTOFDO_PROFILE)$(is-kernel-object)), \
 	$(CFLAGS_AUTOFDO_CLANG))
+_c_flags += $(_autofdo_c_flags)
 _rust_flags += $(if $(patsubst n%,, \
 	$(AUTOFDO_PROFILE_$(target-stem).o)$(AUTOFDO_PROFILE)$(is-kernel-object)), \
 	$(RUSTFLAGS_AUTOFDO_CLANG))
-- 
2.56.0.317.g454f8302e1-goog

Re: [PATCH] kbuild: rust: pass AutoFDO flags to Clang with inline helpers

[Nathan Chancellor]

> When CONFIG_RUST_INLINE_HELPERS=y, cmd_rustc_o_rs and cmd_rustc_library
> compile Rust sources to LLVM bitcode (.bc), link rust/helpers/helpers.bc
> via llvm-link, and then invoke $(CC) to compile the combined bitcode
> into the final ELF object (.o).
> 
> When commit a960c2cdb19e ("kbuild: rust: add AutoFDO support") was
> tested with CONFIG_RUST_INLINE_HELPERS=y and CONFIG_LTO_CLANG_THIN=y,
> cmd_rustc_o_rs skipped the post-llvm-link $(CC) step under
> CONFIG_LTO_CLANG and handed the linked bitcode directly to ld.lld, where
> the ThinLTO backend applied --lto-sample-profile,
> -plugin-opt=-split-machine-functions, and LLD's default function
> sections. Non-LTO builds (!CONFIG_LTO_CLANG), however, ran the
> post-llvm-link $(CC) step without $(CFLAGS_AUTOFDO_CLANG) ever since
> commit 3a2486cc1da5 ("kbuild: rust: provide an option to inline C
> helpers into Rust").
> 
> Subsequently, commit 5febf432df1c ("kbuild: rust: keep Rust objects out
> of Clang LTO with inline helpers") updated cmd_rustc_o_rs and
> rust/Makefile to always invoke $(CC) on the linked bitcode even when
> CONFIG_LTO_CLANG=y. Because that $(CC) invocation only passes
> $(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS)) and does not include
> $(CFLAGS_AUTOFDO_CLANG) (which is normally appended to _c_flags in
> scripts/Makefile.lib), the final code generation step for Rust objects
> is compiled without -ffunction-sections, -fsplit-machine-functions,
> -fprofile-sample-use, or flow-sensitive discriminator flags whenever
> CONFIG_RUST_INLINE_HELPERS=y.
> 
> As a result, Rust crates built with both CONFIG_RUST_INLINE_HELPERS=y
> and CONFIG_AUTOFDO_CLANG=y emit all functions into a single monolithic
> .text section (0 .text.<func> sections instead of 417 in
> rust_binder_main.o and 424 in rust/kernel.o on arm64 with
> CLANG_AUTOFDO_PROFILE set), preventing the linker from performing
> AutoFDO hot/cold function clustering and discarding SamplePGO weights on
> the inlined C helpers during final code generation.
> 
> Factor out _autofdo_c_flags in scripts/Makefile.lib (preserving per-file
> and per-directory AUTOFDO_PROFILE opt-outs) and pass it to the
> post-llvm-link $(CC) invocation in scripts/Makefile.build and
> rust/Makefile.
> 
> Fixes: 3a2486cc1da5 ("kbuild: rust: provide an option to inline C helpers into Rust")
> Fixes: 5febf432df1c ("kbuild: rust: keep Rust objects out of Clang LTO with inline helpers")
> Signed-off-by: Jenhao Chen <jenhaochen@google.com>

I am assuming this will go via the Rust tree.

Reviewed-by: Nathan Chancellor <nathan@kernel.org>

-- 
Cheers,
Nathan

Re: [PATCH] kbuild: rust: pass AutoFDO flags to Clang with inline helpers

[Miguel Ojeda]

On Sat, Oct 31, 2026 at 4:08 AM Jenhao Chen <jenhaochen@google.com> wrote:
>
> Factor out _autofdo_c_flags in scripts/Makefile.lib (preserving per-file
> and per-directory AUTOFDO_PROFILE opt-outs) and pass it to the
> post-llvm-link $(CC) invocation in scripts/Makefile.build and
> rust/Makefile.

This sounds reasonable, but my LLM spotted that Clang will run
`AddDiscriminatorsPass`  again, i.e. on the optimized bitcode that
`rustc` produced. So the profile ends up indexed by Clang's renumbered
discriminators, and then the build that uses the profile will not find
a match for some code (depending on inlining decisions), i.e. that
part of the profile is silently dropped.

It proposes `-mllvm -no-discriminators` in
`CC_FLAGS_RUST_INLINE_HELPERS` so that Clang doesn't add them, i.e.
using the `rustc` ones.

Rong, Han: is that expected?

Cheers,
Miguel

Re: [PATCH] kbuild: rust: pass AutoFDO flags to Clang with inline helpers

[Jenhao Chen]

On Wed, Nov 11, 2026 at 7:48 PM Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> This sounds reasonable, but my LLM spotted that Clang will run
> `AddDiscriminatorsPass`  again, i.e. on the optimized bitcode that
> `rustc` produced. So the profile ends up indexed by Clang's renumbered
> discriminators, and then the build that uses the profile will not find
> a match for some code (depending on inlining decisions), i.e. that
> part of the profile is silently dropped.
>
> It proposes `-mllvm -no-discriminators` in
> `CC_FLAGS_RUST_INLINE_HELPERS` so that Clang doesn't add them, i.e.
> using the `rustc` ones.
>
> Rong, Han: is that expected?

Hi Miguel,

Nice catch -- I tested this on drivers/android/binder/rust_binder_main.o
(arm64, Clang 22 + rustc 1.91) and confirmed that the duplicate
AddDiscriminatorsPass run is real and that passing
`-mllvm -no-discriminators` to the post-llvm-link $(CC) step improves
profile matching in both stages:

1. Why the second AddDiscriminatorsPass renumbers discriminators:
   In Stage 1, rustc runs AddDiscriminatorsPass on pre-inlining IR and
   then runs its optimization/inlining pipeline before emitting .bc.
   When post-llvm-link $(CC) runs AddDiscriminatorsPass again on the
   already-inlined .m.bc, llvm/lib/Transforms/Utils/AddDiscriminators.cpp
   keys blocks by (filename, line) without checking inlinedAt and
   overwrites existing base discriminators via
   cloneWithBaseDiscriminator(). On rust_binder_main.o, that second pass
   increases non-zero DWARF .debug_line discriminator rows from 6,245 to
   9,313 (+3,068 rows, +49%).

   In the profile-use build, rustc's SampleProfileLoaderPass runs before
   rustc's inliner (against pre-inlining discriminators), while Stage 3
   sees a different post-inlining CFG once Stage 1 has applied the
   profile.

2. Measured round-trip impact (42 sampled functions in
   rust_binder_main.o):
   - Stage 1 (rustc -Zprofile-sample-use): annotated branch_weights
     increase from 454 to 490 (+7.9%) when the profile is generated and
     used with `-mllvm -no-discriminators` in Stage 3.
   - Stage 3 ($(CC) -fprofile-sample-use): SampleProfileLoader record
     coverage increases from 39.3% (2,832 / 7,205) to 54.0%
     (3,707 / 6,860, +14.7 percentage points), and sample coverage
     increases from 42.9% (757.8K / 1,767.6K) to 56.4%
     (989.6K / 1,753.4K, +13.6 percentage points; 969 -> 1,035
     branch_weights). Against a hardware ETM profile, Stage 3 matched
     function entry counts also increase by +33.9%.

3. Side-effect checks:
   - C helpers: rust/helpers/helpers*.bc is compiled separately via
     cmd_rust_helper (without CC_FLAGS_RUST_INLINE_HELPERS), so C
     helpers still get their pre-inlining base discriminators before
     llvm-link.
   - Flow-sensitive discriminators: MIRAddFSDiscriminators in
     llvm/lib/CodeGen/MIRFSDiscriminator.cpp checks
     EnableFSDiscriminator and shouldEmitDebugInfoForProfiling() rather
     than NoDiscriminators, so post-link FS discriminators remain
     enabled.

Rong, Han, Miguel, Nathan: would you prefer adding
`-mllvm -no-discriminators` directly to `CC_FLAGS_RUST_INLINE_HELPERS`
in the top-level Makefile, or appending it in scripts/Makefile.autofdo
under `ifdef CONFIG_RUST_INLINE_HELPERS`? Happy to send v2 once we align
on the preferred spot.

Thanks,
Jenhao