From: Aaron Tomlin <atomlin@atomlin.com>
To: peterz@infradead.org, mingo@redhat.com, acme@kernel.org,
namhyung@kernel.org
Cc: mark.rutland@arm.com, alexander.shishkin@linux.intel.com,
jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com,
atomlin@atomlin.com, linux-perf-users@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: [PATCH perf-tools-next v2 0/4] perf ftrace: Support inlined functions and display enhancements in function graph tracer
Date: Tue, 6 Oct 2026 19:27:52 -0400 [thread overview]
Message-ID: <20261006232756.65620-1-atomlin@atomlin.com> (raw)
The Linux kernel's function graph tracer operates at the machine
instruction level via compiler instrumentation
(-fpatchable-function-entry), recording entry and return events for
physical function calls. Consequently, functions inlined by the compiler
are invisible in the resulting call-graph, as no discrete call or return
instructions are emitted for them. While technically expected, this
omission frequently obscures the logical execution flow when analysing
kernel subsystems heavily reliant upon inlining (such as the scheduler,
memory management, locking primitives, and RCU).
When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the
funcgraph-retaddr trace option records the caller's return address on each
function entry, manifested in the trace stream as a comment
(e.g. /* <-wake_up_new_task+0x1d1/0x3e0 */). By interrogating DWARF debug
information from the kernel image (vmlinux) utilising libdw, perf ftrace
resolves this return address to its inlined callchain and synthesises the
intermediate inlined frames directly into the streamed call-graph.
Synthesised inlined functions are rendered with an explicit /* (inline) */
annotation at both entry and exit:
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) 0.856 us | mutex_unlock(lock=0xffffffffbc28bfa0);
0) | wake_up_new_task(p=0xffff8b1f183bac80) {
0) | rb_irq_work_queue() { /* (inline) */
0) | rb_wakeups() { /* (inline) */
0) 1.374 us | housekeeping_any_cpu(type=3);
0) | arch_irq_work_raise() {
0) | apic_wait_icr_idle() { /* (inline) */
0) | x2apic_send_IPI_self(vector=246) {
0) | instr_sysvec_irq_work() { /* (inline) */
0) | irq_enter_rcu() {
0) | instr_sysvec_irq_work() { /* (inline) */
0) 0.673 us | irqtime_account_irq();
0) | } /* instr_sysvec_irq_work (inline) */
0) 1.697 us | }
This series decomposes the feature into a modular, bisectable progression
of four patches:
1. Stack frame optimisation in __cmd_ftrace()
Replaces the 4096-byte stack array in __cmd_ftrace() with dynamic
heap allocation alongside struct strbuf streaming, reducing the
stack frame footprint by over 96% (down to ~160 bytes) while
ensuring complete line-delimited records and robust EOF handling.
2. Return address comment filtering
Introduces ftrace_parse_retaddr() and ftrace_filter_retaddr() with
--filter-retaddr and --graph-opts filter-retaddr to strip caller
comments from trace output, accompanied by an initial unit test
suite in tools/perf/tests/ftrace.c.
3. Inlined function call-graph reconstruction
Introduces --inline and -k or --vmlinux, dynamically synthesising
inlined frames marked with /* (inline) */. Correctly matches bare
closing braces ('}') when funcgraph-tail is disabled, clamps stack
depth to prevent indentation drift, and expands the automated test
suite to verify inline resolution, comment filtering, and stack
saturation recovery.
4. Execution duration suppression
Introduces --graph-opts noduration (and duration=[0|1]) to toggle
the kernel's funcgraph-duration tracing option, omitting the
duration column when the user is focused purely on call-graph
topology and inlined hierarchy.
Changes since v1:
- Modularised the implementation into a 4-patch bisectable series rather
than a monolithic commit
- Resolved potential infinite loop on poll() upon trace_pipe EOF
- Fixed stuck call stack frames when funcgraph-tail is disabled by
supporting bare closing delimiters matching LIFO stack frames
- Prevented unbounded indentation growth on deep call trees by clamping
inline expansion and synchronising cs->inlined_depth to frames
successfully pushed onto cs->stack
- Added dedicated unit tests covering bare closing braces and stack
saturation recovery
- Refined option documentation in perf-ftrace.txt
- Link to v1: https://lore.kernel.org/lkml/20261005211518.26786-1-atomlin@atomlin.com/
Aaron Tomlin (4):
perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming
perf ftrace: Support filtering return address comments
perf ftrace: Support display of inlined functions in function graph
tracer
perf ftrace: Support omitting execution duration in function graph
tracer
tools/perf/Documentation/perf-ftrace.txt | 21 +
tools/perf/builtin-ftrace.c | 126 +++++-
tools/perf/tests/Build | 1 +
tools/perf/tests/builtin-test.c | 1 +
tools/perf/tests/ftrace.c | 240 ++++++++++
tools/perf/tests/tests.h | 1 +
tools/perf/util/Build | 1 +
tools/perf/util/ftrace.c | 537 +++++++++++++++++++++++
tools/perf/util/ftrace.h | 17 +
9 files changed, 933 insertions(+), 12 deletions(-)
create mode 100644 tools/perf/tests/ftrace.c
create mode 100644 tools/perf/util/ftrace.c
--
2.55.0
next reply other threads:[~2026-10-06 23:28 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-06 23:27 Aaron Tomlin [this message]
2026-10-06 23:27 ` [PATCH perf-tools-next v2 1/4] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming Aaron Tomlin
2026-10-06 23:27 ` [PATCH perf-tools-next v2 2/4] perf ftrace: Support filtering return address comments Aaron Tomlin
2026-10-06 23:27 ` [PATCH perf-tools-next v2 3/4] perf ftrace: Support display of inlined functions in function graph tracer Aaron Tomlin
2026-10-06 23:27 ` [PATCH perf-tools-next v2 4/4] perf ftrace: Support omitting execution duration " Aaron Tomlin
2026-10-07 23:24 ` [PATCH perf-tools-next v2 0/4] perf ftrace: Support inlined functions and display enhancements " Namhyung Kim
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261006232756.65620-1-atomlin@atomlin.com \
--to=atomlin@atomlin.com \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=irogers@google.com \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®