From: Steven Rostedt <rostedt@kernel.org>
To: linux-kernel@vger.kernel.org
Cc: Masami Hiramatsu <mhiramat@kernel.org>,
Mark Rutland <mark.rutland@arm.com>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Andrew Morton <akpm@linux-foundation.org>,
Peter Zijlstra <peterz@infradead.org>,
Vineet Gupta <vineet.gupta@linux.dev>,
stable@vger.kernel.org, Andrii Nakryiko <andrii@kernel.org>
Subject: [for-linus][PATCH 13/32] fgraph: Allocate shadow stacks inline with GFP_NOWAIT
Date: Fri, 20 Nov 2026 12:39:05 -0500 [thread overview]
Message-ID: <20261120174116.504279783@kernel.org> (raw)
In-Reply-To: <20261120173852.687189370@kernel.org>
From: Vineet Gupta <vineet.gupta@linux.dev>
When ftrace graphing is turned on, all tasks in the system missing
return stack page are assigned one. This is done in a simplistic
multi-sweep loop of FTRACE_RETSTACK_ALLOC_SIZE (currently 32) tasks
at a time as follows:
start_graph_tracing()
do {
alloc_retstack_tasklist
} while (-EAGAIN);
alloc_retstack_tasklist()
alloc x32 # GFP_KERNEL, may sleep
rcu_read_lock() # preempt off
for_each_process_thread walk N_total, no cond_resched
t->ret_stack = new_page
rcu_read_unlock() # preempt enable but no explicit yield
Each successive iteration of loop invokes for_each_process_thread()
which doesn't support cursor based resume and always restarts from the
init_task. Thus each successive loop needs to skip the tasks assigned
ret_stack in prior sweeps and thus take longer and longer to find the
candidate 32 tasks. And while the loop end calls rcu unlock,
and re-enables preemption briefly, there is no explicit yield.
The total number of iterations turn out to be
N_total * N_null / (2 * FTRACE_RETSTACK_ALLOC_SIZE)
where N_total is the number of threads on the system and N_null the number
of those whose ret_stack is still NULL. After boot with no fgraph user that
is every thread.
This is not a tracing-only path. Since commit 4346ba160409 ("fprobe:
Rewrite fprobe on function-graph tracer") fprobe is built on fgraph,
so an ordinary kprobe_multi BPF attach that flips ftrace_graph_active
from 0 to 1 pays the whole cost inside one bpf() syscall. Commit
2c67dc457bc6 ("tracing: fprobe: optimization for entry only case")
narrows that to return and session probes but does not remove it.
A host running hundreds of thousands of threads, with the current batch
size of 32, turns the ftrace_graph_active 0 -> 1 transition into a
multi-second, and eventually multi-minute, operation which showed up as
RCU stall and softlockup_panic on heavily loaded Meta Fleet machines
with preemption disabled.
rcu: INFO: rcu_sched self-detected stall on CPU
rcu: 0-....: (20718 ticks this GP) (t=21000 jiffies g=913 q=8 ncpus=8)
CPU: 0 UID: 0 PID: 160141 Comm: bpftrace
___slab_alloc+0x549/0xa20
kmem_cache_alloc_noprof+0x16c/0x340
register_ftrace_graph+0x3d7/0x6c0
register_fprobe_ips+0x2d5/0x310
bpf_kprobe_multi_link_attach+0x218/0x7e0
__sys_bpf+0x267a/0x27a0
Fix this by allocating inline with GFP_NOWAIT instead of pre-allocating
a fixed batch. GFP_NOWAIT does not sleep, so it is legal inside the RCU
read-side critical section.
A single sweep now serves every task, so the walk becomes O(N) rather
than O(N_total * N_null), and the retry loop is no longer the normal
path.
The pre-allocated array remains as the fallback when GFP_NOWAIT fails.
The runs below never needed it, so it was exercised with failslab
injection: at a 100% failure rate every task falls back and -EAGAIN
retries ~1560 times for 50000 threads, with no stall or splat.
smp_store_release() is a drop-in replacement for the smp_wmb() and
plain store it replaces.
Measured on a 60-core Sapphire Rapids machine running a PREEMPT_LAZY
kernel (CONFIG_PREEMPT_DYNAMIC=y, mode lazy), timing the write that
drives the ftrace_graph_active 0 -> 1 transition. Threads are spawned
while fgraph is inactive so every one of them has ret_stack == NULL,
and the loop was instrumented to count sweeps:
threads before after
sweeps time sweeps time
100000 3126 15.3 s 1 0.019 s
200000 6253 59.1 s 1 0.047 s
400000 12503 227.8 s 1 0.092 s
The sweep count is 1 throughout. Time now scales linearly with the
thread count at roughly 230 ns per task, which is the allocation and
initialisation of one shadow stack.
Fixes: f201ae2356c7 ("tracing/function-return-tracer: store return stack into task_struct and allocate it dynamically")
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
Tested-by: Vineet Gupta <vineet.gupta@linux.dev>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20261021174406.2891598-2-vineet.gupta@linux.dev
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
kernel/trace/fgraph.c | 39 ++++++++++++++++++++++++---------------
1 file changed, 24 insertions(+), 15 deletions(-)
diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c
index 6d87143b3ddb..59c217b06695 100644
--- a/kernel/trace/fgraph.c
+++ b/kernel/trace/fgraph.c
@@ -1033,13 +1033,16 @@ void ftrace_stub_graph(struct ftrace_graph_ret *trace, struct fgraph_ops *gops,
trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph;
trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub;
-/* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */
+/*
+ * Loop thru all the tasks in the system and assign ret_stack.
+ * The @ret_stack_list is a small pre-allocated fallback, in case
+ * the GFP_NOWAIT allocations here fail.
+ */
static int alloc_retstack_tasklist(unsigned long **ret_stack_list)
{
- int i;
- int ret = 0;
int start = 0, end = FTRACE_RETSTACK_ALLOC_SIZE;
struct task_struct *g, *t;
+ int i, ret = 0;
if (WARN_ON_ONCE(!fgraph_stack_cachep))
return -ENOMEM;
@@ -1056,20 +1059,26 @@ static int alloc_retstack_tasklist(unsigned long **ret_stack_list)
rcu_read_lock();
for_each_process_thread(g, t) {
- if (start == end) {
- ret = -EAGAIN;
- goto unlock;
- }
+ unsigned long *rs;
- if (t->ret_stack == NULL) {
- atomic_set(&t->trace_overrun, 0);
- ret_stack_init_task_vars(ret_stack_list[start]);
- t->curr_ret_stack = 0;
- t->curr_ret_depth = -1;
- /* Make sure the tasks see the 0 first: */
- smp_wmb();
- t->ret_stack = ret_stack_list[start++];
+ if (t->ret_stack)
+ continue;
+
+ rs = kmem_cache_alloc(fgraph_stack_cachep, GFP_NOWAIT);
+ if (!rs) {
+ if (start == end) {
+ ret = -EAGAIN;
+ goto unlock;
+ }
+ rs = ret_stack_list[start++];
}
+
+ atomic_set(&t->trace_overrun, 0);
+ ret_stack_init_task_vars(rs);
+ t->curr_ret_stack = 0;
+ t->curr_ret_depth = -1;
+ /* Make sure the tasks see the 0 first: */
+ smp_store_release(&t->ret_stack, rs);
}
unlock:
--
2.53.0
next prev parent reply other threads:[~2026-11-20 17:41 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 17:38 Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 01/32] ring-buffer: Do not WARN when trace_rb_cpu_prepare() fails to allocate Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 02/32] tracing: Keep copying markers when one instance is stopped Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 03/32] tracing: Restrict perf filters that read kernel strings Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 04/32] tracing/filters: Treat zero-length dynamic strings as empty Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 05/32] tracing: Define common event fields before enabling boot events Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 06/32] tracing: Drop the synthetic event ref when clearing hist triggers Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 07/32] tracing: Avoid undefined shifts for zero histogram map size Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 08/32] tracing/filters: Fix MATCH_END_ONLY regex matching for fixed-size strings Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 09/32] tracing/filters: Handle out-of-range CPU for OP_NE in cpumask comparison Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 10/32] tracing/filters: Check for missing operand after operator in parse_pred() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 11/32] tracing/filters: Fix false success return in process_system_preds() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 12/32] tracing: Free tracer resources on instance creation failure Steven Rostedt
2026-11-20 17:39 ` Steven Rostedt [this message]
2026-11-20 17:39 ` [for-linus][PATCH 14/32] fgraph: Use scoped_guard() for the task list walk Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 15/32] fgraph: Read the task->ret_stack with a smp_load_acquire() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 16/32] tracing/osnoise: Dont start a kthread in timerlat user-space mode Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 17/32] tracing/user_events: Dont free an event that a dynamic event still references Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 18/32] tracing: Wait for marker writers when copy_trace_marker is cleared Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 19/32] tracing: Validate user addresses in faultable buffer copies Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 20/32] fgraph: Do not mark index 0 in every shadow stack frame Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 21/32] ring-buffer: Skip remote reader page swap when on tail Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 22/32] ring-buffer: Check remote reader page commit before meta entries Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 23/32] ring-buffer: Order remote page commit with data and timestamp Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 24/32] ring-buffer: Increment pages_read on remote ring buffer page swap Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 25/32] tracing/hwlat: Do the latency calculations in nanoseconds Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 26/32] tracing: Properly match strings that start with numbers Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 27/32] tracing: Add tests for globbing number-based strings Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 28/32] tracing: Take trace_access_lock() in tracing_buffers_ioctl() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 29/32] tracing: Do not allow user events to add a system name Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 30/32] ring-buffer: Do not check si_mem_available() before deferred init completes Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 31/32] tracing: Add notrace to event functions that hook to ftrace Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 32/32] tracing: Ignore .function on the function trace event Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120174116.504279783@kernel.org \
--to=rostedt@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=andrii@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mathieu.desnoyers@efficios.com \
--cc=mhiramat@kernel.org \
--cc=peterz@infradead.org \
--cc=stable@vger.kernel.org \
--cc=vineet.gupta@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®