mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Vineet Gupta <vineet.gupta@linux.dev>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: mhiramat@kernel.org, mark.rutland@arm.com,
	mathieu.desnoyers@efficios.com, andrii@kernel.org,
	peterz@infradead.org, linux-trace-kernel@vger.kernel.org,
	linux-kernel@vger.kernel.org, bpf@vger.kernel.org,
	kernel-team@meta.com
Subject: Re: [PATCH 0/2] tracing: fgraph: cut the cost of the shadow stack retry loop
Date: Thu, 24 Sep 2026 19:31:53 -0700	[thread overview]
Message-ID: <1cffebd0-57b3-4e59-8595-37a0e157b2a2@linux.dev> (raw)
In-Reply-To: <20260924164512.451a5d97@fedora>

On 9/24/26 13:45, Steven Rostedt wrote:
> Just an FYI. My wife flew into Europe today and I'll be touring Europe
> with her until Oct 3rd. I may get time to look at patches, but it will
> be very limited. And then LPC starts and who knows when I'll get a
> chance.
>
> Thus I may not be very responsive for the next couple of weeks.

Thx for the reply Steve. I understand it is that time of the year with 
travels and such.
No rush, just wanted to push it out of mailbox, to yours :-)
Besides it might all be moot as Peter's solution is much more elegant 
and efficient (as always).
I will reply to that but don't feel pressed to respond.

Thx,
-Vineet

>
> -- Steve
>
>
> On Tue, 22 Sep 2026 15:55:24 -0700
> Vineet Gupta <vineet.gupta@linux.dev> wrote:
>
>> Turning on function-graph tracing, or attaching a kprobe_multi return
>> probe, hands every thread a shadow stack. alloc_retstack_tasklist()
>> does that 32 tasks at a time, and since for_each_process_thread() has
>> no cursor, every sweep restarts from init_task and re-walks the tasks
>> already served. Total work is quadratic O(N^2) on thread count.
>>
>> On a 60-core Sapphire Rapids machine with 400000 idle threads the 0 -> 1
>> transition takes 227 s, inside a single bpf() syscall for the kprobe_multi
>> case. On Meta fleet this showed up as RCU stalls and softlockup panics.
>>
>> Patch 1 raises the batch to 1024, dividing the sweeps by 32:
>> 227 s -> 7.2 s at 400000 threads. It helps on every preemption model.
>>
>> Patch 2 adds a cond_resched() between sweeps. It is supplementary and
>> separable: a no-op on current x86 and arm64, but on !CONFIG_PREEMPTION
>> builds it takes soft lockups from 3-of-3 runs to 0-of-3. Dropping it
>> leaves patch 1 intact.
>>
>> Neither changes the O(N^2) shape; a cursor-based walk would, but
>> task_struct lifetime makes that considerably more involved.
>>
>> Vineet Gupta (2):
>>    tracing: fgraph: Raise FTRACE_RETSTACK_ALLOC_SIZE to 1024
>>    tracing: fgraph: Add a cond_resched() to the shadow stack retry loop
>>
>>   include/linux/ftrace.h | 7 ++++++-
>>   kernel/trace/fgraph.c  | 8 ++++++++
>>   2 files changed, 14 insertions(+), 1 deletion(-)
>>


      reply	other threads:[~2026-09-25  2:31 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-09-22 22:55 Vineet Gupta
2026-09-22 22:55 ` [PATCH 1/2] tracing: fgraph: Raise FTRACE_RETSTACK_ALLOC_SIZE to 1024 Vineet Gupta
2026-09-24 10:00   ` Peter Zijlstra
2026-09-22 22:55 ` [PATCH 2/2] tracing: fgraph: Add a cond_resched() to the shadow stack retry loop Vineet Gupta
2026-09-23  8:47   ` Steven Rostedt
2026-09-24 20:45 ` [PATCH 0/2] tracing: fgraph: cut the cost of " Steven Rostedt
2026-09-25  2:31   ` Vineet Gupta [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1cffebd0-57b3-4e59-8595-37a0e157b2a2@linux.dev \
    --to=vineet.gupta@linux.dev \
    --cc=andrii@kernel.org \
    --cc=bpf@vger.kernel.org \
    --cc=kernel-team@meta.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-trace-kernel@vger.kernel.org \
    --cc=mark.rutland@arm.com \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=mhiramat@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®