From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9D20C433DB for ; Wed, 3 Mar 2021 18:39:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AA0DA64F0B for ; Wed, 3 Mar 2021 18:39:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1386154AbhCCRqt (ORCPT ); Wed, 3 Mar 2021 12:46:49 -0500 Received: from mail.kernel.org ([198.145.29.99]:50620 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242986AbhCCOhu (ORCPT ); Wed, 3 Mar 2021 09:37:50 -0500 Received: from gandalf.local.home (cpe-66-24-58-225.stny.res.rr.com [66.24.58.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1E13964EE1; Wed, 3 Mar 2021 14:26:07 +0000 (UTC) Date: Wed, 3 Mar 2021 09:26:04 -0500 From: Steven Rostedt To: Masami Hiramatsu Cc: "Daniel Xu" , linux-kernel@vger.kernel.org, "bpf@vger.kernel.org" , kuba@kernel.org Subject: Re: Broken kretprobe stack traces Message-ID: <20210303092604.59aea82c@gandalf.local.home> In-Reply-To: <20210303134828.39922eb167524bc7206c7880@kernel.org> References: <1fed0793-391c-4c68-8d19-6dcd9017271d@www.fastmail.com> <20210303134828.39922eb167524bc7206c7880@kernel.org> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 3 Mar 2021 13:48:28 +0900 Masami Hiramatsu wrote: > > > > > I think (can't prove) this used to work: Would be good to find out if it did. > > I'm not sure the bpftrace had correctly handled it or not. > > > > > # bpftrace -e 'kretprobe:__tcp_retransmit_skb { @[kstack()] = count() }' > > Attaching 1 probe... > > ^C > > > > @[ > > kretprobe_trampoline+0 > > ]: 1 > > Would you know how the bpftrace stacktracer rewinds the stack entries? > FYI, ftrace does it in trace_seq_print_sym()@kernel/trace/trace_output.c > The difference between trace events and normal function tracing stack traces is that it keeps its original return address. But kretprobes (and function graph tracing, and some bpf trampolines too) modify the return pointer, and that could possibly cause havoc with the stack trace. -- Steve