From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1757780AbcDMBHY (ORCPT <rfc822;w@1wt.eu>);
	Tue, 12 Apr 2016 21:07:24 -0400
Received: from mail-pf0-f195.google.com ([209.85.192.195]:34074 "EHLO
	mail-pf0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1754931AbcDMBHW (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Tue, 12 Apr 2016 21:07:22 -0400
Date: Wed, 13 Apr 2016 10:06:57 +0900
From: Namhyung Kim <namhyung@kernel.org>
To: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>, David Ahern <dsahern@gmail.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Milian Wolff <milian.wolff@kdab.com>, Wang Nan <wangnan0@huawei.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Re: zeros in frame pointer callchains
Message-ID: <20160413010657.GA10693@danjae.aot.lge.com>
References: <20160412133847.GC10972@kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20160412133847.GC10972@kernel.org>
User-Agent: Mutt/1.6.0 (2016-04-01)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Arnaldo,

On Tue, Apr 12, 2016 at 10:38:47AM -0300, Arnaldo Carvalho de Melo wrote:
> Hi,
> 
> 	Namhyung, while working on supporting callchains in 'perf trace' I
> noticed something I think we discussed at some point, zeros in frame pointer
> callchains...
> 
> 	Oh well figured it out, its probably userspace not having frame
> pointers, grrr anyway, keeping the rest of the message, for reference,
> but now I think we need to have some expressiveness to say that for that
> particular event we want to use perf_event_attr.exclude_callchain_user... 
> 
> First add a special probe point that 'perf trace' uses, if present:
> 
>   # perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
> 
> 	Then lets ask to see just when this thing hits, no syscall
> printing whatsoever ('trace' == 'perf trace'), sometimes just a few
> zeros at the top, sometimes a long series, need to investigate how to
> chop those off in the kernel, not to use the ring buffer needlessly,
> will start chopping these in userspace (in perf trace):

My solution was to look at /proc/sys/vm/mmap_min_addr [1].  For kernel
it'd be simply discarding callchains after seeing an address below the
mmap_min_addr.  For userspace, it requires to carry the mmap_min_addr
via feature bit or something.  Or Maybe we can just discard addresses
under 4096 for simplicity?


[1] https://lkml.org/lkml/2015/11/26/40

Thanks,
Namhyung


> 
>   # trace --no-sys --ev probe:vfs_getname/call-graph=fp/
>   
>  34033.533 probe:vfs_getname:(ffffffff8124c06c) pathname="/usr/share/pixmaps")
>                                        getname_flags+0xfe20009d ([kernel.kallsyms])
>                                        user_path_at_empty+0xfe200023 ([kernel.kallsyms])
>                                        vfs_fstatat+0xfe200066 ([kernel.kallsyms])
>                                        SYSC_newstat+0xfe20002e ([kernel.kallsyms])
>                                        sys_newstat+0xfe20000e ([kernel.kallsyms])
>                                        entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
>                                        __xstat64+0xffff011e90b98015 (/usr/lib64/libc-2.22.so)
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>  36001.681 probe:vfs_getname:(ffffffff8124c06c) pathname="/var/tmp")
>                                        getname_flags+0xfe20009d ([kernel.kallsyms])
>                                        user_path_at_empty+0xfe200023 ([kernel.kallsyms])
>                                        vfs_fstatat+0xfe200066 ([kernel.kallsyms])
>                                        SYSC_newstat+0xfe20002e ([kernel.kallsyms])
>                                        sys_newstat+0xfe20000e ([kernel.kallsyms])
>                                        entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
>                                        __xstat64+0xffff00b816a28015 (/usr/lib64/libc-2.22.so)
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>  36001.696 probe:vfs_getname:(ffffffff8124c06c) pathname="/var/tmp")
>                                        getname_flags+0xfe20009d ([kernel.kallsyms])
>                                        user_path_at_empty+0xfe200023 ([kernel.kallsyms])
>                                        sys_access+0xfe2000b4 ([kernel.kallsyms])
>                                        entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
>                                        __GI___access+0xffff00b816a28007 (/usr/lib64/libc-2.22.so)
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>   8254.597 probe:vfs_getname:(ffffffff8124c06c) pathname="/etc/localtime")
>                                        getname_flags+0xfe20009d ([kernel.kallsyms])
>                                        user_path_at_empty+0xfe200023 ([kernel.kallsyms])
>                                        vfs_fstatat+0xfe200066 ([kernel.kallsyms])
>                                        SYSC_newstat+0xfe20002e ([kernel.kallsyms])
>                                        sys_newstat+0xfe20000e ([kernel.kallsyms])
>                                        entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
>                                        __xstat64+0xffff0104f5696015 (/usr/lib64/libc-2.22.so)
>                                        [0] ([unknown])
>                                        [0] ([unknown])
>   9001.567 probe:vfs_getname:(ffffffff8124c06c) pathname="/etc/firewalld/ipsets")
>                                        getname_flags+0xfe20009d ([kernel.kallsyms])
>                                        user_path_at_empty+0xfe200023 ([kernel.kallsyms])
>                                        sys_inotify_add_watch+0xfe200098 ([kernel.kallsyms])
>                                        entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
>                                        inotify_add_watch+0xffff008bb9432007 (/usr/lib64/libc-2.22.so)
>   9450.181 probe:vfs_getname:(ffffffff8124c06c) pathname="/root")
>                                        getname_flags+0xfe20009d ([kernel.kallsyms])
>                                        user_path_at_empty+0xfe200023 ([kernel.kallsyms])
>                                        sys_inotify_add_watch+0xfe200098 ([kernel.kallsyms])
>                                        entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
>                                        inotify_add_watch+0xffff01b7aa4f2007 (/usr/lib64/libc-2.22.so)