From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Cyrus-Session-Id: sloti22d1t05-1211982-1517542027-2-15658447055481373767 X-Sieve: CMU Sieve 3.0 X-Spam-known-sender: no X-Spam-score: 0.0 X-Spam-hits: BAYES_00 -1.9, HEADER_FROM_DIFFERENT_DOMAINS 0.001, RCVD_IN_DNSWL_NONE -0.0001, LANGUAGES en, BAYES_USED global, SA_VERSION 3.4.0 X-Spam-source: IP='96.67.55.147', Host='shelob.surriel.com', Country='US', FromHeader='org', MailFrom='org' X-Spam-charsets: plain='us-ascii' X-Resolved-to: greg@kroah.com X-Delivered-to: greg@kroah.com X-Mail-from: kernelnewbies-bounces@kernelnewbies.org ARC-Seal: i=1; a=rsa-sha256; cv=none; d=messagingengine.com; s=arctest; t=1517542026; b=pT3AXI7dlG5RJorgx8vTaHbTbYQ8t3WrBLyljgdfcStaBkM fBzL68z5OfJ8obG4OhukkL63z10OL2BP778pXXY1prbxdUDniB0ud1ceaBWrPHLt Vkrt1vbGhca5SwuTsmnYBOIbHpbq0IjSdJdlLcflCuZr7HfBupqaGCaG7PKSJnro BrCdGgssTKLKYPhHEZdnHCzh1uMGIONv1RalJXzL9y1Wt36TKICC+eyV2xbMsHyX A9fqA5KgPu0eZ+RP33z42kFozMmPCOihHlQzQnaIx3p2P9JRd7VKVbDaQbaTobs3 SmQy/rVQ2pISe+kKmybUrdnRKwFFfIyF5UrsK+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=date:from:to:subject:message-id :in-reply-to:references:mime-version:cc:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:content-type :content-transfer-encoding; s=arctest; t=1517542026; bh=WBAC/Q4c eWdn7KlBVyXRy1DBKirHPxXt/aQYB8Qc878=; b=OufH9AUSDlbF0NK3MzOSaaEh xaJsxLLQQ/23fwc8++OFGUt9UMd4mZJH883cL0Hlchz8aQ6MGBRYMgYfNoTPaxj6 7FLU3+/VAhixuBSNxysDGrSq+ddC2XDIYF3Rxyz591rCqHwZs2Una66MCUXuKoVK 1kRqEKIg/z2nCklV96KWR6cQwoXofZBgpL3zi9VxjW9KEkTC4CrkCmPfZ9G7Xis5 0UYV1l5ZsRnz59kDxuUFx6uOIX/hpn/G/OjIXoBYoEkU2JrM7c8+5M68MBBC1zJE RuhrX0u4wr4fyRgDZ/SEBXOHq70QQMZ4dkKzvky7Jrb6nuNZpA/LRTrjCSnECg== ARC-Authentication-Results: i=1; mx2.messagingengine.com; arc=none (no signatures found); dkim=none (no signatures found); dmarc=none (p=none,has-list-id=yes,d=none) header.from=kernel.org; iprev=pass policy.iprev=96.67.55.147 (shelob.surriel.com); spf=permerror smtp.mailfrom=kernelnewbies-bounces@kernelnewbies.org smtp.helo=shelob.surriel.com; x-aligned-from=fail; x-ptr=pass x-ptr-helo=shelob.surriel.com x-ptr-lookup=shelob.surriel.com; x-return-mx=pass smtp.domain=kernelnewbies.org smtp.result=pass smtp_is_org_domain=yes header.domain=kernel.org header.result=pass header_is_org_domain=yes; x-tls=pass version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128 Authentication-Results: mx2.messagingengine.com; arc=none (no signatures found); dkim=none (no signatures found); dmarc=none (p=none,has-list-id=yes,d=none) header.from=kernel.org; iprev=pass policy.iprev=96.67.55.147 (shelob.surriel.com); spf=permerror smtp.mailfrom=kernelnewbies-bounces@kernelnewbies.org smtp.helo=shelob.surriel.com; x-aligned-from=fail; x-ptr=pass x-ptr-helo=shelob.surriel.com x-ptr-lookup=shelob.surriel.com; x-return-mx=pass smtp.domain=kernelnewbies.org smtp.result=pass smtp_is_org_domain=yes header.domain=kernel.org header.result=pass header_is_org_domain=yes; x-tls=pass version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128 DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1FA3C2175D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=mhiramat@kernel.org Date: Tue, 30 Jan 2018 14:43:46 +0900 From: Masami Hiramatsu To: Pintu Kumar Subject: Re: [perf] perf probe fails sometimes on 4.9 Message-Id: <20180130144346.86307346478928ca872473b5@kernel.org> In-Reply-To: References: <20180129231723.524a8c32e1456358f7cb88ce@kernel.org> X-Mailer: Sylpheed 3.5.1 (GTK+ 2.24.31; x86_64-redhat-linux-gnu) Mime-Version: 1.0 X-Mailman-Approved-At: Thu, 01 Feb 2018 22:23:39 -0500 Cc: "Wangnan \(F\)" , Alexis Berlemont , daniel@iogearbox.net, kernelnewbies@kernelnewbies.org, alexander.shishkin@linux.intel.com, linux-kernel@vger.kernel.org, acme@redhat.com, peterz@infradead.org, mingo@redhat.com, huawei.libin@huawei.com, naveen.n.rao@linux.vnet.ibm.com, quentin.monnet@netronome.com, maddy@linux.vnet.ibm.com, davem@davemloft.net X-BeenThere: kernelnewbies@kernelnewbies.org X-Mailman-Version: 2.1.15 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kernelnewbies-bounces@kernelnewbies.org X-getmail-retrieved-from-mailbox: INBOX X-Mailing-List: linux-kernel@vger.kernel.org List-ID: On Mon, 29 Jan 2018 22:00:52 +0530 Pintu Kumar wrote: > Dear Masami, > > Thank you so much for your reply. > Please find some of my answers inline. > > > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu wrote: > > On Mon, 29 Jan 2018 13:40:34 +0530 > > Pintu Kumar wrote: > > > >> Hi All, > >> > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64. > >> # perf probe --add schedule > >> schedule is out of .text, skip it. > >> Error: Failed to add events. > >> > >> If any one have come across this problem please let me know the cause. > > > > Hi Pintu, > > > > Could you run it with --vv? > > > Ok, I will send verbose output by tomorrow. > > >> > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem? > > > > Without it, you can not probe source-level probe nor trace local variable. > > > > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20 > However, I will try to manually include "-g" option during compilation. > > >> However, I manually copied the vmlinux file to /boot/ directory, but > >> still it does not work. > > > > That doesn't work. > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo. > > Without that option, debuginfo is not generated with vmlinux. > > > >> > >> I checked upstream patches until 4.15 but could not find any clue. > >> Please let me know if there is any fixes available for this. > > > > Could you also ensure that you run perf by root user? > > > > Yes I am running with root user. > > My concern is sometimes it works but sometimes it fails. What I thought was that your kernel enables kptr_strict(but maybe not.) Can you also try to find "schedule" and "_etext" functions in /proc/kallsyms? > I still needs to figure out, in which condition it works and which > condition it fails. Yeah, it is important. > Usually I noticed that in fresh reboot case it works. So, after a while, it doesn't work again? If so, it sounds like a daemon process changes settings in background. Thank you, > > > > Thank you, > > > > > >> > >> > >> Thank You! > >> Regards, > >> Pintu > >> > >> > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar wrote: > >> > Hi, > >> > > >> > ** Changed the subject now, since these issues are related to general > >> > perf commands. > >> > > >> > Following are the issues: > >> > > >> > 1) perf probe --add schedule - FAILED > >> > output: > >> > schedule is out of .text, skip it. > >> > Error: Failed to add events. > >> > > >> > what is the issue here? > >> > Sometimes it pass and sometimes it fails... > >> > Similar is the case of 'perf inject' as well. > >> > > >> > 2) perf test - 1 FAILURE > >> > 37.1: Test basic BPF filtering : FAILED! > >> > 37.2: Test BPF prologue generation : Skip > >> > 37.3: Test BPF relocation checker : Skip > >> > > >> > bpf: config program 'func=SyS_epoll_wait' > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null) > >> > bpf: config 'func=SyS_epoll_wait' is ok > >> > Looking at the vmlinux_path (8 entries long) > >> > Using /boot/vmlinux for symbols > >> > Could not open debuginfo. Try to use symbols. > >> > SyS_epoll_wait is out of .text, skip it. > >> > bpf_probe: failed to convert perf probe eventsFailed to add events > >> > selected by BPF > >> > test child finished with -1 > >> > ---- end ---- > >> > Test BPF filter subtest 0: FAILED! > >> > > >> > Looks like both 1,2 are related. > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux > >> > to /boot/ folder. > >> > > >> > ------- > >> > Some more info: > >> > > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build > >> > unset env: KBUILD_OPTS > >> > include option is set to -nostdinc -isystem > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h > >> > set env: NR_CPUS=8 > >> > set env: LINUX_VERSION_CODE=0x40914 > >> > set env: CLANG_EXEC=/usr/bin/clang > >> > set env: CLANG_OPTIONS=-xc > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build > >> > > >> > > >> > If you have any clue about these failure please help me. > >> > > >> > > >> > Thanks, > >> > Pintu > >> > > >> > > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar wrote: > >> >> Hi, > >> >> > >> >> Thanks for your help. > >> >> Yes it was a sub version issue. > >> >> > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank. > >> >> > >> >> make -j8 bindeb-pkg SUBLEVEL=20 > >> >> > >> >> After passing the correct sublevel now the test is working. > >> >> > >> >> But still following are failing: > >> >> > >> >> 16: Try 'import perf' in python, checking link problems : FAILED! > >> >> 37.2: Test BPF prologue generation : FAILED! > >> >> > >> >> > >> >> This is the error I get: > >> >> > >> >> 16: Try 'import perf' in python, checking link problems : > >> >> --- start --- > >> >> test child forked, pid 7637 > >> >> Traceback (most recent call last): > >> >> File "", line 1, in > >> >> ImportError: No module named perf > >> >> test child finished with -1 > >> >> ---- end ---- > >> >> Try 'import perf' in python, checking link problems: FAILED! > >> >> > >> >> > >> >> Looking at the vmlinux_path (8 entries long) > >> >> symsrc__init: cannot get elf header. > >> >> Failed to find the path for kernel: Invalid ELF file > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events > >> >> selected by BPF > >> >> test child finished with -1 > >> >> ---- end ---- > >> >> Test BPF filter subtest 1: FAILED! > >> >> > >> >> > >> >> > >> >> Thanks, > >> >> Pintu > >> >> > >> >> > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) wrote: > >> >>> > >> >>> > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote: > >> >>>> > >> >>>> Hi All, > >> >>>> > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the > >> >>>> kernel version 4.9.20. > >> >>>> I have installed several others packages including: clang, llvm > >> >>>> > >> >>>> But, when I run 'perf test' I get some FAILURE. > >> >>>> Specially, 'perf test LLVM' is failing. > >> >>>> > >> >>>> Please check the below error logs: > >> >>>> > >> >>>> ---------------------------------------------------- > >> >>>> # perf test LLVM > >> >>>> 35: Test LLVM searching and compiling : > >> >>>> 35.1: Basic BPF llvm compiling test : FAILED! > >> >>>> 35.2: Test kbuild searching : Skip > >> >>>> 35.3: Compile source for BPF prologue generation test : Skip > >> >>>> 35.4: Compile source for BPF relocation test : Skip > >> >>>> ---------------------------------------------------- > >> >>>> > >> >>>> When I run with -v I get this: > >> >>>> > >> >>>> ----------------------------------------------------- > >> >>>> # perf test -v LLVM > >> >>>> 35: Test LLVM searching and compiling : > >> >>>> 35.1: Basic BPF llvm compiling test : > >> >>>> --- start --- > >> >>>> test child forked, pid 3304 > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64' > >> >>>> WARNING: unable to get correct kernel building directory. > >> >>>> Hint: Set correct kbuild directory using 'kbuild-dir' option in [llvm] > >> >>>> section of ~/.perfconfig or set it to "" to suppress kbuild > >> >>>> detection. > >> >>>> > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64' > >> >>>> > >> >>>> > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE; > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o > >> >>>> - > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test' > >> >>>> test child finished with -1 > >> >>>> ---- end ---- > >> >>>> Test LLVM searching and compiling subtest 0: FAILED! > >> >>>> ----------------------------------------------------- > >> >>>> > >> >>>> > >> >>>> Basic BPF LLVM compiling test is failing. > >> >>>> It reports that bpf could not able to read the kernel version from uname. > >> >>>> Is it because of this that 'perf test LLVM' is failing ? > >> >>>> > >> >>>> My uname says: > >> >>>> 4.9-xxxx-amd-x86-64 > >> >>> > >> >>> > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to > >> >>> report > >> >>> kernel version number (from your uname string, sublevel can't be > >> >>> determined), > >> >>> please help us improve that function like what we have done for ubuntu > >> >>> (checking /proc/version_signature). > >> >>> > >> >>> > >> >>> > >> >>>> My kernel version is: 4.9.20 (from Makefile) > >> >>>> > >> >>>> So, I dont think there is any problem with kernel version. > >> >>>> > >> >>>> If any body have come across this problem please let me know how to > >> >>>> resolve this issue. > >> >>>> > >> >>>> > >> >>>> > >> >>>> Thank You! > >> >>>> > >> >>>> Regards, > >> >>>> Pintu > >> >>> > >> >>> > >> >>> > > > > > > -- > > Masami Hiramatsu -- Masami Hiramatsu _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies