From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754702AbcITOHV (ORCPT ); Tue, 20 Sep 2016 10:07:21 -0400 Received: from smtprelay0217.hostedemail.com ([216.40.44.217]:54016 "EHLO smtprelay.hostedemail.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752793AbcITOHU (ORCPT ); Tue, 20 Sep 2016 10:07:20 -0400 X-Session-Marker: 726F737465647440676F6F646D69732E6F7267 X-Spam-Summary: 2,0,0,,d41d8cd98f00b204,rostedt@goodmis.org,:::::::::::::::,RULES_HIT:41:355:379:541:599:800:871:960:973:981:982:988:989:1000:1260:1313:1314:1345:1359:1437:1516:1518:1535:1542:1575:1594:1711:1730:1747:1764:1777:1792:2194:2199:2393:2553:2559:2562:2693:2741:3138:3139:3140:3141:3142:3354:3622:3865:3866:3867:3868:3870:3871:3872:3873:3874:4117:5007:6261:6506:7281:7875:7903:10004:10400:10848:10967:11026:11232:11658:11914:12296:12438:12663:12740:13161:13229:13439:13618:14180:14181:14659:14721:21060:21080:21324:21433:21451:30034:30054:30066:30090:30091,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: egg78_3fa3caa983a0f X-Filterd-Recvd-Size: 6040 Date: Tue, 20 Sep 2016 10:07:16 -0400 From: Steven Rostedt To: "Bean Huo (beanhuo)" Cc: "mingo@redhat.com" , "will.deacon@arm.com" , "catalin.marinas@arm.com" , "rfi@lists.rocketboards.org" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "Zoltan Szubbocsev (zszubbocsev)" Subject: Re: ftrace function_graph causes system crash Message-ID: <20160920100716.131d3647@gandalf.local.home> In-Reply-To: <7d0e88fff8e64591ac31be3adbbf763c@SIWEX5A.sing.micron.com> References: <7d0e88fff8e64591ac31be3adbbf763c@SIWEX5A.sing.micron.com> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.30; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/ByyDwrM+nvxqPNmwgt66cDE" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --MP_/ByyDwrM+nvxqPNmwgt66cDE Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline On Tue, 20 Sep 2016 13:10:39 +0000 "Bean Huo (beanhuo)" wrote: > Hi, all > I just use ftrace to do some latency study, found that function_graph can not > Work, as long as enable it, will cause kernel panic. I searched this online. > Found that there are also some cause the same as mine. I am a newer of ftrace. > I want to know who know what root cause? Here is some partial log: > > Can you do a function bisect to find what function this is. This script is used to help find functions that are being traced by function tracer or function graph tracing that causes the machine to reboot, hang, or crash. Here's the steps to take. First, determine if function graph is working with a single function: # cd /sys/kernel/debug/tracing # echo schedule > set_ftrace_filter # echo function_graph > current_tracer If this works, then we know that something is being traced that shouldn't be. # echo nop > current_tracer # cat available_filter_functions > ~/full-file # ftrace-bisect ~/full-file ~/test-file ~/non-test-file # cat ~/test-file > set_ftrace_filter *** Note *** this will take several minutes. Setting multiple functions is an O(n^2) operation, and we are dealing with thousands of functions. So go have coffee, talk with your coworkers, read facebook. And eventually, this operation will end. # echo function_graph > current_tracer If it crashes, we know that ~/test-file has a bad function. Reboot back to test kernel. # cd /sys/kernel/debug/tracing # mv ~/test-file ~/full-file If it didn't crash. # echo nop > current_tracer # mv ~/non-test-file ~/full-file Get rid of the other test file from previous run (or save them off somewhere. # rm -f ~/test-file ~/non-test-file And start again: # ftrace-bisect ~/full-file ~/test-file ~/non-test-file The good thing is, because this cuts the number of functions in ~/test-file by half, the cat of it into set_ftrace_filter takes half as long each iteration, so don't talk so much at the water cooler the second time. Eventually, if you did this correctly, you will get down to the problem function, and all we need to do is to notrace it. The way to figure out if the problem function is bad, just do: # echo > set_ftrace_notrace # echo > set_ftrace_filter # echo function_graph > current_tracer And if it doesn't crash, we are done. -- Steve --MP_/ByyDwrM+nvxqPNmwgt66cDE Content-Type: application/octet-stream; name=ftrace-bisect Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=ftrace-bisect IyEvYmluL2Jhc2gKIwojIEhlcmUncyBob3cgdG8gdXNlIHRoaXM6CiMKIyBTdGFydCBvZmYgYnkg ZG9pbmc6CiMKIyAgIGNhdCAvc3lzL2tlcm5lbC9kZWJ1Zy90cmFjaW5nL2F2YWlsYWJsZV9maWx0 ZXJfZnVuY3Rpb25zID4gZnVsbC1maWxlCiMgICBmdHJhY2UtYmlzZWN0IGZ1bGwtZmlsZSB0ZXN0 LWZpbGUgbm9uLXRlc3QtZmlsZQojICAgY2F0IHRlc3QtZmlsZSA+IC9zeXMva2VybmVsL2RlYnVn L3RyYWNpbmcvc2V0X2Z0cmFjZV9maWx0ZXIKIwojICAgIFsgdGhpcyBtYXkgdGFrZSBzZXZlcmFs IG1pbnV0ZXMgdG8gY29tcGxldGUsIGJlIHBhdGllbnQgXQojCiMgICBlY2hvIGZ1bmN0aW9uID4g L3N5cy9rZXJuZWwvZGVidWcvdHJhY2luZy9jdXJyZW50X3RyYWNlcgojCiMgSWYgaXQgY3Jhc2hl cy9sb2Nrcy11cCB3aGF0ZXZlciwgdGhlbiByZWJvb3QgYW5kIGRvOgojCiMgICBtdiB0ZXN0LWZp bGUgbmV3LWZpbGUKIwojIEVsc2UsIGlmIGl0IGRpZCBub3QgY3Jhc2gsIHRoZW4gZG86CiMKIyAg IG12IG5vbi10ZXN0LWZpbGUgIG5ldy1maWxlCiMKIyBUaGVuIHlvdSBjYW4gY29udGludWUgd2l0 aDoKIwojICAgZnRyYWNlLWJpc2VjdCBuZXctZmlsZSB0ZXN0LWZpbGUgbm9uLXRlc3QtZmlsZQoj ICAgY2F0IHRlc3QtZmlsZSA+IC9zeXMva2VybmVsL2RlYnVnL3RyYWNpbmcvc2V0X2Z0cmFjZV9m aWx0ZXIKIyAgICAgWyBhZ2FpbiwgdGhpcyBjYW4gdGFrZSB0aW1lIHRvIHJ1biBdCiMgICBlY2hv IGZ1bmN0aW9uID4gL3N5cy9rZXJuZWwvZGVidWcvdHJhY2luZy9jdXJyZW50X3RyYWNlcgojCiMg QW5kIHJlcGVhdC4KIwoKCmlmIFsgJCMgLW5lIDMgXTsgdGhlbgogIGVjaG8gJ3VzYWdlOiBmdHJh Y2UtYmlzZWN0IGZ1bGwtZmlsZSB0ZXN0LWZpbGUgIG5vbi10ZXN0LWZpbGUnCiAgZXhpdApmaQoK ZnVsbD0kMQp0ZXN0PSQyCm5vbnRlc3Q9JDMKCng9YGNhdCAkZnVsbCB8IHdjIC1sYAppZiBbICR4 IC1lcSAxIF07IHRoZW4KCWVjaG8gIlRoZXJlJ3Mgb25seSBvbmUgZnVuY3Rpb24gbGVmdCwgbXVz dCBiZSB0aGUgYmFkIG9uZSIKCWNhdCAkZnVsbAoJZXhpdCAwCmZpCgpsZXQgeD0keC8yCmxldCB5 PSR4KzEKCmlmIFsgISAtZiAkZnVsbCBdOyB0aGVuCgllY2hvICIkZnVsbCBkb2VzIG5vdCBleGlz dCIKCWV4aXQgMQpmaQoKaWYgWyAtZiAkdGVzdCBdOyB0aGVuCgllY2hvIC1uICIkdGVzdCBleGlz dHMsIGRlbGV0ZSBpdD8gW3kvTl0iCglyZWFkIGEKCWlmIFsgIiRhIiAhPSAieSIgLWEgIiRhIiAh PSAiWSIgXTsgdGhlbgoJCWV4aXQgMQoJZmkKZmkKCmlmIFsgLWYgJG5vbnRlc3QgXTsgdGhlbgoJ ZWNobyAtbiAiJG5vbnRlc3QgZXhpc3RzLCBkZWxldGUgaXQ/IFt5L05dIgoJcmVhZCBhCglpZiBb ICIkYSIgIT0gInkiIC1hICIkYSIgIT0gIlkiIF07IHRoZW4KCQlleGl0IDEKCWZpCmZpCgpzZWQg LW5lICIxLCR7eH1wIiAkZnVsbCA+ICR0ZXN0CnNlZCAtbmUgIiR5LFwkcCIgJGZ1bGwgPiAkbm9u dGVzdAo= --MP_/ByyDwrM+nvxqPNmwgt66cDE--