From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932849AbcIUHvv convert rfc822-to-8bit (ORCPT ); Wed, 21 Sep 2016 03:51:51 -0400 Received: from mailout.micron.com ([137.201.242.129]:56884 "EHLO mailout.micron.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932754AbcIUHvs (ORCPT ); Wed, 21 Sep 2016 03:51:48 -0400 From: "Bean Huo (beanhuo)" To: Steven Rostedt CC: "Zoltan Szubbocsev (zszubbocsev)" , "catalin.marinas@arm.com" , "will.deacon@arm.com" , "rfi@lists.rocketboards.org" , "linux-kernel@vger.kernel.org" , "mingo@redhat.com" , "linux-arm-kernel@lists.infradead.org" Subject: RE: ftrace function_graph causes system crash Thread-Topic: ftrace function_graph causes system crash Thread-Index: AdITP3reJ8G0agK7SIKsBha9PEINJP//i4MA//5SY9A= Date: Wed, 21 Sep 2016 07:50:58 +0000 Message-ID: <0e38f79fdc1f4671943da74764c15050@SIWEX5A.sing.micron.com> References: <7d0e88fff8e64591ac31be3adbbf763c@SIWEX5A.sing.micron.com> <20160920100716.131d3647@gandalf.local.home> In-Reply-To: <20160920100716.131d3647@gandalf.local.home> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.160.29.124] x-tm-as-product-ver: SMEX-12.0.0.1350-8.000.1202-22590.005 x-tm-as-result: No--50.889500-0.000000-31 x-tm-as-matchedid: 147014-150567-701625-704425-700685-700648-703523-139010-8 53550-700755-851106-704247-106660-708797-700075-110462-702358-863828-705388 -702898-186027-709584-702271-187067-710272-711664-105700-711624-701305-1360 70-702920-861157-121578-700481-121270-706117-708257-186035-700970-702113-11 3131-700702-702020-705947-701298-706249-106470-701594-707395-703747-701461- 705450-700107-702039-700208-136304-703283-148004-148133-20043-42000-42003 x-tm-as-user-approved-sender: Yes x-tm-as-user-blocked-sender: No x-mt-checkinternalsenderrule: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] > On Behalf Of Steven Rostedt > Sent: Dienstag, 20. September 2016 16:07 > To: Bean Huo (beanhuo) > Cc: Zoltan Szubbocsev (zszubbocsev) ; > catalin.marinas@arm.com; will.deacon@arm.com; rfi@lists.rocketboards.org; > linux-kernel@vger.kernel.org; mingo@redhat.com; linux-arm- > kernel@lists.infradead.org > Subject: Re: ftrace function_graph causes system crash > > On Tue, 20 Sep 2016 13:10:39 +0000 > "Bean Huo (beanhuo)" wrote: > > > Hi, all > > I just use ftrace to do some latency study, found that function_graph > > can not Work, as long as enable it, will cause kernel panic. I searched this > online. > > Found that there are also some cause the same as mine. I am a newer of > ftrace. > > I want to know who know what root cause? Here is some partial log: > > > > > > Can you do a function bisect to find what function this is. > > This script is used to help find functions that are being traced by function tracer > or function graph tracing that causes the machine to reboot, hang, or crash. > Here's the steps to take. > > First, determine if function graph is working with a single function: > > # cd /sys/kernel/debug/tracing > # echo schedule > set_ftrace_filter > # echo function_graph > current_tracer > > If this works, then we know that something is being traced that shouldn't be. > > # echo nop > current_tracer > > # cat available_filter_functions > ~/full-file # ftrace-bisect ~/full-file ~/test-file > ~/non-test-file # cat ~/test-file > set_ftrace_filter > > *** Note *** this will take several minutes. Setting multiple functions is an > O(n^2) operation, and we are dealing with thousands of functions. > So go have coffee, talk with your coworkers, read facebook. And eventually, > this operation will end. > > # echo function_graph > current_tracer > > If it crashes, we know that ~/test-file has a bad function. > > Reboot back to test kernel. > > # cd /sys/kernel/debug/tracing > # mv ~/test-file ~/full-file > > If it didn't crash. > > # echo nop > current_tracer > # mv ~/non-test-file ~/full-file > > Get rid of the other test file from previous run (or save them off somewhere. > # rm -f ~/test-file ~/non-test-file > > And start again: > > # ftrace-bisect ~/full-file ~/test-file ~/non-test-file > > The good thing is, because this cuts the number of functions in ~/test-file by half, > the cat of it into set_ftrace_filter takes half as long each iteration, so don't talk > so much at the water cooler the second time. > > Eventually, if you did this correctly, you will get down to the problem function, > and all we need to do is to notrace it. > > The way to figure out if the problem function is bad, just do: > > # echo > set_ftrace_notrace # echo > set_ftrace_filter # > echo function_graph > current_tracer > > And if it doesn't crash, we are done. > > -- Steve Hi, Steve Thanks very much! This is a very useful trace tool, I now know the problem function, It is gt_counter_read, if not trace this function, ftrace function_graph work well. Do you know now how to deeply debug and trace which line is wrong through Ftrace? --Bean