Re: ftrace function_graph causes system crash

[Steven Rostedt <rostedt@goodmis.org>]

[-- Attachment #1: Type: text/plain, Size: 2382 bytes --]

On Tue, 20 Sep 2016 13:10:39 +0000
"Bean Huo (beanhuo)" <beanhuo@micron.com> wrote:

> Hi, all 
> I just use ftrace to do some latency study, found that function_graph can not
> Work, as long as enable it, will cause kernel panic. I searched this online.
> Found that there are also some cause the same as mine. I am a newer of ftrace. 
> I want to know who know what root cause? Here is some partial log:
> 
> 

Can you do a function bisect to find what function this is.

This script is used to help find functions that are being traced by
function tracer or function graph tracing that causes the machine to
reboot, hang, or crash. Here's the steps to take.

First, determine if function graph is working with a single function:

# cd /sys/kernel/debug/tracing
# echo schedule > set_ftrace_filter
# echo function_graph > current_tracer

If this works, then we know that something is being traced that
shouldn't be.

# echo nop > current_tracer

# cat available_filter_functions > ~/full-file
# ftrace-bisect ~/full-file ~/test-file ~/non-test-file
# cat ~/test-file > set_ftrace_filter

*** Note *** this will take several minutes. Setting multiple functions
is an O(n^2) operation, and we are dealing with thousands of functions.
So go have  coffee, talk with your coworkers, read facebook. And
eventually, this operation will end.

# echo function_graph > current_tracer

If it crashes, we know that ~/test-file has a bad function.

   Reboot back to test kernel.

   # cd /sys/kernel/debug/tracing
   # mv ~/test-file ~/full-file

If it didn't crash.

   # echo nop > current_tracer
   # mv ~/non-test-file ~/full-file

Get rid of the other test file from previous run (or save them off
somewhere.
# rm -f ~/test-file ~/non-test-file

And start again:

# ftrace-bisect ~/full-file ~/test-file ~/non-test-file

The good thing is, because this cuts the number of functions in
~/test-file by half, the cat of it into set_ftrace_filter takes half as
long each iteration, so don't talk so much at the water cooler the
second time.

Eventually, if you did this correctly, you will get down to the problem
function, and all we need to do is to notrace it.

The way to figure out if the problem function is bad, just do:

# echo <problem-function> > set_ftrace_notrace
# echo > set_ftrace_filter
# echo function_graph > current_tracer

And if it doesn't crash, we are done.

-- Steve

[-- Attachment #2: ftrace-bisect --]
[-- Type: application/octet-stream, Size: 1487 bytes --]

#!/bin/bash
#
# Here's how to use this:
#
# Start off by doing:
#
#   cat /sys/kernel/debug/tracing/available_filter_functions > full-file
#   ftrace-bisect full-file test-file non-test-file
#   cat test-file > /sys/kernel/debug/tracing/set_ftrace_filter
#
#    [ this may take several minutes to complete, be patient ]
#
#   echo function > /sys/kernel/debug/tracing/current_tracer
#
# If it crashes/locks-up whatever, then reboot and do:
#
#   mv test-file new-file
#
# Else, if it did not crash, then do:
#
#   mv non-test-file  new-file
#
# Then you can continue with:
#
#   ftrace-bisect new-file test-file non-test-file
#   cat test-file > /sys/kernel/debug/tracing/set_ftrace_filter
#     [ again, this can take time to run ]
#   echo function > /sys/kernel/debug/tracing/current_tracer
#
# And repeat.
#


if [ $# -ne 3 ]; then
  echo 'usage: ftrace-bisect full-file test-file  non-test-file'
  exit
fi

full=$1
test=$2
nontest=$3

x=`cat $full | wc -l`
if [ $x -eq 1 ]; then
	echo "There's only one function left, must be the bad one"
	cat $full
	exit 0
fi

let x=$x/2
let y=$x+1

if [ ! -f $full ]; then
	echo "$full does not exist"
	exit 1
fi

if [ -f $test ]; then
	echo -n "$test exists, delete it? [y/N]"
	read a
	if [ "$a" != "y" -a "$a" != "Y" ]; then
		exit 1
	fi
fi

if [ -f $nontest ]; then
	echo -n "$nontest exists, delete it? [y/N]"
	read a
	if [ "$a" != "y" -a "$a" != "Y" ]; then
		exit 1
	fi
fi

sed -ne "1,${x}p" $full > $test
sed -ne "$y,\$p" $full > $nontest