Re: FTRACE_WARN_ON((rec->flags & ~FTRACE_FL_MASK) == 0))

[Steven Rostedt <rostedt@goodmis.org>]

On Mon, 2013-09-30 at 13:12 -0400, Dave Jones wrote:
> On Wed, Aug 28, 2013 at 03:27:45PM -0400, Steven Rostedt wrote:
> 
>  > > [ 6619.050768] WARNING: CPU: 1 PID: 16351 at kernel/trace/ftrace.c:1640 __ftrace_hash_rec_update.part.37+0x20a/0x240()
>  > > [ 6619.053767] Modules linked in: lec snd_seq_dummy bridge stp fuse tun bnep hidp rfcomm nfnetlink ipt_ULOG scsi_transport_iscsi can_bcm can_raw nfc caif_socket caif af_802154 phonet af_rxrpc bluetooth rfkill can llc2 pppoe pppox ppp_generic slhc irda crc_ccitt rds af_key rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 xfs libcrc32c snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc e1000e ptp snd_timer pcspkr pps_core snd soundcore usb_debug
>  > > [ 6619.060523] CPU: 1 PID: 16351 Comm: trinity-child2 Not tainted 3.11.0-rc7+ #31 
>  > > [ 6619.062161]  ffffffff81a21901 ffff8802267b9ce0 ffffffff816f9e4f 0000000000000000
>  > > [ 6619.063733]  ffff8802267b9d18 ffffffff81052dcd 0000000000000000 0000000000000001
>  > > [ 6619.065309]  ffff8802203a3000 0000000000000000 ffff880225e962d0 ffff8802267b9d28
>  > > [ 6619.066895] Call Trace:
>  > > [ 6619.068437]  [<ffffffff816f9e4f>] dump_stack+0x54/0x74
>  > > [ 6619.070046]  [<ffffffff81052dcd>] warn_slowpath_common+0x7d/0xa0
>  > > [ 6619.071642]  [<ffffffff81052eaa>] warn_slowpath_null+0x1a/0x20
>  > > [ 6619.073224]  [<ffffffff81115d1a>] __ftrace_hash_rec_update.part.37+0x20a/0x240
>  > > [ 6619.074817]  [<ffffffff81117e18>] ftrace_shutdown+0xb8/0x160
>  > > [ 6619.076399]  [<ffffffff811182a0>] unregister_ftrace_function+0x30/0x50
>  > > [ 6619.077983]  [<ffffffff81135e57>] perf_ftrace_event_register+0x87/0x150
>  > > [ 6619.079565]  [<ffffffff81135cdc>] perf_trace_destroy+0x2c/0x50
>  > > [ 6619.081180]  [<ffffffff8113df49>] tp_perf_event_destroy+0x9/0x10
>  > > [ 6619.082742]  [<ffffffff81140527>] free_event+0xa7/0x300
>  > > [ 6619.084264]  [<ffffffff81141620>] __perf_event_exit_task+0xe0/0x130
>  > > [ 6619.085792]  [<ffffffff8114a491>] perf_event_exit_task+0x1f1/0x230
>  > > [ 6619.087329]  [<ffffffff810546dd>] do_exit+0x30d/0xcd0
>  > > [ 6619.088860]  [<ffffffff8170cfc0>] ? ftrace_call+0x5/0x2f
>  > > [ 6619.090460]  [<ffffffff8105643c>] do_group_exit+0x4c/0xc0
>  > > [ 6619.092036]  [<ffffffff810564c4>] SyS_exit_group+0x14/0x20
>  > > [ 6619.093614]  [<ffffffff8170d594>] tracesys+0xdd/0xe2
>  >
>  > The first failure disables function tracing completely, which is one of
>  > the causes to return an error, which triggered the warning I added.
>  > 
>  > I was hoping that it failed for another reason and then that would
>  > cause things to break. But you said the hash corruption happened first
>  > (the original bug), so that's not the case.
>  > 
>  > Back to staring at code.
> 
> Steve, any further thoughts on this ? I can't do more than a few hours of fuzz-testing
> without eventually running into this.

I'm currently traveling, but as it seems that you have a pretty good
reproducer, can trace what trinity does up to the point it crashes?

I think we can do that with other buffers. If you download the latest
trace-cmd, you can run:

trace-cmd record -e syscalls -B trinity trinity <args>

This will create a ftrace sub buffer called trinity and shouldn't be
affected by the other ftrace functions. Then it should trace all the
syscalls that trinity does. I guess a strace could be used too.

-- Steve