Re: [QUESTION] Is there a better way to get ftrace dump on guest?

[Steven Rostedt <rostedt@goodmis.org>]

On Wed, 29 Jun 2016 09:57:41 +0900
Minchan Kim <minchan@kernel.org> wrote:

> Hello,
> 
> On Tue, Jun 28, 2016 at 06:46:34PM +0200, Rabin Vincent wrote:
> > On Tue, Jun 28, 2016 at 03:33:18PM +0900, Namhyung Kim wrote:  
> > > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@kernel.org> wrote:  
> > > > I'm running some guest machines for kernel development.  For debugging
> > > > purpose, I use lots of trace_printk() since it's faster than normal
> > > > printk().  When kernel crash happens the trace buffer is printed on
> > > > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > > > don't want to reduce the size of ring buffer as I want to collect the
> > > > debug info as much as possible.  And I also want to see trace from all
> > > > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> > > >
> > > > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > > > trace buffer later.  But it's cumbersome to do it everytime and more
> > > > importantly, I don't want to spend the memory for the crashkernel.  
> > 
> > Assuming you're using QEMU:
> > 
> > QEMU has a dump-guest-memory command which can be used to dump the
> > guest's entire memory to an ELF which can be loaded by the crash utility
> > to extract the trace buffer.  This doesn't require kexec/kdump or any
> > other support from the guest kernel.  
> 
> Thanks for the hint. It's surely handy rather than kexec/kdump.
> 
> A question is that it's possible to capture guest's entire memory
> when guest kernel is oops?
> I mean I don't want to capture alive guest but get snapshot image
> when guest kernel encounters BUG_ON and see event trace from the
> image.
> 
> Anyway, I tried crashtool and load trace.so but failed to load
> extension module 'trace.so' because read_string failed in
> ftrace_get_event_type_name of trace.c.
> Does it work with recent kernel?
> 
> My kernel is 4.7.0-rc4-mm1.

It probably needs another update. I usually send patches to David
Anderson for updates. Fujitsu started that work and was maintaining it
for a while, but I don't think they are anymore. I have no problem
maintaining the trace.so module.

If I get time tomorrow, I'll see if I can get it up to date again.

-- Steve


> 
> > 
> > It's apparently even possible to run QEMU with the guest memory in a
> > file and load that to crash directly, although this is not something
> > I've had a chance to try out myself:
> > 
> > https://github.com/crash-utility/crash/commit/89ed9d0a7f7da4578294a492c1ad857244ce7352