Re: Infrastructure for tracking driver performance events

[Mark Wielaard <mjw@redhat.com>]

Hi,

On Thu, 2009-06-25 at 08:55 -0400, Steven Rostedt wrote:
> On Wed, 24 Jun 2009, Ben Gamari wrote:
> > 3) A large amount of usermode support is necessary (which will likely be
> >    the case for any option; listed here for completeness)
> > 
> > Another option seems to be systemtap. It has already been documented[3]
> > that this option could provide both user-mode and kernel-mode
> > backtraces. The driver could provide a kernel marker at every potential
> > wait point (or a single marker in a function called at each wait point,
> > for that matter) which would be picked up by systemtap and processed in
> > usermode, calling ptrace to acquire a usermode backtrace. This approach
> > seems slightly cleaner as it doesn't require the tracing on the entire
> > machine to catch what should be reasonably rare events (hopefully).
> 
> Enabling the userstacktrace will give userspace stack traces at event
> trace points. The thing is that the userspace utility must be built with 
> frame pointers.

This isn't true for Systemtap. It can unwind through anything since it
contains a dwarf-unwinder that can do backtraces as long as unwind
tables are available for the modules (executables, vdso, shared
libraries, etc.) one wants to unwind through. Systemtap currently gets
these in its "translation" phase and you do need to list them explicitly
atm. There is work underway to make this more flexible and automatic.
Also cross kernel-user-space backtraces need some work (systemtap can
use the dwarf unwinder also in-kernel, but some kernel parts are missing
unwind tables).

Some systemtap bugs to track if you are interested in extending this
functionality:

= Prerequirements for more ubiquitous backtracing
sw#6961 backtrace from non-pt_regs probe context
http://sourceware.org/bugzilla/show_bug.cgi?id=6961
sw#10080 track vdso for process symbols/backtrace
http://sourceware.org/bugzilla/show_bug.cgi?id=10080
sw#10208 Support probing glibc synthesized syscall wrappers
http://sourceware.org/bugzilla/show_bug.cgi?id=10208

NOTE: the above still won't make cross kernel-to-userspace backtracing
fully work since we cannot easily unwind through the kernel-entry/exit
assembly code that doesn't have dwarf unwind tables.

= Make user backtraces more convenient
sw#10228 Add more vma-tracking for user space symbol/backtraces
http://sourceware.org/bugzilla/show_bug.cgi?id=10228
sw#6580 revamp backtrace-related tapset functions
http://sourceware.org/bugzilla/show_bug.cgi?id=6580

Cheers,

Mark