Re: RFC - Approaches to user-space probes

[Arjan van de Ven <arjan@infradead.org>]

On Mon, 2006-03-27 at 12:24 +0530, Prasanna S Panchamukhi wrote:
> Hi All,
> 
> As Andrew Morton suggested, here is a document on user-space probes
> discussing known approaches and design issues.
> 
> Please provide your comments and suggestions.
> 
> Thanks
> Prasanna
> ----
> 
> The basic need is to provide infrastructure for user-space dynamic
> instrumentation. As with kprobes, there is no need to recompile or
> restart the applications for instrumentation, under a debugger for
> instance.
> 
> Some of the use-cases are:
> 
> - To find out the memory leaks dynamically just by inserting probes on
>   malloc and free library routines.

for that you do need to do that from the start of an application, at
which point perfectly good tooling exists already; leak tracking without
full state is, well, not something that'll work too well I suspect.
Also I don't see really why this needs kernel help :)




> - Low overhead and user can have thousands of active probes on the
>   system and detect any instance when the probe was hit including
>   probes on shared library etc.

I suspect this is the only reason for doing it inside the kernel;
anything else still really shouts "do it in userspace via ptrace" to me.


> ===========================================================
> LOCAL PROBES(PER PROCESS) VS GLOBAL PROBES(EXECUTABLE FILE)
> ===========================================================
> 
> - All processes take a trap since the same executable file
>   gets mapped into different address_space.

is that true for breakpoints inserted after start?
The reason I ask because... what if half the processed took a COW on the
page with the instruction you want to trap on. Are you going to edit all
those COW'd pages?


Also you no longer have the option to only do it on a selected subset of
processes (eg the workload vs the system)

> - Compare this with ptrace breakpoints (hence strace and gdb) where
>   tracepoints and breakpoints are localized to a specified set of
>   processes.  To support local probes the text pages are privatized
>   for that process. Global user-probes does not have the side effects
>   (privatization of pages) that ptrace has.

No instead it gets to "deal" with that already having happened ;)


Overall I see only one possible reason to do this in the kernel:
performance. Anything else really suggests that a userspace approach is
more than reasonable to me. (It might not be always be super easy, but
on the other hand you gain a lot back by doing that, for example you
have a lot better backtrace and debuginformation there so that you can
do far more advanced probes like "probe only if the caller is ..." etc)