Re: tracepoint maintainance models

[Ingo Molnar <mingo@elte.hu>]

* Paul Mundt <lethal@linux-sh.org> wrote:

> What exactly are you trying to prove with this? Yes, people aren't 
> opposed to a lightweight marker facility. Ingo made some suggestions 
> regarding that, and others (Andrew, Martin, etc.) have pointed out 
> that this would also be beneficial for certain use cases. I don't see 
> anyone violently opposed to lightweight markers, I see people 
> violently opposed to the ltt-centric breed of static instrumentation 
> (and yes, I'm one of them), let's not confuse the two.

yes. The way i see this whole issue (and what i've been trying argue for 
a long time) is that with dynamic tracers we have a _continuum_ of 
_tracepoint maintainance models_ that maintainers can choose from, each 
of which model gives the same "end-user experience":

  - model #1: we could have all static markers in the main kernel 
    source. No dynamic markups at all.

  - model #2: we could have the least intrusive markers in the main
    kernel source, while the more intrusive ones would still be in the
    upstream kernel, but in scripts/instrumentation/.

  - model #3: we could have the 'hardest' markups in the source, and the 
    'easy' ones as dynamic markups in scripts/instrumentation/.

  - model #4: we could have each and every tracepoint in 
    scripts/intrumentation/ - none in the main source.

Note that each model has a different maintainance tradeoff. In my 
judgement model #2 is the one with the smallest total maintainance cost, 
but we dont _have to_ make a hard decision about this here and now. Not 
having to do a (potentially wrong) maintainance-model decision is always 
good!

These tracepoint models arent even global, they can and should be 
per-subsystem. A seldom changing subsystem could have all its markers 
right embedded in the main kernel source. A subsystem under active 
development will most likely not have many markers (because they are 
just a hindrance when doing high-frequency updates).

The tracepoint model is not only per-subsystem, it can also change in 
time. If a subsystem goes through heavy changes (due to a rewrite), it 
might remove all of its static markups and move all the tracing 
infrastructure into scripts. Once the rate of changes has 'cooled down', 
the tracepoints can move back into the source again.

Furthermore, since there is no end-user visible impact of these "where 
should the markups be" decisions, the decisions will be made on a pure 
technical basis. Nobody will flame anyone about having a particular 
static marker moved to a script, because it's only an implementational 
(performance and maintainance micro-overhead) issue, not a functionality 
issue. In fact, with dynamic tracers, an end-user visible breakage can 
even be fixed _after the main kernel has been released, compiled and 
booted on the end-user's system_. Systemtap scripts can be updated on 
live systems. So there is very, very little maintainance pressure caused 
by dynamic tracing.

On the other hand, if we accept static tracers into the mainline kernel, 
we have to decide in favor of tracepoint-maintainance model #1 
_FOREVER_. It will be a point of no return for a likely long time. 
Moving a static tracepoint or even breaking it will cause end-user pain 
that needs an _upstream kernel fix_. It needs a new stable kernel, etc., 
etc. It is very inflexible, and fundamentally so.

So my argument isnt "dynamic markup vs. static markup", my argument is: 
"we shouldnt force the kernel to carry a 100% set of static markups 
forever". We should allow maintainers to decide the 'mix' of static vs. 
dynamic markups that they prefer in their subsystem.

And i might even be proven wrong in a few years, maybe all tracepoints 
will be static markups in the source. I strongly doubt it, but still 
it's a possibility, it wouldnt be the first time i'm wrong. In that case 
we'd still have the same functionality (sans a few rarer arches that 
done have kprobes, yet). But one thing is sure: if i'm just 20% right, 
we'll be much worse off with all the static tracer dependencies.

> This thread would be much better off talking about how to go about 
> implementing lightweight markers rather than spent on mindless rants.

i agree, as long as it's lightweight markers for _dynamic tracers_, so 
that we keep our options open - as per the arguments above.

	Ingo