Re: [ANNOUNCE] VM Regress - A VM regression and test tool

[Daniel Phillips <phillips@arcor.de>]

On Monday 12 August 2002 03:40, Bernd Eckenfels wrote:
> In article <Pine.LNX.4.44.0208112109110.16360-100000@skynet> you wrote:
> > It works by using kernel modules to get a definite view of what the kernel
> > is at and to provide reliable, reproducible tests. Modules are divided
> > up into 4 catagories. Core modules provide infrastructure for the tool.
> > Sense modules tell what is going on in the VM. Test tests particular
> > features and bench modules (none yet) will benchmark different sections
> > of the VM.
> 
> This sounds more like a micro benchmark tool, which is a good start, but the
> real problem with VM optimizations is, that they have to take into account
> real world load and especially user experience.

We get too hung up on 'real world' world loads, that is not a productive way 
VM developers to spend their time.  Developers need to use tests that focus 
on very specific aspects of VM performance.  Yes, this testing should be 
backed up by 'real world' tests to confirm what the VM developer thinks, that 
improved performance on a subsystem translates into improved overall 
performance, and to keep a watch out for unexpected or undesirable 
interactions.  That's called a 'reality tests'.

If you want to help with 'interactive performance', i.e., user experience, 
then *quantify what contributes to that* and write a micro-measurement tool 
that measures such things.  E.g, latency of response to keyboard events under 
load.  It's not rocket science, it just takes time and effort to set this 
kind of thing up so it's accurate and predictive.

It's an incredible waste of developer's time to be running 'reality tests' 
all the time, and never using more precise measurement methods.  Anyone who 
wants to run reality tests and post the results is more than welcome to, and 
this is valuable.  It's not valuable to throw mud at a testing/measurement 
tool because you think it's not 'realistic'.

-- 
Daniel