Re: 2.4.18pre4aa1

[Daniel Phillips <phillips@bonn-fries.net>]

On January 25, 2002 01:09 am, Andrea Arcangeli wrote:
> On Thu, Jan 24, 2002 at 07:27:43AM +0100, Daniel Phillips wrote:
> > On January 24, 2002 06:23 am, rwhron@earthlink.net wrote:
> > > Benchmarks on 2.4.18pre4aa1 and lots of other kernels at:
> > > http://home.earthlink.net/~rwhron/kernel/k6-2-475.html
> > 
> >   "dbench 64, 128, 192 on ext2fs. dbench may not be the best I/O 
> >   benchmark, but it does create a high load, and may put some pressure on 
> >   the cpu and i/o schedulers. Each dbench process creates about 21 
> >   megabytes worth of files, so disk usage is 1.3 GB, 2.6 GB and 4.0 GB 
> >   for the dbench runs. Big enough so the tests cannot run from the 
> >   buffer/page caches on this box."
> > 
> > Thanks kindly for the testing, but please don't use dbench any more for 
> > benchmarks.  If you are testing stability, fine, but dbench throughput 
> > numbers are not good for much more than wild goose chases.
> > 
> > Even when mostly uncached, dbench still produces flaky results.
> 
> this is not enterely true. dbench has a value.

Yes, but not for benchmarks.  It has value only as a stability test - while 
it may in some cases provide some general indication of performance, its 
variance is far too large, even under controlled conditions, for it to have 
any value as a benchmark.  I'm surprised you'd even suggest this.

Andrea, please, if we want good benchmarks let's at least be clear on what 
tools benchmarkers should/should not be using.

> the only problem with
> dbench is that you can trivially cheat and change the kernel in a broken
> way, but optimal _only_ for dbench, just to get stellar dbench numbers,

No, this is not the only problem.  DBench is just plain *flaky*.  You don't
appear to be clear on why.  In short, dbench has two main flaws:

  - It's extremely sensitive to scheduling.  If one process happens to make
    progress then it gets more heavily cached and its progress becomes even
    greater.  The benchmark completes much more quickly in this case, whereas
    if all processes progress at nearly the same rate (by chance) it runs
    more slowly.

  - It can happen (again by chance) that dbench files get deleted while still
    in cache, and this process completes in a fraction of the time that real 
    disk IO would require.

I've seen successsive runs of dbench *under identical conditions* (that is, 
from a clean reboot etc.) vary by as much as 30%.  Others report even greater 
variance.  Can we please agree that dbench is useless for benchmarks?

-- 
Daniel