Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011.

[Theodore Tso <tytso@MIT.EDU>]

On Mar 2, 2011, at 8:52 PM, CAI Qian wrote:

> Those days, there just too many tests and testing projects for kernel like
> LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and
> then to extract the best?

Part of the problem is that every single testing project has different goals and
priorities.   For example xfstests is maintained by the XFS folks, as well as people
from some of the other file system development efforts (the ext4 one in particular,
thanks to people like Eric Sandeen), to test file systems.

At least at one point, I had heard a complaint that LTP was more focused on 
increasing test coverage as measured by a code coverage tool in the kernel
than it was about about covering edge conditions, or races.  There's nothing 
wrong with that, per se, and I don't know if it was true then or now, but it's a very
different focus from one which is focused increasing the data reliability of file
systems, quickly and efficiently.

And then there's the LSB test suites, which is really code at testing correctness
from a standards perspective, which is a different focus yet again from the LTP
and xfstests approach.

Bottom line, I'm a big fan of having different test suites, with different philosophies.
Each philosophy has its strengths and blind spots, and so a problem that might
be missed by one test suite might get caught by another.

The only real problem is an operational one.   There are some programs which
are used by both LTP and xfstests, and changes that is made in one, don't 
necessarily get propagated to the other unless someone manually does it.
But I think we can solve that without trying to merge all of these tests into a
single Grand Unified Test Suite.

-- Ted