Re: RFC: starting a kernel-testers group for newbies

["Rafael J. Wysocki" <rjw@sisk.pl>]

On Wednesday, 30 of April 2008, Arjan van de Ven wrote:
> On Thu, 1 May 2008 14:30:38 +0300
> Adrian Bunk <bunk@kernel.org> wrote:
> 
> > On Wed, Apr 30, 2008 at 12:03:38AM -0700, Arjan van de Ven wrote:
> > > On Thu, 1 May 2008 03:31:25 +0300
> > > Adrian Bunk <bunk@kernel.org> wrote:
> > > 
> > > > On Wed, Apr 30, 2008 at 01:31:08PM -0700, Linus Torvalds wrote:
> > > > > 
> > > > > 
> > > > > On Wed, 30 Apr 2008, Andrew Morton wrote:
> > > > > > 
> > > > > > <jumps up and down>
> > > > > > 
> > > > > > There should be nothing in 2.6.x-rc1 which wasn't in
> > > > > > 2.6.x-mm1!
> > > > > 
> > > > > The problem I see with both -mm and linux-next is that they
> > > > > tend to be better at finding the "physical conflict" kind of
> > > > > issues (ie the merge itself fails) than the "code looks ok but
> > > > > doesn't actually work" kind of issue.
> > > > > 
> > > > > Why?
> > > > > 
> > > > > The tester base is simply too small.
> > > > > 
> > > > > Now, if *that* could be improved, that would be wonderful, but
> > > > > I'm not seeing it as very likely.
> > > > > 
> > > > > I think we have fairly good penetration these days with the
> > > > > regular -git tree, but I think that one is quite frankly a
> > > > > *lot* less scary than -mm or -next are, and there it has been
> > > > > an absolutely huge boon to get the kernel into the Fedora
> > > > > test-builds etc (and I _think_ Ubuntu and SuSE also started
> > > > > something like that).
> > > > > 
> > > > > So I'm very pessimistic about getting a lot of test coverage
> > > > > before -rc1.
> > > > > 
> > > > > Maybe too pessimistic, who knows?
> > > > 
> > > > First of all:
> > > > I 100% agree with Andrew that our biggest problems are in
> > > > reviewing code and resolving bugs, not in finding bugs (we
> > > > already have far too many unresolved bugs).
> > > 
> > > I would argue instead that we don't know which bugs to fix first.
> > > We're never going to fix all bugs, and to be honest, that's ok.
> > >...
> > 
> > That might be OK.
> > 
> > But our current status quo is not OK:
> > 
> > Check Rafael's regressions lists asking yourself
> > "How many regressions are older than two weeks?" 
> 
> "ext4 doesn't compile on m68k".
> YAWN.
> 
> Wrong question...
> "How many bugs that a sizable portion of users will hit in reality are there?"
> is the right question to ask...
> 
> 
> > 
> > We have unmaintained and de facto unmaintained parts of the kernel
> > where even issues that might be easy to fix don't get fixed.
> 
> And how many people are hitting those issues? If a part of the kernel is really
> important to enough people, there tends to be someone who stands up to either fix
> the issue or start de-facto maintaining that part.
> And yes I know there's parts where that doesn't hold. But to be honest, there's
> not that many of them that have active development (and thus get the biggest
> share of regressions)
> 
> > 
> > >...
> > > So there's a few things we (and you / janitors) can do over time to
> > > get better data on what issues people hit: 
> > > 1) Get automated collection of issues more wide spread. The wider
> > > our net the better we know which issues get hit a lot, and plain
> > > the more data we have on when things start, when they stop, etc
> > > etc. Especially if you get a lot of testers in your project, I'd
> > > like them to install the client for easy reporting of issues. 2) We
> > > should add more WARN_ON()s on "known bad" conditions. If it
> > > WARN_ON()'s, we can learn about it via the automated collection.
> > > And we can then do the statistics to figure out which ones happen a
> > > lot. 3) We need to get persistent-across-reboot oops saving going;
> > > there's some venues for this
> > 
> > No disagreement on this, its just a different issue than our bug
> > fixing problem.
> 
> No it's not! Knowing earlier and better which bugs get hit is NOT different
> to our bug fixing "problem", it's in fact an essential part to the solution of it!

Agreed.

Thanks,
Rafael