Re: Giving developers clue how many testers verified certain kernel version

Re: Giving developers clue how many testers verified certain kernel version

[Blaisorblade]

Adrian Bunk <bunk <at> stusta.de> writes:
> On Thu, Jul 21, 2005 at 09:40:43PM -0500, Alejandro Bonilla wrote:
> > 
> >    How do we know that something is OK or wrong? just by the fact that 
> > it works or not, it doesn't mean like is OK.
> > 
> > There has to be a process for any user to be able to verify and study a 
> > problem. We don't have that yet.

> If the user doesn't notice the difference then there's no problem for 
> him.
Some performance regressions aren't easily noticeable without benchmarks... 
and we've had people claiming unnoticed regressions since 2.6.2 
(http://kerneltrap.org/node/4940)
> If there's a problem the user notices, then the process is to send an 
> email to linux-kernel and/or open a bug in the kernel Bugzilla and 
> follow the "please send the output of foo" and "please test patch bar" 
> instructions.

> What comes nearest to what you are talking about is that you run LTP 
> and/or various benchmarks against every -git and every -mm kernel and 
> report regressions. But this is sinply a task someone could do (and I 
> don't know how much of it is already done e.g. at OSDL), and not 
> something every user could contribute to.

Forgot drivers testing? That is where most of the bugs are hidden, and where 
wide user testing is definitely needed because of the various hardware bugs 
and different configurations existing in real world.

IMHO, I think that publishing statistics about kernel patches downloads would 
be a very Good Thing(tm) to do. Peter, what's your opinion? I think that was 
even talked about at Kernel Summit (or at least I thought of it there), but 
I've not understood if this is going to happen.
-- 
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

Re: Giving developers clue how many testers verified certain kernel version

[David Lang]

On Sat, 23 Jul 2005, Blaisorblade wrote:

> IMHO, I think that publishing statistics about kernel patches downloads would
> be a very Good Thing(tm) to do. Peter, what's your opinion? I think that was
> even talked about at Kernel Summit (or at least I thought of it there), but
> I've not understood if this is going to happen.

remember that most downloads will be from mirrors, and they don't get 
stats from them.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: Giving developers clue how many testers verified certain kernel version

[H. Peter Anvin]

David Lang wrote:
> On Sat, 23 Jul 2005, Blaisorblade wrote:
> 
>> IMHO, I think that publishing statistics about kernel patches 
>> downloads would
>> be a very Good Thing(tm) to do. Peter, what's your opinion? I think 
>> that was
>> even talked about at Kernel Summit (or at least I thought of it 
>> there), but
>> I've not understood if this is going to happen.
> 
> 
> remember that most downloads will be from mirrors, and they don't get 
> stats from them.
> 
> David Lang
> 

That, plus there is http+ftp+rsync (not to mention git downloads, etc.) 
and the noise caused by other sites mirroring *us*.

	-hpa

Re: Giving developers clue how many testers verified certain kernel version

[Alejandro Bonilla]

Blaisorblade wrote:

>Adrian Bunk <bunk <at> stusta.de> writes:
>  
>
>>On Thu, Jul 21, 2005 at 09:40:43PM -0500, Alejandro Bonilla wrote:
>>    
>>
>>>   How do we know that something is OK or wrong? just by the fact that 
>>>it works or not, it doesn't mean like is OK.
>>>
>>>There has to be a process for any user to be able to verify and study a 
>>>problem. We don't have that yet.
>>>      
>>>
>
>  
>
>>If the user doesn't notice the difference then there's no problem for 
>>him.
>>    
>>
>Some performance regressions aren't easily noticeable without benchmarks... 
>and we've had people claiming unnoticed regressions since 2.6.2 
>(http://kerneltrap.org/node/4940)
>  
>
I will get flames for this, but my laptop boots faster and sometimes 
responds faster in 2.4.27 than in 2.6.12. Sorry, but this is the fact 
for me. IBM T42.

>>If there's a problem the user notices, then the process is to send an 
>>email to linux-kernel and/or open a bug in the kernel Bugzilla and 
>>follow the "please send the output of foo" and "please test patch bar" 
>>instructions.
>>    
>>
The thing is, I might not be able to know there *are* issues. I most 
just notice that something is strange. And then wait for a new kernel 
version because i might think it is something silly.

>
>  
>
>>What comes nearest to what you are talking about is that you run LTP 
>>and/or various benchmarks against every -git and every -mm kernel and 
>>report regressions. But this is sinply a task someone could do (and I 
>>don't know how much of it is already done e.g. at OSDL), and not 
>>something every user could contribute to.
>>    
>>
>
>Forgot drivers testing? That is where most of the bugs are hidden, and where 
>wide user testing is definitely needed because of the various hardware bugs 
>and different configurations existing in real world.
>  
>
This is my opinion too. If someone could do a simple script or 
benchmarking file, then users would be able to report most common 
important differences from previous kernel versions on their systems.

i.e. i would run the script that checks the write speed, CPU, latencys, 
and I don't know how many more tests and then compare it with the 
results that were with the previous git or full kernel release. 
Sometimes the users don't even know the commands to benchmark this parts 
of the systems. I don't know them.

>IMHO, I think that publishing statistics about kernel patches downloads would 
>be a very Good Thing(tm) to do. Peter, what's your opinion? I think that was 
>even talked about at Kernel Summit (or at least I thought of it there), but 
>I've not understood if this is going to happen.
>  
>
What can we do here? Can we probably create a project like the janitors 
so that we can report this kind of thing? Should we report here? How can 
we make a script to really benchmark the system and then say, since this 
guy sent a patch for the Pentium M CPU's, things are running slower? Or 
my SCSI drive is running slower since rc2, but not with rc1.

At least if the user notices this kind of things, then one will be able 
to google for patches for your controller for the last weeks and see if 
someone screwed up with a change they sent to the kernel.

In other words, kernel testing is not really easy for normal users, it 
can only really be benchmarked by the one that knows... Which are many, 
but not everyone.

And I really want to give my 2 cent on this.

.Alejandro

Re: Giving developers clue how many testers verified certain kernel version

[Lee Revell]

On Fri, 2005-07-22 at 20:07 -0500, Alejandro Bonilla wrote:
> I will get flames for this, but my laptop boots faster and sometimes 
> responds faster in 2.4.27 than in 2.6.12. Sorry, but this is the fact 
> for me. IBM T42.

Sorry dude, but there's just no way that any automated process can catch
these.

You will have to provide a detailed bug report (with numbers) like
everyone else so we can fix it.  "Waiting for it to fix itself" is the
WORST thing you can do.

If you find a regression vs. an earlier kernel, please assume that
you're the ONLY one to notice it and respond accordingly.

Lee

Re: Giving developers clue how many testers verified certain kernel version

[Alejandro Bonilla]

Lee Revell wrote:

>On Fri, 2005-07-22 at 20:07 -0500, Alejandro Bonilla wrote:
>  
>
>>I will get flames for this, but my laptop boots faster and sometimes 
>>responds faster in 2.4.27 than in 2.6.12. Sorry, but this is the fact 
>>for me. IBM T42.
>>    
>>
>
>Sorry dude, but there's just no way that any automated process can catch
>these.
>  
>
I'm not looking for an automated process for this. But for all in 
general, when moving from 2.6.11 to 2.6.12 or from any version to 
another. (At least in the same kernel branch)

>You will have to provide a detailed bug report (with numbers) like
>everyone else so we can fix it.  "Waiting for it to fix itself" is the
>WORST thing you can do.
>  
>
I never do this, believe me, but I could if I don't really see a 
problem. But there could really be one behind.

>If you find a regression vs. an earlier kernel, please assume that
>you're the ONLY one to notice it and respond accordingly.
>  
>
OK, I will, but I first of all need to learn how to tell if benchmarks 
are better or worse.

>Lee
>
>  
>
.Alejandro

Re: Giving developers clue how many testers verified certain kernel version

[Lee Revell]

On Fri, 2005-07-22 at 21:15 -0500, Alejandro Bonilla wrote:
> OK, I will, but I first of all need to learn how to tell if benchmarks 
> are better or worse.

Con's interactivity benchmark looks quite promising for finding
scheduler related interactivity regressions.  It certainly has confirmed
what we already knew re: SCHED_FIFO performance, if we extend that to
SCHED_OTHER which is a more interesting problem then there's serious
potential for improvement.  AFAIK no one has posted any 2.4 vs 2.6
interbench results yet...

I suspect a lot of the boot time issue is due to userspace.  But, it
should be trivial to benchmark this one, just use the TSC or whatever to
measure the time from first kernel entry to execing init().

Lee 

Re: Giving developers clue how many testers verified certain kernel version

[Alejandro Bonilla]

Lee Revell wrote:

>On Fri, 2005-07-22 at 21:15 -0500, Alejandro Bonilla wrote:
>  
>
>>OK, I will, but I first of all need to learn how to tell if benchmarks 
>>are better or worse.
>>    
>>
>
>Con's interactivity benchmark looks quite promising for finding
>scheduler related interactivity regressions.  It certainly has confirmed
>what we already knew re: SCHED_FIFO performance, if we extend that to
>SCHED_OTHER which is a more interesting problem then there's serious
>potential for improvement.  AFAIK no one has posted any 2.4 vs 2.6
>interbench results yet...
>  
>
I will give it a try.

>I suspect a lot of the boot time issue is due to userspace.  But, it
>should be trivial to benchmark this one, just use the TSC or whatever to
>measure the time from first kernel entry to execing init().
>  
>
You got it! As a laptop user, I think it just takes too much more. I 
think it is maybe hotplugs fault with the kernel? I don't know how much 
is done by the kernel or userspace but it definitely takes longer.

I could do some sort of benchmarks, but believe me, I hate to say this, 
but I use 2.6 because of much more power managements features in it. 
Else I like 2.4 a lot more. Is like, the feels is sharper. Sometimes 
when I got into a tty1, it takes some time after I put my username in to 
prompt me for a password. This does not occur when I boot with 2.4.27. 
Strange huh?

I don't want to be an ass and say that 2.4 is better, instead I want to 
help and let determine why is it that I feel 2.6 slower.

.Alejandro

>Lee 
>
>
>  
>

Re: Giving developers clue how many testers verified certain kernel version

[Linus Torvalds]

On Fri, 22 Jul 2005, Lee Revell wrote:
> 
> Con's interactivity benchmark looks quite promising for finding
> scheduler related interactivity regressions.

I doubt that _any_ of the regressions that are user-visible are
scheduler-related. They all tend to be disk IO issues (bad scheduling or
just plain bad drivers), and then sometimes just VM misbehaviour.

People are looking at all these RT patches, when the thing is that most
nobody will ever be able to tell the difference between 10us and 1ms
latencies unless it causes a skip in audio.

		Linus

Re: Giving developers clue how many testers verified certain kernel version

[Alejandro Bonilla]

Linus Torvalds wrote:

>On Fri, 22 Jul 2005, Lee Revell wrote:
>  
>
>>Con's interactivity benchmark looks quite promising for finding
>>scheduler related interactivity regressions.
>>    
>>
>
>I doubt that _any_ of the regressions that are user-visible are
>scheduler-related. They all tend to be disk IO issues (bad scheduling or
>just plain bad drivers), and then sometimes just VM misbehaviour.
>
>People are looking at all these RT patches, when the thing is that most
>nobody will ever be able to tell the difference between 10us and 1ms
>latencies unless it causes a skip in audio.
>  
>
True, and I just couldn't agree more with Lee that lots of the delays 
that one looks at is because of user space. Still, I have some doubt on 
how faster 2.6 is sometimes, where 2.4 is faster in other things.

i.e. As my newbie view, I can see 2.6 running faster in X, Compiling and 
stuff, but I see 2.4 working much faster when running commands, response 
and interaction in the console. But then again, this could be only me...


>		Linus
>
>  
>
.Alejandro

Re: Giving developers clue how many testers verified certain kernel version

[Lee Revell]

On Fri, 2005-07-22 at 20:31 -0700, Linus Torvalds wrote:
> 
> On Fri, 22 Jul 2005, Lee Revell wrote:
> > 
> > Con's interactivity benchmark looks quite promising for finding
> > scheduler related interactivity regressions.
> 
> I doubt that _any_ of the regressions that are user-visible are
> scheduler-related. They all tend to be disk IO issues (bad scheduling or
> just plain bad drivers), and then sometimes just VM misbehaviour.
> 
> People are looking at all these RT patches, when the thing is that most
> nobody will ever be able to tell the difference between 10us and 1ms
> latencies unless it causes a skip in audio.

I agree re: the RT patches, but what makes Con's benchmark useful is
that it also tests interactivity (measuring in msecs vs. usecs) with
everything running SCHED_NORMAL, which is a much better approximation of
a desktop load.  And the numbers do go well up into the range where
people would notice, tens and hundreds of ms.

Lee

Re: Giving developers clue how many testers verified certain kernel version

[Con Kolivas]

On Sat, 23 Jul 2005 01:34 pm, Lee Revell wrote:
> On Fri, 2005-07-22 at 20:31 -0700, Linus Torvalds wrote:
> > On Fri, 22 Jul 2005, Lee Revell wrote:
> > > Con's interactivity benchmark looks quite promising for finding
> > > scheduler related interactivity regressions.
> >
> > I doubt that _any_ of the regressions that are user-visible are
> > scheduler-related. They all tend to be disk IO issues (bad scheduling or
> > just plain bad drivers), and then sometimes just VM misbehaviour.
> >
> > People are looking at all these RT patches, when the thing is that most
> > nobody will ever be able to tell the difference between 10us and 1ms
> > latencies unless it causes a skip in audio.
>
> I agree re: the RT patches, but what makes Con's benchmark useful is
> that it also tests interactivity (measuring in msecs vs. usecs) with
> everything running SCHED_NORMAL, which is a much better approximation of
> a desktop load.  And the numbers do go well up into the range where
> people would notice, tens and hundreds of ms.

Indeed, and the purpose of the benchmark is to quantify something rather than 
leave it to subjective feeling. Fortunately if I was to quantify the current 
kernel's situation I would say everything is fine.

Cheers,
Con

Re: Giving developers clue how many testers verified certain kernel version

[Lee Revell]

On Sat, 2005-07-23 at 19:05 +1000, Con Kolivas wrote:
> Indeed, and the purpose of the benchmark is to quantify something rather than 
> leave it to subjective feeling. Fortunately if I was to quantify the current 
> kernel's situation I would say everything is fine.

Agreed.  Unfortunately everything in userspace is not fine.  I think a
lot of these interactivity problems are due to X and broken/bloated
apps.

Lee

RE: Giving developers clue how many testers verifiedcertain kernel version

[Al Boldi]

Lee Revell wrote: {
On Fri, 2005-07-22 at 21:15 -0500, Alejandro Bonilla wrote:
> OK, I will, but I first of all need to learn how to tell if benchmarks 
> are better or worse.

Con's interactivity benchmark looks quite promising for finding scheduler
related interactivity regressions.
}

Scheduler performance does not imply net system performance.

In fact, a well tuned scheduler hides absolute performance-related issues!

--Al

Re: Giving developers clue how many testers verified certain kernel version

[Adrian Bunk]

On Fri, Jul 22, 2005 at 09:15:14PM -0500, Alejandro Bonilla wrote:
> Lee Revell wrote:
> 
> >On Fri, 2005-07-22 at 20:07 -0500, Alejandro Bonilla wrote:
> > 
> >>I will get flames for this, but my laptop boots faster and sometimes 
> >>responds faster in 2.4.27 than in 2.6.12. Sorry, but this is the fact 
> >>for me. IBM T42.
> >
> >Sorry dude, but there's just no way that any automated process can catch
> >these.
> >
> I'm not looking for an automated process for this. But for all in 
> general, when moving from 2.6.11 to 2.6.12 or from any version to 
> another. (At least in the same kernel branch)
>...

You send:
- a problem description X
- tell that the last working kernel was Y
- tell that it is broken in kernel Z

The probability of any kernel developer being interested in your problem 
increases:
- the better the description X is
- the nearer versions Y and Z are together
- the more recent version Y is

Ideally, you are able to say that patch A in the latest -mm kernel
broke it.

It's perfectly OK to send a description X that says:
- with version Y and the following workload B, everything is working
  perfectly
- with version Z and the same workload B, XMMS is stuttering

If any kernel developer is interested in your bug report, he will tell 
you which data might be interesting for debugging the problem.

The problem is that debugging a problem often requires knowledge about 
possible causes and changes between versions Y and Z in this area. Even 
a kernel developer who perfectly knows one part of the kernel might not 
be able to debug a problem in a completely different area of the kernel.

> .Alejandro

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Giving developers clue how many testers verified certain kernel version

[Jesper Krogh]

I gmane.linux.kernel, skrev Blaisorblade:
>  Forgot drivers testing? That is where most of the bugs are hidden, and where 
>  wide user testing is definitely needed because of the various hardware bugs 
>  and different configurations existing in real world.

A way that could raise the testing upon a particular kernel, would be to
provide; (debian example follows):
... example .. 
An apt-repository with the newest tagged kernel build modular for the
architecture. 

Just drop all tagged kernels in a common repository that the users can
follow, then I'd be happy to test a new kernel on every reboot on my
system. I'd probably still would respond if anything was broken in the
new kernel.. 

Then it wouldn't be: "try this patch and see if that solves anything"
but do:

apt-get install kernel-image-386-torvalds-linux-2.6-v2.6.13-rc3

(automatically build from the "torvalds/linux-2.6"-branch with tag
"v2.6.13-rc3" using a modular kernel-configuration similar to the one
used in the stock debian kernels.  

Then I find and report something and "Pavel Machek" releases a "try-fix", by
tagging a branch ind a tree and tells me to try
kernel-image-386-pavel-good-2.6-v2.6.13-rc3 
instead. 

(and variations.. acip/no-acip smp, etc. etc. )

... example end .. 


It would be quite a lot central kernel-building, but as far as I can
see, it can be fully automated. 

It would defininately lower the barrier for being able to paticipate in
testing, but I am not the one to decide if that would be a desirable
goal?  Or for that matter, worth the work. 

Jesper
-- 
./Jesper Krogh, jesper@krogh.cc, Jabber ID: jesper@jabbernet.dk