Introduce a New Metrics to measure Load average.

Introduce a New Metrics to measure Load average.

[sena seneviratne]

Dear Friends,
Please give us your valuable comments on this important change to introduce 
a new Metric to measure Load average.
Currently /proc/loadavg reports only the resultant value.

We are doing a scheduling in the Grid project. As a part of that we had to 
do some changes to the kernel

The  problem with the load metric of current Linux/Unix is that it measures 
CPU load and Disk load without indicating the true nature of the load, 
thereby creating some confusion among the readers. For example, if a CPU 
bound task switches on to read a large chunk of disk data, then the load 
average value would still continue to indicate this activity as a load, yet 
the true CPU load during this period would have been zero. This situation 
triggered us to make necessary additions to the kernel so that CPU load and 
Disk load could be reported separately. Further the specialisation of load 
helped our model to perform predictions when there is interference between 
CPU and Disk IO loads.

In the user mode, a new proc file called /proc/loadavgus would collect the 
new data according to a new format which would look like the following,

                 CPU    Disk
Root            0.7     0
User1   0.9     1
User2   0.9     0
User3   1.03    1
User4   0.93    0
User5   1.0     0

What do you think about this change?

Thanks
Sena Seneviratne
Computer Engineering Lab
School of Electrical and Information Engineering
Sydney University
Australia

Re: Introduce a New Metrics to measure Load average.

[Erik Mouw]

On Wed, Jun 14, 2006 at 03:12:29PM +1000, sena seneviratne wrote:
> The  problem with the load metric of current Linux/Unix is that it measures 
> CPU load and Disk load without indicating the true nature of the load, 
> thereby creating some confusion among the readers. For example, if a CPU 
> bound task switches on to read a large chunk of disk data, then the load 
> average value would still continue to indicate this activity as a load, yet 
> the true CPU load during this period would have been zero.

Right, we've seen such things with busy servers.

> This situation 
> triggered us to make necessary additions to the kernel so that CPU load and 
> Disk load could be reported separately. Further the specialisation of load 
> helped our model to perform predictions when there is interference between 
> CPU and Disk IO loads.

OK.

> In the user mode, a new proc file called /proc/loadavgus would collect the 
> new data according to a new format which would look like the following,
> 
>                 CPU    Disk
> Root            0.7     0
> User1   0.9     1
> User2   0.9     0
> User3   1.03    1
> User4   0.93    0
> User5   1.0     0

The kernel doesn't know about user names, only uids. So the layout
should be something like:

		CPU	Disk
0		0.7	0
500		0.9	1
501		0.9	0

> What do you think about this change?

Why do you want to tell the load per user? Just the CPU and disk load
should be sufficient.


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: Introduce a New Metrics to measure Load average.

[sena seneviratne]

Hi Erik,

(1) Yes layout was just like you have mentioned, yet I put user names so 
that it is easy read in the e-mail

-Why do you want to tell the load per user? Just the CPU and disk load
-should be sufficient.

This is the answer:
The reason for the division of load signal along the "user lines" is that 
in the Grid computing the users submit jobs separately.
Therefore if we are to collect a historical data of a particular user, then 
we have to collect it separately free from system/root load.

In a grid, which is implemented through Globus, a user will have to 
register himself with the provider and needs to be allocated a separate 
login account. Thus the user is only allowed to submit his HPC jobs to this 
particular account. Therefore if we can measure the load under that 
particular user-login, such collection of historical load profiles could 
show very high epochal behavior, which is related to the user's operational 
habits. This fact further emphasizes the importance of the Division of the 
load signal.
After applying the Division of load at the kernel level, we have developed 
a Prediction model called "Free load profile model for load and run time 
prediction"
Erik our load prediction system has given very good results as well.
(2) Now about the tests
As I have documented all this yet need to perform some standard tests for 
the sake of completion.
What tests should I carry out to prove that the system is still intact?

Please tell me whether the below is correct?

(a) As suggested by the http://kernel-perf.sourceforge.net/ the lmbench and 
re-aim-7 test packages can be used to test the performance of the kernel 
before making changes and after. (Not done as yet)

(b) Further tests have been carried out to check the response time of short 
tasks before making changes and after making changes. The results indicated 
that there was no difference in the response time after introducing the 
changes to the kernel (done)

(c) Thereafter the tests have been carried out to check the runtime of long 
tasks before and after making changes. The results of the tests revealed 
that there is no change in reported runtime in both occasions.(done)

Thanks
Sena Seneviratne
Computer Engineering Lab
School of Electrical and Information Engineering
Sydney University
Australia






At 02:48 PM 6/14/2006 +0200, you wrote:
>On Wed, Jun 14, 2006 at 03:12:29PM +1000, sena seneviratne wrote:
> > The  problem with the load metric of current Linux/Unix is that it 
> measures
> > CPU load and Disk load without indicating the true nature of the load,
> > thereby creating some confusion among the readers. For example, if a CPU
> > bound task switches on to read a large chunk of disk data, then the load
> > average value would still continue to indicate this activity as a load, 
> yet
> > the true CPU load during this period would have been zero.
>
>Right, we've seen such things with busy servers.
>
> > This situation
> > triggered us to make necessary additions to the kernel so that CPU load 
> and
> > Disk load could be reported separately. Further the specialisation of load
> > helped our model to perform predictions when there is interference between
> > CPU and Disk IO loads.
>
>OK.
>
> > In the user mode, a new proc file called /proc/loadavgus would collect the
> > new data according to a new format which would look like the following,
> >
> >                 CPU    Disk
> > Root            0.7     0
> > User1   0.9     1
> > User2   0.9     0
> > User3   1.03    1
> > User4   0.93    0
> > User5   1.0     0
>
>The kernel doesn't know about user names, only uids. So the layout
>should be something like:
>
>                 CPU     Disk
>0               0.7     0
>500             0.9     1
>501             0.9     0
>
> > What do you think about this change?
>
>Why do you want to tell the load per user? Just the CPU and disk load
>should be sufficient.
>
>
>Erik
>
>--
>+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
>| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands