[PATCH] Documentation update sched-stat.txt

[PATCH] Documentation update sched-stat.txt

[Joachim Deguara]

While learning about schedstats I found that the documentation in the tree is 
old.  I updated it and found some interesting stuff like schedstats version 
14 is the same as version and version 13 never saw a kernel release!  Also 
there are 6 fields in the current schedstats that are not used anymore.  Nick 
had made them irrelevant in commit 476d139c218e44e045e4bc6d4cc02b010b343939 
but never removed them.

Thanks to Rick's perl script who I borrowed some of the updated descriptions 
from.

-Joachim

--
Updating schedstats documentation from version 10 to 14.

Signed-off-by: Joachim Deguara <joachim.deguara@amd.com>

Index: kernel/Documentation/sched-stats.txt
===================================================================
--- kernel.orig/Documentation/sched-stats.txt
+++ kernel/Documentation/sched-stats.txt
@@ -1,10 +1,11 @@
-Version 10 of schedstats includes support for sched_domains, which
-hit the mainline kernel in 2.6.7.  Some counters make more sense to be
-per-runqueue; other to be per-domain.  Note that domains (and their 
associated
-information) will only be pertinent and available on machines utilizing
-CONFIG_SMP.
+Version 14 of schedstats includes support for sched_domains, which hit the
+mainline kernel in 2.6.20 although it is identical to the stats from version
+12 which was in the kernel from 2.6.13-2.6.19 (version 13 never saw a kernel
+release).  Some counters make more sense to be per-runqueue; other to be
+per-domain.  Note that domains (and their associated information) will only
+be pertinent and available on machines utilizing CONFIG_SMP.
 
-In version 10 of schedstat, there is at least one level of domain
+In version 14 of schedstat, there is at least one level of domain
 statistics for each cpu listed, and there may well be more than one
 domain.  Domains have no particular names in this implementation, but
 the highest numbered one typically arbitrates balancing across all the
@@ -27,7 +28,7 @@ to write their own scripts, the fields a
 
 CPU statistics
 --------------
-cpu<N> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 
27 28
+cpu<N> 1 2 3 4 5 6 7 8 9 10 11 12
 
 NOTE: In the sched_yield() statistics, the active queue is considered empty
     if it has only one process in it, since obviously the process calling
@@ -39,48 +40,20 @@ First four fields are sched_yield() stat
      3) # of times just the expired queue was empty
      4) # of times sched_yield() was called
 
-Next four are schedule() statistics:
-     5) # of times the active queue had at least one other process on it
-     6) # of times we switched to the expired queue and reused it
-     7) # of times schedule() was called
-     8) # of times schedule() left the processor idle
-
-Next four are active_load_balance() statistics:
-     9) # of times active_load_balance() was called
-    10) # of times active_load_balance() caused this cpu to gain a task
-    11) # of times active_load_balance() caused this cpu to lose a task
-    12) # of times active_load_balance() tried to move a task and failed
-
-Next three are try_to_wake_up() statistics:
-    13) # of times try_to_wake_up() was called
-    14) # of times try_to_wake_up() successfully moved the awakening task
-    15) # of times try_to_wake_up() attempted to move the awakening task
-
-Next two are wake_up_new_task() statistics:
-    16) # of times wake_up_new_task() was called
-    17) # of times wake_up_new_task() successfully moved the new task
+Next three are schedule() statistics:
+     5) # of times we switched to the expired queue and reused it
+     6) # of times schedule() was called
+     7) # of times schedule() left the processor idle
 
-Next one is a sched_migrate_task() statistic:
-    18) # of times sched_migrate_task() was called
-
-Next one is a sched_balance_exec() statistic:
-    19) # of times sched_balance_exec() was called
+Next two are try_to_wake_up() statistics:
+     8) # of times try_to_wake_up() was called
+     9) # of times try_to_wake_up() was called to wake up the local cpu
 
 Next three are statistics describing scheduling latency:
-    20) sum of all time spent running by tasks on this processor (in ms)
-    21) sum of all time spent waiting to run by tasks on this processor (in 
ms)
-    22) # of tasks (not necessarily unique) given to the processor
-
-The last six are statistics dealing with pull_task():
-    23) # of times pull_task() moved a task to this cpu when newly idle
-    24) # of times pull_task() stole a task from this cpu when another cpu
-	was newly idle
-    25) # of times pull_task() moved a task to this cpu when idle
-    26) # of times pull_task() stole a task from this cpu when another cpu
-	was idle
-    27) # of times pull_task() moved a task to this cpu when busy
-    28) # of times pull_task() stole a task from this cpu when another cpu
-	was busy
+    10) sum of all time spent running by tasks on this processor (in jiffies)
+    11) sum of all time spent waiting to run by tasks on this processor (in
+        jiffies)
+    12) # of timeslices run on this cpu
 
 
 Domain statistics
@@ -89,65 +62,95 @@ One of these is produced per domain for 
 CONFIG_SMP is not defined, *no* domains are utilized and these lines
 will not appear in the output.)
 
-domain<N> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+domain<N> <cpumask> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 
23 24 25 26 27 28 29 30 31 32 33 34 35 36
 
 The first field is a bit mask indicating what cpus this domain operates over.
 
-The next fifteen are a variety of load_balance() statistics:
-
-     1) # of times in this domain load_balance() was called when the cpu
-	was idle
-     2) # of times in this domain load_balance() was called when the cpu
-	was busy
-     3) # of times in this domain load_balance() was called when the cpu
-	was just becoming idle
-     4) # of times in this domain load_balance() tried to move one or more
-	tasks and failed, when the cpu was idle
-     5) # of times in this domain load_balance() tried to move one or more
-	tasks and failed, when the cpu was busy
-     6) # of times in this domain load_balance() tried to move one or more
-	tasks and failed, when the cpu was just becoming idle
-     7) sum of imbalances discovered (if any) with each call to
-	load_balance() in this domain when the cpu was idle
-     8) sum of imbalances discovered (if any) with each call to
-	load_balance() in this domain when the cpu was busy
-     9) sum of imbalances discovered (if any) with each call to
-	load_balance() in this domain when the cpu was just becoming idle
-    10) # of times in this domain load_balance() was called but did not find
-	a busier queue while the cpu was idle
-    11) # of times in this domain load_balance() was called but did not find
-	a busier queue while the cpu was busy
-    12) # of times in this domain load_balance() was called but did not find
-	a busier queue while the cpu was just becoming idle
-    13) # of times in this domain a busier queue was found while the cpu was
-	idle but no busier group was found
-    14) # of times in this domain a busier queue was found while the cpu was
-	busy but no busier group was found
-    15) # of times in this domain a busier queue was found while the cpu was
-	just becoming idle but no busier group was found
-
-Next two are sched_balance_exec() statistics:
-    17) # of times in this domain sched_balance_exec() successfully pushed
-	a task to a new cpu
-    18) # of times in this domain sched_balance_exec() tried but failed to
-	push a task to a new cpu
-
-Next two are try_to_wake_up() statistics:
-    19) # of times in this domain try_to_wake_up() tried to move a task based
-	on affinity and cache warmth
-    20) # of times in this domain try_to_wake_up() tried to move a task based
-	on load balancing
+The next 24 are a variety of load_balance() statistics in grouped into types
+of idleness (idle, busy, and newly idle):
 
+     1) # of times in this domain load_balance() was called when the
+        cpu was idle
+     2) # of times in this domain load_balance() checked but found
+        the load did not require balancing when the cpu was idle
+     3) # of times in this domain load_balance() tried to move one or
+        more tasks and failed, when the cpu was idle
+     4) sum of imbalances discovered (if any) with each call to
+        load_balance() in this domain when the cpu was idle
+     5) # of times in this domain pull_task() was called when the cpu
+        was idle
+     6) # of times in this domain pull_task() was called even though
+        the target task was cache-hot when idle
+     7) # of times in this domain load_balance() was called but did
+        not find a busier queue while the cpu was idle
+     8) # of times in this domain a busier queue was found while the
+        cpu was idle but no busier group was found
+
+     9) # of times in this domain load_balance() was called when the
+        cpu was busy
+    10) # of times in this domain load_balance() checked but found the
+        load did not require balancing when busy
+    11) # of times in this domain load_balance() tried to move one or
+        more tasks and failed, when the cpu was busy
+    12) sum of imbalances discovered (if any) with each call to
+        load_balance() in this domain when the cpu was busy
+    13) # of times in this domain pull_task() was called when busy
+    14) # of times in this domain pull_task() was called even though the
+        target task was cache-hot when busy
+    15) # of times in this domain load_balance() was called but did not
+        find a busier queue while the cpu was busy
+    16) # of times in this domain a busier queue was found while the cpu
+        was busy but no busier group was found
+
+    17) # of times in this domain load_balance() was called when the
+        cpu was just becoming idle
+    18) # of times in this domain load_balance() checked but found the
+        load did not require balancing when the cpu was just becoming idle
+    19) # of times in this domain load_balance() tried to move one or more
+        tasks and failed, when the cpu was just becoming idle
+    20) sum of imbalances discovered (if any) with each call to
+        load_balance() in this domain when the cpu was just becoming idle
+    21) # of times in this domain pull_task() was called when newly idle
+    22) # of times in this domain pull_task() was called even though the
+        target task was cache-hot when just becoming idle
+    23) # of times in this domain load_balance() was called but did not
+        find a busier queue while the cpu was just becoming idle
+    24) # of times in this domain a busier queue was found while the cpu
+        was just becoming idle but no busier group was found
+
+   Next three are active_load_balance() statistics:
+    25) # of times active_load_balance() was called
+    26) # of times active_load_balance() tried to move a task and failed
+    27) # of times active_load_balance() successfully moved a task
+
+   Next three are sched_balance_exec() statistics:
+    28) sbe_cnt is not used
+    29) sbe_balanced is not used
+    30) sbe_pushed is not used
+
+   Next three are sched_balance_fork() statistics:
+    31) sbf_cnt is not used
+    32) sbf_balanced is not used
+    33) sbf_pushed is not used
+
+   Next three are try_to_wake_up() statistics:
+    34) # of times in this domain try_to_wake_up() awoke a task that
+        last ran on a different cpu in this domain
+    35) # of times in this domain try_to_wake_up() moved a task to the
+        waking cpu because it was cache-cold on its own cpu anyway
+    36) # of times in this domain try_to_wake_up() started passive balancing
 
 /proc/<pid>/schedstat
 ----------------
 schedstats also adds a new /proc/<pid/schedstat file to include some of
 the same information on a per-process level.  There are three fields in
-this file correlating to fields 20, 21, and 22 in the CPU fields, but
-they only apply for that process.
+this file correlating for that process to:
+     1) time spent on the cpu
+     2) time spent waiting on a runqueue
+     3) # of timeslices run on this cpu
 
 A program could be easily written to make use of these extra fields to
 report on how well a particular process or set of processes is faring
 under the scheduler's policies.  A simple version of such a program is
 available at
-    http://eaglet.rain.com/rick/linux/schedstat/v10/latency.c
+    http://eaglet.rain.com/rick/linux/schedstat/v12/latency.c

Re: [PATCH] Documentation update sched-stat.txt

[Nick Piggin]

On Wed, Jul 18, 2007 at 11:11:30AM +0200, Joachim Deguara wrote:
> While learning about schedstats I found that the documentation in the tree is 
> old.  I updated it and found some interesting stuff like schedstats version 
> 14 is the same as version and version 13 never saw a kernel release!  Also 
> there are 6 fields in the current schedstats that are not used anymore.  Nick 
> had made them irrelevant in commit 476d139c218e44e045e4bc6d4cc02b010b343939 
> but never removed them.
> 
> Thanks to Rick's perl script who I borrowed some of the updated descriptions 
> from.

Ah, thanks, I actually didn't realise there was such good documentation
there. Patch looks good.

BTW. I have a simple program to do a basic statistical summary of the
multiprocessor balancing if you are interested and haven't seen it.

Acked-by: Nick Piggin <npiggin@suse.de>

> 
> -Joachim
> 
> --
> Updating schedstats documentation from version 10 to 14.
> 
> Signed-off-by: Joachim Deguara <joachim.deguara@amd.com>
> 
> Index: kernel/Documentation/sched-stats.txt
> ===================================================================
> --- kernel.orig/Documentation/sched-stats.txt
> +++ kernel/Documentation/sched-stats.txt
> @@ -1,10 +1,11 @@
> -Version 10 of schedstats includes support for sched_domains, which
> -hit the mainline kernel in 2.6.7.  Some counters make more sense to be
> -per-runqueue; other to be per-domain.  Note that domains (and their 
> associated
> -information) will only be pertinent and available on machines utilizing
> -CONFIG_SMP.
> +Version 14 of schedstats includes support for sched_domains, which hit the
> +mainline kernel in 2.6.20 although it is identical to the stats from version
> +12 which was in the kernel from 2.6.13-2.6.19 (version 13 never saw a kernel
> +release).  Some counters make more sense to be per-runqueue; other to be
> +per-domain.  Note that domains (and their associated information) will only
> +be pertinent and available on machines utilizing CONFIG_SMP.
>  
> -In version 10 of schedstat, there is at least one level of domain
> +In version 14 of schedstat, there is at least one level of domain
>  statistics for each cpu listed, and there may well be more than one
>  domain.  Domains have no particular names in this implementation, but
>  the highest numbered one typically arbitrates balancing across all the
> @@ -27,7 +28,7 @@ to write their own scripts, the fields a
>  
>  CPU statistics
>  --------------
> -cpu<N> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 
> 27 28
> +cpu<N> 1 2 3 4 5 6 7 8 9 10 11 12
>  
>  NOTE: In the sched_yield() statistics, the active queue is considered empty
>      if it has only one process in it, since obviously the process calling
> @@ -39,48 +40,20 @@ First four fields are sched_yield() stat
>       3) # of times just the expired queue was empty
>       4) # of times sched_yield() was called
>  
> -Next four are schedule() statistics:
> -     5) # of times the active queue had at least one other process on it
> -     6) # of times we switched to the expired queue and reused it
> -     7) # of times schedule() was called
> -     8) # of times schedule() left the processor idle
> -
> -Next four are active_load_balance() statistics:
> -     9) # of times active_load_balance() was called
> -    10) # of times active_load_balance() caused this cpu to gain a task
> -    11) # of times active_load_balance() caused this cpu to lose a task
> -    12) # of times active_load_balance() tried to move a task and failed
> -
> -Next three are try_to_wake_up() statistics:
> -    13) # of times try_to_wake_up() was called
> -    14) # of times try_to_wake_up() successfully moved the awakening task
> -    15) # of times try_to_wake_up() attempted to move the awakening task
> -
> -Next two are wake_up_new_task() statistics:
> -    16) # of times wake_up_new_task() was called
> -    17) # of times wake_up_new_task() successfully moved the new task
> +Next three are schedule() statistics:
> +     5) # of times we switched to the expired queue and reused it
> +     6) # of times schedule() was called
> +     7) # of times schedule() left the processor idle
>  
> -Next one is a sched_migrate_task() statistic:
> -    18) # of times sched_migrate_task() was called
> -
> -Next one is a sched_balance_exec() statistic:
> -    19) # of times sched_balance_exec() was called
> +Next two are try_to_wake_up() statistics:
> +     8) # of times try_to_wake_up() was called
> +     9) # of times try_to_wake_up() was called to wake up the local cpu
>  
>  Next three are statistics describing scheduling latency:
> -    20) sum of all time spent running by tasks on this processor (in ms)
> -    21) sum of all time spent waiting to run by tasks on this processor (in 
> ms)
> -    22) # of tasks (not necessarily unique) given to the processor
> -
> -The last six are statistics dealing with pull_task():
> -    23) # of times pull_task() moved a task to this cpu when newly idle
> -    24) # of times pull_task() stole a task from this cpu when another cpu
> -	was newly idle
> -    25) # of times pull_task() moved a task to this cpu when idle
> -    26) # of times pull_task() stole a task from this cpu when another cpu
> -	was idle
> -    27) # of times pull_task() moved a task to this cpu when busy
> -    28) # of times pull_task() stole a task from this cpu when another cpu
> -	was busy
> +    10) sum of all time spent running by tasks on this processor (in jiffies)
> +    11) sum of all time spent waiting to run by tasks on this processor (in
> +        jiffies)
> +    12) # of timeslices run on this cpu
>  
>  
>  Domain statistics
> @@ -89,65 +62,95 @@ One of these is produced per domain for 
>  CONFIG_SMP is not defined, *no* domains are utilized and these lines
>  will not appear in the output.)
>  
> -domain<N> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
> +domain<N> <cpumask> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 
> 23 24 25 26 27 28 29 30 31 32 33 34 35 36
>  
>  The first field is a bit mask indicating what cpus this domain operates over.
>  
> -The next fifteen are a variety of load_balance() statistics:
> -
> -     1) # of times in this domain load_balance() was called when the cpu
> -	was idle
> -     2) # of times in this domain load_balance() was called when the cpu
> -	was busy
> -     3) # of times in this domain load_balance() was called when the cpu
> -	was just becoming idle
> -     4) # of times in this domain load_balance() tried to move one or more
> -	tasks and failed, when the cpu was idle
> -     5) # of times in this domain load_balance() tried to move one or more
> -	tasks and failed, when the cpu was busy
> -     6) # of times in this domain load_balance() tried to move one or more
> -	tasks and failed, when the cpu was just becoming idle
> -     7) sum of imbalances discovered (if any) with each call to
> -	load_balance() in this domain when the cpu was idle
> -     8) sum of imbalances discovered (if any) with each call to
> -	load_balance() in this domain when the cpu was busy
> -     9) sum of imbalances discovered (if any) with each call to
> -	load_balance() in this domain when the cpu was just becoming idle
> -    10) # of times in this domain load_balance() was called but did not find
> -	a busier queue while the cpu was idle
> -    11) # of times in this domain load_balance() was called but did not find
> -	a busier queue while the cpu was busy
> -    12) # of times in this domain load_balance() was called but did not find
> -	a busier queue while the cpu was just becoming idle
> -    13) # of times in this domain a busier queue was found while the cpu was
> -	idle but no busier group was found
> -    14) # of times in this domain a busier queue was found while the cpu was
> -	busy but no busier group was found
> -    15) # of times in this domain a busier queue was found while the cpu was
> -	just becoming idle but no busier group was found
> -
> -Next two are sched_balance_exec() statistics:
> -    17) # of times in this domain sched_balance_exec() successfully pushed
> -	a task to a new cpu
> -    18) # of times in this domain sched_balance_exec() tried but failed to
> -	push a task to a new cpu
> -
> -Next two are try_to_wake_up() statistics:
> -    19) # of times in this domain try_to_wake_up() tried to move a task based
> -	on affinity and cache warmth
> -    20) # of times in this domain try_to_wake_up() tried to move a task based
> -	on load balancing
> +The next 24 are a variety of load_balance() statistics in grouped into types
> +of idleness (idle, busy, and newly idle):
>  
> +     1) # of times in this domain load_balance() was called when the
> +        cpu was idle
> +     2) # of times in this domain load_balance() checked but found
> +        the load did not require balancing when the cpu was idle
> +     3) # of times in this domain load_balance() tried to move one or
> +        more tasks and failed, when the cpu was idle
> +     4) sum of imbalances discovered (if any) with each call to
> +        load_balance() in this domain when the cpu was idle
> +     5) # of times in this domain pull_task() was called when the cpu
> +        was idle
> +     6) # of times in this domain pull_task() was called even though
> +        the target task was cache-hot when idle
> +     7) # of times in this domain load_balance() was called but did
> +        not find a busier queue while the cpu was idle
> +     8) # of times in this domain a busier queue was found while the
> +        cpu was idle but no busier group was found
> +
> +     9) # of times in this domain load_balance() was called when the
> +        cpu was busy
> +    10) # of times in this domain load_balance() checked but found the
> +        load did not require balancing when busy
> +    11) # of times in this domain load_balance() tried to move one or
> +        more tasks and failed, when the cpu was busy
> +    12) sum of imbalances discovered (if any) with each call to
> +        load_balance() in this domain when the cpu was busy
> +    13) # of times in this domain pull_task() was called when busy
> +    14) # of times in this domain pull_task() was called even though the
> +        target task was cache-hot when busy
> +    15) # of times in this domain load_balance() was called but did not
> +        find a busier queue while the cpu was busy
> +    16) # of times in this domain a busier queue was found while the cpu
> +        was busy but no busier group was found
> +
> +    17) # of times in this domain load_balance() was called when the
> +        cpu was just becoming idle
> +    18) # of times in this domain load_balance() checked but found the
> +        load did not require balancing when the cpu was just becoming idle
> +    19) # of times in this domain load_balance() tried to move one or more
> +        tasks and failed, when the cpu was just becoming idle
> +    20) sum of imbalances discovered (if any) with each call to
> +        load_balance() in this domain when the cpu was just becoming idle
> +    21) # of times in this domain pull_task() was called when newly idle
> +    22) # of times in this domain pull_task() was called even though the
> +        target task was cache-hot when just becoming idle
> +    23) # of times in this domain load_balance() was called but did not
> +        find a busier queue while the cpu was just becoming idle
> +    24) # of times in this domain a busier queue was found while the cpu
> +        was just becoming idle but no busier group was found
> +
> +   Next three are active_load_balance() statistics:
> +    25) # of times active_load_balance() was called
> +    26) # of times active_load_balance() tried to move a task and failed
> +    27) # of times active_load_balance() successfully moved a task
> +
> +   Next three are sched_balance_exec() statistics:
> +    28) sbe_cnt is not used
> +    29) sbe_balanced is not used
> +    30) sbe_pushed is not used
> +
> +   Next three are sched_balance_fork() statistics:
> +    31) sbf_cnt is not used
> +    32) sbf_balanced is not used
> +    33) sbf_pushed is not used
> +
> +   Next three are try_to_wake_up() statistics:
> +    34) # of times in this domain try_to_wake_up() awoke a task that
> +        last ran on a different cpu in this domain
> +    35) # of times in this domain try_to_wake_up() moved a task to the
> +        waking cpu because it was cache-cold on its own cpu anyway
> +    36) # of times in this domain try_to_wake_up() started passive balancing
>  
>  /proc/<pid>/schedstat
>  ----------------
>  schedstats also adds a new /proc/<pid/schedstat file to include some of
>  the same information on a per-process level.  There are three fields in
> -this file correlating to fields 20, 21, and 22 in the CPU fields, but
> -they only apply for that process.
> +this file correlating for that process to:
> +     1) time spent on the cpu
> +     2) time spent waiting on a runqueue
> +     3) # of timeslices run on this cpu
>  
>  A program could be easily written to make use of these extra fields to
>  report on how well a particular process or set of processes is faring
>  under the scheduler's policies.  A simple version of such a program is
>  available at
> -    http://eaglet.rain.com/rick/linux/schedstat/v10/latency.c
> +    http://eaglet.rain.com/rick/linux/schedstat/v12/latency.c
> 

Re: [PATCH] Documentation update sched-stat.txt

[Joachim Deguara]

On Friday 20 July 2007 09:25:22 Nick Piggin wrote:
> On Wed, Jul 18, 2007 at 11:11:30AM +0200, Joachim Deguara wrote:
> > While learning about schedstats I found that the documentation in the
> > tree is old.  I updated it and found some interesting stuff like
> > schedstats version 14 is the same as version and version 13 never saw a
> > kernel release!  Also there are 6 fields in the current schedstats that
> > are not used anymore.  Nick had made them irrelevant in commit
> > 476d139c218e44e045e4bc6d4cc02b010b343939 but never removed them.
> >
> > Thanks to Rick's perl script who I borrowed some of the updated
> > descriptions from.
>
> Ah, thanks, I actually didn't realise there was such good documentation
> there. Patch looks good.
>
> BTW. I have a simple program to do a basic statistical summary of the
> multiprocessor balancing if you are interested and haven't seen it.

Yes I am interested.  Actually I started down this road looking to find out if 
task migration could be tracked and I saw that got kicked out from early 
versions.
Your script could come in useful to link to in the documentation.  Rick has a 
great page but hasn't been updated in a little while (though still up-to-date 
as version 12==14) and his email bounced (though just a config error).

-Joachim

Re: [PATCH] Documentation update sched-stat.txt

[Nick Piggin]

On Fri, Jul 20, 2007 at 09:56:03AM +0200, Joachim Deguara wrote:
> On Friday 20 July 2007 09:25:22 Nick Piggin wrote:
> > On Wed, Jul 18, 2007 at 11:11:30AM +0200, Joachim Deguara wrote:
> > > While learning about schedstats I found that the documentation in the
> > > tree is old.  I updated it and found some interesting stuff like
> > > schedstats version 14 is the same as version and version 13 never saw a
> > > kernel release!  Also there are 6 fields in the current schedstats that
> > > are not used anymore.  Nick had made them irrelevant in commit
> > > 476d139c218e44e045e4bc6d4cc02b010b343939 but never removed them.
> > >
> > > Thanks to Rick's perl script who I borrowed some of the updated
> > > descriptions from.
> >
> > Ah, thanks, I actually didn't realise there was such good documentation
> > there. Patch looks good.
> >
> > BTW. I have a simple program to do a basic statistical summary of the
> > multiprocessor balancing if you are interested and haven't seen it.
> 
> Yes I am interested.  Actually I started down this road looking to find out if 
> task migration could be tracked and I saw that got kicked out from early 
> versions.

What do you mean by that? You mean if you can check information on the
migration events that a particular task has experienced?


> Your script could come in useful to link to in the documentation.  Rick has a 
> great page but hasn't been updated in a little while (though still up-to-date 
> as version 12==14) and his email bounced (though just a config error).

Here it is, its a bit ugly and I think it may still have a bug somewhere,
but I haven't looked at it for a while.

---
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#define SCHEDSTAT_VERSION 14
struct rq_stats {
	/* sys_sched_yield stats */
	unsigned long long yld_both_empty;
	unsigned long long yld_act_empty;
	unsigned long long yld_exp_empty;
	unsigned long long yld_cnt;
										
	/* schedule stats */
	unsigned long long sched_active; //new
	unsigned long long sched_switch;
	unsigned long long sched_cnt;
	unsigned long long sched_idle;

	/* wake stats */
	unsigned long long ttwu_cnt;
	unsigned long long ttwu_local;

	/* latency stats */
	unsigned long long cpu_time;
	unsigned long long delay_time;
	unsigned long long pcnt;
};

struct domain_stats {
	unsigned long long lb_cnt[3];
	unsigned long long lb_balanced[3];
	unsigned long long lb_failed[3];
	unsigned long long lb_pulled[3];
	unsigned long long lb_hot_pulled[3];
	unsigned long long lb_imbalance[3];
	unsigned long long lb_nobusyq[3];
	unsigned long long lb_nobusyg[3];
										
	/* Active load balancing */
	unsigned long long alb_cnt;
	unsigned long long alb_failed;
	unsigned long long alb_pushed;
	
	/* Wake ups */
	unsigned long long ttwu_wake_remote;

	/* Passive load balancing */
	unsigned long long ttwu_move_balance;
										
	/* Affine wakeups */
	unsigned long long ttwu_move_affine;
										
	/* SD_BALANCE_EXEC */
	unsigned long long sbe_cnt;
	unsigned long long sbe_balanced;
	unsigned long long sbe_pushed;

	/* SD_BALANCE_FORK */
	unsigned long long sbf_cnt;
	unsigned long long sbf_balanced;
	unsigned long long sbf_pushed;
};

enum idle_type {
	IDLE,
	NOT_IDLE,
	NEWLY_IDLE,
};

#define MAXDOMAINS	4
#define MAXCPUS		32
#define HZ		100UL

static void parse_file(FILE *file, unsigned long long *ts,
		int *cpus, int *domains,
		struct rq_stats rq_stats[MAXCPUS],
		struct domain_stats domain_stats[MAXCPUS][MAXDOMAINS])
{
	int i, j;
	int ret, cpu, domain;
	int tmp;

	*domains = -1;

	ret = fscanf(file, "version %d\n", &tmp);
	if (ret == 0 || ret == EOF) {
		fprintf(stderr, "file format error 0\n");
		exit(1);
	}
	if (tmp != SCHEDSTAT_VERSION) {
		fprintf(stderr, "wrong file format version\n");
		exit(1);
	}

	ret = fscanf(file, "timestamp %llu\n", ts);
	if (ret == 0 || ret == EOF) {
		fprintf(stderr, "file format error 1\n");
		exit(1);
	}

	for (i = 0; i < MAXCPUS; i++) {
		struct rq_stats *rs = &rq_stats[i];

		ret = fscanf(file, "cpu%d ", &cpu);
		if (ret == EOF)
			break;
		if (ret == 0 || cpu != i) {
			fprintf(stderr, "file format error 2\n");
			exit(1);
		}

		ret = fscanf(file, "%llu %llu %llu %llu %llu %llu %llu %llu %llu %llu %llu %llu",
			&rs->yld_both_empty, &rs->yld_act_empty,
			&rs->yld_exp_empty, &rs->yld_cnt,
			&rs->sched_switch, &rs->sched_cnt, &rs->sched_idle,
			&rs->ttwu_cnt, &rs->ttwu_local,
			&rs->cpu_time, &rs->delay_time, &rs->pcnt);
		if (ret == 0 || ret == EOF) {
			fprintf(stderr, "file format error 3\n");
			exit(1);
		}
		
		for (j = 0; j < MAXDOMAINS; j++) {
			int k;
			struct domain_stats *ds = &domain_stats[i][j];
			
			/* We discard the domain's cpumask for now */
			ret = fscanf(file, " domain%d %*s", &domain);
			if (ret == 0 || ret == EOF)
				break;
			if (domain != j) {
				fprintf(stderr, "file format error 4\n");
				exit(1);
			}

			for (k = 0; k < 3; k++) {
				ret = fscanf(file, "%llu %llu %llu %llu %llu %llu %llu %llu",
					&ds->lb_cnt[k], &ds->lb_balanced[k],
					&ds->lb_failed[k], &ds->lb_imbalance[k],
					&ds->lb_pulled[k], &ds->lb_hot_pulled[k],
					&ds->lb_nobusyq[k], &ds->lb_nobusyg[k]);
				if (ret == 0 || ret == EOF) {
					fprintf(stderr, "file format error 5\n");
					exit(1);
				}
			}

			ret = fscanf(file, " %llu %llu %llu %llu %llu %llu %llu %llu %llu %llu %llu %llu",
				&ds->alb_cnt, &ds->alb_failed, &ds->alb_pushed,
				&ds->sbe_cnt, &ds->sbe_balanced, &ds->sbe_pushed,
				&ds->sbf_cnt, &ds->sbf_balanced, &ds->sbf_pushed,
				&ds->ttwu_wake_remote, &ds->ttwu_move_affine,
				&ds->ttwu_move_balance);
			if (ret == 0 || ret == EOF) {
				fprintf(stderr, "file format error 6\n");
				exit(1);
			}
		}

		if (*domains != -1 && *domains != j) {
			fprintf(stderr, "domains mismatch within file\n");
			exit(1);
		}
		*domains = j;

		ret = fscanf(file, "\n");
		if (ret == EOF)
			break;
	}
	
	*cpus = i;
}

static void find_stats_delta(struct rq_stats rq_pre[MAXCPUS],
			struct rq_stats rq_post[MAXCPUS],
			struct rq_stats *rq_delta,
			struct domain_stats domain_pre[MAXCPUS][MAXDOMAINS],
			struct domain_stats domain_post[MAXCPUS][MAXDOMAINS],
			struct domain_stats domain_delta[MAXDOMAINS])
{
	int i;

	memset(rq_delta, 0, sizeof(struct rq_stats));
	memset(domain_delta, 0, sizeof(struct domain_stats)*MAXDOMAINS);
	
	for (i = 0; i < MAXCPUS; i++) {
		unsigned int j, k;

		/* No problem because they're all unsigned long long */
		for (j = 0; j < sizeof(struct rq_stats)/sizeof(unsigned long long); j++) {
			*((unsigned long long *)rq_delta + j) +=
			*((unsigned long long *)&rq_post[i] + j) -
			*((unsigned long long *)&rq_pre[i] + j);
		}
		
		for (j = 0; j < MAXDOMAINS; j++) {
			for (k = 0; k < sizeof(struct domain_stats)/sizeof(unsigned long long); k++) {
				*((unsigned long long *)&domain_delta[j] + k) +=
				*((unsigned long long *)&domain_post[i][j] + k) -
				*((unsigned long long *)&domain_pre[i][j] + k);
			}
		}
	}
}

static void show_stats(unsigned long long time_delta, int cpus, int domains,
		struct rq_stats *rq_stats,
		struct domain_stats domain_stats[MAXDOMAINS])
{
	unsigned long long ttwu_remote;
	double s, tmp;
	int i;

	/* Ensures we don't get 0 time delta */
	s = ((double)0.5 + time_delta) / HZ;
	printf("sample period: %.3fs\n", s);
	
	/* TODO add the runqueue stats */
	tmp = (double)rq_stats->sched_cnt / s;
	printf("%.3f calls to schedule / s\n", tmp);

	tmp = (double)rq_stats->cpu_time / rq_stats->pcnt;
	printf("%.3fms average timeslice\n", tmp);

	tmp = (double)rq_stats->delay_time / rq_stats->pcnt;
	printf("%.3fms average runqueue delay\n", tmp);

	printf("\n--- wakeup statistics ---\n");
	tmp = (double)rq_stats->ttwu_cnt / s;
	printf("  %.3f task wakes / s\n", tmp);
	tmp = (double)100 * rq_stats->ttwu_local / rq_stats->ttwu_cnt;
	printf("    %.3f%% of them from the local CPU\n", tmp);

	ttwu_remote = rq_stats->ttwu_cnt - rq_stats->ttwu_local;

	for (i = 0; i < domains; i++) {
		tmp = (double)100 * domain_stats[i].ttwu_wake_remote / ttwu_remote;
		printf("    %.3f%% of remote wakeups come from domain%d\n", tmp, i);

		tmp = (double)100 * domain_stats[i].ttwu_move_balance / domain_stats[i].ttwu_wake_remote;
		printf("      %.3f%% are moved to the local CPU via passive load balancing\n", tmp);

		tmp = (double)100 * domain_stats[i].ttwu_move_affine / domain_stats[i].ttwu_wake_remote;
		printf("      %.3f%% are moved to the local CPU via affine wakeups\n", tmp);
	}

	printf("\n--- load balancing statistics ---\n");

	for (i = 0; i < domains; i++) {
		unsigned long long total_lb = 0;
		unsigned long long total_pulled = 0;
		int j;

		printf("  for domain%d\n", i);

		for (j = 0; j < 3; j++) {
			total_lb += domain_stats[i].lb_cnt[j];
			total_pulled += domain_stats[i].lb_pulled[j];
		}

		tmp = (double)total_lb / s;
		printf("    %.3f load balance calls / s", tmp);
		tmp = (double)total_pulled / s;
		printf(" move %.3f tasks / s\n", tmp);

		for (j = 0; j < 3; j++) {
			unsigned long long lb = domain_stats[i].lb_cnt[j];
			unsigned long long pulled = domain_stats[i].lb_pulled[j];
			tmp = (double)100 * lb / total_lb;
			printf("      %.3f%% calls and", tmp);
			tmp = (double)100 * pulled / total_pulled;
			printf(" %.3f%% task moves came from ", tmp);
			if (j == 0)
				printf("idle balancing\n");
			else if (j == 1)
				printf("busy balancing\n");
			else if (j == 2)
				printf("new-idle balancing\n");

			if (lb) {
				tmp = (double)100 * (lb - domain_stats[i].lb_balanced[j]) / lb;
				printf("        %.3f%% were imbalanced", tmp);

				tmp = (double)domain_stats[i].lb_imbalance[j] / (lb - domain_stats[i].lb_balanced[j]);
				printf(" with an average imbalance of %.3f\n", tmp);

				tmp = (double)100 * domain_stats[i].lb_failed[j] / lb;
				printf("        %.3f%% found an imbalance but failed\n", tmp);
			}

			if (pulled) {
				tmp = (double)100 * domain_stats[i].lb_hot_pulled[j] / pulled;
				printf("        %.3f%% of tasks moved were cache hot\n", tmp);
			}
		}
		
		tmp = (double)domain_stats[i].alb_cnt / s;
		printf("    %.3f active balances / s ", tmp);
		
		tmp = (double)domain_stats[i].alb_pushed / s;
		printf(" move %.3f tasks / s\n", tmp);

		if (domain_stats[i].alb_cnt) {
			tmp = (double)100 * domain_stats[i].alb_failed / domain_stats[i].alb_cnt;
			printf("      %%%.3f attempts failed\n", tmp);
		}

		tmp = (double)domain_stats[i].sbe_cnt / s;
		printf("    %.3f exec balances / s ", tmp);
		
		tmp = (double)domain_stats[i].sbe_pushed / s;
		printf(" move %.3f tasks / s\n", tmp);

		if (domain_stats[i].sbe_cnt) {
			tmp = (double)100 * domain_stats[i].sbe_balanced / domain_stats[i].sbe_cnt;
			printf("      %%%.3f found no imbalance\n", tmp);
		}

		tmp = (double)domain_stats[i].sbf_cnt / s;
		printf("    %.3f fork balances / s ", tmp);
		
		tmp = (double)domain_stats[i].sbf_pushed / s;
		printf(" move %.3f tasks / s\n", tmp);

		if (domain_stats[i].sbf_cnt) {
			tmp = (double)100 * domain_stats[i].sbf_balanced / domain_stats[i].sbf_cnt;
			printf("      %%%.3f found no imbalance\n", tmp);
		}


		printf("\n");
	}
}

static unsigned long long pre_ts, post_ts;
static int pre_cpus, post_cpus;
static int pre_domains, post_domains;
static struct rq_stats pre_rq_stats[MAXCPUS];
static struct rq_stats post_rq_stats[MAXCPUS];
static struct rq_stats delta_rq_stats;
static struct domain_stats pre_domain_stats[MAXCPUS][MAXDOMAINS];
static struct domain_stats post_domain_stats[MAXCPUS][MAXDOMAINS];
static struct domain_stats delta_domain_stats[MAXDOMAINS];

int main(int argc, char *argv[])
{
	FILE *pre, *post;

	if (argc < 2) {
		fprintf(stderr, "Usage: %s <starts before> <stats after>\n",
				argv[0]);
		exit(1);
	}

	pre = fopen(argv[1], "r");
	if (pre == NULL)
		perror("fopen pre file"), exit(1);
	post = fopen(argv[2], "r");
	if (post == NULL)
		perror("fopen post file"), exit(1);

	parse_file(pre, &pre_ts, &pre_cpus, &pre_domains,
			pre_rq_stats, pre_domain_stats);
	parse_file(post, &post_ts, &post_cpus, &post_domains,
			post_rq_stats, post_domain_stats);
	if (pre_cpus != post_cpus || pre_domains != post_domains) {
		fprintf(stderr, "pre and post file formats mismatch\n");
		exit(1);
	}

	find_stats_delta(pre_rq_stats, post_rq_stats, &delta_rq_stats,
		pre_domain_stats, post_domain_stats, delta_domain_stats);

	show_stats(post_ts - pre_ts, pre_cpus, pre_domains,
			&delta_rq_stats, delta_domain_stats);

	exit(0);
}

Re: [PATCH] Documentation update sched-stat.txt

[Rick Lindsley]

Yes, I noticed the version updated without the docs a while ago, and
I've had it on my list for a time to update the docs. Thanks for
actually providing the patch!

I will update the table on the website with a new column for v14.  I
think v13 occurred in (and was removed from) the -mm tree at one point
during some interim scheduler changes.

There was a paper presented at OLS suggesting that under some loads,
schedstats could impose a performance penalty due to the cache hits
incurred by accruing the statistics. In light of the changes in v12/v14,
and the issues raised by that paper, it is probably worthwhile to review
schedstats for usefulness.  Is it still useful and, perhaps more to the
point, is it still measuring the right stuff?  Should counters be added
or deleted?  (That discussion should be separate from this patch.)

Rick

Acked-by: Rick Lindsley <ricklind@us.ibm.com>
    
Index: kernel/Documentation/sched-stats.txt
===================================================================
--- kernel.orig/Documentation/sched-stats.txt
+++ kernel/Documentation/sched-stats.txt
@@ -1,10 +1,11 @@
-Version 10 of schedstats includes support for sched_domains, which
-hit the mainline kernel in 2.6.7.  Some counters make more sense to be
-per-runqueue; other to be per-domain.  Note that domains (and their 
associated
-information) will only be pertinent and available on machines utilizing
-CONFIG_SMP.
+Version 14 of schedstats includes support for sched_domains, which hit the
+mainline kernel in 2.6.20 although it is identical to the stats from version
+12 which was in the kernel from 2.6.13-2.6.19 (version 13 never saw a kernel
+release).  Some counters make more sense to be per-runqueue; other to be
+per-domain.  Note that domains (and their associated information) will only
+be pertinent and available on machines utilizing CONFIG_SMP.
 
-In version 10 of schedstat, there is at least one level of domain
+In version 14 of schedstat, there is at least one level of domain
 statistics for each cpu listed, and there may well be more than one
 domain.  Domains have no particular names in this implementation, but
 the highest numbered one typically arbitrates balancing across all the
@@ -27,7 +28,7 @@ to write their own scripts, the fields a
 
 CPU statistics
 --------------
-cpu<N> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 
27 28
+cpu<N> 1 2 3 4 5 6 7 8 9 10 11 12
 
 NOTE: In the sched_yield() statistics, the active queue is considered empty
     if it has only one process in it, since obviously the process calling
@@ -39,48 +40,20 @@ First four fields are sched_yield() stat
      3) # of times just the expired queue was empty
      4) # of times sched_yield() was called
 
-Next four are schedule() statistics:
-     5) # of times the active queue had at least one other process on it
-     6) # of times we switched to the expired queue and reused it
-     7) # of times schedule() was called
-     8) # of times schedule() left the processor idle
-
-Next four are active_load_balance() statistics:
-     9) # of times active_load_balance() was called
-    10) # of times active_load_balance() caused this cpu to gain a task
-    11) # of times active_load_balance() caused this cpu to lose a task
-    12) # of times active_load_balance() tried to move a task and failed
-
-Next three are try_to_wake_up() statistics:
-    13) # of times try_to_wake_up() was called
-    14) # of times try_to_wake_up() successfully moved the awakening task
-    15) # of times try_to_wake_up() attempted to move the awakening task
-
-Next two are wake_up_new_task() statistics:
-    16) # of times wake_up_new_task() was called
-    17) # of times wake_up_new_task() successfully moved the new task
+Next three are schedule() statistics:
+     5) # of times we switched to the expired queue and reused it
+     6) # of times schedule() was called
+     7) # of times schedule() left the processor idle
 
-Next one is a sched_migrate_task() statistic:
-    18) # of times sched_migrate_task() was called
-
-Next one is a sched_balance_exec() statistic:
-    19) # of times sched_balance_exec() was called
+Next two are try_to_wake_up() statistics:
+     8) # of times try_to_wake_up() was called
+     9) # of times try_to_wake_up() was called to wake up the local cpu
 
 Next three are statistics describing scheduling latency:
-    20) sum of all time spent running by tasks on this processor (in ms)
-    21) sum of all time spent waiting to run by tasks on this processor (in 
ms)
-    22) # of tasks (not necessarily unique) given to the processor
-
-The last six are statistics dealing with pull_task():
-    23) # of times pull_task() moved a task to this cpu when newly idle
-    24) # of times pull_task() stole a task from this cpu when another cpu
-	was newly idle
-    25) # of times pull_task() moved a task to this cpu when idle
-    26) # of times pull_task() stole a task from this cpu when another cpu
-	was idle
-    27) # of times pull_task() moved a task to this cpu when busy
-    28) # of times pull_task() stole a task from this cpu when another cpu
-	was busy
+    10) sum of all time spent running by tasks on this processor (in jiffies)
+    11) sum of all time spent waiting to run by tasks on this processor (in
+        jiffies)
+    12) # of timeslices run on this cpu
 
 
 Domain statistics
@@ -89,65 +62,95 @@ One of these is produced per domain for 
 CONFIG_SMP is not defined, *no* domains are utilized and these lines
 will not appear in the output.)
 
-domain<N> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+domain<N> <cpumask> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 
23 24 25 26 27 28 29 30 31 32 33 34 35 36
 
 The first field is a bit mask indicating what cpus this domain operates over.
 
-The next fifteen are a variety of load_balance() statistics:
-
-     1) # of times in this domain load_balance() was called when the cpu
-	was idle
-     2) # of times in this domain load_balance() was called when the cpu
-	was busy
-     3) # of times in this domain load_balance() was called when the cpu
-	was just becoming idle
-     4) # of times in this domain load_balance() tried to move one or more
-	tasks and failed, when the cpu was idle
-     5) # of times in this domain load_balance() tried to move one or more
-	tasks and failed, when the cpu was busy
-     6) # of times in this domain load_balance() tried to move one or more
-	tasks and failed, when the cpu was just becoming idle
-     7) sum of imbalances discovered (if any) with each call to
-	load_balance() in this domain when the cpu was idle
-     8) sum of imbalances discovered (if any) with each call to
-	load_balance() in this domain when the cpu was busy
-     9) sum of imbalances discovered (if any) with each call to
-	load_balance() in this domain when the cpu was just becoming idle
-    10) # of times in this domain load_balance() was called but did not find
-	a busier queue while the cpu was idle
-    11) # of times in this domain load_balance() was called but did not find
-	a busier queue while the cpu was busy
-    12) # of times in this domain load_balance() was called but did not find
-	a busier queue while the cpu was just becoming idle
-    13) # of times in this domain a busier queue was found while the cpu was
-	idle but no busier group was found
-    14) # of times in this domain a busier queue was found while the cpu was
-	busy but no busier group was found
-    15) # of times in this domain a busier queue was found while the cpu was
-	just becoming idle but no busier group was found
-
-Next two are sched_balance_exec() statistics:
-    17) # of times in this domain sched_balance_exec() successfully pushed
-	a task to a new cpu
-    18) # of times in this domain sched_balance_exec() tried but failed to
-	push a task to a new cpu
-
-Next two are try_to_wake_up() statistics:
-    19) # of times in this domain try_to_wake_up() tried to move a task based
-	on affinity and cache warmth
-    20) # of times in this domain try_to_wake_up() tried to move a task based
-	on load balancing
+The next 24 are a variety of load_balance() statistics in grouped into types
+of idleness (idle, busy, and newly idle):
 
+     1) # of times in this domain load_balance() was called when the
+        cpu was idle
+     2) # of times in this domain load_balance() checked but found
+        the load did not require balancing when the cpu was idle
+     3) # of times in this domain load_balance() tried to move one or
+        more tasks and failed, when the cpu was idle
+     4) sum of imbalances discovered (if any) with each call to
+        load_balance() in this domain when the cpu was idle
+     5) # of times in this domain pull_task() was called when the cpu
+        was idle
+     6) # of times in this domain pull_task() was called even though
+        the target task was cache-hot when idle
+     7) # of times in this domain load_balance() was called but did
+        not find a busier queue while the cpu was idle
+     8) # of times in this domain a busier queue was found while the
+        cpu was idle but no busier group was found
+
+     9) # of times in this domain load_balance() was called when the
+        cpu was busy
+    10) # of times in this domain load_balance() checked but found the
+        load did not require balancing when busy
+    11) # of times in this domain load_balance() tried to move one or
+        more tasks and failed, when the cpu was busy
+    12) sum of imbalances discovered (if any) with each call to
+        load_balance() in this domain when the cpu was busy
+    13) # of times in this domain pull_task() was called when busy
+    14) # of times in this domain pull_task() was called even though the
+        target task was cache-hot when busy
+    15) # of times in this domain load_balance() was called but did not
+        find a busier queue while the cpu was busy
+    16) # of times in this domain a busier queue was found while the cpu
+        was busy but no busier group was found
+
+    17) # of times in this domain load_balance() was called when the
+        cpu was just becoming idle
+    18) # of times in this domain load_balance() checked but found the
+        load did not require balancing when the cpu was just becoming idle
+    19) # of times in this domain load_balance() tried to move one or more
+        tasks and failed, when the cpu was just becoming idle
+    20) sum of imbalances discovered (if any) with each call to
+        load_balance() in this domain when the cpu was just becoming idle
+    21) # of times in this domain pull_task() was called when newly idle
+    22) # of times in this domain pull_task() was called even though the
+        target task was cache-hot when just becoming idle
+    23) # of times in this domain load_balance() was called but did not
+        find a busier queue while the cpu was just becoming idle
+    24) # of times in this domain a busier queue was found while the cpu
+        was just becoming idle but no busier group was found
+
+   Next three are active_load_balance() statistics:
+    25) # of times active_load_balance() was called
+    26) # of times active_load_balance() tried to move a task and failed
+    27) # of times active_load_balance() successfully moved a task
+
+   Next three are sched_balance_exec() statistics:
+    28) sbe_cnt is not used
+    29) sbe_balanced is not used
+    30) sbe_pushed is not used
+
+   Next three are sched_balance_fork() statistics:
+    31) sbf_cnt is not used
+    32) sbf_balanced is not used
+    33) sbf_pushed is not used
+
+   Next three are try_to_wake_up() statistics:
+    34) # of times in this domain try_to_wake_up() awoke a task that
+        last ran on a different cpu in this domain
+    35) # of times in this domain try_to_wake_up() moved a task to the
+        waking cpu because it was cache-cold on its own cpu anyway
+    36) # of times in this domain try_to_wake_up() started passive balancing
 
 /proc/<pid>/schedstat
 ----------------
 schedstats also adds a new /proc/<pid/schedstat file to include some of
 the same information on a per-process level.  There are three fields in
-this file correlating to fields 20, 21, and 22 in the CPU fields, but
-they only apply for that process.
+this file correlating for that process to:
+     1) time spent on the cpu
+     2) time spent waiting on a runqueue
+     3) # of timeslices run on this cpu
 
 A program could be easily written to make use of these extra fields to
 report on how well a particular process or set of processes is faring
 under the scheduler's policies.  A simple version of such a program is
 available at
-    http://eaglet.rain.com/rick/linux/schedstat/v10/latency.c
+    http://eaglet.rain.com/rick/linux/schedstat/v12/latency.c