[ANNOUNCE] (Resend) Tools to analyse PM and scheduling behaviour

[ANNOUNCE] (Resend) Tools to analyse PM and scheduling behaviour

[Amit Kucheria]

(Apologies for the resend. Gmail plain-text mode still sends multipart
messages that includes text/html and vger doesn't like that.)

Received: by 10.107.1.149 with HTTP; Fri, 8 Aug 2014 06:54:01 -0700 (PDT)
Date: Fri, 8 Aug 2014 19:24:01 +0530
Delivered-To: amit.kucheria@linaro.org
Message-ID: <CAP245DUSAHO_VBoBMy4KQaaA6_0a-i3Y3u4utnK8ji3UMVniyA@mail.gmail.com>
Subject: [ANNOUNCE] Tools to analyse PM and scheduling behaviour
From: Amit Kucheria <amit.kucheria@linaro.org>
To: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@kernel.org>, "Rafael J. Wysocki" <rjw@rjwysocki.net>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, Linux PM list <linux-pm@vger.kernel.org>
Cc: "Gross\, Mark" <mark.gross@intel.com>, Paul McKenney <Paul.McKenney@us.ibm.com>, Morten Rasmussen <Morten.Rasmussen@arm.com>, Dietmar Eggemann <Dietmar.Eggemann@arm.com>, Preeti U Murthy <preeti@linux.vnet.ibm.com>, Daniel Lezcano <daniel.lezcano@linaro.org>, Vincent Guittot <vincent.guittot@linaro.org>, Mike Turquette <mturquette@linaro.org>, Juri Lelli <juri.lelli@arm.com>, sched-tools@linaro.org

Hi all,

At the Energy-Aware Scheduling workshop at Kernel Summit last year[1],
maintainers identified a need for tools that would make it easier to
analyse changes to scheduling and power management behaviour of a system.
In order to satisfy that need, the Power Management working group at Linaro
has created idlestat[2]. The key design principles of idlestat are to
minimize measurement overhead during a trace capture and be accurate in
capturing P-state and C-state transitions while being architecture-agnostic.

Another requirement was to give maintainers the ability to benchmark kernel
changes against typical mobile workloads without having to run a system
like Android. In order to satisfy that need, we’ve created a workload
generator[3] (by extending rt-app) to allow simulation of typical mobile
use cases[4].

An example usage of both tools together would be:
 sudo ./idlestat --trace -f /tmp/mytrace -t 100 -- rt-app mp3.json

This should allow us to do repeatable analysis of the power management and
scheduling behaviour of the system with well-crafted workloads.

We’re soliciting early feedback from community on the direction of idlestat
and rt-app and enhancements that would make the tools more useful. I also
want to propose idlestat as a candidate for inclusion into kernel tools
directory.

Idlestat Details
----------------
Idlestat uses FTRACE to capture traces related to C-state and P-state
transitions of the CPU and wakeups (IRQ, IPI) on the system and then
post-processes the data to print statistics. It is designed to be used
non-interactively. Idlestat can deduce the idle time for a cluster as an
intersection between the idle times of all the cpus belonging to the same
cluster. This data is useful to analyse and optimise scheduling behaviour.
The tool will also list how many times the menu governor mis-predicts
target residency in a C-state.

In the future, we plan to add a ‘diff mode’ to allow comparison of two
trace files (useful for regression tracking) and to allow inputting an
energy model for the CPU so that the tool can output a rough energy
consumption estimate for the workload. See the wiki[2] for other ideas for
new features.

A sample run on 24-core Intel XEON with HT disabled and all but two cpus in
each cluster offlined (for the sake of keeping the output short) looks as
follows:

sudo ./idlestat --trace -f /tmp/mytrace -t 10 -p -c -w
Log is 12.934396 secs long with 148153 events
--------------------------------------------------------------------------------
| C-state  |   min    |   max    |   avg    |   total  | hits  |  over |
under |
--------------------------------------------------------------------------------
| clusterA
    |
--------------------------------------------------------------------------------
|   C6-IVT |      0us |  21.02ms |    364us | 244.85ms |   673 |     0 |
  0 |
--------------------------------------------------------------------------------
|             cpu0
    |
--------------------------------------------------------------------------------
|   C1-IVT |     31us |     42us |     37us |    111us |     3 |     0 |
  0 |
|  C1E-IVT |     21us |    112us |     52us |    155us |     3 |     2 |
  0 |
|   C3-IVT |     40us |    276us |    178us |   1.07ms |     6 |     2 |
  0 |
|   C6-IVT |      2us |  21.13ms |    580us | 260.23ms |   449 |   318 |
  0 |
--------------------------------------------------------------------------------
|             cpu1
    |
--------------------------------------------------------------------------------
|     POLL |     17us |     17us |     17us |     17us |     1 |     0 |
  1 |
|   C1-IVT |      6us |     95us |     49us |   1.09ms |    22 |     0 |
  3 |
|  C1E-IVT |      4us |    166us |    108us |   2.16ms |    20 |     3 |
  4 |
|   C3-IVT |      6us |    274us |    167us |   4.51ms |    27 |     7 |
  0 |
|   C6-IVT |      1us |  52.56ms |    372us |   11.07s | 29794 | 23533 |
  0 |
--------------------------------------------------------------------------------
| clusterB
    |
--------------------------------------------------------------------------------
|   C6-IVT |      0us |  85.65ms |   2.21ms |   11.33s |  5121 |     0 |
  0 |
--------------------------------------------------------------------------------
|             cpu6
    |
--------------------------------------------------------------------------------
|   C1-IVT |     44us |     44us |     44us |     44us |     1 |     0 |
  0 |
|  C1E-IVT |    164us |    164us |    164us |    164us |     1 |     0 |
  1 |
|   C3-IVT |    258us |    258us |    258us |    258us |     1 |     0 |
  0 |
|   C6-IVT |      1us |  85.65ms |   2.24ms |   11.43s |  5113 |  1358 |
  0 |
--------------------------------------------------------------------------------
|             cpu7
    |
--------------------------------------------------------------------------------
|   C1-IVT |      9us |      9us |      9us |      9us |     1 |     0 |
  0 |
|   C3-IVT |    298us |    298us |    298us |    298us |     1 |     0 |
  0 |
|   C6-IVT |     22us |    1.50s | 279.02ms |   12.83s |    46 |     3 |
  0 |
--------------------------------------------------------------------------------

----------------------------------------------------------------
| P-state  |   min    |   max    |   avg    |   total  | hits  |
----------------------------------------------------------------
|             cpu0                                             |
----------------------------------------------------------------
|  2.60GHz |      0us |   12.56s |  59.79ms |   12.62s |   211 |
|  2.20GHz |  12.01ms |  12.01ms |  12.01ms |  12.01ms |     1 |
|  1.50GHz |      0us |   2.66ms |     95us |   3.53ms |    37 |
|  1.20GHz |      1us |  14.24ms |    368us |  26.85ms |    73 |
----------------------------------------------------------------
|             cpu1                                             |
----------------------------------------------------------------
|  2.60GHz |      0us |   6.43ms |     57us | 757.08ms | 13341 |
|  2.50GHz |      0us |   6.82ms |     79us |  31.93ms |   404 |
|  2.40GHz |      0us |   6.59ms |    111us |  11.43ms |   103 |
|  2.30GHz |      0us |   5.82ms |     29us |  29.27ms |   999 |
|  2.20GHz |      0us |   7.80ms |     49us |  79.03ms |  1618 |
|  2.10GHz |      0us |   4.96ms |     66us |  30.72ms |   465 |
|  2.00GHz |      0us |   9.01ms |     33us |  40.57ms |  1246 |
|  1.90GHz |      0us |   6.16ms |     57us |  24.81ms |   434 |
|  1.80GHz |      0us |  11.56ms |    101us |  77.03ms |   764 |
|  1.70GHz |      0us |  12.97ms |     96us |  58.47ms |   606 |
|  1.60GHz |      0us |  12.94ms |     98us |  85.31ms |   867 |
|  1.50GHz |      0us |  11.99ms |    166us |  35.10ms |   211 |
|  1.40GHz |      0us |   3.64ms |     62us |  26.97ms |   433 |
|  1.30GHz |   7.99ms |   7.99ms |   7.99ms |   7.99ms |     1 |
|  1.20GHz |      0us |  15.29ms |     64us | 531.01ms |  8291 |
----------------------------------------------------------------
|             cpu6                                             |
----------------------------------------------------------------
|  2.60GHz |      1us |   6.10ms |    215us | 658.53ms |  3059 |
|  2.30GHz |     20us |   5.57ms |    205us |  21.16ms |   103 |
|  2.20GHz |     18us |   7.42ms |    201us |  26.49ms |   132 |
|  2.00GHz |     21us |    367us |    159us |   4.77ms |    30 |
|  1.90GHz |     21us |   8.96ms |    259us |  18.42ms |    71 |
|  1.80GHz |     72us |   6.09ms |    280us |  11.49ms |    41 |
|  1.70GHz |      2us |   5.40ms |    173us |  38.69ms |   224 |
|  1.60GHz |      2us |   3.95ms |    188us |  11.50ms |    61 |
|  1.50GHz |      2us |  11.66ms |    323us |  38.39ms |   119 |
|  1.40GHz |      2us |  11.94ms |    364us |  61.85ms |   170 |
|  1.30GHz |      5us |  10.72ms |    315us |  24.27ms |    77 |
|  1.20GHz |      2us |  15.83ms |    608us | 569.34ms |   937 |
----------------------------------------------------------------
|             cpu7                                             |
----------------------------------------------------------------
|  2.60GHz |      5us |  75.74ms |   8.43ms |  75.88ms |     9 |
|  1.70GHz |   1.41ms |   1.41ms |   1.41ms |   1.41ms |     1 |
|  1.20GHz |     10us |   4.87ms |    181us |   7.07ms |    39 |
----------------------------------------------------------------

--------------------------------------------
| Wakeup |  #  |       Name      |  Count  |
--------------------------------------------
|             cpu0                         |
--------------------------------------------
| irq    | 117 |     eth0-TxRx-6 |     355 |
| ipi    | --- |      RESCHEDULE |       2 |
--------------------------------------------
|             cpu1                         |
--------------------------------------------
| irq    | 115 |     eth0-TxRx-4 |   29654 |
| irq    | 130 |     eth1-TxRx-1 |       4 |
| irq    | 111 |     eth0-TxRx-0 |       1 |
| irq    | 112 |     eth0-TxRx-1 |       3 |
--------------------------------------------
|             cpu6                         |
--------------------------------------------
| ipi    | --- |      RESCHEDULE |     274 |
--------------------------------------------
|             cpu7                         |
--------------------------------------------
| ipi    | --- |      RESCHEDULE |       4 |
--------------------------------------------

The IPI statistics require support for generic IPI tracing[4] to land
upstream.

Workload Generator Details
--------------------------
The Workload Generator is based on rt-app. Shared resources, waits, signals
and locking order can be specified for each thread in a JSON description to
construct complex interdependencies. This allows simulation of interesting
use-cases. The repository contains examples of a mp3 and web-browsing
workload.

The method to construct the JSON description is quite manual currently. We
capture the trace of a workload of interest (say, on Android) and analyse
it in kernelshark. We then construct the JSON description by hand until its
resultant trace looks similar to the trace from the real workload.

The immediate goal is to expand the library to cover the common mobile
workloads. We’re also working with the rt-app maintainer to get our changes
merged.

Thanks to ARM for help in specifying the mobile workloads and reviewing the
results.

Please play with the tools and let us know what you think. You can write to
sched-tools@linaro.org if you want to contact us off-list.

Regards,
Amit

[1] http://lwn.net/Articles/571414/
[2]
https://wiki.linaro.org/WorkingGroups/PowerManagement/Resources/Tools/Idlestat
[3]
https://wiki.linaro.org/WorkingGroups/PowerManagement/Resources/Tools/WorkloadGen
[4] https://lkml.org/lkml/2014/1/7/355
[5] https://lkml.org/lkml/2014/7/25/610

Re: [ANNOUNCE] (Resend) Tools to analyse PM and scheduling behaviour

[Sundar]

Hi Amit,

On Tue, Aug 19, 2014 at 11:11 AM, Amit Kucheria
<amit.kucheria@linaro.org> wrote:
>
> We’re soliciting early feedback from community on the direction of idlestat

Nice :)

> Idlestat Details
> ----------------
> Idlestat uses FTRACE to capture traces related to C-state and P-state
> transitions of the CPU and wakeups (IRQ, IPI) on the system and then
> post-processes the data to print statistics. It is designed to be used
> non-interactively. Idlestat can deduce the idle time for a cluster as an
> intersection between the idle times of all the cpus belonging to the same
> cluster. This data is useful to analyse and optimise scheduling behaviour.
> The tool will also list how many times the menu governor mis-predicts
> target residency in a C-state.

We discussed this in the energy aware scheduling workshop this week @
the Kernel Summit. A few notes:

1. We need to really understand the co-relation of this tool w.r.t
actual hardware states.
It is usually likely that the software "thinks" it is in a low power
state, but the actual
hardware might not be. What is the coverage for these kind of cases here.

2. I understand that C/P states are a direct metric of how well the
workload behaved w.r.t power;
but I am not sure that relates to a direct measure of how the
scheduler performed. The C/P states
could be maintained whilst giving away performance or power at the
expense of additional components
on the SoC and platform like DDR IOs, fabric states etc.

Quick Summary of what I discussed with Daniel @ the workshop about idlestat:

1. There might be usually platform specific tools to get residencies
for P/C states.
PowerTop & Turbostat are two that first come to mind. Any specific
item apart from prediction logic
that idlestat differs from these two?

2. To me debugging performance or power, C/P states provide the
direction that something is wrong.
But they still dont tell me "what" is wrong "if" the issue is somehow
in the kernel as opposed to a more
easily fixable software code (traceable at hardware/software level for
best optimizations). How do I
conclude that my scheduler is the culprit apart from the points where
it took a decision to select the
right idle states based on predicted sleep times? In my opinion, that
would boil down to if the scheduler
was invoking too much load balancing calls, moving my threads across
cores too much, data being
thrashed across caches, cores too much etc.

I think a tool for scheduler metrics must be based on more inner
details like the above, finally culminating
into C/P states. as opposed to C/P states being the metric to be relied.

Let me know your thoughts.

Cheers!

-- these are my personal thoughts and do not represent my employers'

Re: [ANNOUNCE] (Resend) Tools to analyse PM and scheduling behaviour

[Amit Kucheria]

On Sat, 23 Aug 2014 at 07:44 +0530, Sundar <sunder.svit@gmail.com> wrote:
> Hi Amit,
>
> On Tue, Aug 19, 2014 at 11:11 AM, Amit Kucheria
> <amit.kucheria@linaro.org> wrote:
>>
>> We’re soliciting early feedback from community on the direction of idlestat
>
> Nice :)
>
>> Idlestat Details
>> ----------------
>> Idlestat uses FTRACE to capture traces related to C-state and P-state
>> transitions of the CPU and wakeups (IRQ, IPI) on the system and then
>> post-processes the data to print statistics. It is designed to be used
>> non-interactively. Idlestat can deduce the idle time for a cluster as an
>> intersection between the idle times of all the cpus belonging to the same
>> cluster. This data is useful to analyse and optimise scheduling behaviour.
>> The tool will also list how many times the menu governor mis-predicts
>> target residency in a C-state.
>
> We discussed this in the energy aware scheduling workshop this week @
> the Kernel Summit. A few notes:
>
> 1. We need to really understand the co-relation of this tool w.r.t
> actual hardware states.
> It is usually likely that the software "thinks" it is in a low power
> state, but the actual
> hardware might not be. What is the coverage for these kind of cases here.

You are right, it does not represent the actual state of the HW, only
the 'requested' state.

There are various platform-dependent ways to knowing the actual HW
state. Some examples are:
 - Through an external HW signal (e.g. a GPIO that is toggled when clock
 to the CPU is cut off)
 - Measuring power on the power rails and correlating those well-known
 values (CPU ON, retention, OFF) to the traces
 - Reading some register (like MSR on x86)

This is not the main focus of the tool.

> 2. I understand that C/P states are a direct metric of how well the
> workload behaved w.r.t power;
> but I am not sure that relates to a direct measure of how the
> scheduler performed.

Consider the following examples:

*On a given platform*, we see the same benchmark scores with and
without patchset ABC, but including patchset ABC leads to better "power
behaviour" i.e. requests of deeper idle states and/or lower frequencies.

Consider another example where the benchmark score dramatically improves
with patchset XYZ while the idle and frequency requests are marginally
worse (shallower idle, reduced residency or increased frequency requests).

In both cases, it is left to platforms to do real measurements to confirm that
this is indeed the case. The latter example might not even be possible
on some platforms, given some platform constraints e.g. the platform
thermal envelope.

Idlestat is not a replacement for real measurements. It is a tool to
allow maintainers (scheduler, PM) to judge if any further investigation
is needed and request such numbers from people running the code on
various architectures before merging the patches.

> The C/P states
> could be maintained whilst giving away performance or power at the
> expense of additional components
> on the SoC and platform like DDR IOs, fabric states etc.

True.

> Quick Summary of what I discussed with Daniel @ the workshop about idlestat:
>
> 1. There might be usually platform specific tools to get residencies
> for P/C states.
> PowerTop & Turbostat are two that first come to mind. Any specific
> item apart from prediction logic
> that idlestat differs from these two?

First, idlestat is designed to be architecture-independent. It only
depends on what the kernel knows.
Second, it is created with benchmarking in mind - non-interactive and
minimal overhead.
Third, it was designed for maintainers to be able to quickly tell if a
patchset changes OS behaviour dramatically and request deeper
analysis on various architectures.
Fourth, it has the prediction logic which calculates the intersection of
C-state requests by several cpus in a cluster to determine the cluster
state.

On top of this, we have two WIP additions:
 - an experimental "energy model" patch for idlestat that lets a SoC
 vendor provide the cost of various states as input and idlestat will
 output the "energy cost" of a workload.
 - a 'diff mode' to show the diff between two traces

> 2. To me debugging performance or power, C/P states provide the
> direction that something is wrong.
>
> But they still dont tell me "what" is wrong "if" the issue is somehow
> in the kernel as opposed to a more

Correct. At the moment, idlestat can only provide an indication if
something might be wrong.

> easily fixable software code (traceable at hardware/software level for
> best optimizations). How do I
> conclude that my scheduler is the culprit apart from the points where
> it took a decision to select the
> right idle states based on predicted sleep times? In my opinion, that
> would boil down to if the scheduler
> was invoking too much load balancing calls, moving my threads across
> cores too much, data being
> thrashed across caches, cores too much etc.

These would show up as regressions in benchmark results. Fengguang's
excellent benchmark report[1] already captures such "changes". Does it
make sense to recapture that in a tool?

We're open to tracking more metrics if it is felt they are useful.

> I think a tool for scheduler metrics must be based on more inner
> details like the above, finally culminating
> into C/P states. as opposed to C/P states being the metric to be relied.

One of the tenets of energy-aware scheduling is "improving energy
efficiency with little or no performance regression". idlestat tells us
about possible regressions on the energy front and benchmarks should
tell us if we are regressing on performance. Hence the focus on
C/P-states for now.

Regards,
Amit
[1] https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg703826.html

Re: [ANNOUNCE] (Resend) Tools to analyse PM and scheduling behaviour

[Sundar]

Hi Amit,

On Tue, Aug 26, 2014 at 11:02 AM, Amit Kucheria
<amit.kucheria@linaro.org> wrote:

> Consider the following examples:
>
> *On a given platform*, we see the same benchmark scores with and
> without patchset ABC, but including patchset ABC leads to better "power
> behaviour" i.e. requests of deeper idle states and/or lower frequencies.
>
> Consider another example where the benchmark score dramatically improves
> with patchset XYZ while the idle and frequency requests are marginally
> worse (shallower idle, reduced residency or increased frequency requests).
>
> In both cases, it is left to platforms to do real measurements to confirm that
> this is indeed the case. The latter example might not even be possible
> on some platforms, given some platform constraints e.g. the platform
> thermal envelope.
>
> Idlestat is not a replacement for real measurements. It is a tool to
> allow maintainers (scheduler, PM) to judge if any further investigation
> is needed and request such numbers from people running the code on
> various architectures before merging the patches.

As I mentioned, it is very much possible for a workload to preserve the CPU
C/P states but damage some other system metric like memory/soc bandwidth,
cache characteristics because the scheduler was probably doing more aggressive
task placements. I agree that no tool (within room for
errors/approximations) can
replace a physical measurement; my only query/concern being is C/P correlation
the direct or primary metric for scheduler behavior (not PM behavior).

> First, idlestat is designed to be architecture-independent. It only
> depends on what the kernel knows.
> Second, it is created with benchmarking in mind - non-interactive and
> minimal overhead.
> Third, it was designed for maintainers to be able to quickly tell if a
> patchset changes OS behaviour dramatically and request deeper
> analysis on various architectures.
> Fourth, it has the prediction logic which calculates the intersection of
> C-state requests by several cpus in a cluster to determine the cluster
> state.
>
> On top of this, we have two WIP additions:
>  - an experimental "energy model" patch for idlestat that lets a SoC
>  vendor provide the cost of various states as input and idlestat will
>  output the "energy cost" of a workload.
>  - a 'diff mode' to show the diff between two traces

I see this as no different from powertop; would it not be easier to
add the prediction
logic and investigate energy models integration? I dont mind a
different tool to be
doing almost same things, but is there really a need for one?

> Correct. At the moment, idlestat can only provide an indication if
> something might be wrong.

And that's where I think I see an immense value for idlestat to stick to
scheduler details beyond the traditional C/P state statistics.

> These would show up as regressions in benchmark results. Fengguang's
> excellent benchmark report[1] already captures such "changes". Does it
> make sense to recapture that in a tool?
> [1] https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg703826.html

I am yet to digest that report, so apologies :)

> We're open to tracking more metrics if it is felt they are useful.
>
> One of the tenets of energy-aware scheduling is "improving energy
> efficiency with little or no performance regression". idlestat tells us
> about possible regressions on the energy front and benchmarks should
> tell us if we are regressing on performance. Hence the focus on
> C/P-states for now.

I would like to know your views on adding additional scheduler metrics
like task thrashing,
irregular placements, increased load balancing into the tool to be
able to zero in the
scheduler for efficiency losses. There might be more critical metrics
that I am missing...

Cheers!
-- 
---------
The views expressed in this email are personal and do not necessarily
echo my employers.