mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Ken Chen <kenchen@google.com>
To: Ingo Molnar <mingo@elte.hu>, Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: sched: wakeup profile for sd-wake-affine optimization
Date: Mon, 15 Dec 2008 23:01:02 -0800	[thread overview]
Message-ID: <b040c32a0812152301t6718aabbua3820a710801df02@mail.gmail.com> (raw)

[-- Attachment #1: Type: text/plain, Size: 1898 bytes --]

Hi,

We continue seeing problems with wake-affine load balance heuristics on our
workloads.  I did some instrumentation on application wake-up profile and I
thought it would be good to share the result with the kernel community.

First attached is a 3D surface plot of wake-up initiator, target, and their
occurrence.  (wakeup-surface.png). x-axis is the source of wakeup initiator
(the one that sends the wakeup event), y-axis is the target of the wakeup
(the one that got woken up).  z-axis is the occurrence count.  From the
graph, it is clear that the distribution of wakeup initiators is non-uniform.
There is clearly one small cluster of sources waking up one small cluster
of targets.  Besides the two peaks in the surface graph, the other area are
not flat either, further indicates non-uniformity in wakeup distribution
profile.

2nd graph is a flattened 3d surface with the same data.  Each line is a
source of wakeup initiator, x-axis is the target and y-axis is number of
wakeup events.  This graph distinctively showed that there is one wakeup
initiator (irq context) wakes up two target threads (tid 6149 and 6191)
while there are no other initiator wakes up these two targets.

With sd-wake-affine, what happened is that these two target threads got
pulled to only *one* cpu due to wakeup initiator are more likely to be
executed only on a small set of CPU in the system.  And there are no other
source to load balance it back and spread these two threads execution on
multiple CPUs.  The end result is that we see idle time as the system
couldn't load balance all runnable threads on all available CPUs.

I was hoping that smart people here on LKML can brainstorm a solution.
My solution earlier was to introduce a sysctl to control sd-wake-affine
behavior, but that requires user to know what their application behavior
is and is not a desirable solution for general usage.

- Ken

[-- Attachment #2: wakeup-surface.png --]
[-- Type: image/png, Size: 7105 bytes --]

[-- Attachment #3: wakeup-2d.png --]
[-- Type: image/png, Size: 6662 bytes --]

                 reply	other threads:[~2008-12-16  7:01 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=b040c32a0812152301t6718aabbua3820a710801df02@mail.gmail.com \
    --to=kenchen@google.com \
    --cc=a.p.zijlstra@chello.nl \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®