From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Fengguang Wu <fengguang.wu@intel.com>
Cc: Dave Hansen <dave.hansen@intel.com>,
LKML <linux-kernel@vger.kernel.org>,
lkp@01.org
Subject: Re: [rcu] 5057f55e543: dmesg.BUG:soft_lockup-CPU_stuck_for_s
Date: Mon, 6 Oct 2014 03:57:12 -0700 [thread overview]
Message-ID: <20141006105712.GX5015@linux.vnet.ibm.com> (raw)
In-Reply-To: <20141006091716.GA1608@wfg-t540p.sh.intel.com>
On Mon, Oct 06, 2014 at 05:17:16PM +0800, Fengguang Wu wrote:
> On Mon, Oct 06, 2014 at 01:54:56PM +0800, Fengguang Wu wrote:
> > On Mon, Oct 06, 2014 at 01:50:24PM +0800, Fengguang Wu wrote:
> > > Hi Paul,
> > >
> > > FYI, we noticed a number of ups and downs for commit
> > >
> > > 5057f55e543b7859cfd26bc281291795eac93f8a ("rcu: Bind RCU grace-period kthreads if NO_HZ_FULL")
> >
> > Here is an overview of the performance/power/latency/kernel size
> > index. The baseline (71a9b26963f8c2d) number is 100, the larger, the better.
> >
> > 96 perf-index 5057f55e543b7859cfd26bc281291795eac93f8a
> > 99 power-index 5057f55e543b7859cfd26bc281291795eac93f8a
> > 101 latency-index 5057f55e543b7859cfd26bc281291795eac93f8a
> > 102 size-index 5057f55e543b7859cfd26bc281291795eac93f8a
>
> The performance changes seem to have a strong correlation with the
> time.involuntary_context_switches changes.
I bet that if you booted with additional CPUs not in nohz mode that
the numbers of involuntary context switches would come down. By
default, only CPU 0 is non-nohz, so all of the RCU kthreads get bound
to CPU 0.
Thanx, Paul
> 71a9b26963f8c2d 5057f55e543b7859cfd26bc28 time.involuntary_context_switches
> --------------- ------------------------- ------------------------------------
> 1209498 ± 1% -6.5% 1131376 ± 1% lkp-a04/netperf/900s-200%-TCP_STREAM
> 31677 ± 0% +42.5% 45147 ± 1% lkp-a05/iperf/300s-tcp
> 116081 ± 0% +12.5% 130610 ± 1% lkp-a06/qperf/600s
> 546 ±24% +2329.3% 13280 ± 8% lkp-sb03/nepim/300s-100%-tcp
> 863 ±45% +1260.1% 11737 ±12% lkp-sb03/nepim/300s-100%-tcp6
> 343 ±18% +12507.6% 43294 ± 2% lkp-sb03/nepim/300s-100%-udp6
> 633 ±23% +1292.7% 8816 ±15% lkp-sb03/nepim/300s-25%-tcp
> 417 ± 5% +1714.5% 7572 ±11% lkp-sb03/nepim/300s-25%-tcp6
> 364 ±20% +9569.9% 35198 ± 7% lkp-sb03/nepim/300s-25%-udp
> 312 ± 0% +12223.0% 38521 ± 1% lkp-sb03/nepim/300s-25%-udp6
> 308 ± 0% +6197.7% 19418 ± 0% lkp-sb03/nuttcp/300s
> 418 ± 4% +4948.2% 21101 ± 1% lkp-sb03/thrulay/300s
> 1.062e+09 ± 0% -2.9% 1.031e+09 ± 0% lkp-snb01/hackbench/50%-threads-pipe
> 18870 ± 0% +265.8% 69025 ± 0% lkp-snb01/will-it-scale/open2
> 20813 ± 0% +95.2% 40618 ± 0% xps/ftrace_onoff/5m
>
> iperf tcp:
>
> iperf.tcp.sender.bps
>
> 2.2e+10 ++-----*-*---------*-*--------------------------------------------+
> | + + + .*. .* |
> 2.1e+10 ++ .* + .* *.*.*..*.*. .* *.. .*.*. + .*.. |
> *.*. *.*. *. .* * *.*
> | *.*. |
> 2e+10 ++ |
> | |
> 1.9e+10 ++ |
> | |
> 1.8e+10 ++ O |
> | O O O |
> O O O O O O O |
> 1.7e+10 ++O O O O |
> | O O O |
> 1.6e+10 ++----------------------------------------------------------------+
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
>
> iperf.tcp.receiver.bps
>
> 2.2e+10 ++-----*-*---------*-*--------------------------------------------+
> | + + + .*. .* |
> 2.1e+10 ++ .* + .* *.*.*..*.*. .* *.. .*.*. + .*.. |
> *.*. *.*. *. .* * *.*
> | *.*. |
> 2e+10 ++ |
> | |
> 1.9e+10 ++ |
> | |
> 1.8e+10 ++ O |
> | O O O |
> O O O O O O O |
> 1.7e+10 ++O O O O |
> | O O O |
> 1.6e+10 ++----------------------------------------------------------------+
>
>
> time.involuntary_context_switches
>
> 10000 ++----------------------------------O-------------------------------+
> 9000 ++ |
> | O O O O O O O O |
> 8000 O+O O O O O |
> 7000 ++ O O O |
> | |
> 6000 ++ |
> 5000 ++ |
> 4000 ++ |
> | |
> 3000 ++ |
> 2000 ++*..* .*.* *
> * + .*.*. + .*.*..*.*.. .*.*..*.*.. .*.*.*.. .*.. +|
> 1000 ++ *. *. * *.*. * *.*..* |
> 0 ++------------------------------------------------------------------+
>
>
> qperf:
> time.involuntary_context_switches
>
> 25000 ++------------------------------------------------------------------+
> | |
> | O O OO OO O OO |
> 20000 O+O OO O O O O OO O O O O O OO O O OO OO |
> | O O O O O
> | |
> 15000 ++ OO O O O |
> | O O |
> 10000 ++ |
> | |
> | |
> 5000 ++ |
> |.**.* .**.*.* .* .*. *.* .* .*. *.**.** |
> * * * * * * * **.**.*.* |
> 0 ++------------------------------------------------------------------+
>
>
> qperf.udp.send_bw
>
> 2.2e+09 *+**--*----*--**----**-*-----*-**----*---*------------------------+
> | * ** * ** **.* ** *.* *.** |
> 2e+09 ++ |
> 1.8e+09 ++ |
> | O O O O
> 1.6e+09 ++ O O O OO OO O OO OO O O O O |
> | O O O O O O |
> 1.4e+09 ++ |
> | |
> 1.2e+09 ++ |
> 1e+09 ++ |
> | O |
> 8e+08 O+ O O O O O |
> | O OO O OO O O O O O |
> 6e+08 ++----------------------------------------------------------------+
>
>
> qperf.udp.recv_bw
>
> 2.2e+09 *+**--*----*--**----**-*-----*-**----*---*------------------------+
> | * ** * ** **.* ** *.* *.** |
> 2e+09 ++ |
> 1.8e+09 ++ |
> | O O O
> 1.6e+09 ++ O O OO OO OO O OO OO O O O O |
> | O O O O O O |
> 1.4e+09 ++ |
> | |
> 1.2e+09 ++ |
> 1e+09 ++ |
> | O |
> 8e+08 O+ O O O O O |
> | O OO O OO O O O O O |
> 6e+08 ++----------------------------------------------------------------+
>
>
> will-it-scale unlink1:
>
> time.voluntary_context_switches
>
> 40000 ++--O---------O-----------------------------------------------------+
> | O |
> 35000 O+ O O O O O O |
> 30000 ++ O O O O O O |
> | O O O O
> 25000 ++ |
> | |
> 20000 ++ |
> | |
> 15000 ++ |
> 10000 ++ ..*..*.. |
> | .*...*..*... .*...*. . ..*..*...|
> 5000 *+..*..*...*. *...*..*...*. *..*. *
> | |
> 0 ++------------------------------------------------------------------+
>
>
> time.involuntary_context_switches
>
> 60000 ++------------------------------------------------------------------+
> | O O O O |
> 50000 O+ O O O O O O O O O O O O O O O
> | |
> | |
> 40000 ++ |
> | |
> 30000 ++ |
> | |
> 20000 ++ |
> | |
> | *.. |
> 10000 ++ .. *.. |
> | . . |
> 0 *+--*--*---*--*---*--*---*---*--*---*--*---*----------*--*---*--*---*
>
> Thanks,
> Fengguang
>
prev parent reply other threads:[~2014-10-06 10:57 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-10-06 5:50 Fengguang Wu
2014-10-06 5:54 ` Fengguang Wu
2014-10-06 9:17 ` Fengguang Wu
2014-10-06 10:57 ` Paul E. McKenney [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20141006105712.GX5015@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=dave.hansen@intel.com \
--cc=fengguang.wu@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lkp@01.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome