Re: 2.6.27-rc5 OLTP performance regression

Re: 2.6.27-rc5 OLTP performance regression

[Peter Zijlstra]

On Mon, 2008-09-08 at 10:58 -0700, Ma, Chinang wrote:
> We found the group scheduler in 2.6.27-rc5 has negative performance
> impact on TPC Online Transaction Processing workload. The test was
> conducted on a dual-socket quad-core Xeon server. OLTP workload is
> disk i/o intensive and we have over 200 database shadow processes
> running in the server during this test.  Enabling group scheduler
> (CONFIG_GROUP_SCHED=y) reduced performance by 2.0%. Oprofile data
> indicates significant amount of cycles are spent in tg_shares_up().
>  This is new regression as we did not find the same issue with 2.6.26
> kernel group scheduler. Is anybody looking into group scheduler
> performance and any idea for reducing the performance impact? 

Because the .26 group scheduler wasn't SMP aware. The extra cost comes
from the fact that .27 is.

Re: 2.6.27-rc5 OLTP performance regression

[Peter Zijlstra]

On Mon, 2008-09-08 at 20:00 +0200, Peter Zijlstra wrote:
> On Mon, 2008-09-08 at 10:58 -0700, Ma, Chinang wrote:
> > We found the group scheduler in 2.6.27-rc5 has negative performance
> > impact on TPC Online Transaction Processing workload. The test was
> > conducted on a dual-socket quad-core Xeon server. OLTP workload is
> > disk i/o intensive and we have over 200 database shadow processes
> > running in the server during this test.  Enabling group scheduler
> > (CONFIG_GROUP_SCHED=y) reduced performance by 2.0%. Oprofile data
> > indicates significant amount of cycles are spent in tg_shares_up().
> >  This is new regression as we did not find the same issue with 2.6.26
> > kernel group scheduler. Is anybody looking into group scheduler
> > performance and any idea for reducing the performance impact? 
> 
> Because the .26 group scheduler wasn't SMP aware. The extra cost comes
> from the fact that .27 is.

What you can do it increase the /proc/sys/kernel/sched_shares_ratelimit
value and thereby decrease the accuracy of the SMP fairness of the group
scheduler.

RE: 2.6.27-rc5 OLTP performance regression

[Ma, Chinang]

Peter,
        We increased sched_shares_ratelimit and were able to recover the 2% performance regression.
Thanks,

-----Original Message-----
From: Peter Zijlstra [mailto:peterz@infradead.org]
Sent: Monday, September 08, 2008 11:04 AM
To: Ma, Chinang
Cc: Ingo Molnar; Srivatsa Vaddagiri; Mike Galbraith; Gregory Haskins; Steven Rostedt; Nick Piggin; Siddha, Suresh B; Wilcox, Matthew R; Tripathi, Sharad C; Chilukuri, Harita; linux-kernel@vger.kernel.org
Subject: Re: 2.6.27-rc5 OLTP performance regression

On Mon, 2008-09-08 at 20:00 +0200, Peter Zijlstra wrote:
> On Mon, 2008-09-08 at 10:58 -0700, Ma, Chinang wrote:
> > We found the group scheduler in 2.6.27-rc5 has negative performance
> > impact on TPC Online Transaction Processing workload. The test was
> > conducted on a dual-socket quad-core Xeon server. OLTP workload is
> > disk i/o intensive and we have over 200 database shadow processes
> > running in the server during this test.  Enabling group scheduler
> > (CONFIG_GROUP_SCHED=y) reduced performance by 2.0%. Oprofile data
> > indicates significant amount of cycles are spent in tg_shares_up().
> >  This is new regression as we did not find the same issue with 2.6.26
> > kernel group scheduler. Is anybody looking into group scheduler
> > performance and any idea for reducing the performance impact?
>
> Because the .26 group scheduler wasn't SMP aware. The extra cost comes
> from the fact that .27 is.

What you can do it increase the /proc/sys/kernel/sched_shares_ratelimit
value and thereby decrease the accuracy of the SMP fairness of the group
scheduler.

Re: 2.6.27-rc5 OLTP performance regression

[Ingo Molnar]

* Ma, Chinang <chinang.ma@intel.com> wrote:

> Peter,
>         We increased sched_shares_ratelimit and were able to recover 
> the 2% performance regression. [...]

great. Could you please post a patch for that?

	Ingo

RE: 2.6.27-rc5 OLTP performance regression

[Ma, Chinang]

I increased the /proc/sys/kernel/sched_shares_ratelimit from 1 millisecond to one second to reduce the frequency of shares update. That was okay for OLTP workload. As Peter mentioned, a larger value will reduce the accuracy of SMP fairness for group scheduler and the ratelimit and using a larger default value may not be good for general case.

Thanks,
Chinang
-----Original Message-----
From: Ingo Molnar [mailto:mingo@elte.hu]
Sent: Wednesday, September 10, 2008 5:26 AM
To: Ma, Chinang
Cc: Peter Zijlstra; Srivatsa Vaddagiri; Mike Galbraith; Gregory Haskins; Steven Rostedt; Nick Piggin; Siddha, Suresh B; Wilcox, Matthew R; Tripathi, Sharad C; Chilukuri, Harita; linux-kernel@vger.kernel.org
Subject: Re: 2.6.27-rc5 OLTP performance regression


* Ma, Chinang <chinang.ma@intel.com> wrote:

> Peter,
>         We increased sched_shares_ratelimit and were able to recover
> the 2% performance regression. [...]

great. Could you please post a patch for that?

        Ingo

Re: 2.6.27-rc5 OLTP performance regression

[Ingo Molnar]

* Ma, Chinang <chinang.ma@intel.com> wrote:

> I increased the /proc/sys/kernel/sched_shares_ratelimit from 1 
> millisecond to one second to reduce the frequency of shares update. 
> That was okay for OLTP workload. As Peter mentioned, a larger value 
> will reduce the accuracy of SMP fairness for group scheduler and the 
> ratelimit and using a larger default value may not be good for general 
> case.

does 10 msecs already get you most of the 2%? 1000 millisecond sounds 
very (and probably unacceptable) coarse.

	Ingo

RE: 2.6.27-rc5 OLTP performance regression

[Ma, Chinang]

When set to 10 ms the regression is about 0.4%. We get most of the 2%.

Chinang
-----Original Message-----
From: Ingo Molnar [mailto:mingo@elte.hu]
Sent: Wednesday, September 10, 2008 10:35 AM
To: Ma, Chinang
Cc: Peter Zijlstra; Srivatsa Vaddagiri; Mike Galbraith; Gregory Haskins; Steven Rostedt; Nick Piggin; Siddha, Suresh B; Wilcox, Matthew R; Tripathi, Sharad C; Chilukuri, Harita; linux-kernel@vger.kernel.org
Subject: Re: 2.6.27-rc5 OLTP performance regression


* Ma, Chinang <chinang.ma@intel.com> wrote:

> I increased the /proc/sys/kernel/sched_shares_ratelimit from 1
> millisecond to one second to reduce the frequency of shares update.
> That was okay for OLTP workload. As Peter mentioned, a larger value
> will reduce the accuracy of SMP fairness for group scheduler and the
> ratelimit and using a larger default value may not be good for general
> case.

does 10 msecs already get you most of the 2%? 1000 millisecond sounds
very (and probably unacceptable) coarse.

        Ingo

RE: 2.6.27-rc5 OLTP performance regression

[Ma, Chinang]

We also tested sched_shares_ratelimit = 100 ms and reduce the OLTP regression to 0.2%. (very small return in performance with 10x the value). Can we count on this sched_shares_ratelimit tunable be always accessible for workload that does not depend on group fairness?

Chinang

-----Original Message-----
From: Ma, Chinang
Sent: Wednesday, September 10, 2008 5:08 PM
To: Ingo Molnar
Cc: Peter Zijlstra; Srivatsa Vaddagiri; Mike Galbraith; Gregory Haskins; Steven Rostedt; Nick Piggin; Siddha, Suresh B; Wilcox, Matthew R; Tripathi, Sharad C; Chilukuri, Harita; linux-kernel@vger.kernel.org
Subject: RE: 2.6.27-rc5 OLTP performance regression

When set to 10 ms the regression is about 0.4%. We get most of the 2%.

Chinang
-----Original Message-----
From: Ingo Molnar [mailto:mingo@elte.hu]
Sent: Wednesday, September 10, 2008 10:35 AM
To: Ma, Chinang
Cc: Peter Zijlstra; Srivatsa Vaddagiri; Mike Galbraith; Gregory Haskins; Steven Rostedt; Nick Piggin; Siddha, Suresh B; Wilcox, Matthew R; Tripathi, Sharad C; Chilukuri, Harita; linux-kernel@vger.kernel.org
Subject: Re: 2.6.27-rc5 OLTP performance regression


* Ma, Chinang <chinang.ma@intel.com> wrote:

> I increased the /proc/sys/kernel/sched_shares_ratelimit from 1
> millisecond to one second to reduce the frequency of shares update.
> That was okay for OLTP workload. As Peter mentioned, a larger value
> will reduce the accuracy of SMP fairness for group scheduler and the
> ratelimit and using a larger default value may not be good for general
> case.

does 10 msecs already get you most of the 2%? 1000 millisecond sounds
very (and probably unacceptable) coarse.

        Ingo