From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932210AbcF3Nwd (ORCPT ); Thu, 30 Jun 2016 09:52:33 -0400 Received: from bombadil.infradead.org ([198.137.202.9]:51559 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932115AbcF3Nwb (ORCPT ); Thu, 30 Jun 2016 09:52:31 -0400 Date: Thu, 30 Jun 2016 15:20:37 +0200 From: Peter Zijlstra To: Frederic Weisbecker Cc: LKML , Ingo Molnar , Mike Galbraith , Thomas Gleixner Subject: Re: [PATCH 2/3] sched: Unloop sched avg decaying Message-ID: <20160630132037.GE30921@twins.programming.kicks-ass.net> References: <1465918082-27005-1-git-send-email-fweisbec@gmail.com> <1465918082-27005-3-git-send-email-fweisbec@gmail.com> <20160614155842.GJ30921@twins.programming.kicks-ass.net> <20160630125225.GA32568@lerouge> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160630125225.GA32568@lerouge> User-Agent: Mutt/1.5.23.1 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jun 30, 2016 at 02:52:26PM +0200, Frederic Weisbecker wrote: > On Tue, Jun 14, 2016 at 05:58:42PM +0200, Peter Zijlstra wrote: > > Why not add the division to the nohz exit path only? > > It would be worse I think because we may exit much more often from nohz > than we reach a sched_avg_period(). > > So the only safe optimization I can do for now is: How about something like this then? --- kernel/sched/core.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 3387e4f14fc9..fd1ae4c4105f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -665,9 +665,23 @@ bool sched_can_stop_tick(struct rq *rq) void sched_avg_update(struct rq *rq) { - s64 period = sched_avg_period(); + s64 delta, period = sched_avg_period(); - while ((s64)(rq_clock(rq) - rq->age_stamp) > period) { + delta = (s64)(rq_clock(rq) - rq->age_stamp); + if (likely(delta < period)) + return; + + if (unlikely(delta > 3*period)) { + int pending; + u64 rem; + + pending = div64_u64_rem(delta, period, &rem); + rq->age_stamp += delta - rem; + rq->rt_avg >>= pending; + return; + } + + while (delta > period) { /* * Inline assembly required to prevent the compiler * optimising this loop into a divmod call. @@ -675,6 +689,7 @@ void sched_avg_update(struct rq *rq) */ asm("" : "+rm" (rq->age_stamp)); rq->age_stamp += period; + delta -= period; rq->rt_avg /= 2; } }