mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Valentin Schneider <valentin.schneider@arm.com>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@kernel.org>,
	linux-kernel <linux-kernel@vger.kernel.org>,
	Morten Rasmussen <morten.rasmussen@foss.arm.com>,
	Brendan Jackman <brendan.jackman@arm.com>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>
Subject: Re: [PATCH v5 1/3] sched: Stop nohz stats when decayed
Date: Wed, 21 Feb 2018 13:13:52 +0000	[thread overview]
Message-ID: <a351f2de-3a00-1cf8-b16f-59493c5b67f5@arm.com> (raw)
In-Reply-To: <CAKfTPtAbr2vUvw2DvVCuqCXNnSA-7LuO2EaJ59pP=oJTgQ3R_g@mail.gmail.com>

On 02/16/2018 01:44 PM, Vincent Guittot wrote:
> On 16 February 2018 at 13:13, Valentin Schneider
> <valentin.schneider@arm.com> wrote:
>> On 02/14/2018 03:26 PM, Vincent Guittot wrote:
>>> Stopped the periodic update of blocked load when all idle CPUs have fully
>>> decayed. We introduce a new nohz.has_blocked that reflect if some idle
>>> CPUs has blocked load that have to be periodiccally updated. nohz.has_blocked
>>> is set everytime that a Idle CPU can have blocked load and it is then clear
>>> when no more blocked load has been detected during an update. We don't need
>>> atomic operation but only to make cure of the right ordering when updating
>>> nohz.idle_cpus_mask and nohz.has_blocked.
>>>
>>> Suggested-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>>> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
>>> ---
>>>  kernel/sched/fair.c  | 122 ++++++++++++++++++++++++++++++++++++++++++---------
>>>  kernel/sched/sched.h |   1 +
>>>  2 files changed, 102 insertions(+), 21 deletions(-)
>>>
>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>> index 7af1fa9..5a6835e 100644
>>> --- a/kernel/sched/fair.c
>>> +++ b/kernel/sched/fair.c
>>>
>>> [...]
>>> @@ -9383,11 +9452,16 @@ static bool nohz_idle_balance(struct rq *this_rq, enum cpu_idle_type idle)
>>>                * work being done for other cpus. Next load
>>>                * balancing owner will pick it up.
>>>                */
>>> -             if (need_resched())
>>> -                     break;
>>> +             if (need_resched()) {
>>> +                     has_blocked_load = true;
>>> +                     goto abort;
>>> +             }
>>>
>>>               rq = cpu_rq(balance_cpu);
>>>
>>
>> I'd say it's safe to do the following here. The flag is raised in
>> nohz_balance_enter_idle() before the smp_mb(), so we won't skip a CPU
>> that just got added to nohz.idle_cpus_mask.
> 
> rq->has_blocked_load will be set before the barrier only if
> nohz_tick_stopped is not already set,
> Otherwise, we skip cpumask update and the barrier in  nohz_balance_enter_idle
> 

Right, forgot about that bit. I think it's still fine because:
- nohz_balance_enter_idle() can't be called before the last running task is
dequeued, which requires the rq lock.
- update_blocked_averages takes the rq lock and clears rq->has_blocked_load
with the lock held.

So even though we could have some very unlikely scenario where a CPU quickly
goes out/in of idle after nohz.idle_cpus_mask has been read, the blocked load
itself is safe so rq->has_blocked_load will end up being set correctly.
(Took me a while to see it that way)


BTW, with the current set on Peter's sched/testing, update_nohz_stats()
is called here, which doesn't do the update if !rq->has_blocked_load
(Although that check is done without lock/barrier, so maybe we could not see
a CPU that just went idle ?)

I have one more question on that bit:


		has_blocked_load |= update_nohz_stats(rq, true);

		/*
		 * If time for next balance is due,
		 * do the balance.
		 */
		if (time_after_eq(jiffies, rq->next_balance)) {
			struct rq_flags rf;

			rq_lock_irqsave(rq, &rf);
			update_rq_clock(rq);
			cpu_load_update_idle(rq);
			rq_unlock_irqrestore(rq, &rf);

			if (flags & NOHZ_BALANCE_KICK)
				rebalance_domains(rq, CPU_IDLE);
		}

		if (time_after(next_balance, rq->next_balance)) {
			next_balance = rq->next_balance;
			update_next_balance = 1;
		}


Now that I think about it, shouldn't we always have a 'continue' after
the blocked load update if (flags & NOHZ_KICK_MASK) == NOHZ_STATS_KICK ?
AFAICT we don't want to push the next_balance forward, only the next_blocked.
That would also take care of not doing the load balance.
>>
>>                 /*
>>                  * This cpu doesn't have any remaining blocked load, skip it.
>>                  * It's sane to do this because this flag is raised in
>>                  * nohz_balance_enter_idle()
>>                  */
>>                 if ((flags & NOHZ_KICK_MASK) == NOHZ_STATS_KICK &&
>>                     !rq->has_blocked_load)
>>                         continue;
>>
>>> +             update_blocked_averages(rq->cpu);
>>> +             has_blocked_load |= rq->has_blocked_load;
>>> +

  reply	other threads:[~2018-02-21 13:13 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-02-14 15:26 [PATCH v5 0/3] sched: Update blocked load Vincent Guittot
2018-02-14 15:26 ` [PATCH v5 1/3] sched: Stop nohz stats when decayed Vincent Guittot
2018-02-15 15:22   ` Patrick Bellasi
2018-02-15 16:50     ` Peter Zijlstra
2018-02-15 16:59       ` Patrick Bellasi
2018-02-15 17:06     ` Vincent Guittot
2018-02-16 12:13   ` Valentin Schneider
2018-02-16 13:44     ` Vincent Guittot
2018-02-21 13:13       ` Valentin Schneider [this message]
2018-02-21 13:48         ` Valentin Schneider
2018-02-22  8:37         ` Vincent Guittot
2018-02-22 10:04           ` Valentin Schneider
2018-02-16 12:53   ` Valentin Schneider
2018-02-16 17:02     ` Vincent Guittot
2018-02-16 19:23       ` Valentin Schneider
2018-02-14 15:26 ` [PATCH v5 2/3] sched: reduce the periodic update duration Vincent Guittot
2018-02-14 15:26 ` [PATCH v5 3/3] sched: update blocked load when newly idle Vincent Guittot
2018-02-20 11:59   ` Peter Zijlstra
2018-03-09  9:06   ` [tip:sched/core] sched/fair: Update " tip-bot for Vincent Guittot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a351f2de-3a00-1cf8-b16f-59493c5b67f5@arm.com \
    --to=valentin.schneider@arm.com \
    --cc=brendan.jackman@arm.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=morten.rasmussen@foss.arm.com \
    --cc=peterz@infradead.org \
    --cc=vincent.guittot@linaro.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®