mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Vincent Guittot <vincent.guittot@linaro.org>
To: Josh Don <joshdon@google.com>
Cc: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com,
	 dietmar.eggemann@arm.com, rostedt@goodmis.org,
	bsegall@google.com,  mgorman@suse.de, bristot@redhat.com,
	vschneid@redhat.com,  linux-kernel@vger.kernel.org,
	zhangqiao22@huawei.com
Subject: Re: [PATCH 1/4] sched/fair: make sure to try to detach at least one movable task
Date: Wed, 20 Mar 2024 17:57:50 +0100	[thread overview]
Message-ID: <CAKfTPtDSC25N8TvszDAjseqdLdGy4qiDnwobNThkt8piSL_5Pw@mail.gmail.com> (raw)
In-Reply-To: <CABk29NsQf_xStzWg8bB_hpNpPC_LduMs-M058LjdhnDG16wN_A@mail.gmail.com>

Hi Josh,

Sorry for the late reply.

On Mon, 12 Feb 2024 at 21:29, Josh Don <joshdon@google.com> wrote:
>
> Hi Vincent,
>
> On Thu, Aug 25, 2022 at 5:27 AM Vincent Guittot
> <vincent.guittot@linaro.org> wrote:
> >
> > During load balance, we try at most env->loop_max time to move a task.
> > But it can happen that the loop_max LRU tasks (ie tail of
> > the cfs_tasks list) can't be moved to dst_cpu because of affinity.
> > In this case, loop in the list until we found at least one.
>
> We had a user recently trigger a hard lockup which we believe is due
> to this patch. The user in question had O(10k) threads affinitized to
> a cpu; seems like the process had an out of control thread spawning
> issue, and was in the middle of getting killed. However, that was
> being slowed down due to the fact that load balance was iterating all

Does it mean that it was progressing but not as fast as you would like

> these threads and bouncing the rq lock (and making no progress due to
> ALL_PINNED). Before this patch, load balance would quit after hitting
> loop_max.
>
> Even ignoring that specific instance, it seems pretty easy for this
> patch to cause a softlockup due to a buggy or malicious process.

The fact that the rq is released regularly should prevent a
softlockup. And we could even fasten can_migrate() which does a lot of
useless stuff for task affined to 1 cpu.

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e8270e2e15cb..15bc1067c69d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8920,6 +8920,8 @@ int can_migrate_task(struct task_struct *p,
struct lb_env *env)

        lockdep_assert_rq_held(env->src_rq);

+       if (p->nr_cpus_allowed == 1)
+               return 0;
        /*
         * We do not migrate tasks that are:
         * 1) throttled_lb_pair, or

>
> For the tradeoff you were trying to make in this patch (spend more
> time searching in the hopes that there's something migratable further
> in the list), perhaps it would be better to adjust
> sysctl.sched_nr_migrate instead of baking this into the kernel?

That could be a solution but this increases the iterations for all
cases including those which are more time consuming to sort out and
the number of tasks that you will migrate in one lb. The latter is the
one which consumes time

Vincent

>
> Best,
> Josh
>
> >
> > The maximum of detached tasks remained the same as before.
> >
> > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > ---
> >  kernel/sched/fair.c | 12 +++++++++---
> >  1 file changed, 9 insertions(+), 3 deletions(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index da388657d5ac..02b7b808e186 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -8052,8 +8052,12 @@ static int detach_tasks(struct lb_env *env)
> >                 p = list_last_entry(tasks, struct task_struct, se.group_node);
> >
> >                 env->loop++;
> > -               /* We've more or less seen every task there is, call it quits */
> > -               if (env->loop > env->loop_max)
> > +               /*
> > +                * We've more or less seen every task there is, call it quits
> > +                * unless we haven't found any movable task yet.
> > +                */
> > +               if (env->loop > env->loop_max &&
> > +                   !(env->flags & LBF_ALL_PINNED))
> >                         break;
> >
> >                 /* take a breather every nr_migrate tasks */
> > @@ -10182,7 +10186,9 @@ static int load_balance(int this_cpu, struct rq *this_rq,
> >
> >                 if (env.flags & LBF_NEED_BREAK) {
> >                         env.flags &= ~LBF_NEED_BREAK;
> > -                       goto more_balance;
> > +                       /* Stop if we tried all running tasks */
> > +                       if (env.loop < busiest->nr_running)
> > +                               goto more_balance;
> >                 }
> >
> >                 /*
> > --
> > 2.17.1
> >

  reply	other threads:[~2024-03-20 16:58 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-08-25 12:27 [PATCH 0/4] sched/fair: fixes in presence of lot of sched_idle tasks Vincent Guittot
2022-08-25 12:27 ` [PATCH 1/4] sched/fair: make sure to try to detach at least one movable task Vincent Guittot
2022-09-12  8:44   ` Dietmar Eggemann
2022-09-13  8:54     ` Vincent Guittot
2022-09-15 14:24   ` [tip: sched/core] sched/fair: Make " tip-bot2 for Vincent Guittot
2024-02-12 20:28   ` [PATCH 1/4] sched/fair: make " Josh Don
2024-03-20 16:57     ` Vincent Guittot [this message]
2024-03-21 20:25       ` Josh Don
2024-03-22 17:16         ` Vincent Guittot
2024-03-22 19:49           ` Josh Don
2022-08-25 12:27 ` [PATCH 2/4] sched/fair: cleanup loop_max and loop_break Vincent Guittot
2022-09-12  8:45   ` Dietmar Eggemann
2022-09-13  8:37     ` Vincent Guittot
2022-09-15 14:24   ` [tip: sched/core] sched/fair: Cleanup " tip-bot2 for Vincent Guittot
2022-08-25 12:27 ` [PATCH 3/4] sched/fair: move call to list_last_entry() in detach_tasks Vincent Guittot
2022-09-12  8:46   ` Dietmar Eggemann
2022-09-15 14:24   ` [tip: sched/core] sched/fair: Move " tip-bot2 for Vincent Guittot
2022-08-25 12:27 ` [PATCH 4/4] sched/fair: limit sched slice duration Vincent Guittot
2022-09-09 11:14   ` Peter Zijlstra
2022-09-09 14:03     ` Vincent Guittot
2022-09-09 15:05       ` Vincent Guittot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAKfTPtDSC25N8TvszDAjseqdLdGy4qiDnwobNThkt8piSL_5Pw@mail.gmail.com \
    --to=vincent.guittot@linaro.org \
    --cc=bristot@redhat.com \
    --cc=bsegall@google.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=joshdon@google.com \
    --cc=juri.lelli@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=vschneid@redhat.com \
    --cc=zhangqiao22@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®