From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90897C433E7 for ; Tue, 20 Oct 2020 01:44:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5703122244 for ; Tue, 20 Oct 2020 01:44:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="AxgB6+vW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728459AbgJTBoY (ORCPT ); Mon, 19 Oct 2020 21:44:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34636 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390197AbgJTBoR (ORCPT ); Mon, 19 Oct 2020 21:44:17 -0400 Received: from mail-il1-x143.google.com (mail-il1-x143.google.com [IPv6:2607:f8b0:4864:20::143]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D657EC0613D1 for ; Mon, 19 Oct 2020 18:44:16 -0700 (PDT) Received: by mail-il1-x143.google.com with SMTP id p9so331672ilr.1 for ; Mon, 19 Oct 2020 18:44:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tzFeLd3Ul9UYvJhyhWguM7eMGuk5M7WHuEML0jXsWiQ=; b=AxgB6+vWNXD/bGD3Ct/8etWge7ecjLIazUQe0x+mzC3OIenqFcllfkG+/QuW8j7gtQ FCLZ8GgCGR6r6lzRTRdXFw5RmiZ/M9OI+xnx9hNMHt/TCqxUJJe/KsvtZL8hcJ2WtgGp DT/Tv3JIWAr8ypKXLG/MvlIuikBrAs5qbNRYQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tzFeLd3Ul9UYvJhyhWguM7eMGuk5M7WHuEML0jXsWiQ=; b=HmFWakXrqxd/ups8F1IQhrlBsG0QXEnoY8ONpzYB/el5ZY9eNboSK7HaomsEIArdeq HPEsXwL838Fq6b5waObRyq0EU7o6+OfplHZGYsWv5PXo55p2k6hX+pb3lcbZ0MOW0UzV 3YcJnU8Ob2zUSJYrM1m1fOO+V82HlZg9fNmKETGCLQEbvzQ0aS1ZRjfjZKTx5U/95u/U 5McxIngD9THx0LEYdvjsvaVn/ybGr0TmeDSoQqpDON6iKgUHcFeuVsZihICaeI4AxmPa UvjExGwpaRtg4ehYUOw8ySB/4fo1LAMVw7TEfYPFbyDOE1QgMsXZ9HPgl8UBRBNQSGXt 2WFA== X-Gm-Message-State: AOAM5303w3TyKxCbm49aUwUQGhPSv2OFQRirmLgJniFETfYQMAqUYZJ+ Ulbe3aHGTi8OPRmQnsf0LQiNiQ== X-Google-Smtp-Source: ABdhPJxtJiEorwgIw5K821aXhnwzt1vki2c5A1qI+pe/JbpZI6STW4minAhTmEbI8tb3/z/84gmSmQ== X-Received: by 2002:a92:dcc7:: with SMTP id b7mr252188ilr.210.1603158256201; Mon, 19 Oct 2020 18:44:16 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:cad3:ffff:feb3:bd59]) by smtp.gmail.com with ESMTPSA id t22sm485306ili.9.2020.10.19.18.44.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Oct 2020 18:44:15 -0700 (PDT) From: "Joel Fernandes (Google)" To: Nishanth Aravamudan , Julien Desfossez , Peter Zijlstra , Tim Chen , Vineeth Pillai , Aaron Lu , Aubrey Li , tglx@linutronix.de, linux-kernel@vger.kernel.org Cc: mingo@kernel.org, torvalds@linux-foundation.org, fweisbec@gmail.com, keescook@chromium.org, kerrnel@google.com, Phil Auld , Valentin Schneider , Mel Gorman , Pawan Gupta , Paolo Bonzini , joel@joelfernandes.org, vineeth@bitbyteword.org, Chen Yu , Christian Brauner , Agata Gruza , Antonio Gomez Iglesias , graf@amazon.com, konrad.wilk@oracle.com, dfaggioli@suse.com, pjt@google.com, rostedt@goodmis.org, derkling@google.com, benbjiang@tencent.com, Alexandre Chartre , James.Bottomley@hansenpartnership.com, OWeisse@umich.edu, Dhaval Giani , Junaid Shahid , jsbarnes@google.com, chris.hyser@oracle.com, Aubrey Li , Aubrey Li , Vineeth Remanan Pillai , "Paul E. McKenney" , Tim Chen Subject: [PATCH v8 -tip 10/26] sched: migration changes for core scheduling Date: Mon, 19 Oct 2020 21:43:20 -0400 Message-Id: <20201020014336.2076526-11-joel@joelfernandes.org> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201020014336.2076526-1-joel@joelfernandes.org> References: <20201020014336.2076526-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Aubrey Li - Don't migrate if there is a cookie mismatch Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU. - Select cookie matched idle CPU In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU. - Find cookie matched idlest CPU In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie - Don't migrate task if cookie not match For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie Tested-by: Julien Desfossez Signed-off-by: Aubrey Li Signed-off-by: Tim Chen Signed-off-by: Vineeth Remanan Pillai --- kernel/sched/fair.c | 64 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 29 ++++++++++++++++++++ 2 files changed, 88 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9cae08c3fca1..93a3b874077d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1912,6 +1912,15 @@ static void task_numa_find_cpu(struct task_numa_env *env, if (!cpumask_test_cpu(cpu, env->p->cpus_ptr)) continue; +#ifdef CONFIG_SCHED_CORE + /* + * Skip this cpu if source task's cookie does not match + * with CPU's core cookie. + */ + if (!sched_core_cookie_match(cpu_rq(cpu), env->p)) + continue; +#endif + env->dst_cpu = cpu; if (task_numa_compare(env, taskimp, groupimp, maymove)) break; @@ -5846,11 +5855,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this /* Traverse only the allowed CPUs */ for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) { + struct rq *rq = cpu_rq(i); + +#ifdef CONFIG_SCHED_CORE + if (!sched_core_cookie_match(rq, p)) + continue; +#endif + if (sched_idle_cpu(i)) return i; if (available_idle_cpu(i)) { - struct rq *rq = cpu_rq(i); struct cpuidle_state *idle = idle_get_state(rq); if (idle && idle->exit_latency < min_exit_latency) { /* @@ -6108,8 +6123,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t for_each_cpu_wrap(cpu, cpus, target) { if (!--nr) return -1; - if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) - break; + + if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) { +#ifdef CONFIG_SCHED_CORE + /* + * If Core Scheduling is enabled, select this cpu + * only if the process cookie matches core cookie. + */ + if (sched_core_enabled(cpu_rq(cpu)) && + p->core_cookie == cpu_rq(cpu)->core->core_cookie) +#endif + break; + } } time = cpu_clock(this) - time; @@ -7495,8 +7520,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) * We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or - * 3) running (obviously), or - * 4) are cache-hot on their current CPU. + * 3) task's cookie does not match with this CPU's core cookie + * 4) running (obviously), or + * 5) are cache-hot on their current CPU. */ if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu)) return 0; @@ -7531,6 +7557,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) return 0; } +#ifdef CONFIG_SCHED_CORE + /* + * Don't migrate task if the task's cookie does not match + * with the destination CPU's core cookie. + */ + if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p)) + return 0; +#endif + /* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED; @@ -8757,6 +8792,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr)) continue; +#ifdef CONFIG_SCHED_CORE + if (sched_core_enabled(cpu_rq(this_cpu))) { + int i = 0; + bool cookie_match = false; + + for_each_cpu(i, sched_group_span(group)) { + struct rq *rq = cpu_rq(i); + + if (sched_core_cookie_match(rq, p)) { + cookie_match = true; + break; + } + } + /* Skip over this group if no cookie matched */ + if (!cookie_match) + continue; + } +#endif + local_group = cpumask_test_cpu(this_cpu, sched_group_span(group)); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 58f741b52103..d0c7a7f87d73 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1111,6 +1111,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq) bool cfs_prio_less(struct task_struct *a, struct task_struct *b); +/* + * Helper to check if the CPU's core cookie matches with the task's cookie + * when core scheduling is enabled. + * A special case is that the task's cookie always matches with CPU's core + * cookie if the CPU is in an idle core. + */ +static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p) +{ + bool idle_core = true; + int cpu; + + /* Ignore cookie match if core scheduler is not enabled on the CPU. */ + if (!sched_core_enabled(rq)) + return true; + + for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) { + if (!available_idle_cpu(cpu)) { + idle_core = false; + break; + } + } + + /* + * A CPU in an idle core is always the best choice for tasks with + * cookies. + */ + return idle_core || rq->core->core_cookie == p->core_cookie; +} + extern void queue_core_balance(struct rq *rq); #else /* !CONFIG_SCHED_CORE */ -- 2.29.0.rc1.297.gfa9743e501-goog