From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLACK,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E426C433ED for ; Tue, 6 Apr 2021 23:37:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 45D12610FA for ; Tue, 6 Apr 2021 23:37:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242705AbhDFXhe (ORCPT ); Tue, 6 Apr 2021 19:37:34 -0400 Received: from mga14.intel.com ([192.55.52.115]:2472 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233184AbhDFXhc (ORCPT ); Tue, 6 Apr 2021 19:37:32 -0400 IronPort-SDR: dwIzZrz9ewKjJDzaCAaijLNrbr44NxBvKcvBNDU3zs2SxHq/3T8YsEGLCG3elSAGv59M+5bkDD bCzQ+DVsHD/w== X-IronPort-AV: E=McAfee;i="6000,8403,9946"; a="192718245" X-IronPort-AV: E=Sophos;i="5.82,201,1613462400"; d="scan'208";a="192718245" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Apr 2021 16:37:23 -0700 IronPort-SDR: g8Qp21V4eKDGDOkBrafrvZ9n3nX6o8xhpNCxqvk379jd98aq+SFYvSzfgdeH6kzoZZh4YQykx2 7K5bnIKGic2Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,201,1613462400"; d="scan'208";a="441108616" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga004.fm.intel.com with ESMTP; 06 Apr 2021 16:37:23 -0700 Date: Tue, 6 Apr 2021 16:36:52 -0700 From: Ricardo Neri To: Vincent Guittot Cc: "Peter Zijlstra (Intel)" , Ingo Molnar , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Len Brown , Srinivas Pandruvada , Tim Chen , Aubrey Li , "Ravi V. Shankar" , Ricardo Neri , Quentin Perret , "Joel Fernandes (Google)" , linux-kernel , Aubrey Li , Ben Segall , Daniel Bristot de Oliveira Subject: Re: [PATCH 2/4] sched/fair: Introduce arch_sched_asym_prefer_early() Message-ID: <20210406233652.GD27195@ranerica-svr.sc.intel.com> References: <20210406041108.7416-1-ricardo.neri-calderon@linux.intel.com> <20210406041108.7416-3-ricardo.neri-calderon@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Apr 06, 2021 at 04:31:35PM +0200, Vincent Guittot wrote: > On Tue, 6 Apr 2021 at 06:11, Ricardo Neri > wrote: > > > > Introduce arch_sched_asym_prefer_early() so that architectures with SMT > > can delay the decision to label a candidate busiest group as > > group_asym_packing. > > > > When using asymmetric packing, high priority idle CPUs pull tasks from > > scheduling groups with low priority CPUs. The decision on using asymmetric > > packing for load balancing is done after collecting the statistics of a > > candidate busiest group. However, this decision needs to consider the > > state of SMT siblings of dst_cpu. > > > > Cc: Aubrey Li > > Cc: Ben Segall > > Cc: Daniel Bristot de Oliveira > > Cc: Dietmar Eggemann > > Cc: Joel Fernandes (Google) > > Cc: Mel Gorman > > Cc: Quentin Perret > > Cc: Srinivas Pandruvada > > Cc: Steven Rostedt > > Cc: Tim Chen > > Reviewed-by: Len Brown > > Signed-off-by: Ricardo Neri > > --- > > include/linux/sched/topology.h | 1 + > > kernel/sched/fair.c | 11 ++++++++++- > > 2 files changed, 11 insertions(+), 1 deletion(-) > > > > diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h > > index 8f0f778b7c91..663b98959305 100644 > > --- a/include/linux/sched/topology.h > > +++ b/include/linux/sched/topology.h > > @@ -57,6 +57,7 @@ static inline int cpu_numa_flags(void) > > #endif > > > > extern int arch_asym_cpu_priority(int cpu); > > +extern bool arch_sched_asym_prefer_early(int a, int b); > > > > struct sched_domain_attr { > > int relax_domain_level; > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > > index 4ef3fa0d5e8d..e74da853b046 100644 > > --- a/kernel/sched/fair.c > > +++ b/kernel/sched/fair.c > > @@ -106,6 +106,15 @@ int __weak arch_asym_cpu_priority(int cpu) > > return -cpu; > > } > > > > +/* > > + * For asym packing, early check if CPUs with higher priority should be > > + * preferred. On some architectures, more data is needed to make a decision. > > + */ > > +bool __weak arch_sched_asym_prefer_early(int a, int b) > > +{ > > + return sched_asym_prefer(a, b); > > +} > > + > > /* > > * The margin used when comparing utilization with CPU capacity. > > * > > @@ -8458,7 +8467,7 @@ static inline void update_sg_lb_stats(struct lb_env *env, > > if (!local_group && env->sd->flags & SD_ASYM_PACKING && > > env->idle != CPU_NOT_IDLE && > > sgs->sum_h_nr_running && > > - sched_asym_prefer(env->dst_cpu, group->asym_prefer_cpu)) { > > + arch_sched_asym_prefer_early(env->dst_cpu, group->asym_prefer_cpu)) { > > If itmt set arch_sched_asym_prefer_early to true all groups will be > set as group_asym_packing unconditionally which is wrong. The state > has to be set only when we want asym packing migration Thanks for your quick review Vincent! Indeed, sgs->group_asym_packing should only be set when we want asym_packing migration. However, for ITMT we also need to know the state of the SMT siblings of dst_cpu if any. update_sg_lb_stats() does not anything about the local group. Thus, arch_sched_asym_prefer_early() intends to give an early decision that can be revoked later when statistics of the local group are known. I also thought about checking the state of the SMT siblings from update_sg_lb_stats() but that would duplicate the functionality of update_sg_lb_stats() when called on the local group. The group can still be classified as group_overloaded and group_imbalanced as they take precedence over group_asym_packing. We would miss the group_has_spare and group_fully_busy classifications, though. I can look into fixing that. Thanks and BR, Ricardo