From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F23D17BCA for ; Fri, 26 Jun 2026 00:10:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.19 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782432617; cv=none; b=bB2PCXqqTGv5g9hUJO7jMQsoUsQ4GDy02upv3lvCfv6QaPFJWqrOD4kwrmhEZv4F2sp31EzsZBb1CKuKlpGrCzus3DNdbEnqP9V18cjnqCb2zP1LyC11HO1LwzR6ZUBVJCofbEEArod+0zoyPRN+bMj+wJLOUfYWpHV0nyeInI4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782432617; c=relaxed/simple; bh=GvWgmalwXW7W9C9gJb+13a28E1P4vnRMwV8i6KAAxI0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ITcan7C5kGlaJZu1StwePA21tbrpniLzucRgQQRPDaQ9W8Zo+Azax1PgQS4ZoUR1o1uwrP9rduk4x0ecLwTcwTMcwpgA0Aee9A8tj6Mqj6q/6wd9XY873xnoTE3y2E5fPK/6aycpmo4PJMiNNcyjnv9AQcBUZ18uSSBXewubRvg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=LwGE6Mj0; arc=none smtp.client-ip=198.175.65.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="LwGE6Mj0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1782432617; x=1813968617; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=GvWgmalwXW7W9C9gJb+13a28E1P4vnRMwV8i6KAAxI0=; b=LwGE6Mj0EVyWLPb/JYwUdnAuALpzb9Q4nhaO7IwHJNI3vhfF4L5LEuH8 PNSEupk56b9tF93Jl5nXloOTrs/aIiqhhjaVDEnt7h8lwYksigJg2QpRS aD+xz9a9DrhhFfmqtULjENLjVDmi1SmwBo1SHMp4YfMj4G1gI9VVvk4es KouCpIpBNyvuPonuXbF3ihDw1uAk5F894ke5VwQXwXFKAJiRn/P6ndQIi dT9wVEDVoacOIsZ0XwgmOz3HZgsvf7CZT3uEwNUdt8Bw8TQHB9MU924+/ hpkx69XpyLhJpR1OBXjjud1sXPaAoz5aZRmTWW61GYy8BfFMJ8CykorS3 w==; X-CSE-ConnectionGUID: y6TOx/0vSYiHnG1b+jc6bg== X-CSE-MsgGUID: 6zPUilBJRn6l3RyDrl85qQ== X-IronPort-AV: E=McAfee;i="6800,10657,11828"; a="83239041" X-IronPort-AV: E=Sophos;i="6.24,225,1774335600"; d="scan'208";a="83239041" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jun 2026 17:10:16 -0700 X-CSE-ConnectionGUID: kttt7p3RSm+ecMXMYdIi/Q== X-CSE-MsgGUID: 9PI9PvxqSm2uQKZ8nXbV4g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.24,225,1774335600"; d="scan'208";a="253078622" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmviesa004.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jun 2026 17:10:15 -0700 Date: Thu, 25 Jun 2026 17:19:37 -0700 From: Ricardo Neri To: Vincent Guittot Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Tim C Chen , Chen Yu , Christian Loehle , K Prateek Nayak , Barry Song , "Rafael J. Wysocki" , Andrea Righi , Len Brown , ricardo.neri@intel.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH v5 6/6] sched/topology: Do not clear SD_PREFER_SIBLING in domains with clusters Message-ID: <20260626001937.GB19085@ranerica-svr.sc.intel.com> References: <20260622-rneri-fix-cas-clusters-v5-0-19968f2d1497@linux.intel.com> <20260622-rneri-fix-cas-clusters-v5-6-19968f2d1497@linux.intel.com> <20260624051457.GA16816@ranerica-svr.sc.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260624051457.GA16816@ranerica-svr.sc.intel.com> User-Agent: Mutt/1.9.4 (2018-02-28) On Tue, Jun 23, 2026 at 10:14:57PM -0700, Ricardo Neri wrote: > On Tue, Jun 23, 2026 at 09:26:57AM +0200, Vincent Guittot wrote: > > On Tue, 23 Jun 2026 at 01:55, Ricardo Neri > > wrote: > > > > > > Some topologies have scheduling domains that contain CPUs of asymmetric > > > capacity, grouped into two or more clusters of equal-capacity CPUs > > > sharing an L2 cache. When CONFIG_SCHED_CLUSTER is enabled, load must be > > > balanced across these clusters. > > > > > > Do not clear SD_PREFER_SIBLING in the child domains to indicate to the > > > load balancer that it should spread load among cluster siblings. > > > > > > Checks for capacity in update_sd_pick_busiest(), > > > sched_balance_find_src_group(), and sched_balance_find_src_rq() prevent > > > migrations from high- to low-capacity CPUs if the busiest group is not > > > overloaded. > > > > > > CPUs with spare capacity, big or small, have always helped overloaded > > > groups. Once the overloading condition disappears, misfit load will still > > > be used to move high-utilization tasks to bigger CPUs if they have spare > > > capacity. > > > > > > Adding the SD_PREFER_SIBLING flag shifts load balancing in shared-LLC > > > domains from equalizing the number of idle CPUs to equalizing the number > > > of running tasks. This also enables migrations among clusters from newly- > > > idle load balance, where the outgoing task is already dequeued but the CPU > > > has not yet transitioned to idle. > > > > > > Reviewed-by: Tim Chen > > > Tested-by: Christian Loehle > > > Signed-off-by: Ricardo Neri > > > --- > > > Changes in v5: > > > * Improved inline comments for accuracy. > > > * Added Tested-by tag from Christian. Thanks! > > > > > > Changes in v4: > > > * Added Reviewed-by tag from Tim. Thanks! > > > > > > Changes in v3: > > > * Updated documentation of SD_PREFER_SIBLING. > > > * Expanded the patch description to explain the behavior when overloaded > > > groups are involved. > > > > > > Changes in v2: > > > * Reworded the patch description for clarity. > > > * Kept parentheses around bitwise operators for clarity. > > > --- > > > include/linux/sched/sd_flags.h | 3 ++- > > > kernel/sched/topology.c | 14 ++++++++++++-- > > > 2 files changed, 14 insertions(+), 3 deletions(-) > > > > > > diff --git a/include/linux/sched/sd_flags.h b/include/linux/sched/sd_flags.h > > > index 42839cfa2778..f9a46fb8cacf 100644 > > > --- a/include/linux/sched/sd_flags.h > > > +++ b/include/linux/sched/sd_flags.h > > > @@ -147,7 +147,8 @@ SD_FLAG(SD_ASYM_PACKING, SDF_NEEDS_GROUPS) > > > * Prefer to place tasks in a sibling domain > > > * > > > * Set up until domains start spanning NUMA nodes. Close to being a SHARED_CHILD > > > - * flag, but cleared below domains with SD_ASYM_CPUCAPACITY. > > > + * flag, but cleared below domains with SD_ASYM_CPUCAPACITY unless those child > > > + * domains have clusters of CPUs sharing cache. > > > * > > > * NEEDS_GROUPS: Load balancing flag. > > > */ > > > diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c > > > index 622e2e01974c..261b407d0936 100644 > > > --- a/kernel/sched/topology.c > > > +++ b/kernel/sched/topology.c > > > @@ -1995,8 +1995,18 @@ sd_init(struct sched_domain_topology_level *tl, > > > /* > > > * Convert topological properties into behaviour. > > > */ > > > - /* Don't attempt to spread across CPUs of different capacities. */ > > > - if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child) > > > + /* > > > + * Don't attempt to spread across CPUs of different capacities. > > > + * > > > + * If the child domain has clusters of CPUs sharing L2 cache, keep the > > > + * flag to spread tasks across clusters of identical capacity. Checks in > > > + * the load balancer prevent task migrations from high- to low-capacity > > > + * CPUs unless the source group is overloaded. Migrations to a lower- > > > + * capacity CPU can happen if a higher-capacity group is overloaded and > > > + * a lower-capacity CPU has spare capacity. > > > + */ > > > + if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child && > > > + !(sd->child->flags & SD_CLUSTER)) > > > sd->child->flags &= ~SD_PREFER_SIBLING; > > > > Last time I looked at this patch I was balanced between your proposal > > above and simply keeping SD_PREFER_SIBLING for all HMP topologies. As > > added in the comment: > > " Checks in > > * the load balancer prevent task migrations from high- to low-capacity > > * CPUs unless the source group is overloaded. > > " > > So, why should we bother for (SD_ASYM_CPUCAPACITY && !SD_CLUSTER) topology ? > > No reason, AFAICS. I just wanted to restrict the change to the target > topology of this patchset. > > But you raise a good point: given the checks in place in the load balancer, > it should be OK to keep SD_PREFER_SIBLING in all asymmetric topologies. I > will run a few experiments to confirm. I ran a few experiments with and without CONFIG_CLUSTER_SCHED. I ran N threads where N < nproc to ensure that sched groups were classified as has_spare or fully_busy. The threads saturated the CPUs to minimize task placement decisions at wake up. I observed these threads to remain on the CPUs with highest capacity; no spreading. I repeated the experiment with EAS enabled and threads ramping up utilization. EAS kept them on small CPUs and later duly moved to CPUs of higher capacity as they became misfits. I will update my patch to keep SD_PREFER_SIBLING regardless of asymmetric capacity.