From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE8E830B50F for ; Fri, 26 Jun 2026 14:54:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=209.85.208.42 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782485675; cv=pass; b=trp4fx1sntEqvCGKQrIfUao2wTe6EFa8YucptizHnbxGObnk4F0CwloerJgJQLNfrroHZgnwmERpZSXWTc97zOVBLx34C8qcUd4rexq8wImAs1tR4T+OLMgv7UJTbCnQQB/0o7O3WYv2ZBLYhWZ2LKaFX6DjOhxQJ8Kt1fuBxYc= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782485675; c=relaxed/simple; bh=Ad0N20l91V+13sM0OuyvpIwVxYhVd5EGQpRxHBNpVtA=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=M0JROPrSitBYbtUGnNV/WH3JRQjH9CVZ6Bc2eToPZbO57qBXgO4Iamhiu7UijOhnCnvc7JBBiysXK8UdpGEHsle7zIU6kejqjBveYzMLdTF6F3lVDP1ybcQNsgWOPlfDl0KYNyoUs3n+ez3B2OzWYJUnihh8Uj90giu9SkqNMzk= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=o/D2sdU/; arc=pass smtp.client-ip=209.85.208.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="o/D2sdU/" Received: by mail-ed1-f42.google.com with SMTP id 4fb4d7f45d1cf-693c69b97e7so2052714a12.2 for ; Fri, 26 Jun 2026 07:54:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1782485672; cv=none; d=google.com; s=arc-20260327; b=eVE5KubGXLXDBvRO2B/zN7qp4sQ0jiRbS+yyc2YmIvT+aitDvmnPP8rtf+RBaVA3W0 akpuL/etPtRAb5u2a13nHGFN/oA7K1yCuPXRpoEK7mSnuydAXdDONIHxSMje2kJB+naM 4jvrAQtXMMyyBBWvqe/FilJBQLH7Ob762z2g7rKldpw0kqd64ALkO/UF5kT21b+Ewew3 f7+9c+tWSsL0zSr7mU/Sx3KZobiaHf8N53oW9RoysaqJ8m0RWQbEC5xgRHsHk/S5m6yM k5NmJE6ELIx+xqGf+5r95ViX4a3wJ91KVU/7n9Z3dCKim2dgCLDubUZ9e06Z6SEnRn2g 0keQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=tj7Ev36QgtaqaSaH9aNVYeG6F2L/NsjWPcNFjnX8a+M=; fh=qHgBzHDj5VukdYyuWyaKiqZ0ivo9hmeh5D5xCyhQ9VA=; b=k9xWb/Nowx9xbO3bztsAb4yzY/wSuD17BAkVTGUV0M2hFb0Xq6ws+juQXBwbuQsC09 ARvYT3wE81pm16gddUu1mzL+EUI4Tfu10WztxBnMlaqerX1vDgzUSWM+2cq4BasaEcg0 dDI08YtFbm+0yKAM+s4KzaznS5rpFxeEiL+PeBkaRseXnyCupAr8xZ+ifwd98Uyzg7fj IThhzEfVZsLx0svCxbcwBqOMWT7D75xi9AO0E+OuNz5trunkvOYzU+NyCzC0kiA1JalV l4JdsOtILx8wtpnpI9TZM2cHVwdx3FAsRAynbvyWcxvIl3Pr8V6PS8H9tM9qohi5f5SE vS9Q==; darn=vger.kernel.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1782485672; x=1783090472; darn=vger.kernel.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=tj7Ev36QgtaqaSaH9aNVYeG6F2L/NsjWPcNFjnX8a+M=; b=o/D2sdU/imd3p6bnjmJZrfLemImmxcebAFMIeRJ+NYoGHurnRJnshSNd8+7npZD/Xm vZVhD/sZ6PkZWTTGBjSmwZchbtR9h5ZkF1DX1ofT/EK/9mSv/sO2QTtIVxVcTBxbGi6h xO/nFE4+Hz+alTcAP9kRh6lmIuons4bxVtT4xZ/HIDJhQ+5Guru/mJDYJPySwJFgeMUI ITkABmLibh8yxWiW0jKgCDA5njIuS1MNVxfCT3MIWmLXB+Vmyhr+eHTgzcbiYuV3oB2l Gq5R6BllGkWpuQD0pJjD7LgqQ2Ch4HJoikkqBVksrDk0yJdVFitNYqJ6CK2iNIiRqSt0 V49w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782485672; x=1783090472; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=tj7Ev36QgtaqaSaH9aNVYeG6F2L/NsjWPcNFjnX8a+M=; b=hSVrnGeL/86HbJLhtntNjqBy3plbvhTHbU0FicharPIviT69Mnw+8C16DrOwuFVem+ hDa627jLdj9rpr+inf5aOBqLAuTM4mWxOT70OkdJyQsQkSVxqls/JSTUvdWA5v23c9ya B3KItaikkRt0Au7yIDXNU2c9iGemAZbs/EE56BaOUU1CAPTCnqOSmFwG+PvLIshYqxeG g7Fn8yuHvktJVSYOaahrKYtvHDdc31uo97R0jmA7hsmNDJ790oB/JSfeh3kCF+bFYQzs XgTAFmf0PW1eVCHi59GdE/AsZd0xYcYU7rdEDaUwnYaJ5EQqX4v8JkU0J1OA4eJyRCQB iK1g== X-Forwarded-Encrypted: i=1; AHgh+RrLzGeCEfCpd4oFxm3kcODPXzIgTn3MvnSE85QGoqRlyfOYUifgLNsJhYtTHEClfi6aHWRIgpTEVxg/XJs=@vger.kernel.org X-Gm-Message-State: AOJu0Yw3zKIx1xW7rwFQJ3D7AKeBxFMF8/Pz5Fy3uL4LotbKBYAiS0D8 7dWiytTq0WtLCsKdm2QC5iFynK5zoCEUXB46jhLEZYif0w2Zqhkn/z5Jc1z6/+UIYo5vB9ofeVZ +2mbZwDM9fII2bqiUKt+oPUhNhZFr7uPL/48tfiOIcg== X-Gm-Gg: AfdE7cnYPxgC9/fyj2YQtjOor3QrqzqiBN4pGqHZcj1oAWKQu5hcCr5P8LXPdkP3Znq outEQrwrZAAuWseMf4NVVmYsaE8LKJry64SKhlyid96TAJFGq+3PLpSbJn+Zk5vFJXSi3R+CP3C cxhY6UqDiM4vTWS6w61JewNOj5PmyWQqlxTRzZ0kUAUCCNnNz0wpf/7BkKVvl9QElHtgV/UddQ8 hs7BZbRc4etSVDQh4W5krPy0L7OF75Oz3H5W3TzrWb3V5/Av+JPqZzOzIFK3Oyx7hjJfLj8hpXK Y7rnkHdGcNIXg6eFwHWOtG0lCA== X-Received: by 2002:a05:6402:d0e:b0:698:f7d:4e11 with SMTP id 4fb4d7f45d1cf-69810a17aeamr2303193a12.1.1782485672210; Fri, 26 Jun 2026 07:54:32 -0700 (PDT) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20260622-rneri-fix-cas-clusters-v5-0-19968f2d1497@linux.intel.com> <20260622-rneri-fix-cas-clusters-v5-6-19968f2d1497@linux.intel.com> <20260624051457.GA16816@ranerica-svr.sc.intel.com> <20260626001937.GB19085@ranerica-svr.sc.intel.com> In-Reply-To: <20260626001937.GB19085@ranerica-svr.sc.intel.com> From: Vincent Guittot Date: Fri, 26 Jun 2026 16:54:18 +0200 X-Gm-Features: AVVi8CeThMpp1fxBCgQwwhpUNB_MCEAUKYIaJPBuUX0AdbqlN6XbysmIHH7PgRU Message-ID: Subject: Re: [PATCH v5 6/6] sched/topology: Do not clear SD_PREFER_SIBLING in domains with clusters To: Ricardo Neri Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Tim C Chen , Chen Yu , Christian Loehle , K Prateek Nayak , Barry Song , "Rafael J. Wysocki" , Andrea Righi , Len Brown , ricardo.neri@intel.com, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" On Fri, 26 Jun 2026 at 02:10, Ricardo Neri wrote: > > On Tue, Jun 23, 2026 at 10:14:57PM -0700, Ricardo Neri wrote: > > On Tue, Jun 23, 2026 at 09:26:57AM +0200, Vincent Guittot wrote: > > > On Tue, 23 Jun 2026 at 01:55, Ricardo Neri > > > wrote: > > > > > > > > Some topologies have scheduling domains that contain CPUs of asymmetric > > > > capacity, grouped into two or more clusters of equal-capacity CPUs > > > > sharing an L2 cache. When CONFIG_SCHED_CLUSTER is enabled, load must be > > > > balanced across these clusters. > > > > > > > > Do not clear SD_PREFER_SIBLING in the child domains to indicate to the > > > > load balancer that it should spread load among cluster siblings. > > > > > > > > Checks for capacity in update_sd_pick_busiest(), > > > > sched_balance_find_src_group(), and sched_balance_find_src_rq() prevent > > > > migrations from high- to low-capacity CPUs if the busiest group is not > > > > overloaded. > > > > > > > > CPUs with spare capacity, big or small, have always helped overloaded > > > > groups. Once the overloading condition disappears, misfit load will still > > > > be used to move high-utilization tasks to bigger CPUs if they have spare > > > > capacity. > > > > > > > > Adding the SD_PREFER_SIBLING flag shifts load balancing in shared-LLC > > > > domains from equalizing the number of idle CPUs to equalizing the number > > > > of running tasks. This also enables migrations among clusters from newly- > > > > idle load balance, where the outgoing task is already dequeued but the CPU > > > > has not yet transitioned to idle. > > > > > > > > Reviewed-by: Tim Chen > > > > Tested-by: Christian Loehle > > > > Signed-off-by: Ricardo Neri > > > > --- > > > > Changes in v5: > > > > * Improved inline comments for accuracy. > > > > * Added Tested-by tag from Christian. Thanks! > > > > > > > > Changes in v4: > > > > * Added Reviewed-by tag from Tim. Thanks! > > > > > > > > Changes in v3: > > > > * Updated documentation of SD_PREFER_SIBLING. > > > > * Expanded the patch description to explain the behavior when overloaded > > > > groups are involved. > > > > > > > > Changes in v2: > > > > * Reworded the patch description for clarity. > > > > * Kept parentheses around bitwise operators for clarity. > > > > --- > > > > include/linux/sched/sd_flags.h | 3 ++- > > > > kernel/sched/topology.c | 14 ++++++++++++-- > > > > 2 files changed, 14 insertions(+), 3 deletions(-) > > > > > > > > diff --git a/include/linux/sched/sd_flags.h b/include/linux/sched/sd_flags.h > > > > index 42839cfa2778..f9a46fb8cacf 100644 > > > > --- a/include/linux/sched/sd_flags.h > > > > +++ b/include/linux/sched/sd_flags.h > > > > @@ -147,7 +147,8 @@ SD_FLAG(SD_ASYM_PACKING, SDF_NEEDS_GROUPS) > > > > * Prefer to place tasks in a sibling domain > > > > * > > > > * Set up until domains start spanning NUMA nodes. Close to being a SHARED_CHILD > > > > - * flag, but cleared below domains with SD_ASYM_CPUCAPACITY. > > > > + * flag, but cleared below domains with SD_ASYM_CPUCAPACITY unless those child > > > > + * domains have clusters of CPUs sharing cache. > > > > * > > > > * NEEDS_GROUPS: Load balancing flag. > > > > */ > > > > diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c > > > > index 622e2e01974c..261b407d0936 100644 > > > > --- a/kernel/sched/topology.c > > > > +++ b/kernel/sched/topology.c > > > > @@ -1995,8 +1995,18 @@ sd_init(struct sched_domain_topology_level *tl, > > > > /* > > > > * Convert topological properties into behaviour. > > > > */ > > > > - /* Don't attempt to spread across CPUs of different capacities. */ > > > > - if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child) > > > > + /* > > > > + * Don't attempt to spread across CPUs of different capacities. > > > > + * > > > > + * If the child domain has clusters of CPUs sharing L2 cache, keep the > > > > + * flag to spread tasks across clusters of identical capacity. Checks in > > > > + * the load balancer prevent task migrations from high- to low-capacity > > > > + * CPUs unless the source group is overloaded. Migrations to a lower- > > > > + * capacity CPU can happen if a higher-capacity group is overloaded and > > > > + * a lower-capacity CPU has spare capacity. > > > > + */ > > > > + if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child && > > > > + !(sd->child->flags & SD_CLUSTER)) > > > > sd->child->flags &= ~SD_PREFER_SIBLING; > > > > > > Last time I looked at this patch I was balanced between your proposal > > > above and simply keeping SD_PREFER_SIBLING for all HMP topologies. As > > > added in the comment: > > > " Checks in > > > * the load balancer prevent task migrations from high- to low-capacity > > > * CPUs unless the source group is overloaded. > > > " > > > So, why should we bother for (SD_ASYM_CPUCAPACITY && !SD_CLUSTER) topology ? > > > > No reason, AFAICS. I just wanted to restrict the change to the target > > topology of this patchset. > > > > But you raise a good point: given the checks in place in the load balancer, > > it should be OK to keep SD_PREFER_SIBLING in all asymmetric topologies. I > > will run a few experiments to confirm. > > I ran a few experiments with and without CONFIG_CLUSTER_SCHED. I ran N > threads where N < nproc to ensure that sched groups were classified as > has_spare or fully_busy. The threads saturated the CPUs to minimize task > placement decisions at wake up. > > I observed these threads to remain on the CPUs with highest capacity; no > spreading. > > I repeated the experiment with EAS enabled and threads ramping up > utilization. EAS kept them on small CPUs and later duly moved to CPUs of > higher capacity as they became misfits. > > I will update my patch to keep SD_PREFER_SIBLING regardless of asymmetric > capacity. Thanks