From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CY7PR03CU001.outbound.protection.outlook.com (mail-westcentralusazon11010003.outbound.protection.outlook.com [40.93.198.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EB36274670 for ; Tue, 28 Apr 2026 05:17:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.198.3 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777353470; cv=fail; b=ZX5lfLqPXm5CPi928TNvoNtvzqYIJj5lys9cLYuAinNtWEyOW6/cgbgydQFZslV5pvMOfje5wkevfr5nLLvreuWE/hOHYT3KgRdLI6Wwjt0joSOQX8PoXAtMU3BI/IiKRyse80HFAbOUyNsbedHaR/6q/mmYT45UgRfrYtYc/YI= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777353470; c=relaxed/simple; bh=obrD27iNzK6Tnxn1GwVHnJnAIBZiSCnJ8KqljsYv8vQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=t+tOLZGgeuWUJBXwZAJkAsr1fvSEQ0jGWJuiTBP0uHh8W+97lp4dxnXrSTibOGztvHpHj3FtULsId8hTbcaJ4I/Tl7kRhrHN7corQpE87/j9VaG0D58a0HnIB1FzLZo5lp0Nbmu8PNCL0sm5d+JFMhJ/xTy/aGz5tMoVa1Yr2DY= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=HWoN2GHt; arc=fail smtp.client-ip=40.93.198.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="HWoN2GHt" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hIp2iqXwQbUzJQCe/puyHP7WoXmOueyW7nUYNzKwAOErhIGq7VSj2jND/UuyXffO8kWW0Q+iKGZoK3cyCeGBmqvGHJUvyvagQMFXmQ5DSyYRXG71cuBBoDwOVFepLa+LfWUFBcFesddA8oc2OSwtUwGS+oxVMKJ3PDd6SBRg1/C//wFXMh7rlkNRVxChtTvHftoNNGXAQZxd651fXn7CnUUf4W52aI0TfX+/YvY2o1cU2Xfs5LmXGwMyTmnT1dJidvPnggYK6EjvQnFxr7bUycA3P8UpUXZMmgv1ZTyulTOzMKyO9qKoReXVh5ko1Ma2DbiiGIlZFm3ee7HzAOavag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vSAtNK9m0kEL6zmFe5F4NEWSrf8DnHLY3XXNppLxjBM=; b=uopP/hNuSWo+Jzcf9UKFbVjphu5HBaCx8tY9iZE2RK+p0POpITLy81kyAPhM4GLimq/eeAiVDAQ2spQPxO4unTPdEDtydoiP6wG9loAGGUQraxWtPc/F105l7+2DrhQPQbcsFZmJZ/+/02u6TDVKv16ecFxFoSRcjEMWSafo6WTMxKhtsGjD5opyDSw9PMQGjPARcH3dknJqNNy7KDHe/SISe5dKD22ulYudf0bq7sfgqZf+Ipc2F1YKJS2tUOIiFhcA/N+TgVS/mnotMpl4hiNlS60MKjjB9XQ3dc6RZUVdUSeJa+Evp3vjQjurkaZ0+hDU7hxuAH8b1VDK03oPAw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vSAtNK9m0kEL6zmFe5F4NEWSrf8DnHLY3XXNppLxjBM=; b=HWoN2GHt0GDlfw6ClifwAU+iBiue/SZlzDuaWGzN+EMIspgiW8YlArWD71GsyLiMQZgZuEeHaPXFx9mPsz2R5Wisqf+Znh6UputX5jHQkOWDLXtFKfSw+HZr/oPi4dsU673sM3SSVNk7TlpqVRzlj7clevR0kVM2Bq+L3nNfpDzRH+DkR3BBBP2lCzimYsHSNMdvVJaip2z6x0LkpF5NVc3mweZDPZZO1awYmLGL1jW/VB1cWD7JBmvJqddINckqpIUUHdfKu9vCHZ0p/COjUFWy0eM7EN4ooheixNUczWI/RNUpy7JbX7tz3cZnxNUyUtuo+VaJOkDAzt3czwBRGg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by CY8PR12MB8337.namprd12.prod.outlook.com (2603:10b6:930:7d::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.16; Tue, 28 Apr 2026 05:17:43 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9870.013; Tue, 28 Apr 2026 05:17:43 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , linux-kernel@vger.kernel.org Subject: [PATCH 2/6] sched/fair: Attach sched_domain_shared to sd_asym_cpucapacity Date: Tue, 28 Apr 2026 07:16:36 +0200 Message-ID: <20260428051720.3180182-3-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260428051720.3180182-1-arighi@nvidia.com> References: <20260428051720.3180182-1-arighi@nvidia.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: BYAPR08CA0041.namprd08.prod.outlook.com (2603:10b6:a03:117::18) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|CY8PR12MB8337:EE_ X-MS-Office365-Filtering-Correlation-Id: f82e56cf-8d6a-484c-241b-08dea4e579b9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: 2VCd3fVgB+bpznNsnwyiodebk/Xq18KYZvM2yztHDDI9mfMiD36M+USeObGta/j7qtj67+HVBVh0J/6xLbyaIkj/FZGqCk452bNrRfMsYak5ET4xpVex/abDXDmmn/IwsK9NdkGsgvpWKgq+FoJSCFYsJIv+O8VjBYU56A9M6fEjRurVlD0a795+UCFZHg1Do+PEWLrXtt92M99v1dpKElhztLi7MEQHLndBXWmQOhdDCVPlIHz04hjPRyjS9g8QNbmhZ31MyVcamf1t/hZ44hOvyHS0xyLoKFyMWtAEWOyIabvrhWNNmTYRuKu6b0Y+SWJZ9S42A7rKzUozRnupHqXcmtB9M1fahmkubQdYwIDYa1bo5jFfqcMWNweA8Lg/y21fOltKoMMkEMzBXucjpymNPPHDg+P/60cNZ1Kz87mranaNOSp+2Z6/neVHHfRIGdqddX0Z4+sB6tD6csIipICkMdk4br4LDh6HTw6gi0gjvIPG2KitE2oJN2utkf/EqgNBhCIFv031qCG4BjuKg+rPYTHSHhxKY1y77PuwzxhVm+ORIXhA4aBKtQa5kRZWXxRsugA67ZhPaku6xGhY2aE0E36nN7/lFHlo2VMhD0Ln3KLEg/zvXWxDYckFxDeAergZh1WHMUQOhaelw1vMMoUNk3hMHb0BLqJKyD8RKR5EwD839HOzSlp9EyTHeq1al8xzCk9E3PM0VgSdYiE6IgDZ/INKPiMa+7KagJEMkIs= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(7416014)(1800799024)(376014)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?zxHbOiccq79DuenJKQ29BIBo4E4IggvKCkBaBIXHDCgAOf9Jcdhyo5coAhMW?= =?us-ascii?Q?CkdFX5B9/tc+dy6qTeWruc9azml2Fsp2BAagyzH4rxHg47wa3GkVqfZ+fbuZ?= =?us-ascii?Q?/ThIQyRbm6bz3/sZgO7Te0wgxXoQ1G7gmSsqhF/NuQ3yIJuvJvpqymVZWIns?= =?us-ascii?Q?pR4nKCdFVpBDGDYADtQ9al2U/sZhyZvLx9EKUeco5/X49MuVgPbHgI9GyB1S?= =?us-ascii?Q?LH/P0UrKdGyv684R2K5DoDD4UMqfT92diL33PcQpe7bnvkrI/NKv/HaZ5xJP?= =?us-ascii?Q?BZWyPF50sBoC9IyrykS8arL3wvtoChmlyTQL6AXE1ttYjm7vcV8pW1gMgXwf?= =?us-ascii?Q?YdjDa1p2FpUQUBEM9Nj5q4wMfTcrL7784YRpvUMiT0iRZD1CyEBMvhcfAZFG?= =?us-ascii?Q?rEteJdbk66a0tNrhn6DwiVrjdg5b20X87cUOnkbplMU+xGlhlyaPcX3sFZtE?= =?us-ascii?Q?Utlld9zLS0u6R6P4i31ZhTaR3v+T3NYEsaTcNxW/u4Qsr4kanIXq7jgKxTQh?= =?us-ascii?Q?vw71eARoNnicWhdT9TF7vkSMfyqglBuwb+zPCeDYZMfTHWZCsRosjggwPyNT?= =?us-ascii?Q?BFDQTFBie9jImTkHFqxor1dp1uD7uK3Wrzf1SvpOb08uhekwsXs8XmphknSG?= =?us-ascii?Q?pOnHWAK9FKTgZ/iz5YSvzfS6cJtpJc3/QS0IXvZ/TBrRs36nqUt76I6bLFPX?= =?us-ascii?Q?B/j20rc98GBzKj0RGO8e61BFDtw5HB8/RdGIhbO23Z+hoTh2bjiuSXxAIvvx?= =?us-ascii?Q?Krnn1FiiJC2RbXiNkvwCH8zwUvymJruS/wxaFxXfk6xgsMw+WvKYuId22yEI?= =?us-ascii?Q?6YDOJLmSaGQx7uD+I6PyjXIUficWFEpKhVLbBkwTnYTRomnHKKoJsIIqmhYl?= =?us-ascii?Q?Xd6TY3iXMc1W9JA6GFnVWjyrpWanf1Y6XqdKPgV6xpSdNtPJOhnydA1AhAqB?= =?us-ascii?Q?YB7WcN6zclHC+xDbiv6bpXa1F/4DCuDqy5zwOYyV3MtWbaN2veZ3esqoipyc?= =?us-ascii?Q?qFuiZVZ20eRBT6tc+PbyG/fGNpSh3Dz9WA4LZHGklwW6X8irY6G2prVIzTx7?= =?us-ascii?Q?mBFhKZoyQzq1orEeJ9TY373l6ZyPnRsz1XpyluxvK4eWCIOQD9fxpfSJ7S0m?= =?us-ascii?Q?RgB3WUAfTF1nO8sfUKMXhShMOhHVf6UWWfmQ2Gb51N860n4NHBGRwtLSF3A+?= =?us-ascii?Q?oaWXyniiVmGjB3nv1D5YDjGQvBjiyDKrP8u+XKlPHU/8Jonb4DSg2ELzKlsw?= =?us-ascii?Q?SFsIL78fpGzHdl1FTENS0/pSIuFuG7HN3LSp+LaFXo8zTwiUeLmBmLLnGi8G?= =?us-ascii?Q?YDLWPeQhYR0xPY4iAQJrw1KQAcLQTCWyMfdosZkz3fXUQPhXQEpOiqRtkCzA?= =?us-ascii?Q?60pkMo3hioU3wlwXeSkj+9EWJFAVowePCrN7z5eYvj6O1Yy450a05oZgRbfH?= =?us-ascii?Q?tcKJaczYKm5TMV+mN0BkbXkOzufnvU9P7JED2LtTReHcl4jzqNyzxkjkq88R?= =?us-ascii?Q?pFA+lz2DtmM3gJVguWh4fosPUVyAaRYgaeT4q23NrbQD8LmWAca69eCfnc58?= =?us-ascii?Q?PsCLcIeF7/20R6+4RdYN7jWJxEiKySLjt8O3I33T1JGt0OcTmpi+15MD8pvN?= =?us-ascii?Q?Rg3/Dr7B2aBQWDpaJSu4YWdoIWxZd7yWiJfR71ixtHLclHV3/fXXdCzQqKzs?= =?us-ascii?Q?2cQoaJQIEX+7QQV4jH8q3HjOm8evfWRR3Ea8zM6+RiGyAtpfJR5yEx9rByt2?= =?us-ascii?Q?0lQAkxCbng=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f82e56cf-8d6a-484c-241b-08dea4e579b9 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Apr 2026 05:17:43.3256 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CjgQ8UKk1uxjiRW4f1mmeCWMyrnS31Ot1kkkQYmEd9lZIdRDqJi33QTvJdWbfLl2CLcy0O65DzpgkMOzxnSxYA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8337 From: K Prateek Nayak On asymmetric CPU capacity systems, the wakeup path uses select_idle_capacity(), which scans the span of sd_asym_cpucapacity rather than sd_llc. The has_idle_cores hint however lives on sd_llc->shared, so the wakeup-time read of has_idle_cores operates on an LLC-scoped blob while the actual scan/decision spans the asym domain; nr_busy_cpus also lives in the same shared sched_domain data, but it's never used in the asym CPU capacity scenario. Therefore, move the sched_domain_shared object to sd_asym_cpucapacity whenever the CPU has a SD_ASYM_CPUCAPACITY_FULL ancestor and that ancestor is non-overlapping (i.e., not built from SD_NUMA). In that case the scope of has_idle_cores matches the scope of the wakeup scan. Fall back to attaching the shared object to sd_llc in three cases: 1) plain symmetric systems (no SD_ASYM_CPUCAPACITY_FULL anywhere); 2) CPUs in an exclusive cpuset that carves out a symmetric capacity island: has_asym is system-wide but those CPUs have no SD_ASYM_CPUCAPACITY_FULL ancestor in their hierarchy and follow the symmetric LLC path in select_idle_sibling(); 3) exotic topologies where SD_ASYM_CPUCAPACITY_FULL lands on an SD_NUMA-built domain. init_sched_domain_shared() keys the shared blob off cpumask_first(span), which on overlapping NUMA domains would alias unrelated spans onto the same blob. Keep the shared object on the LLC there; select_idle_capacity() gracefully skips the has_idle_cores preference when sd->shared is NULL. While at it, also rename the per-CPU sd_llc_shared to sd_balance_shared, as it is no longer strictly tied to the LLC. Co-developed-by: Andrea Righi Signed-off-by: Andrea Righi Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 20 +++++---- kernel/sched/sched.h | 2 +- kernel/sched/topology.c | 91 +++++++++++++++++++++++++++++++++++------ 3 files changed, 91 insertions(+), 22 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index fc0828150c780..ece3a26f59c27 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7790,7 +7790,7 @@ static inline void set_idle_cores(int cpu, int val) { struct sched_domain_shared *sds; - sds = rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); + sds = rcu_dereference_all(per_cpu(sd_balance_shared, cpu)); if (sds) WRITE_ONCE(sds->has_idle_cores, val); } @@ -7799,7 +7799,7 @@ static inline bool test_idle_cores(int cpu) { struct sched_domain_shared *sds; - sds = rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); + sds = rcu_dereference_all(per_cpu(sd_balance_shared, cpu)); if (sds) return READ_ONCE(sds->has_idle_cores); @@ -7808,7 +7808,7 @@ static inline bool test_idle_cores(int cpu) /* * Scans the local SMT mask to see if the entire core is idle, and records this - * information in sd_llc_shared->has_idle_cores. + * information in sd_balance_shared->has_idle_cores. * * Since SMT siblings share all cache levels, inspecting this limited remote * state should be fairly cheap. @@ -7925,7 +7925,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask); int i, cpu, idle_cpu = -1, nr = INT_MAX; - if (sched_feat(SIS_UTIL)) { + if (sched_feat(SIS_UTIL) && sd->shared) { /* * Increment because !--nr is the condition to stop scan. * @@ -12759,7 +12759,7 @@ static bool nohz_balancer_needs_kick(struct rq *rq) return false; } - sds = rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); + sds = rcu_dereference_all(per_cpu(sd_balance_shared, cpu)); if (sds) { /* * If there is an imbalance between LLC domains (IOW we could @@ -12841,10 +12841,13 @@ static void set_cpu_sd_state_busy(int cpu) guard(rcu)(); sd = rcu_dereference_all(per_cpu(sd_llc, cpu)); - if (!sd || !sd->nohz_idle) + /* + * sd->nohz_idle only pairs with nr_busy_cpus on sd->shared; if this LLC + * domain has no shared object there is nothing to clear or account. + */ + if (!sd || !sd->shared || !sd->nohz_idle) return; sd->nohz_idle = 0; - atomic_inc(&sd->shared->nr_busy_cpus); } @@ -12868,7 +12871,8 @@ static void set_cpu_sd_state_idle(int cpu) guard(rcu)(); sd = rcu_dereference_all(per_cpu(sd_llc, cpu)); - if (!sd || sd->nohz_idle) + /* See set_cpu_sd_state_busy(): nohz_idle is only used with sd->shared. */ + if (!sd || !sd->shared || sd->nohz_idle) return; sd->nohz_idle = 1; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9f63b15d309d1..330f5893c4561 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2170,7 +2170,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_llc); DECLARE_PER_CPU(int, sd_llc_size); DECLARE_PER_CPU(int, sd_llc_id); DECLARE_PER_CPU(int, sd_share_id); -DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); +DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_balance_shared); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_numa); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 5847b83d9d552..1e6ce369a4bbc 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -665,7 +665,7 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc); DEFINE_PER_CPU(int, sd_llc_size); DEFINE_PER_CPU(int, sd_llc_id); DEFINE_PER_CPU(int, sd_share_id); -DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); +DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_balance_shared); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); @@ -680,20 +680,39 @@ static void update_top_cache_domain(int cpu) int id = cpu; int size = 1; + sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); + /* + * The shared object is attached to sd_asym_cpucapacity only when the + * asym domain is non-overlapping (i.e., not built from SD_NUMA). + * On overlapping (NUMA) asym domains we fall back to letting the + * SD_SHARE_LLC path own the shared object, so sd->shared may be NULL + * here. + */ + if (sd && sd->shared) + sds = sd->shared; + + rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); + sd = highest_flag_domain(cpu, SD_SHARE_LLC); if (sd) { id = cpumask_first(sched_domain_span(sd)); size = cpumask_weight(sched_domain_span(sd)); - /* If sd_llc exists, sd_llc_shared should exist too. */ - WARN_ON_ONCE(!sd->shared); - sds = sd->shared; + /* + * If sd_asym_cpucapacity didn't claim the shared object, + * sd_llc must have one linked. + */ + if (!sds) { + WARN_ON_ONCE(!sd->shared); + sds = sd->shared; + } } rcu_assign_pointer(per_cpu(sd_llc, cpu), sd); per_cpu(sd_llc_size, cpu) = size; per_cpu(sd_llc_id, cpu) = id; - rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds); + + rcu_assign_pointer(per_cpu(sd_balance_shared, cpu), sds); sd = lowest_flag_domain(cpu, SD_CLUSTER); if (sd) @@ -711,9 +730,6 @@ static void update_top_cache_domain(int cpu) sd = highest_flag_domain(cpu, SD_ASYM_PACKING); rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd); - - sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); - rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); } /* @@ -2650,6 +2666,49 @@ static void adjust_numa_imbalance(struct sched_domain *sd_llc) } } +static void init_sched_domain_shared(struct s_data *d, struct sched_domain *sd) +{ + int sd_id = cpumask_first(sched_domain_span(sd)); + + sd->shared = *per_cpu_ptr(d->sds, sd_id); + atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); + atomic_inc(&sd->shared->ref); +} + +/* + * For asymmetric CPU capacity, attach sched_domain_shared on the innermost + * SD_ASYM_CPUCAPACITY_FULL ancestor of @cpu's base domain when that ancestor is + * not an overlapping NUMA-built domain (then LLC should claim shared). + * + * A CPU may lack any FULL ancestor (e.g., exclusive cpuset symmetric island), + * then LLC must claim shared instead. + * + * Note: SD_ASYM_CPUCAPACITY_FULL is only set when multiple distinct capacities + * exist in the domain span, so the asym domain we attach to cannot degenerate + * into a single-capacity group. The relevant edge cases are instead covered by + * the caveats above. + * + * Return true if this CPU's asym path claimed sd->shared, false otherwise. + */ +static bool claim_asym_sched_domain_shared(struct s_data *d, int cpu) +{ + struct sched_domain *sd = *per_cpu_ptr(d->sd, cpu); + struct sched_domain *sd_asym; + + if (!sd) + return false; + + sd_asym = sd; + while (sd_asym && !(sd_asym->flags & SD_ASYM_CPUCAPACITY_FULL)) + sd_asym = sd_asym->parent; + + if (!sd_asym || (sd_asym->flags & SD_NUMA)) + return false; + + init_sched_domain_shared(d, sd_asym); + return true; +} + /* * Build sched domains for a given set of CPUs and attach the sched domains * to the individual CPUs @@ -2708,20 +2767,26 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att } for_each_cpu(i, cpu_map) { + bool asym_claimed = false; + sd = *per_cpu_ptr(d.sd, i); if (!sd) continue; + if (has_asym) + asym_claimed = claim_asym_sched_domain_shared(&d, i); + /* First, find the topmost SD_SHARE_LLC domain */ while (sd->parent && (sd->parent->flags & SD_SHARE_LLC)) sd = sd->parent; if (sd->flags & SD_SHARE_LLC) { - int sd_id = cpumask_first(sched_domain_span(sd)); - - sd->shared = *per_cpu_ptr(d.sds, sd_id); - atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); - atomic_inc(&sd->shared->ref); + /* + * Initialize the sd->shared for SD_SHARE_LLC unless + * the asym path above already claimed it. + */ + if (!asym_claimed) + init_sched_domain_shared(&d, sd); /* * In presence of higher domains, adjust the -- 2.54.0