From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from DM5PR21CU001.outbound.protection.outlook.com (mail-centralusazon11011033.outbound.protection.outlook.com [52.101.62.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9CF6175A7B for ; Sat, 16 May 2026 05:59:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.62.33 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778911142; cv=fail; b=Ezz10aZHXwC4i3dzGjyEpkK0M/CyyZNp0dv0JrxTy0q4KSGlPjpF71wnhYqcWrRxFTrVFCV73cR8+psfdE99vMPpSfe3KDcoyidsj9vGRUbrU8fU13mWrX770BOsUwAWxpzn/MqA/1Lb511axGql2qZSNxuV52ObwXUNxL8Z0oI= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778911142; c=relaxed/simple; bh=pbxCucoKN8AoTrEc0BTg9h/W/ul15GofyHdq/UCFL98=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Rb6uUDX0umkAru9B0iCa0ULaYxqGGiobQWy7wosRBRDhfmhcASuqwG394XeZTGT6EfaNuloHNUUtoYCsELT72i3lVJqmsxbpHv/9WwvYoSXX1lLe0A9DPi1RbYY/bm7YWG946mfu6BCUfpnIYeWI1Uuvy+giPrUViVm7T7vCuXU= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=fW0MNnFo; arc=fail smtp.client-ip=52.101.62.33 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="fW0MNnFo" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=o2+wHyX4NackqRKFnCsAg84Aygxfp7FXi2MG/FunA4QuTQLT5q57+MABOfr7h6nSWVBn25HgWQ9rUvEK8JLccDmlRpYuU4N3dXfsr/MQttxw0xZ4klzU1oN3LwNh+J6VnTdYo1a/yyiL7YbaQdrxx29YNZ97liN83XMtxxnHHcdBQecSBn0U1xeEyVR45XSZxkQQX0s/gaNm2DoFzEnk4EKRSnQWylt02HyEmA2Sop7dCD7HxQjMkXWRSMxp8W1C2WSqXKR3gQToKrTQclJvpoYoW8jKvH1z1Dll1PjJB1AIkMwaNhIV0HIbsOixh2o7u8Jxz+Ps6rCXCTaw4PIMEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7nqDBBO0Gw1OWjks/wcV4dT973QstBI0zO7OSLyx3E8=; b=XiHd58t52HiO8g7qweY2cvduLUg1DwoIpMTlAKdLFeSsy3/fC77d8hyavF6fGNDUtoXg74B1+lKZOeuXiqoX4jCrdkHuKM8wC0Rwl89I2Kxv2MPonvKjnFM4HWQcNWD8Y0bm5GoB4G8N6QVys9LOLJL9T/JrNC6czdztVQurEmotEc0biVl4uh9jqHCFaS6GTAfw9g9I/hC1fEyGL8I0aeODxkkLzszJVgHdLUnOfVY2O4wqmKuJdkJNg6pZJvq6UjtYfCVA2b0jkvcm/llkrIg6keLsvS6l/s9p2MkPn1M2qO6OeOqOvBNlev6Qksr/9x3n2/1BQVKnWknXgtTVGw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7nqDBBO0Gw1OWjks/wcV4dT973QstBI0zO7OSLyx3E8=; b=fW0MNnFoUOBMKb2KZcKv+vHEO7qA8vxo2R35Qj/sOqiN50VeQtBDJ+4S6WqDBFObZfDbQnzAeglINj+YktRT/xAvdt257Kx/IPWDdUPUjHJLJa00Wd0XnvwsldTr6GrAM8Yxy60s77NrbfxJM54WOWvnmTd01ytmdk+CaakN8LHBv/cv7QQa4ISvvJpGeo8NcZw3K7BfUDszHQ3y0KdEf7PhWDPkTCqnJqC21h7bYhyt986vfExsS0FSkTc8Q8tn1BnbhBz/p/Dmvckp6VD+RzKswnbHDSYhwHxWi9kqYai3FhxKR3FMYgBEHg6BQzJkFCFlpga5DcPovm3nY+3irw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB4827.namprd12.prod.outlook.com (2603:10b6:5:1d6::14) by IA4PR12MB9785.namprd12.prod.outlook.com (2603:10b6:208:55b::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9913.11; Sat, 16 May 2026 05:58:57 +0000 Received: from DM6PR12MB4827.namprd12.prod.outlook.com ([fe80::6261:3040:864b:159c]) by DM6PR12MB4827.namprd12.prod.outlook.com ([fe80::6261:3040:864b:159c%4]) with mapi id 15.20.9913.009; Sat, 16 May 2026 05:58:57 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Phil Auld , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , linux-kernel@vger.kernel.org Subject: [PATCH v2 2/5] sched/fair: Attach sched_domain_shared to sd_asym_cpucapacity Date: Sat, 16 May 2026 07:58:50 +0200 Message-ID: <20260516055850.1345932-1-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: MI1PEPF000008C4.ITAP293.PROD.OUTLOOK.COM (2603:10a6:298:1::426) To DM6PR12MB4827.namprd12.prod.outlook.com (2603:10b6:5:1d6::14) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB4827:EE_|IA4PR12MB9785:EE_ X-MS-Office365-Filtering-Correlation-Id: c01abbd0-654f-4a87-bf04-08deb31037e1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014|11063799003|22082099003|18002099003|56012099003; X-Microsoft-Antispam-Message-Info: P2TaynDJvMw4yB/im//TJd4hqE3/hlhmFQtfbjAl/hqfpvbTsuKNGL1CzGMas0HggwkE6BaggwYvpFntvwPQQZznvtIMp5qAziRo+t75P9qPoCbrzEiUb3hpZRptfwsMRbURMVm3PyI7Pj7sY1zGw0CSqOQlE9SHZa9hBL8dC0gpJotVSaUYvEPKYFnWUqHq6wjJsuttHaKOHT7/PjvgdZeT9E+tRf1xX8xxoOw5vzIVvTA5Mz3gSKU6zzeixUWv8fEDDD3dXAM5B9Wb4KSY5TEw6XiMHeDaLvBbmmYHDAJUibamCXLVd1kjEZwa7JYI2uJOJQMeWe+KlbffsPKsFwCI/KtGae5I4u6ht+ubiLWnVZ+Oy53Nr0UtmLrPb2a37laI3OHN7MVqyqlAvDABwLLFetjtGRUEIU056iYusXgVIZ8ijxi3/BspNRb77GUCc5yJDyf5yaYADS549EwB1fJwNQJLgxJnssLLtS1MKtQIcPr/p1Edlqyg8xXw5UpDjvA2VNFZ4GJyVrfminz6wFRW35SkRKj11CQkCr4M9aisClQWEaoGCHhDzNTdJWOzk///ucuUj14NpcEBMZSR2ZNnF3P4xXcR+YMTOc8L9spfKghi24SPKYdokKAgLWfzzQUeXRu0NY2CfyKwIB4TY3VhxQVg/3m/BIT6Y/khYjp2SnWqgs7bLfD/UAfXINSH X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB4827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(7416014)(1800799024)(376014)(11063799003)(22082099003)(18002099003)(56012099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?d/QnscV9D8xlGDzjuuiZtCqLj+BUWwvxL5d/pM7PNOkBNxC0TecdZWC+TfhL?= =?us-ascii?Q?fW/sc6JVq5MabNYRaYQ5tFnYxI+HEigjyQy0Qyalx7a6NvaMapyqQsb8ay2o?= =?us-ascii?Q?cX/LaPvPUFKRBGUjWc3neC1z2Z5DC5UfZo8VBvcXxPPTr367sO20BLGCYMH2?= =?us-ascii?Q?miYaLB0wx8ud4ZEfAvN20m23zE/5ESlbtHpObQQWzfRzC085T3VKDlJlUKPp?= =?us-ascii?Q?JLjBmTSlA2F8JqAa+58+aPHAisfKUgJ5IsOQgv+pZ4Nil2PBq/uX5w3M07Q1?= =?us-ascii?Q?Dx/dLDvVjyPEREKV80jC7HlXSNwBFrJulSoQ5+HZBWNwJR2AzpIFQu//W/Qk?= =?us-ascii?Q?YP9gYbZj17+rWbpRGlaDXCZsrGNQrSN9SFc+pNqqqi2QslkpNPFMjfMGjF0J?= =?us-ascii?Q?yeMe7xaQMw+hJ/Y8RGAViB5LLHRomK/M0GrRsbfwcFc3huaE6CjTUgBTHWZU?= =?us-ascii?Q?+NzggvjbHdT9pMJwVySJJFspstWIO9tYYyxtsPfIh0VJWcj8gDebBx/ruBS8?= =?us-ascii?Q?JnL76MVDRCFsmrTRLzSpff9DyisSZDK+iJTbHqjBtVgEaiTXXIBYadSaXhCh?= =?us-ascii?Q?Gwbu1ME+DinRlp365n9jiBTYTPm11CfYd5jjBsyt/P8ZOWwoc1K+K/TLMLQG?= =?us-ascii?Q?EfgfJ4yLA9QW9dwjK8ijm8I9CeLDOflkDfM0I+p2b5XZTTIxV2USsZI4wn1S?= =?us-ascii?Q?YATO0WtAeL7Nqf1Gr6aCiDjOlFQwVmefEAQ+LCzYg+nRDhYEk0KNYZYW+/JO?= =?us-ascii?Q?PAtRoTP6AyIQzYFuxrLlYe8HlEcB39OQk/6bkk2MoPaLV32Jc9f3XCtbb8wO?= =?us-ascii?Q?REoeJ3d4sMFVFeKnNAVwo5ebUtsTfDdhFpg8gQ841O7WpJNWjZskL7Xq3PRE?= =?us-ascii?Q?1toSHaDx1o30C5ZQ+RDNiP8ttDsXgM3M2erBDuzg/Y6T3p/ssW0qP8lneJz6?= =?us-ascii?Q?BsbTWL/WoBFDDqfhq+gMXxEmau1Js+JpUxQaviTLK9FiDDCk3VfamgbcwXMc?= =?us-ascii?Q?DHko86QvLFMpUbTObwKc3MJOtN5T5XZNjNlHW30eqfMU8eIeAjEZUYCgSlP7?= =?us-ascii?Q?lhdo7uaPzxks9iwGpnqnRjm7hS74eApgfh7vdrVeA8d6BJ/Wqv36qedzUCcC?= =?us-ascii?Q?GVOixDIdKkcPW8WdvVOJwQg9xfrWwH6oU6L+13PAywVMmYzKNw7Js/06C5cV?= =?us-ascii?Q?klUB6SkjB9Uf204gwbJ/XSN+bu47BykBJMprddKTNc24Y//n6PY9eI4JPBet?= =?us-ascii?Q?5qkr0PSBY6FeMhaJGCXbfYFMpHvftFgqGi/jHFzfYl3xQskQ2gnNuNh9P+v/?= =?us-ascii?Q?5TWbpq/ig28FlFhYckkXy/tdeqN7D+caFskRCBdvl3CEQ9v17ZSE28buJMMJ?= =?us-ascii?Q?SpHo5qTG5i6cNXm0Yv7rVJQ71PrLDj2pJsO7tE/3+CLsxJpNtF6J9ay/nbXN?= =?us-ascii?Q?1NAVCP34HIPEn3orxHc2LNe3fqcSdYOqH2vM/kvFiDlRF5J422wGm3BzNsGC?= =?us-ascii?Q?Sf+v36oG4XN0gJ2raYf2nkoOv6PVQyTpw+RARG774xsNwmccfOW1EeapTape?= =?us-ascii?Q?6kcmMQtxAeRPNqx5PoVe0H42LVy3SuMBuA6k4H+ZMxtCC0yYvyRLladVFWxg?= =?us-ascii?Q?MSEN/UTqhAtqEmTuR8Tr117TRdXshEomU84u70CX4JtByb0VuOIhiQhw+O3y?= =?us-ascii?Q?VrntXvwRM0BbQ3dxbiIhAKdNeORvVDH2xti/X1EP93J//+2K?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c01abbd0-654f-4a87-bf04-08deb31037e1 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB4827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2026 05:58:57.4863 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: y4Re8ETBMftgPhlUyRSknnrQHZJ255aAgqO1hT24Tw64ejigyOHkM0V++WPx48petMcfzzkTF8dpGmXq4B1Mig== X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA4PR12MB9785 From: K Prateek Nayak On asymmetric CPU capacity systems, the wakeup path uses select_idle_capacity(), which scans the span of sd_asym_cpucapacity rather than sd_llc. The has_idle_cores hint however lives on sd_llc->shared, so the wakeup-time read of has_idle_cores operates on an LLC-scoped blob while the actual scan/decision spans the asym domain; nr_busy_cpus also lives in the same shared sched_domain data, but it's never used in the asym CPU capacity scenario. Therefore, move the sched_domain_shared object to sd_asym_cpucapacity whenever the CPU has a SD_ASYM_CPUCAPACITY_FULL ancestor and that ancestor is non-overlapping (i.e., not built from SD_NUMA). In that case the scope of has_idle_cores matches the scope of the wakeup scan. Fall back to attaching the shared object to sd_llc in three cases: 1) plain symmetric systems (no SD_ASYM_CPUCAPACITY_FULL anywhere); 2) CPUs in an exclusive cpuset that carves out a symmetric capacity island: has_asym is system-wide but those CPUs have no SD_ASYM_CPUCAPACITY_FULL ancestor in their hierarchy and follow the symmetric LLC path in select_idle_sibling(); 3) exotic topologies where SD_ASYM_CPUCAPACITY_FULL lands on an SD_NUMA-built domain. init_sched_domain_shared() keys the shared blob off cpumask_first(span), which on overlapping NUMA domains would alias unrelated spans onto the same blob. Keep the shared object on the LLC there; select_idle_capacity() gracefully skips the has_idle_cores preference when sd->shared is NULL. While at it, also rename the per-CPU sd_llc_shared to sd_balance_shared, as it is no longer strictly tied to the LLC. Cc: Dietmar Eggemann Acked-by: Vincent Guittot Co-developed-by: Andrea Righi Signed-off-by: Andrea Righi Signed-off-by: K Prateek Nayak --- Changes in v2: - update comment referencing to the old sd_llc->shared->has_idle_cores (Shrikanth Hegde) kernel/sched/fair.c | 22 ++++++---- kernel/sched/sched.h | 2 +- kernel/sched/topology.c | 95 +++++++++++++++++++++++++++++++++++------ 3 files changed, 97 insertions(+), 22 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6b059ee80b631..4ef028605cecf 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7819,7 +7819,7 @@ static inline void set_idle_cores(int cpu, int val) { struct sched_domain_shared *sds; - sds = rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); + sds = rcu_dereference_all(per_cpu(sd_balance_shared, cpu)); if (sds) WRITE_ONCE(sds->has_idle_cores, val); } @@ -7828,7 +7828,7 @@ static inline bool test_idle_cores(int cpu) { struct sched_domain_shared *sds; - sds = rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); + sds = rcu_dereference_all(per_cpu(sd_balance_shared, cpu)); if (sds) return READ_ONCE(sds->has_idle_cores); @@ -7837,7 +7837,7 @@ static inline bool test_idle_cores(int cpu) /* * Scans the local SMT mask to see if the entire core is idle, and records this - * information in sd_llc_shared->has_idle_cores. + * information in sd_balance_shared->has_idle_cores. * * Since SMT siblings share all cache levels, inspecting this limited remote * state should be fairly cheap. @@ -7867,7 +7867,8 @@ void __update_idle_core(struct rq *rq) /* * Scan the entire LLC domain for idle cores; this dynamically switches off if * there are no idle cores left in the system; tracked through - * sd_llc->shared->has_idle_cores and enabled through update_idle_core() above. + * sd_balance_shared->has_idle_cores and enabled through update_idle_core() + * above. */ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus, int *idle_cpu) { @@ -7954,7 +7955,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask); int i, cpu, idle_cpu = -1, nr = INT_MAX; - if (sched_feat(SIS_UTIL)) { + if (sched_feat(SIS_UTIL) && sd->shared) { /* * Increment because !--nr is the condition to stop scan. * @@ -12834,7 +12835,7 @@ static void nohz_balancer_kick(struct rq *rq) goto out; } - sds = rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); + sds = rcu_dereference_all(per_cpu(sd_balance_shared, cpu)); if (sds) { /* * If there is an imbalance between LLC domains (IOW we could @@ -12862,7 +12863,11 @@ static void set_cpu_sd_state_busy(int cpu) struct sched_domain *sd; sd = rcu_dereference_all(per_cpu(sd_llc, cpu)); - if (!sd || !sd->nohz_idle) + /* + * sd->nohz_idle only pairs with nr_busy_cpus on sd->shared; if this + * domain has no shared object there is nothing to clear or account. + */ + if (!sd || !sd->shared || !sd->nohz_idle) return; sd->nohz_idle = 0; @@ -12887,7 +12892,8 @@ static void set_cpu_sd_state_idle(int cpu) struct sched_domain *sd; sd = rcu_dereference_all(per_cpu(sd_llc, cpu)); - if (!sd || sd->nohz_idle) + /* See set_cpu_sd_state_busy(): nohz_idle is only used with sd->shared. */ + if (!sd || !sd->shared || sd->nohz_idle) return; sd->nohz_idle = 1; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9f63b15d309d1..330f5893c4561 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2170,7 +2170,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_llc); DECLARE_PER_CPU(int, sd_llc_size); DECLARE_PER_CPU(int, sd_llc_id); DECLARE_PER_CPU(int, sd_share_id); -DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); +DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_balance_shared); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_numa); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 5847b83d9d552..9bc4d11dd6a98 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -665,7 +665,7 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc); DEFINE_PER_CPU(int, sd_llc_size); DEFINE_PER_CPU(int, sd_llc_id); DEFINE_PER_CPU(int, sd_share_id); -DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); +DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_balance_shared); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); @@ -680,20 +680,38 @@ static void update_top_cache_domain(int cpu) int id = cpu; int size = 1; + sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); + /* + * The shared object is attached to sd_asym_cpucapacity only when the + * asym domain is non-overlapping (i.e., not built from SD_NUMA). + * On overlapping (NUMA) asym domains we fall back to letting the + * SD_SHARE_LLC path own the shared object, so sd->shared may be NULL + * here. + */ + if (sd && sd->shared) + sds = sd->shared; + + rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); + sd = highest_flag_domain(cpu, SD_SHARE_LLC); if (sd) { id = cpumask_first(sched_domain_span(sd)); size = cpumask_weight(sched_domain_span(sd)); - /* If sd_llc exists, sd_llc_shared should exist too. */ - WARN_ON_ONCE(!sd->shared); - sds = sd->shared; + /* + * If sd_asym_cpucapacity didn't claim the shared object, + * sd_llc must have one linked. + */ + if (!sds) { + WARN_ON_ONCE(!sd->shared); + sds = sd->shared; + } } rcu_assign_pointer(per_cpu(sd_llc, cpu), sd); per_cpu(sd_llc_size, cpu) = size; per_cpu(sd_llc_id, cpu) = id; - rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds); + rcu_assign_pointer(per_cpu(sd_balance_shared, cpu), sds); sd = lowest_flag_domain(cpu, SD_CLUSTER); if (sd) @@ -711,9 +729,6 @@ static void update_top_cache_domain(int cpu) sd = highest_flag_domain(cpu, SD_ASYM_PACKING); rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd); - - sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); - rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); } /* @@ -2650,6 +2665,54 @@ static void adjust_numa_imbalance(struct sched_domain *sd_llc) } } +static void init_sched_domain_shared(struct s_data *d, struct sched_domain *sd) +{ + int sd_id = cpumask_first(sched_domain_span(sd)); + + sd->shared = *per_cpu_ptr(d->sds, sd_id); + /* + * nr_busy_cpus is consumed only by the NOHZ kick path via + * sd_balance_shared; on the asym-capacity path it is initialized but + * never read. + */ + atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); + atomic_inc(&sd->shared->ref); +} + +/* + * For asymmetric CPU capacity, attach sched_domain_shared on the innermost + * SD_ASYM_CPUCAPACITY_FULL ancestor of @cpu's base domain when that ancestor is + * not an overlapping NUMA-built domain (then LLC should claim shared). + * + * A CPU may lack any FULL ancestor (e.g., exclusive cpuset symmetric island), + * then LLC must claim shared instead. + * + * Note: SD_ASYM_CPUCAPACITY_FULL is only set when all CPU capacity values + * are present in the domain span, so the asym domain we attach to cannot + * degenerate into a single-capacity group. The relevant edge cases are instead + * covered by the caveats above. + * + * Return true if this CPU's asym path claimed sd->shared, false otherwise. + */ +static bool claim_asym_sched_domain_shared(struct s_data *d, int cpu) +{ + struct sched_domain *sd = *per_cpu_ptr(d->sd, cpu); + struct sched_domain *sd_asym; + + if (!sd) + return false; + + sd_asym = sd; + while (sd_asym && !(sd_asym->flags & SD_ASYM_CPUCAPACITY_FULL)) + sd_asym = sd_asym->parent; + + if (!sd_asym || (sd_asym->flags & SD_NUMA)) + return false; + + init_sched_domain_shared(d, sd_asym); + return true; +} + /* * Build sched domains for a given set of CPUs and attach the sched domains * to the individual CPUs @@ -2708,20 +2771,26 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att } for_each_cpu(i, cpu_map) { + bool asym_claimed = false; + sd = *per_cpu_ptr(d.sd, i); if (!sd) continue; + if (has_asym) + asym_claimed = claim_asym_sched_domain_shared(&d, i); + /* First, find the topmost SD_SHARE_LLC domain */ while (sd->parent && (sd->parent->flags & SD_SHARE_LLC)) sd = sd->parent; if (sd->flags & SD_SHARE_LLC) { - int sd_id = cpumask_first(sched_domain_span(sd)); - - sd->shared = *per_cpu_ptr(d.sds, sd_id); - atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); - atomic_inc(&sd->shared->ref); + /* + * Initialize the sd->shared for SD_SHARE_LLC unless + * the asym path above already claimed it. + */ + if (!asym_claimed) + init_sched_domain_shared(&d, sd); /* * In presence of higher domains, adjust the -- 2.54.0