From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazon11012014.outbound.protection.outlook.com [40.107.200.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF22837DEB6 for ; Tue, 30 Jun 2026 15:28:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.200.14 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782833308; cv=fail; b=VSVm3+oU+73Doh4tDBqoZyHkDYuJUDeL+IYgBmmfxiOoMSQs4oMthiG1V6gzpUBenLx5xuwWp3pMvBJE0qIUJ1N9lWkipzza0ltUT4kT+9jTNkj6m3pVJzRNYKDhrwSWToJYAM9TrzFwZWAnVhTCFVJPUnHlojEvbpYRjHMLvu4= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782833308; c=relaxed/simple; bh=kcSITMxXVNJoWZCsMRpmyY0CzUcR5RiQcEjGBZpm+Ek=; h=From:To:Cc:Subject:Date:Message-ID:Content-Type:MIME-Version; b=f22qXDSsnbSD9lKyU7PfHyitye7nZGeKcmFcudS3v9Jw1yjDgiwvVRyZ2te3AfMUfh06WgNBxAUAkJHQbaHW0QxcnVC9zw7bDsz+Ty+23DLZP8g8FQl2me1NLBoD0l5mv5W45Csj9cG25o0I/vTFw4+JC4zE2OiQWnLh0bgRwbw= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=hSjJoxpk; arc=fail smtp.client-ip=40.107.200.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="hSjJoxpk" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cvcclGWTh06QIxluaTrNfY/0V2OUz/Yyua7EtRXQUgED55NTLH6ytVRIAmvk8TxPFHCR38Y4/xJcNehnbxUR1ORreY5XRRoaALIOnQhAKhqxEcSjtpjfKn0Kguq26/mOO4xQ8T6n0tPsoC/hrU2Mg1buD6t9602qRk2a+Fyqizc9oQbhcWAk0suk2QLc4xQ7S2697wligIXjmtaxfIC8AsTx/YjKCQxlZqDXOhQTAxJKSuSxPvGiseg2hz0pMFTvievypg4EsmVvt6FGtzdOPnwerBk45Fnq9a2qPtW93XhPkBKXSJNinP737pGcLxkJ6FAlYaKYgljdmpw+yfI0Zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SPg1OpBD6hZ7M+bo6I1lmHfHMGB49Um2e44Ax66nMwg=; b=tR4AFst2TBHvfaPQZ6AXLWifL4IMxOtMWEghQxUOjzEsveANxYdv6+iYcH2DNUuIiLHtKtDCoER/EPjbeJXwC1PPQE0sPj+9pVFCeUmlnC34OlT+nDp1CUUjVam343tW1/1HYaYMXx99xiTvtVc9aESvsJaJOlKmWNqdUq/XmynG86WXQC8xo4hQ0vCcU9jTZLJlgw8xrbQbggCSJFnRT7e4rZc7j9SWSbdP/f56x5Sg7jRHGgy6oGAZG3yoTwxLI903HvMfTVTYVKq3YRqwWh/40QsWxL97FAuqHH2rEZWKmATzMnGzTgTX3gCdJqDC1EQTIAuBVfbvtPG5Ohw0Tg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SPg1OpBD6hZ7M+bo6I1lmHfHMGB49Um2e44Ax66nMwg=; b=hSjJoxpkviEZTOIFEgrET3CZfePuIuq5UoieYW3cgpcaFi7EyfoYPVOJZiiY5ehQ640u4YyqEGEyy40n05xR8OViBDFkUGlPIV3fqxd2GyVxkTZM6vZlJ1tk+Ui4d1AOzbnx8yHbDT7ugfASGPQp3TNjp/0dtYhMhHpzzytBISQxUIQduR3PkxDUZAet6dSNkl/Un0f2bVl7OOQn1Ykk18I+DSVx36J4urQLeCoOtvEEMzyZ5SX0SaskdP+QsOKxwLZLFbS+yEH+BmPTVNRPhZ8xtumv5FhKsRIlwIzY8twnPN717fv7r2glmYvLeU/tIbZiGDspB6mz3Wh0Fur1Vg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB4827.namprd12.prod.outlook.com (2603:10b6:5:1d6::14) by CY8PR12MB7754.namprd12.prod.outlook.com (2603:10b6:930:86::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.8; Tue, 30 Jun 2026 15:28:04 +0000 Received: from DM6PR12MB4827.namprd12.prod.outlook.com ([fe80::6261:3040:864b:159c]) by DM6PR12MB4827.namprd12.prod.outlook.com ([fe80::6261:3040:864b:159c%3]) with mapi id 15.21.0159.018; Tue, 30 Jun 2026 15:27:56 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Ricardo Neri , Christian Loehle , Shrikanth Hegde , Felix Abecassis , Joel Fernandes , Phil Auld , linux-kernel@vger.kernel.org Subject: [PATCH] sched/fair: Stabilize idle SMT core selection with asym-capacity Date: Tue, 30 Jun 2026 17:27:47 +0200 Message-ID: <20260630152747.128746-1-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: MI2PEPF00000B84.ITAP293.PROD.OUTLOOK.COM (2603:10a6:298:1::41b) To DM6PR12MB4827.namprd12.prod.outlook.com (2603:10b6:5:1d6::14) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB4827:EE_|CY8PR12MB7754:EE_ X-MS-Office365-Filtering-Correlation-Id: 19e62c17-e205-4e01-e915-08ded6bc28d7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|23010399003|1800799024|366016|18002099003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info: HnVCBsFtqfctfTtjyw0caBsrwBx7fTzByVft6zbd96AfeYHyeIhXKNzBCAXAzPU2b3XiCBHSQ8Nu0pGVFzPOk7EepDyGYUHvK22FrqZhlD0Jvkuyrqf2UsnTOm2cMy2FjueUzFSDIuYzCNPPqyTikn8+vnHw5Q169n8incfpz+c9TCmNdwk7SUs4KNRBUgwUp8irbH9L1ZEXfQP5Mytk2pejCxSGfucZ7Z19fy0bMof/XDUjbM0J0zULNHh314urBQHTH/nZBAZKsR0YgzFnHF23ZgZonGmoeYcmaq6AEc7NI/HjQxOjkzm2Xm3Dstpe+4Us+VyDkERGsDUD0Bs9LPFQsTZhPdu6gu2sSOqvaKmzhT8DzhuoBtwfdHeQZ6sEAlGnEehMkmJzVBFK7gEhh7kCv9B34hr8oeyqUTbF55O4yX9851U4v47i3Ngyk1yYP5Rou6BL5AWysBUZSX99diJVplHUCZkzwwD2uskLF+8zTcZFSAKYnH1uWOnUowQNJi0jf9eHVkBR7qMDeEJrTmC/VeCda5uFgUtqwq1ZGiMCNe0hVKXXBJ5OtrgACGWyeAxHZTX4O8AwTIvU1hvhcNYADU4NFa7oJjTHdclDuZYR5IkObgCJHZpCberLJkA5l+VcCz+8mDhKcczBW4IFugE1gFPhaQHDWcfLoNnqt8A= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB4827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(23010399003)(1800799024)(366016)(18002099003)(11063799006)(56012099006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?GbhvC0p1zIX9xIlbM+CkRBzV+FUtSZ4m9fPq3KmC5Uc71XkYM4tVGynFjv30?= =?us-ascii?Q?D14QPdskoi0Rv6LwMt9aHVvSi4I3kXQNEJGzkjuEqSjzOhAq7Ou22qiZnQvO?= =?us-ascii?Q?JTsB8EXR0VfSQ+z+fh2zF/fGQ+C8n1phbuHdo6tS6Lr0ra0CfUk/PB+ZLQ9g?= =?us-ascii?Q?cQ/jutGZu29jlwYs6trSoG9F78cPTmidAlHm/3qVkOUkKVEFuJaztD+hrQIQ?= =?us-ascii?Q?SfOauOnlqD1QIkas4ZZ46sXPVzOOzg009xUXTiJ5BZ0eQMZrEGkyQ6DXin0A?= =?us-ascii?Q?E6NGuqfpeTPhd2TLoSVlcZiYp4M8p3Z/moIsh7wKjm2v+eUrwEUOy8hD39kT?= =?us-ascii?Q?zzUvPZBAAVWHitb2hCaxPEPK/4J1iVQMBbfxvkxw1qI8jyh9D4Rf0vQPvVpE?= =?us-ascii?Q?JL1229nAVghzqCxsu9okdFDRZjBmZAOwVgvIw4SneUbMZbEf3fBZs5CledIn?= =?us-ascii?Q?OilXqrPSlzIVABdfsahc6vRLBfdt0c0tqQOwLOfw1XohyWICDpDgt67vN5cz?= =?us-ascii?Q?yB1CSOSseWuh4vD9SOs+H/LH3b2VMyt38ilFvQqEJ+2ejdLkMBbCzPYCUON9?= =?us-ascii?Q?0CbDXKSg38uOy5f/LQ/2vRrypasRDLF/nwHlwoFRaZQWj//H0DWLUSlaIn01?= =?us-ascii?Q?M+k6QR1q9+xdWHlN7UjQQ8Pjp3sPv1E1/wB7s+6/q7fw+c/yiqbDOpQUr7yE?= =?us-ascii?Q?7yxF49qGIOqH+5nisBtuRhQAjVbuFfE//8TXvynnDRPDnljh7wMISf/I5kiw?= =?us-ascii?Q?RP5jwLUCywu2ReEl8QDJ69MqxjPzVD8VcdR02C5AnC3eA8rFZmyHWEqONoFT?= =?us-ascii?Q?Ag98MWybfA8MR+xZTNUhUxqdmGCvKL0zAkZdKPxKT+1ocXfwD4E5P6UEae6s?= =?us-ascii?Q?HhMK6I6gynzbWCGNKrmJ9kZRYTA8V6lOvc4hyFe/9OY8+9hoX1XXjgbSpaYN?= =?us-ascii?Q?T5WU5HmdeHwqRxsnVmeG7C++kCb4WwWnyZge9gkwVNOeQoPaPnGL2oDr4nN4?= =?us-ascii?Q?J9OSrJqfXoNMPMOk93pA0C2hBeHS/gSJ1/sDZvzZnrTqFDDn5OFkOsdkr9ph?= =?us-ascii?Q?Wy7+QyIf0VdfCZDjzaDqRiYklB6lwFoTLEUrnMvzkjn1z95x4qBNjnhaPCsN?= =?us-ascii?Q?fu5auHlW84K+p4idea0l5lPt+mDXxU5i345/Ki7P4FE0DY37adnkIbjv1FKv?= =?us-ascii?Q?nlPloNwY7gSx4ldKCKf3STaabm4mhUCd1Muzoh+LHb8rr/DRkgNjfmCAZGqs?= =?us-ascii?Q?E8d1o0Zur4YX+IpapU8879Rxg7ZR0uMKRFTa7HV1zeiI6acO1CWzB+7QRdtb?= =?us-ascii?Q?cIhCEffouEl9wE1lAdh4ZGDWdkjTpxOtELVsWohku1DrKm4tAx3MmKYMbZWM?= =?us-ascii?Q?42CyF1CKgoQs8TFbDoh/CXgQqHWdBExsYbUbQ/sCi/TLV0Ndbzq7ZeXliJUg?= =?us-ascii?Q?lYC7YovAAlXMRmDcGZYlQ+4FBqULukvOeoOsKyMPD8McsSXSRBL6qjdZzvah?= =?us-ascii?Q?BE/UZm+gK5JsC/Qz/+dJ/Y88kKxr2bVXxO3uTUzzPxzdA2K78mmEeBabh7C0?= =?us-ascii?Q?OXroRh4JomGOP3EqoIpUJgACOrSpEOcyluzzwfYWNwdZxlyz6lcnkPvxo6P7?= =?us-ascii?Q?EK3QdBRkx2v5k4idIxsdOdh5pz8MW933GnHnlXjDH8nTi1O+2Xg86eyFvSAP?= =?us-ascii?Q?RGHLWw0bZwtDgyDu2Q2a48ejgsoPQNmYLCyxQ8bhoUMFG/Kdmin1s7ZIy28s?= =?us-ascii?Q?D1XNJgI/PQ=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 19e62c17-e205-4e01-e915-08ded6bc28d7 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB4827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jun 2026 15:27:56.2085 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yKOMgRbiQETq7ouocUAqEWJ0/qjhoYXwHkkKE3lWH2STFi3xp9GXJ9ESPzeTZFi3jzbTzOR/Amyh5C2nuIE+NQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7754 select_idle_capacity() scans all logical CPUs also when it is looking for a fully idle SMT core. Two concurrent wakeups can therefore observe the same core as idle, encounter different siblings first, and place one task on each sibling while another core remains unused. Make every logical CPU of a selected idle core resolve to the same stable CPU representative within the scan's existing affinity and scheduling-domain mask. If the first task is enqueued before the next scan examines the core, that scan rejects the now-busy core. If both scans observe the core as idle, they select the same runqueue even if the first enqueue becomes visible before the second scan finishes, exposing the imbalance to the load balancer. The symmetric CPU idle selection path is subject to the same race, but normally returns as soon as select_idle_core() finds a fully idle core, reducing the conflict window. The per-CPU capacity scan can retain an idle-core candidate while evaluating other CPUs, giving concurrent wakeups more opportunity to select different siblings of the same SMT core. Therefore, limit the normalization to the asym-capacity path, where this behavior has a measurable impact. On NVIDIA Vera Rubin (arm64, 176 CPUs/88 cores per NUMA node), a CPU-intensive NVPL SGEMM workload restricted to 88 threads (one per core) showed a consistent 23% increase in mean throughput across multiple runs. For comparison, DCPerf MediaWiki running at system saturation (with all SMT siblings busy) showed neither a benefit nor a regression: throughput and Nginx request latency remained within measurement error. Likewise, schbench under partially idle conditions showed no material change in wakeup latency, request latency, or throughput (within 0.1%). Tail wakeup latency was more consistent across runs with this change applied. Signed-off-by: Andrea Righi --- kernel/sched/fair.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee13..f846fbe7379f4 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8647,6 +8647,16 @@ enum asym_fits_state { ASYM_IDLE_CORE_BIAS = -3, }; +/* + * Return a stable CPU representative of @cpu's SMT core within @cpus. + */ +static int select_idle_core_cpu(int cpu, const struct cpumask *cpus) +{ + int sibling = cpumask_first_and(cpu_smt_mask(cpu), cpus); + + return sibling < nr_cpu_ids ? sibling : cpu; +} + /* * Scan the asym_capacity domain for idle CPUs; pick the first idle one on which * the task fits. If no CPU is big enough, but there are idle ones, try to @@ -8661,6 +8671,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) * collapses to the plain capacity scan. */ bool has_idle_core = sched_smt_active() && test_idle_cores(target); + bool best_idle_core = false; unsigned long task_util, util_min, util_max, best_cap = 0; int fits, best_fits = ASYM_IDLE_THREAD_MISFIT; int cpu, best_cpu = -1; @@ -8686,7 +8697,8 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) } for_each_cpu_wrap(cpu, cpus, target) { - bool preferred_core = !has_idle_core || is_core_idle(cpu); + bool idle_core = !sched_smt_active() || is_core_idle(cpu); + bool preferred_core = !has_idle_core || idle_core; unsigned long cpu_cap = capacity_of(cpu); /* @@ -8709,7 +8721,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) * immediately. */ if (fits > 0 && preferred_core) - return cpu; + return idle_core ? select_idle_core_cpu(cpu, cpus) : cpu; /* * Only the min performance hint (i.e. uclamp_min) doesn't fit. * Look for the CPU with best capacity. @@ -8750,6 +8762,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) best_cap = cpu_cap; best_cpu = cpu; best_fits = fits; + best_idle_core = idle_core; } } @@ -8765,6 +8778,8 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) */ if (has_idle_core && best_fits > ASYM_IDLE_COMPLETE_MISFIT) set_idle_cores(target, false); + else if (best_idle_core) + best_cpu = select_idle_core_cpu(best_cpu, cpus); return best_cpu; } -- 2.54.0