From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SN4PR0501CU005.outbound.protection.outlook.com (mail-southcentralusazon11011041.outbound.protection.outlook.com [40.93.194.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 473892F12D4 for ; Sun, 29 Mar 2026 21:36:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.194.41 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774820177; cv=fail; b=Gf51UaEvfO2FEWLNKAfvpl+AwIUrw5MPtuuX/r2FvvnHkAPFyPuHo9oidZk9tq3ptpyxYDTWF6c5fIOAu3mKpvO0Xjs3IeSk3NAJEDf+CCQGMLA8a6S4bCpYkD9jTMQDWWmpq3BzIueqLeEo4iks2u6t4GCWx3kABua8qtkbMpg= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774820177; c=relaxed/simple; bh=NPaeeKTxG3YGlNkA05pNwTuWaaBR7UpQrkSFNGChCXA=; h=Message-ID:Date:Subject:To:Cc:References:From:In-Reply-To: Content-Type:MIME-Version; b=bQ/tQMVm0bhQ1i0QkJaDjeal4P+vr8MGVN7qfumhLprJEj09/s7vV1uIT87xanz6/z5jBg+WiBbwgY3tevgGiT54MM9EKYhffruOzMoyCVQcwolE8Yw3xHxsN6FkRuFeBPDvx105b+VTzfOQa0/Ik+evv60UDFUo+8AJ4Dl+yiE= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=OrYpvq0K; arc=fail smtp.client-ip=40.93.194.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="OrYpvq0K" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xEGxo1BAJ4+9ZLugzPaLN47ZDhNC6sgKoBtk7SPBF3UbdWw/4j9ouCdh8TccPthXEgS4DkZxpR6dQdGN0/8w2QWyYGDMSsFDR0ol126XDglORAhoKyOZ1AVkNVixB1ymKc/N7zrYNcfhxUJmLsQ59yy47QzSdPx+qC6W6msO6/P/aEgIyqeTWhEWaTfjXjnNr+u5uFj4ARUpKuFwvSg5u/jlBjT5KckMUideFBVcA3YreccKVzhBzeWIfjrpNOUqPvlPqFt0U8a0Pda+xRgMIp/tXfPYNisq/7Qi+Ds+heiVMn3kaFICUjiTb2Qww3NoKkS2lHZ1t6CoMsW5lGMQqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bMT6778BUr3k675nXebaqFcfyFqQKRWnRGSy+XZyP0c=; b=ekkaVryskbBI0SoeZlpYnT8v8o6jFzbe9OS8hyNC/v8ZEyTAPngqpzhws5NqY4/vzpfw3ibgJvpdh/tKFdrFfMhJ+ALlYVk9RWn6E/5iJY9UpgbSHFZtU27P/TMmG5XpLzWnyg1cAwMMlCOy8lSbSv5Vb2wKcYD8D63WkQcliXxDnc/1KVvhSn97Cf7AjDZ5mhuJJJUI4k/TC67H8EOn6mNN7VDZyXvjfU6AiQT7h+6yPqTlAOrZ3wYuRnLctbVunPYZEw/ICgbEqy5v45mn15kxRq0lJ8P/ZeLW3Ls72pP9lhijKp8v27ydqqL9Z0wneWosdnY3gt0U85lyhVfsZg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bMT6778BUr3k675nXebaqFcfyFqQKRWnRGSy+XZyP0c=; b=OrYpvq0KHcQbS7O/h93BQMlwFF3dETgs8Ll5IQP+l2a/+pfz6AJi5dqdydXw7c62h9BhUJhV0lzJnzd8EQjWjic+TQSle0gPaPF/T+GYxkY+MZVGpdfVcW6RxgK99HIrfW27V2sNjoK6nN23NEgqgIW97loVP381pvGMQv4WQ9G0dczf1CVnQLLuWU8iobXIHUfNXmvcmITRBovVRRpQBiyKzkCIj8StTtKH8r8DVw39EjbTjWE7HN58xy1sVItWsgiWPQBTdraDy8oHqiq24/OfaKnl+2/Vn/HytCXoSZUtKTTd48L761oc31PMmZlaHNh4iPoPoZEP100TxOEb8w== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by CH3PR12MB9430.namprd12.prod.outlook.com (2603:10b6:610:1cd::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.8; Sun, 29 Mar 2026 21:36:11 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::2920:e6d9:4461:e2b4]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::2920:e6d9:4461:e2b4%5]) with mapi id 15.20.9769.014; Sun, 29 Mar 2026 21:36:11 +0000 Message-ID: <6316c687-6a51-4d76-8c15-029d0e8ca2d8@nvidia.com> Date: Mon, 30 Mar 2026 08:36:05 +1100 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/4] sched/fair: SMT-aware asymmetric CPU capacity To: Andrea Righi Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , Koba Ko , Felix Abecassis , linux-kernel@vger.kernel.org References: <20260326151211.1862600-1-arighi@nvidia.com> Content-Language: en-US From: Balbir Singh In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: SJ0PR13CA0003.namprd13.prod.outlook.com (2603:10b6:a03:2c0::8) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|CH3PR12MB9430:EE_ X-MS-Office365-Filtering-Correlation-Id: 3e61c8a1-6f89-44b5-2ad1-08de8ddb3230 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016|56012099003|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: EBKk5UUvAi4tF56iLMz4fzlSevefOzOwd0GEa/JgRUI5QGvWwDTe+KFEv2F6l+959J/jCIWYoId3gUC0QIFr7odcORKRQFRXjeL55epdb3j/LiwNL29ZH1AKBeu5bRXa+a0UFM7BD2wS8j450xhDEvTiyLOEnvb1JhFevp/03Zr18VIwJp17LJB4Loj4FxTqJ3RmtZEkaL4YutFnpy1Uf5ttfQf5E62x1V4ZMHTpWr/zhBwYzq+aA27VmBQFXmgb4Eumzcb4W4LQBP/MjrhN887RbAM8Ml9Dbd0HZm9NShu1gNR3OJtYm0kumjPBCvyBmKuAjK9g9Oq4ot+IzVMSzeNbC1tksyDETBuawtv5XnYM6lk6PI0wi96q2GIEJyaOMJjhdutQ375zJQX6+bceJqF327bzfNQkPlruL+T9K2+53YxuZk3X2wN63i9VzxUt/Z8MXrVoISApC9klUk+SpuKYTk8nPamqyCMDxvltISgjS06XrZbT5xcKhRya5giqOBB01Ma7HIwzhr4KBv2Jt9L9Bq0nkkgRlTzrdz/ysImMEo6OBGOmBcETh2nXnkUgfT1xjx6ZvuLMYVx66ctnB6ABlakudio+Kw+RVkc7OOLJuhSfrTdIm1jH3/SAPDJrGjEIk21A8W/fChWqVcJntT2xpUqoTU1U9KlwgdYRgjcia8Yaci+JCFsG8T2pi60SyMJmevb2fjf0EzXdKkQafyakAUVYMQXi5PcMBgV8cas= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016)(56012099003)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Wm9QK2FRRzRrdFR5NkhYZ0xzRFhseGNsWVgweVRnMC9leFhuSXRJZTZtVnBQ?= =?utf-8?B?VGZqWnZxUFNlc2lKdWgvZFI2OWEzVWtNRjZsRzVXbndHWEpYc3dpamVybVdT?= =?utf-8?B?WmY4enpYZTAzVlp4UFJENkNGQkgvSklKREJlVVEzZVZIbWNTTWhxdTRvTHNy?= =?utf-8?B?VU80aERTVkczU2c1ZkZpR09pWWF0Tkl6S2F6T3pESzFzcUVuS0JoV3Awdm5E?= =?utf-8?B?bTRPUUlxa1ByNGJsKzZRYzAwNmVHczlwcm1qUkJKNlpPMjYzQzB1amlGbldI?= =?utf-8?B?RmRqRFhhUFVZQmFBdzd4S1V6d3dDdXptWlJteVlNckFHWTFtbTNsNjV5Y3RE?= =?utf-8?B?VDMvSFU0OUs5anoyL3U2Tk5Rd0VVTTAwL09yM0Z0L0JtS0luQlRJdURYcDZx?= =?utf-8?B?VXlBUUpReVBHdjNMbDF3U2dDdUduR2ZoQVlLQWNFVWQwV1NsbjlxcEN3V1dm?= =?utf-8?B?dmtzYWdEcUo2RUUzODBiN0IxSUpCd3ZFR0xFVEFZa1FUdERMNGU3d3lxZWNy?= =?utf-8?B?d28yeTRJOWdXZTZZVmV1dXFYbG9OVXMzT0x2UmY0eHhtVmRHOURsNGhvNWRy?= =?utf-8?B?TVpKTWV5QVh3WGloeG4zMkFpQzcvYlZ3dUxwK0gzelZOUjJuUTA5RUZ4SjYx?= =?utf-8?B?d29CMFZOeGpzN3lEV0h4WjVwaDF2bzFRdUIwWHBYOERoZ2NBT2xSU2NZVmto?= =?utf-8?B?a0ZpV3l6M0JzaDgzSGo4ZEk0Y0ljMTdyb2lpNEhpdXI2MjNkS2pQYTVoaVp5?= =?utf-8?B?YitURUVudHl2TkJjZUZtKzZXVEluam83MWJab3BYWXVIR2RTZTQrMGVIaDNS?= =?utf-8?B?Qm1jbVhHY2hjcGRleFFETGV6YzgyaFM4cmNMTFV1TkVTT1F0cmtqQUEyTFRC?= =?utf-8?B?ZUJ1UEtZZUdKNit6ejUwcENTQjBMYTNpZVRtSElyRzg1OVYwVjZtU0o1a241?= =?utf-8?B?WVBVcW8zTGhpVkk4dHJqK1B0ak9TOElyWFBtdGxpUm1NeTExa0ppSEdkUDlP?= =?utf-8?B?YUlIMjBhT2I2V25tUWRHMXI5LzI0NFJoeDd1R0dvdFU3ZUlSVm1NVmEwd3Bh?= =?utf-8?B?M0pwR2FzMXpBYVluTXBzRkNkaUgydGFMV3B3OUM5ZGpnNGZ6Rk5xSGJUQVNP?= =?utf-8?B?aElnK0VoYncybEFpNSt3T0tnMGxoNlN5SVN4QVVpRkwyd0xkYThlYk1DTTRB?= =?utf-8?B?alVhQXhnNWpUNTVNN0x0VXlXd1h4S0ZQYy9OL1RIMGhUZUY4N0ZLVmZEbnBw?= =?utf-8?B?OTBZRVd4a0pFOXEyTkM3REFlTXBPakRRT3d6OWVyTnZ3aXppamJQeE5hTE5V?= =?utf-8?B?NzZHSk1FUm8xNGo3dWJveFhQZVFjc1RFNXF0ZmE3Y2p6S3dHSUdURzB3azFO?= =?utf-8?B?bFJIQ0VCU0kwWGJyVmFNK3hjNXlNZTFnbmdkS1RjUHBkRWlnRXpVWGs0cDJp?= =?utf-8?B?MGFCeG9XWWhqK2ZsWFRLMUZxOGpYMHh3RDJZdXltNERBWGZsd3BCYTkrWFBm?= =?utf-8?B?WnNzY0Z6eUh5S21ZeS8zVi9jUkJuMEg1b2dVWlBYNnFGd0I1VXpFbTAxSnRF?= =?utf-8?B?dFllbXhXN2pDM1RFODNORUh3VC82elFZUzkyb0JMdkJQR3ZybkRSMjV5dEk5?= =?utf-8?B?NXVrWC9xTXl2RjRSd2dzbytEV3RpQ2VYSnFNa0djUUw0cUxvNVVPM242WnE3?= =?utf-8?B?UDgrbWI1NkF2ZVpML3BNRnRnekhzR3J6UlUxNW1DUnBkeHl5QTJYWjVmR3hl?= =?utf-8?B?ZEl2dU94YTZqYVhFeE5uN1pwZ3k0K001SUdFTndyaWdTOWp4Y1JhRy85RHU2?= =?utf-8?B?cktEWG1Cb3Z5NGxLL2NqZzU1bWMwL24vdXFiaVFqcExGREhCSENYdytSbFFq?= =?utf-8?B?ZllXVVRtTXNHRmpRSnRaM3pjN0NSZENPN1F3QlJWejgvWUl2SFNvbUJXK29B?= =?utf-8?B?ZEFBNDhjODlheUk1MC8zMWZKdEIxd0szSDRMZkY3cmRXbHJaTWxsRHlJTCt0?= =?utf-8?B?M1A1ckV3V3JidzY0TVdaNkV6Y0FvZkYyRk5MdXVKS04rdC9sTThOU3IvY1ZJ?= =?utf-8?B?RjEyOENwdVNUQ2hhR0F5SkNGeVMvVmU5Mmg5cDAxR21zNXZpejdVemE3cGNm?= =?utf-8?B?SUgzR09UVkh6M1JaL1Q0UWxyelpJVi9zNE04Um5VbVd1enBpLzRVbmJJR1p5?= =?utf-8?B?SnFEczRXR09raXFGY2NJdTMwbUtjK0hBTlVSZWp4YWJXU0hrZ3RNZDFTNGdQ?= =?utf-8?B?Wm12Mnd1eXBaRG41NXJGUHNINWszRy9HQ2ZHSXBVa0tTUlFWdW5aOXZYaEY4?= =?utf-8?B?L25kdlpUU2JzV1BLbFFJK3pnYmNMaVJMcmVvSFpvandtRG1GMGQydz09?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3e61c8a1-6f89-44b5-2ad1-08de8ddb3230 X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Mar 2026 21:36:11.4852 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4ySVwp6FY0tuAhk97VcwPOuUxKhmsC3gKcs2prEgfuS17ZanPeX5+/HDXlv0nDiWfpmZ9/AygsvKCPTVojlUWA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB9430 On 3/29/26 09:50, Andrea Righi wrote: > Hi Balbir, > > On Sun, Mar 29, 2026 at 12:03:19AM +1100, Balbir Singh wrote: >> On 3/27/26 02:02, Andrea Righi wrote: >>> This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by >>> introducing SMT awareness. >>> >>> = Problem = >>> >>> Nominal per-logical-CPU capacity can overstate usable compute when an SMT >>> sibling is busy, because the physical core doesn't deliver its full nominal >>> capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs >>> that are not actually good destinations. >>> >>> = Proposed Solution = >>> >>> This patch set aligns those paths with a simple rule already used >>> elsewhere: when SMT is active, prefer fully idle cores and avoid treating >>> partially idle SMT siblings as full-capacity targets where that would >>> mislead load balance. >> >> In kernel/sched/topology.c >> >> /* Don't attempt to spread across CPUs of different capacities. */ >> if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child) >> sd->child->flags &= ~SD_PREFER_SIBLING; >> >> Should handle the selection, but I guess this does not work for SMT level sd's? > > IIUC, SD_PREFER_SIBLING steers load balance toward sibling_imbalance() > (spread runnables across child/sibling domains), it doesn't encode the > fully-idle core first logic. In practice it doesn't give us SMT-aware > destination choice when a sibling is busy and this series is trying to > cover that gap in the palcement path. > Thanks, so we care about idle selection, not necessarily balancing and yes I did see that sd->child needs to be set for SD_PEFER_SIBLING to be cleared. > BTW, on Vera the hierarchy is SMT -> MC -> NUMA: > > root@localhost:~# grep . /sys/kernel/debug/sched/domains/cpu0/domain*/flags > /sys/kernel/debug/sched/domains/cpu0/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_CPUCAPACITY SD_SHARE_LLC SD_PREFER_SIBLING > /sys/kernel/debug/sched/domains/cpu0/domain1/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_SHARE_LLC > /sys/kernel/debug/sched/domains/cpu0/domain2/flags:SD_BALANCE_NEWIDLE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SERIALIZE SD_NUMA > > And domain1/groups_flags (child / SMT flags on the sched groups used at the > MC level) still has SD_PREFER_SIBLING together with SD_SHARE_CPUCAPACITY. > > root@localhost:~# cat /sys/kernel/debug/sched/domains/cpu0/domain1/groups_flags > SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_CPUCAPACITY SD_SHARE_LLC SD_PREFER_SIBLING > > So, prefer-sibling is still in play for SMT (including via MC > groups_flags). On machines where asymmetry attaches immediately above SMT, > topology may strip that flag and reduce this branch of behavior, but > explicit SMT-aware placement still matters. > >>> >>> Patch set summary: >>> >>> - [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection >>> >>> Prefer fully-idle SMT cores in asym-capacity idle selection. In the >>> wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so >>> idle selection can prefer CPUs on fully idle cores, with a safe fallback. >>> >>> - [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity >>> >>> Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. >>> Provided for consistency with PATCH 1/4. >>> >>> - [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems >>> >>> Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for >>> consistency with PATCH 1/4. I've also tested with/without >>> /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't >>> noticed any regression. >>> >>> - [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer >>> >>> When choosing the housekeeping CPU that runs the idle load balancer, >>> prefer an idle CPU on a fully idle core so migrated work lands where >>> effective capacity is available. >>> >>> The change is still consistent with the same "avoid CPUs with busy >>> sibling" logic and it shows some benefits on Vera, but could have >>> negative impact on other systems, I'm including it for completeness >>> (feedback is appreciated). >>> >>> This patch set has been tested on the new NVIDIA Vera Rubin platform, where >>> SMT is enabled and the firmware exposes small frequency variations (+/-~5%) >>> as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set. >>> >> >> Are you referring to nominal_freq? >> > > Correct. > Thanks, Balbir