From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11010012.outbound.protection.outlook.com [52.101.61.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CC4E389462 for ; Sat, 28 Mar 2026 13:03:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.12 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774703011; cv=fail; b=p/msSC6tYmDbdNBz/pflT6iglc6EI1KJxiRALLJ5RudtdfadubVpuQV62fAGp7gHbpzjd/VEpzxEw4NYcPNkg7ebPtxiogIFBrB12WBBQsB+Ol4shNm7q6nDf598IlkZUA/DSZClqSZXt5WTpXAiZX9tSxw6xPqM9xNcUfnnuNU= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774703011; c=relaxed/simple; bh=4jEP1LqrkGuWiIA0cwlT19Hpcm7hZ6dEo7Z5do2cAnM=; h=Message-ID:Date:Subject:To:Cc:References:From:In-Reply-To: Content-Type:MIME-Version; b=PeYe2PCtLlrURVDxG5OGYvKhJzgy3h7LvwwnsMvbKMLdtB1j9pJBgmm7KrIF/FqpHfUw8+DLoZ2R+DThRpb6B7b3JDJ2RqJKVeWht1Kz/ADUE9rvqLM2F3irhT7d0nKgobT8OfXaYeUksu2BXIpUJBhnF67x18bDOC4dccFdIRE= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=FTL17wlb; arc=fail smtp.client-ip=52.101.61.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="FTL17wlb" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LhYGpK1uu54tE8QsUSWEiP4FmMtP64K80VOpc/lCu/478OrZcxfxVJp5jFg6Ji9rq5WgJh1cC6bxxJNDkm3OiiECgZ84VOauXC0kM4Q2lQ5bA6hpofu/LjB2peC3TXJZZ3gfRt/gm/3UIyOzpH1TwEyeHlcFG4uIWOZDABG8ZwgxMPxk4mf+9PsLnmo29Xt0NZpJsE3XUYzQwwnpUBXAaFab+rpScD+1bu8Vp4xjp+xzPKSruq5qdUekaXul2W/7RhFLpm+j9O9WaLQTDbU4sNcFGg8zqfiMX34r/e6HgWqL5IaN0jN7qw0eeb1kgBhrfC/cPG3PhHhgCg+hmDiO4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HFA2rO59jdGvzJTy1P4YfPhSqNOyB/qE4H4pdtTlzcs=; b=Lq/I7lY/qemJcLPSKpt1dHFHw8IEbOIA7a52Zd+R0aqO87snnDeUF68KRRTMlzkw7i+tIbQk5ck9VPlu72qELzMTfRBWJB3xiwclHrdlivuLwS1/TYx8U42qP6x9LtI14lXJOlodMsrfeB95upZSQvYN/A0U2IvmHru6GNH3YLoPrX0sL3Z999Oww6bAsH4uLG7sCyxF/JWdQx5iucegq/I6/wYeURpLiMCjbknVv8UneIJ2zOm/66L9JXoB0dm9ahFPcA/mmSPqJ67kC3WS4ScP0bAdwYfTPLXtzpMJCg68b44E9ObaIPtVvAQM15HIChRuGdiUm8UXjsx5PgfGiQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HFA2rO59jdGvzJTy1P4YfPhSqNOyB/qE4H4pdtTlzcs=; b=FTL17wlbtEKj1mTrqb4JH4LpUJpFjWOCTApYl6AEgH/e+ASSWYIunz0LmkrD1d5PAGv+LMavaE0CdLGR4Ru3/i90E0EU1hali1UU8KGpDkRsSP8+kcQbm5aA33aeA+T+vnll6f3lMPzHMJLuBXAlRRAs/RjQ2sNPyuO0lVQxWXLYPQdxsRfWesvPTk2Pc5Wwvh09H/Pdb0dTgkGHTWfkGcsLM68VV9CA4YIhqzIdxMkNPZ4p55GGo69vqDJQYsIVAaA7u44YYYofcPY5mjJJdOcYNKsppFw8P2SpDUeo8ZV4ps/0oevj5G+H4A6+Q63f4AM16HCi4VgmHm+VmcKWmA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by DM4PR12MB6374.namprd12.prod.outlook.com (2603:10b6:8:a3::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.12; Sat, 28 Mar 2026 13:03:26 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::2920:e6d9:4461:e2b4]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::2920:e6d9:4461:e2b4%5]) with mapi id 15.20.9769.011; Sat, 28 Mar 2026 13:03:25 +0000 Message-ID: Date: Sun, 29 Mar 2026 00:03:19 +1100 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/4] sched/fair: SMT-aware asymmetric CPU capacity To: Andrea Righi , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , Koba Ko , Felix Abecassis , linux-kernel@vger.kernel.org References: <20260326151211.1862600-1-arighi@nvidia.com> Content-Language: en-US From: Balbir Singh In-Reply-To: <20260326151211.1862600-1-arighi@nvidia.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: BY5PR03CA0027.namprd03.prod.outlook.com (2603:10b6:a03:1e0::37) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|DM4PR12MB6374:EE_ X-MS-Office365-Filtering-Correlation-Id: b015010c-e86a-4059-6efc-08de8cca6601 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|7416014|376014|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: N5pFW2NGz+cm+HZcPp8qt8SdAIpp962NDTkeCdhyb3F370wf9iGW2qhgsujJ2OtlFJm8IoIfd3c2rOUefukspxGm7S+brDbHOMRIrFIYRFcQ3H12C97vntTGZlLEYqJTda7gxhz0d5ErhjxAQ2YxylhZ/QLjyzDgjQQTrCvdTtSTdJgPwT4q3On8eR9WdoEOF9cge7mxH1vzsRdfn5cW74WS4LKlo1XPTw/p1ZhfYps0dCCuqn7RprwTD04vTa8GuPifN2ehppDCN8bbRg17vGnnDCFOI5htA+m7DrbM5SoJ0T40KHOap5GcnXWAdSqkewsAcrdVu6BFhnrD1cEvGvHuJip/TcfTJG7OTc8Kp7/duCKJ4s2AQUsDy0VlTRCS997xpGy7ugxhP+eQpDWMd7NRDS6JtYjETlD00mVhsSq9dIfOkZSvOOCrtpOSo7EeQZHtoP5zGl3IYDvjEmytm7zEe3JcCXK2PSgiPSvKjzcnhFEb35UHEDtkq3wdwRwvJMruolQtv7DhIOG+UcCQuX8h/SwE7qTpvJExfLlfMKacpjTWO9/nmWRLKkx5RbkgNkBA8AsOmIVrDfv/Wwri4rch7M0kqNZtkO2OpU4K+5Pc4Z67M+Qi9p1n9Qqer+cqPMYd2/98H7O9LYqX8AbYSVtjin7AgrKYTonX3wG8nYAFGdiCSscMDUWXQi2B2WzuKeQ/UcVNFxSwOINVJ6Tt+Lnbv1i1a74ypuxGTBbl6DeIzVQP7++C7kmDTEy63fP6 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(7416014)(376014)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?c2FRem5rSkpSTUhHREswbVhTbDFpNWpGSW4rZ3p4VXdZZ0JvRUhrazZlY2Qx?= =?utf-8?B?ZmtKQnI5eFU0dTZlRWVPSHQvcVdyZlZjSENaZGpEYWR0TnhRdWl6S0VWUVcw?= =?utf-8?B?cVVnNitDblRRdVhjZEJPOWYybFlrTkRYakFNRG51Q29hRzl0MlJTQXlpdUMr?= =?utf-8?B?V0x6UzVlbXFpQk55UGNRUUNUWnMyWHVmS3VMVit1eW5IWU83SjcxY0Z4ZHJz?= =?utf-8?B?UzZmNEsySVFDdHlNb0QrVlB3cFpnUzNIa1drR0lxVUkzY1R6SW1PL0FlK3F4?= =?utf-8?B?TklQUUZzVzNOa3dyMG5BYytxbU1zWTNnS2NzMWE3K21XWlg3QTJJR2FuSUcv?= =?utf-8?B?cVY5d3BLSHE1SWlFWE1JWmVFUTNHcDIyNEhTcDFramt1czVrZnBWYUgxZ2FE?= =?utf-8?B?RERrbldGc2x3emJ4bm5yWnhZdjdqeUlnblk3US9BWExUUEZpZjcxeVY3Nmpj?= =?utf-8?B?WHY2a2VEVm84T0hLYzJOK0g2NEVTbTZZeUV6eGJhYjY2M0hVMDcxc3d3T09E?= =?utf-8?B?YW1RMmlRL0Q2cmNMOGN4M2h2d3Zlc2hmbjhQd1FuZWkxbW5GRTJ0QmpiSVA5?= =?utf-8?B?bkh5VzV5LzJjVEFRSFg3VkhWVlViSkE5U1d6enltN2s0bUszMzNVM0U5V29T?= =?utf-8?B?d3A2VDhOZmxjTDFFbVE4R1BWbnNNQUZhTlU0dExvWEdYdExhRER6b213S3lu?= =?utf-8?B?RnFtclBaR0lkaGhleDRlMS8rZWVSdFlIcEtmMmR1dmdBZTNuOG9MWDQrY1Z1?= =?utf-8?B?UVA3Z2hsK3NoMGV0NTgxSkI0T1hReHRuTWlPa3FhU3llSTBpQXNJRHFRTXQ3?= =?utf-8?B?NnV3akM5NFNCbUVJSG9yZ1ZLLzVFMzVhT2ZWUWJGaVk4d04zdDJkT21KR0dF?= =?utf-8?B?TXlubXNQcjdPZ21XMzZTVmw2bXdHUUV5UnM4K1J1eVA2RjZJV1psTkluam5q?= =?utf-8?B?RW11dlJkaTRjbXpOMitWZXlzamtJUkk5TWxidkllZnJOSkhKU21YNUFOVXVG?= =?utf-8?B?WkhJWEtWVHVrMnA4ZXVRK1BXRytEdmtTUVEwZ0pvZlBPcWwrVTRLbCt5WDNQ?= =?utf-8?B?bHVxUGlDVElJNHFRQkkrVEdPdkJvWWlwOFl5bmo0TGVCY1hMR1Q1dDNRUzFO?= =?utf-8?B?YnIwTnhITGdseUVJWUkxNmcyMU9GNUx0RlJ4VVlwanJ6K2hWRjVGMVZBRUlY?= =?utf-8?B?ck5VWTl5OWpveGlpTHdrNFE5MzRIUDN4U1NVZERRc2ZKL2VObzJsUE5GZElp?= =?utf-8?B?QWJNWFB4dEFMUXdVeElLVUFMUlJ2L3VSRGFyUzE3T0JoV2lIc0NVQ2s2a3NT?= =?utf-8?B?UnFoV3pnRmNEalowK1VIN3dFM2o2TVRCeXZtaFMzQlNJZmczSkt5UTlydlM4?= =?utf-8?B?eER2WHRyYjdsV0FqUzc5Kzg1ZlFEOVBoU3lvOWZsNm1JZDNwbVJ0ZDVmZlo5?= =?utf-8?B?SVlnNnNERDQzZ1o5S1g5dGFlc2xEd2VZcWpoUWphcUxPUGowSURsRnhBMno3?= =?utf-8?B?NXlERk1uZUc3WWYwUFpJRi96TmNnV3JpUHFQU1VlNFFxYXFkK0lYTjdTbXQz?= =?utf-8?B?Z3RWc2F2MS9Mb1hjTm5YZURNN3BwRmh5Vk1GWlhPYiswSUQ0a0dRY05JNkVK?= =?utf-8?B?RUNrM1F5ZmcvZkR4OXVpM29oWVpBUXdBUVJOdGxyQjloQ25zcUZsMlhoNmR2?= =?utf-8?B?OExvZ1hZUWFzZFJQcUxlRkJiZ2JKUW9xTlF3NTRqRU5oZHpuMDNNYWNOMDl6?= =?utf-8?B?amIzRGlWbndkeWtEU29KZHhjU2s4TFRlNm5KeGU3cG82YnkvMXY0cm1pT3I5?= =?utf-8?B?d0JHSnZxcSthVkliMXlqK1p5ZXZ4Mjh6ZUdheVVLaWozVm9WNTBoZ053MUJU?= =?utf-8?B?QnlFNEk4di91ZVgxSFJkVzFrQUpRaHVjYVJOM2dCR0FSb3JJaEpYQVZycDcv?= =?utf-8?B?UWtaOVB6TEpldFVqaWRpT0ova1JGVDk4VHR5TzE4S1Z1UmgzeE51M2hKM0k2?= =?utf-8?B?ZTMvc0pTM1pzRlhUQ0NuVVhZRVZIQnVxYTZWak5LcStPV3E2YThPUjVSWXFy?= =?utf-8?B?V1RQc3ZrQTFzVUZ6MzBxMzlCRFhMZ05zYzJESWtqdE5QYVJraUVEVVJ1ZGJD?= =?utf-8?B?SXFmSUJLNG5wS3ovM2ovMndBSWVZUGpuQzBmWmtYMGlPNExIVnBPWEJZSWNK?= =?utf-8?B?azZpQnVBZlJ5Y3FNU2huUDNHaXNaOThrenRyTXM4c1krVThaRjg5Y25Pdms2?= =?utf-8?B?UEJQdUFrcmVaWVdXZ2lrR3ZodFpybzNOT0FEeitDYlhXK2c2bWQzYk10T3kz?= =?utf-8?B?S3h4R3dDSHp2aWtqbEloenN0TDBrUmg1VU43NWpQNkl4UlIxYnhIUT09?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: b015010c-e86a-4059-6efc-08de8cca6601 X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Mar 2026 13:03:25.7722 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FKky51bhjDa+hs4DEtbighxlr6bTbZzI1U9Ya8ZfR/aQ8RLwsKFlKkq+THHTSPbZLD+FS4BGUyT1XoYUPHY9hA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6374 On 3/27/26 02:02, Andrea Righi wrote: > This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by > introducing SMT awareness. > > = Problem = > > Nominal per-logical-CPU capacity can overstate usable compute when an SMT > sibling is busy, because the physical core doesn't deliver its full nominal > capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs > that are not actually good destinations. > > = Proposed Solution = > > This patch set aligns those paths with a simple rule already used > elsewhere: when SMT is active, prefer fully idle cores and avoid treating > partially idle SMT siblings as full-capacity targets where that would > mislead load balance. In kernel/sched/topology.c /* Don't attempt to spread across CPUs of different capacities. */ if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child) sd->child->flags &= ~SD_PREFER_SIBLING; Should handle the selection, but I guess this does not work for SMT level sd's? > > Patch set summary: > > - [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection > > Prefer fully-idle SMT cores in asym-capacity idle selection. In the > wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so > idle selection can prefer CPUs on fully idle cores, with a safe fallback. > > - [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity > > Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. > Provided for consistency with PATCH 1/4. > > - [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems > > Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for > consistency with PATCH 1/4. I've also tested with/without > /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't > noticed any regression. > > - [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer > > When choosing the housekeeping CPU that runs the idle load balancer, > prefer an idle CPU on a fully idle core so migrated work lands where > effective capacity is available. > > The change is still consistent with the same "avoid CPUs with busy > sibling" logic and it shows some benefits on Vera, but could have > negative impact on other systems, I'm including it for completeness > (feedback is appreciated). > > This patch set has been tested on the new NVIDIA Vera Rubin platform, where > SMT is enabled and the firmware exposes small frequency variations (+/-~5%) > as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set. > Are you referring to nominal_freq? > Without these patches, performance can drop up to ~2x with CPU-intensive > workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not > account for busy SMT siblings. > > Alternative approaches have been evaluated, such as equalizing CPU > capacities, either by exposing uniform values via firmware (ACPI/CPPC) or > normalizing them in the kernel by grouping CPUs within a small capacity > window (+-5%) [1][2], or enabling asympacking [3]. > > However, adding SMT awareness to SD_ASYM_CPUCAPACITY has shown better > results so far. Improving this policy also seems worthwhile in general, as > other platforms in the future may enable SMT with asymmetric CPU > topologies. > > [1] https://lore.kernel.org/lkml/20260324005509.1134981-1-arighi@nvidia.com > [2] https://lore.kernel.org/lkml/20260318092214.130908-1-arighi@nvidia.com > [3] https://lore.kernel.org/all/20260325181314.3875909-1-christian.loehle@arm.com/ > > Andrea Righi (4): > sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection > sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity > sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems > sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer > > kernel/sched/fair.c | 163 +++++++++++++++++++++++++++++++++++++++++++----- > kernel/sched/topology.c | 9 --- > 2 files changed, 147 insertions(+), 25 deletions(-) Thanks, Balbir