From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SN4PR2101CU001.outbound.protection.outlook.com (mail-southcentralusazon11012068.outbound.protection.outlook.com [40.93.195.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D851B26E71F for ; Fri, 3 Apr 2026 14:46:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.195.68 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775227569; cv=fail; b=PSJZKEdve6uWbZjM7Vm31d3X0s1/kZEQC6xtvUMNki2GPHgKXYtth1YXFhfTlxKFzY66o8zxRL+DEoTrIlooIMNcYi2bvzU58LmwjiaRnJt47ix0Yw8opJkrtDDCb6P0ISfBPqFDb5cUnmMjN/HC53wBoNyP1ijja4kzMSV0XwY= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775227569; c=relaxed/simple; bh=PRgHOR4G6ZcF9oEZVwRuYzXGA6l2jXcxZ0LmCJkkQbk=; h=Date:From:To:Cc:Subject:Message-ID:References:Content-Type: Content-Disposition:In-Reply-To:MIME-Version; b=sxN/SeIrcygdSCSHPSAegMFIIcwoEPn5p4Wxhr8liouLEQVXzMZ0+UDeJnV390JZ1WOE0nOLUzgjd02ucyuibQcWupIk0If8Mi86vE5yizaOiYfMIddH0p6OqYXVtLymBEz+rucfGVWnvG+waNwG6nrsIJC8iV9fn+XPWrdHdYY= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=JkwQP+aM; arc=fail smtp.client-ip=40.93.195.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="JkwQP+aM" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=YpzzHVz5l8hFq76z/DWBLJ+L4ylzu+CyBWPDVXFMOoI8+9fca11q9P4aVRCedqiJIklEFNyDU2/ia16y3M5GI3rClCipsC2+vxIM/8DMiC2b3Kub9WoJFiNxA+9iLDsLW7dIDuFYOhNUXq8o1fVjktGWO/9nHhAK1tc+WBh3peEmIRwxwguPVab8vvlJEil4UHdZM+7YWysZ//+DG2dh9u2n7sq/e6MaqGFs00Xwd6cjCMNkQZyZ75r3xNNlDCwF9NLjtuTJdth9Hdi3UwGVM0b+1l2kKEh1FENebXKlCTT3yE6F8L0kXBDAitg3oK9dqAQoVSYIu5YCsGH7OqUIKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QAjQ4u/z/+A/6ltOR8Gp7qzT8i+dOlc3pLp9PSzTRe0=; b=O4JPB4D+3FCUoihGkREW/n16kKyUdDLCkZtqSvWYv2Bpcp5Y12ISQQ8NxwliBm1b7/vC5Y1dvvOnvjb8Qy8xLU6+07bhDXx57EaSdGAm9ApgaLJ5ybNPWiLj5qjxcroxDbi9kRxY15ChDt42A1cRn6OocGJ/22pWi9oA52YMHo7ptX1c2gwPUuGggmxiEioigK8T2+Y0IOVR5VyL2Eityh4UsOl1Hq+8an8NKnCqc1NQnW6OpQfCada251VBCW1liA1cF2/BTpntDVw+d4KnQctHQGJM/FfmGtMK27zMASUgU3ubLxpkg26103HLazhw4OoGH/EqmsMTZ33JgAeQFw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QAjQ4u/z/+A/6ltOR8Gp7qzT8i+dOlc3pLp9PSzTRe0=; b=JkwQP+aMorX1QC3swDX70LXXIbSZT0V5Z7ao24Hye5rKXloYGFMbnuhCOE8vsLoJ6GGeu1/p1v/1L9/B1UNkuC+i7dMqDCjoZTXUiwI1iZrdfEfTU7myBGAtkdb42m6ONoU/rv5puPiV4IIGFbotauEQoAf88TZ6tR/99YL2MkyCJdFPjpObx9MyfadeDxNAwYEPd5yy0HNM/B+SVKaDJllvFn8H+gUVHFy9AIigEQPp/wLXdVqybqJpsw71adF7hc8iLascz6+pjqdgM74SUbhJwmEPCSuo5pu1cpis+Gl42C+6CrEXc1nsL43NqJ9kBtCX9qVZZRF5KJIt8j/q3g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS2PR12MB9615.namprd12.prod.outlook.com (2603:10b6:8:275::18) by SJ0PR12MB6879.namprd12.prod.outlook.com (2603:10b6:a03:484::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.15; Fri, 3 Apr 2026 14:46:02 +0000 Received: from DS2PR12MB9615.namprd12.prod.outlook.com ([fe80::f4e9:9ad6:cb62:2c15]) by DS2PR12MB9615.namprd12.prod.outlook.com ([fe80::f4e9:9ad6:cb62:2c15%6]) with mapi id 15.20.9769.018; Fri, 3 Apr 2026 14:46:02 +0000 Date: Fri, 3 Apr 2026 16:45:50 +0200 From: Andrea Righi To: Dietmar Eggemann Cc: Vincent Guittot , Ingo Molnar , Peter Zijlstra , Juri Lelli , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , Koba Ko , Felix Abecassis , Balbir Singh , linux-kernel@vger.kernel.org Subject: Re: [PATCH 0/4] sched/fair: SMT-aware asymmetric CPU capacity Message-ID: References: <20260326151211.1862600-1-arighi@nvidia.com> <193f1cd1-ced3-4b37-83af-ea43a7e5e3d0@arm.com> <9886a7d3-fb54-4637-8b4c-1f35272f4882@arm.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9886a7d3-fb54-4637-8b4c-1f35272f4882@arm.com> X-ClientProxiedBy: MI2P293CA0010.ITAP293.PROD.OUTLOOK.COM (2603:10a6:290:45::20) To DS2PR12MB9615.namprd12.prod.outlook.com (2603:10b6:8:275::18) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PR12MB9615:EE_|SJ0PR12MB6879:EE_ X-MS-Office365-Filtering-Correlation-Id: 065e06f3-0ca6-4948-7896-08de918fb9ba X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016|22082099003|18002099003|56012099003; X-Microsoft-Antispam-Message-Info: mrRCeHZ6X4M33ia+g9PkT81knFOFpacxbCQp7FslZowSx6mxWcMVPplyQ0FjXXDrqLCVuUbF+ovcGXGweDYbP8DrX+TUnKxdmn66wsB1q2ji279pKDbTfF9hqMDkCZuIP0LSJiSiuqS480yugq8uygrPGOgeTQ44NEyC0of+cR7Coicj+c9zxPAZoQESr1pptd5n11OyxFBBuvotsCh8NM0KkkwoSvKWmWw3o+PMsWhTo8Ibm5x/rTSDxD6KbzS0nB/drfBQsrCvLqBtt2HseOpRcgrt1OOzrH/mlr6lUigmM13T3Fm6H4WkILn6pL0HMKEp4JLZpI83WkRrxWue1VI8Kfz/ai8BqZLq5eOHcEQGVjB0G7IflT6PWTYfcxSV22muzypJNYyzQyZ1RUREloNm2RBw3vUKSsu17PZOBIli6InEgQNGaQwaQ4gaGDs+OCiSTtALErN+23BpY+8/mH7FhkVrPRUUd0Z/HTat0uzZmSayN8kBlmw2m9n0LXEXDPXdyBTaTAavEGtkeg0kKO7T7+zE2qrk2Qec8fHoGpwVrHBkdpBy/DgPNqflBLZTLXVG+KMl2c5CjxfCrpv3rt3Mi+El2hye+LJ2DPyvGSEGJ2Hr3HYBSr0qcLZeXzfk/2kPkqLNMpsOnaBCTBbCEBnCU7LcJVIMs7LFypsF1ej9FOIHZgPBP2hTmFlebySCWDMFjPmE68oFdT46XM+OZSdLIeBeg/fvVOqwu9ROoqU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS2PR12MB9615.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016)(22082099003)(18002099003)(56012099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?RsZEv1iJGH+EKKVgblu+VJFxA66eKHnieFwjilOAnI/MTxWvCyKCnawWu4rs?= =?us-ascii?Q?zz9dE9opOogOcIcsruPAbmA8zPC8LbOQu/TH0jVpXfmhUw1zb8dFOxA5IPw+?= =?us-ascii?Q?AJrNxZvqODOYC5jAfdF1T35kp9lZ4Yr2kK01hJoDpnTiWeQqY2L4DfPqaQCe?= =?us-ascii?Q?qXPhklxsvao9WBBhXUzfPbEs9pXWhAetK/Vyb44KQuG50Vpe8Mfl803nNqNc?= =?us-ascii?Q?L27lhMHmYw/YqYNaCiKt7wMT9oWCbpYic2p6m7NkJVjWLMz/iRqduhF7mQib?= =?us-ascii?Q?fYTswnr1P+jrkIjepXRuZWOHnqoLJ7kF4uLzCWG5AvFh6yCR/P2t7gnVHU2u?= =?us-ascii?Q?LWyTnOtUkbDITLQpGHgEZu0+DKDBx6L3+wH7+HNQatlmxrZMjSrYsljTmO0y?= =?us-ascii?Q?Uay/2RUvjAbLeXKfV7RM3xW8wsr5mtuWP2i+KfIKQlV+zaLXpo26ZYiGSJkT?= =?us-ascii?Q?c0HmmipgDSiCRYZSfof6JclOqCTMvogOmUr2wvbJsIehadBLrpbiupXdQ2FH?= =?us-ascii?Q?AdxKnI/fjJ5ck7Y1aPOSOo+xl83AxMRyQy8mRrFqDQOFXNejN2krSQ+UJCi2?= =?us-ascii?Q?58cXgRTN/bKVgy257GvBgB9myQ0L9ID34PzJcTDBrSoJNiyco2d6TlM+efmg?= =?us-ascii?Q?h4shukjO4+kyXNI9d79TFFRFqZ9q/47CBO1H4QLSv0rHW6cmpFQ9OVuVZdGy?= =?us-ascii?Q?B/m20LDBdjEGLjF9wJy5s4WfNb7BmuZYe+T5Yvl8dJOIhAKUy/FmvJhiQege?= =?us-ascii?Q?3HQ455fmBrNQ+9X4YUG91N63aQqK64S7LjSe4UWFYa6FR3nuGXbZBjzF/hNB?= =?us-ascii?Q?hknrR29NXTfZqv4zETgSagi0KU+nzUX8I77c5WIh6LIcXmsvO0FPos3suRWn?= =?us-ascii?Q?gKd4uLa+ken3qADXgxmjGUsslvn1kpzrdFGidHKE+mH+YGNGvVsT0MAUyDTx?= =?us-ascii?Q?Tr0krKTyTZclUmuVNFt1srsL2zigppNkroC9+wvn4rYFlp5Bsr1eNDFRKaqO?= =?us-ascii?Q?HaGzqbmHcm2RB93uF8BenH0y69bhHcWxEYbab95D0FT2qTbh3LYpWUxedwv2?= =?us-ascii?Q?Fgy1H6UEuOqa2ZjS+51FPyeb5PmIwaI7fdBT0/dUtymiSZZOZkZ3hVH3Dp5l?= =?us-ascii?Q?IcL+2bnpSWOLrSEmN8fUkfdGxHRBrkP5sTNMrSHxttZKY5ZkUSgbt+cH9RAy?= =?us-ascii?Q?+6HhFCCWY/4ZzTnWz6951tK+G+vPOvzQyDEfOpBEyvZ5bunuSRVtCHLVTsY2?= =?us-ascii?Q?9GJnapp0KgJ2Apn1+HsFmKwdb9EW0jlHrZ7q+mB1+s0DJJ9E21tHKB0HCC8s?= =?us-ascii?Q?+M7untMAwBn8TrKHyMGjZVEuimoDzMJzAb4J5XFj5B4LsDhTOG2nNmuGcLMv?= =?us-ascii?Q?2VInGmHRbCKKzmuoeSrduLsuKJOf/6gwN+A4djj1cq8xiZtgJ9DGKlmw0Dk+?= =?us-ascii?Q?5I4txLTTEO4kvUBAdiJ5sjCt/O4aj2+F1vifMcRNFILvhQVhqNQWfrRsleSL?= =?us-ascii?Q?6Xqijv7r2SPt0hDk8+PYWqMw7sIEW1oZYCfk9X9/y3Y3dslLfMUDh2MEPSDW?= =?us-ascii?Q?m8OgPGB+0CeociOf4mInYWeSGBv9rMRtQX99fU3S4Ij2La5OWiBXLfdyqEW/?= =?us-ascii?Q?Bs3AJ97lRcXF07lGIrZ1k3NOvF4VJM7lftdXWFzI5zjA7hmikYEU1Au4EQAb?= =?us-ascii?Q?NR/1b6xicdKS7VOo002HI5xJEr32IX7MN3dCIKC+14Evg3nuNF5FvBb7dLfB?= =?us-ascii?Q?YOhxwH1/mQ=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 065e06f3-0ca6-4948-7896-08de918fb9ba X-MS-Exchange-CrossTenant-AuthSource: DS2PR12MB9615.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2026 14:46:02.0684 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: oR05HSD1YGKJd3rwTBFWorGO9cyX75wZBnETmC2Yok1WBSXzUZDjFCntONvAO3pZQlFu1pxeiHhj+6UFFFR1kA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6879 Hi Dietmar, On Fri, Apr 03, 2026 at 01:47:17PM +0200, Dietmar Eggemann wrote: > On 01.04.26 15:12, Andrea Righi wrote: > > On Wed, Apr 01, 2026 at 02:42:34PM +0200, Andrea Righi wrote: > >> On Wed, Apr 01, 2026 at 02:08:27PM +0200, Vincent Guittot wrote: > >>> On Wed, 1 Apr 2026 at 13:57, Dietmar Eggemann wrote: > >>>> > >>>> On 31.03.26 11:04, Andrea Righi wrote: > >>>>> Hi Dietmar, > >>>>> > >>>>> On Tue, Mar 31, 2026 at 12:30:55AM +0200, Dietmar Eggemann wrote: > >>>>>> Hi Andrea, > >>>>>> > >>>>>> On 26.03.26 16:02, Andrea Righi wrote: > > [...] > > > Just finished running some tests with DCPerf MediaWiki on Vera as well > > (sorry, it took a while, I did mutliple runs to rule out potential flukes): > > > > +---------------------------------+--------+--------+--------+--------+ > > | Configuration | rps | p50 | p95 | p99 | > > Just to make sure: rps -> "Wrk RPS" and pXX -> "Nginx PXX time" in > run_details.json ? Correct, rps == "Wrk RPS", p50 == "Nginx P50 time", etc. > > > +---------------------------------+--------+--------+--------+--------+ > > | NO ASYM + SIS_UTIL | 8113 | 0.067 | 0.184 | 0.225 | > > | NO ASYM + NO_SIS_UTIL | 8093 | 0.068 | 0.184 | 0.223 | > > Thanks for the test results! Ok, so SIS_UTIL doesn't seem to play a role > here. This workload should have #runnable tasks > #CPUs. > > Still trying to grasp why 'sic() + smt' is better than 'sis() + smt' for > NVBLAS? Same... > > There is a subtle differences in the start cpu for iterating: > > sis(): for_each_cpu_wrap(cpu, cpus, target + 1) > ^^^ > sic(): for_each_cpu_wrap(cpu, cpus, target) > > Not sure if this makes all the difference? I quickly matching the wrap start (both ways), but still doesn't make any difference: sic() is still slightly better than sis(). So the performance gap doesn't seem to be in the wrap origin. > > > | | | | | | > > | ASYM + SMT + SIS_UTIL | 8129 | 0.076 | 0.149 | 0.188 | > > | ASYM + SMT + NO_SIS_UTIL | 8138 | 0.076 | 0.148 | 0.186 | > > This should be the same, right? SIS_UTIL is only for sis() so when using > sic() this shouldn't differ. Or did you code SIS_UTIL into sic()? No, you're right, it should be the same, SIS_UTIL is irrelevant here. > > > | | | | | | > > | ASYM + ILB SMT + SIS_UTIL | 8189 | 0.075 | 0.150 | 0.189 | > > | ASYM + SMT + ILB SMT + SIS_UTIL | 8185 | 0.076 | 0.151 | 0.190 | > > +---------------------------------+--------+--------+--------+--------+ > > So with '#tasks > #CPUs' smt doesn't make a difference. Correct. At saturation there's no benefit with the SMT awareness, which makes sense, all CPUs/siblings are busy, so there's no preferred fully-idle SMT core to prioritize. > > > Looking at the data: > > - SIS_UTIL doesn't seem relevant in this case (differences are within > > error range), > > - ASYM_CPU_CAPACITY seems to provide a small throughput gain, but it seems > > more beneficial for tail latency reduction, > > - the ILB SMT patch seems to slightly improve throughput, but the biggest > > benefit is still coming from ASYM_CPU_CAPACITY. > > > Overall, also in this case it seems beneficial to use ASYM_CPU_CAPACITY > > rather than equalizing the capacities. > > > > That said, I'm still not sure why ASYM is helping. The frequency asymmetry > > OK, I still would be more comfortable with this when I would now why > this is :-) Working on this. :) Thanks, -Andrea