From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CO1PR03CU002.outbound.protection.outlook.com (mail-westus2azon11010032.outbound.protection.outlook.com [52.101.46.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55232361651 for ; Thu, 2 Apr 2026 04:41:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.46.32 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775104889; cv=fail; b=luCMYkM0WSc5cOkXelS7fB5kQeL+SZkxr2ituL5s6C7H6IZfFBB4WyGGrSfYc2RR4zJWvFkcqqy4EYOj0/5nIsG9LqtDYUb8/3YCZueTWOafk+1YvfgfaupfqPgfRXP+hMzHD5V2htIfW/3XyRJVS5Bxak0tXT+AtiSYPdyGt8o= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775104889; c=relaxed/simple; bh=TTk5/RujKYUHJ7qvgyfno2zOqTGtMvJUs1C3s02s0/s=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=kUyyBoydpEY1XJXJmJyemfJ6rFpnvbzbMR1AEtOZPCM1G8hoANTYGs5xgd7kn4SPCSRJh28S7fNNmJW8NXQrj7Ay+Ux4JXJfQ5T8kpr83UqH3SC20moC+f4aJOSPMOPwPVlU2qZnAufyA28gRkEfkIqJIjJIwMVhgE/JZShRwXc= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=ot4XLbCu; arc=fail smtp.client-ip=52.101.46.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="ot4XLbCu" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CpkzHnoHpQ+UNP3SZC4Qs1LStfLh9GVU3sLFpnRNIOEQRa/UZqqDXS74ZgkeqMRzkkMJEUmcXXKHuRMfW3u0Z6Q7N2MR3p4OFaZ0HZxnrAec8s3CzqaVj3E5nX0LPNAm8bUDK4QJUKNQMy+9Zr8JIFfshCXCgmQzSPPDYc4x4zCS5hrSoqtic39xKPxLwm9UxkLpZrMwy+PjfjMHod4BJEUxiLhnoa/uutaAuF4viRq13SaWQjT95iHx5AakkLi5r7QBN39r5gBlTmWWCBMhV/HMIVXDLdK//nrpCjmBfB1p5ZySVDIRzAK8Ac8RYGSWDfLQykYEAEgZgQnAToiOsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tFBs0VvJk9oEtUJIRK41y0JXCCqHdEpxLROF4K4yQ9g=; b=bj/bxcxZeCmQ27WF9RJi0NEfQN54R0HaD/68sf8rIWFFJwLTOxtOOyq5SuLearlNNr3MTsFQIeNINCV5F7BkZlSWlDCseTDQ1OTOORmJfHt/dbx/jn+Pe+STPa1Mv8rQO3gWYgSa0uYp5y7SyGrtbRfTZ69F18T/ajzETsEVbYgNCC8ibGvCdnPG8rX4KK3MeKdQOpZxiqoIIHLOpP+HA17DxdGEmT6RYLP4N05zozjmPEGiQb51yFbyxSNG3Tczi51yVFc4p05s0boqykVJ1rGxqIxXL/XRSiBZZEW6s0EqtuQFFiW1G/+TLYwmHEPlWmpSOMCu7ekBBeXna8LVfA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=intel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tFBs0VvJk9oEtUJIRK41y0JXCCqHdEpxLROF4K4yQ9g=; b=ot4XLbCujbYNOqpwCMZIRBhSFC65IoLlFBzsIMIV7V7SK1KuSVrSFrV/NTnCOXQ8WfkqakU/9KgMLdP1HI0042i8wNv3aVoGCGBYUW7TeFHiFHEJFPRD2Bj2kc35Fu5NvFyzBoocrFe3S5OBOTWt/NDSlQJKOgi0ldi837ADEls= Received: from SJ0PR05CA0054.namprd05.prod.outlook.com (2603:10b6:a03:33f::29) by LV8PR12MB9690.namprd12.prod.outlook.com (2603:10b6:408:296::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.17; Thu, 2 Apr 2026 04:41:20 +0000 Received: from SJ1PEPF000023D5.namprd21.prod.outlook.com (2603:10b6:a03:33f:cafe::2b) by SJ0PR05CA0054.outlook.office365.com (2603:10b6:a03:33f::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9769.16 via Frontend Transport; Thu, 2 Apr 2026 04:41:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by SJ1PEPF000023D5.mail.protection.outlook.com (10.167.244.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9791.0 via Frontend Transport; Thu, 2 Apr 2026 04:41:19 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.2562.17; Wed, 1 Apr 2026 23:41:19 -0500 Received: from satlexmb07.amd.com (10.181.42.216) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 1 Apr 2026 23:41:19 -0500 Received: from [10.136.42.52] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Wed, 1 Apr 2026 23:41:17 -0500 Message-ID: Date: Thu, 2 Apr 2026 10:11:11 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 4/4] sched/rt: Split cpupri_vec->cpumask to per NUMA node to reduce contention To: "Chen, Yu C" , Peter Zijlstra , Tim Chen CC: Pan Deng , , , References: <20260320124003.GU3738786@noisy.programming.kicks-ass.net> <63a095f02428700a7ff2623b8ea81e524a406834.camel@linux.intel.com> <20260324120008.GB3738010@noisy.programming.kicks-ass.net> <138c3f9d-309f-41e6-aa72-a3f6bd713bf0@intel.com> <22072ef8-5aec-49ac-9cc4-8a80bec14261@amd.com> <64649c85-29ab-4f70-a0c4-3c83cbdae2fc@intel.com> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <64649c85-29ab-4f70-a0c4-3c83cbdae2fc@intel.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Received-SPF: None (SATLEXMB03.amd.com: kprateek.nayak@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF000023D5:EE_|LV8PR12MB9690:EE_ X-MS-Office365-Filtering-Correlation-Id: 9b7404b8-e84f-4d4a-9e14-08de907215a7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|376014|1800799024|82310400026|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: afrKMLDm8wrVPgt6blb1mTk25rrxbylhhFqPEJXY+VBn6CabytMaPi9eINY4B+s0VNXWki4vyVkT0N9T8ztuc/IQlduV2wKlHnZ8xlnjqBygNZDqR99ZVHS20bnS8E06fnlhjK30A7QneK/fpbwwh/O3RCp8Js8reZQ4HD1NbECk/vqx5tM9V/fUZvyn59ZRxu+eWHOI/aT7bGhf5aICd0KUIMnMVKKF2HSPUVwexAkGNRsSKYtxt6gQrF10XAzFCPQBc+pxjPmsn9ErM2eCcb7LMbSqJywxqOXA/mj/UGFeX7QTfHo7mTBDkStmwWHTzbyzQA+2JE/DTcIDZenxpuptMvxSx9AwzH3qkWOUOXcXYMstb6zS138D8+P9OvsuU1kgduBLl18Y9ir6XvJsaxocjpwXClIjQlrAa2wchRGBhp2hsl9Y2clYmY5aenHl1NMb+YbNoUP6ooq4hHW+rN3DYx+7BbPJrMMilTrCToTFQHmE0F6J7R2UMNK6U1pUqRu7MmVMSWLWAQotN+/pcfCYaugA8BxuG+DeUM4nAYhDE35GRgN4GM34lSEBdTxR7jUDrk4Kx5pc7WLPhqp20zEJkYLY4KxLhJnarf8sIjc+bTOZg9X3pHtwOqXAJVCCCTqrAfHZvrJvZcZC5evtrAAOxPBC/YAkzVMXNWE9eKw2RMgVUH2/sTgDkdbv0CLVzizPs1bTXW7cJ2cGTdpe8RRAdXX/f90pF3JKmipvwZ7FMn1sMa/DAG7WVmistiNnFtFwe568V6TaL8yojeS4ZQ== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(376014)(1800799024)(82310400026)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: eoe5A0zLhfCYq0Bf/kostHkbmoMv8Yj7BWrZb50FkY42HQSf3SJPKG/nEXeiFJ9OXoCoRw7W0Skk8L2acPrRjx2aqA2KnxCbI+CQVi4GKpDoNFI661QHTS6wKAYA93Iq4Hl9KalyKo8lspGyg0eemR1115sGF4jcdPV0jTi1oJXzcGM6l5v6wcbreEvzxeFPLD77x/qgN6v8eujZpoNB0M5B2slyczGINpMXtN4ds94Pt9DmJOfHAR0Q+Hmchl9rKh1RO+1WKEQFQgKVEwiTbZVRe0d45ehOuZPNrICmIRqXmVlv5aO0YgkdBT5rBQ8hnwjuyq7zzJQluE47m0BSVsaNeSJ54xvHSSRUiXuQWp8uORU9ZkugH3j9DNEM9s9DChGGE7YRISyVQ3W4YuHddeA43C6ds+bzkbHG0PfZPJQG7ed+Qt+8EbdtAkl8jJSr X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Apr 2026 04:41:19.6074 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9b7404b8-e84f-4d4a-9e14-08de907215a7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF000023D5.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9690 Hello Chenyu, Thank you for testing the changes! Much appreciated. On 4/2/2026 8:45 AM, Chen, Yu C wrote: > One suspicion is that with sbm enabled(without your patch), more > tasks are "aggregated" onto the first CPU(or maybe the front part) > in nohz.sbm, because sbm_for_each_set_bit() always picks the first > idle CPU to pull work. As we already know, hackbench on our > platform strongly prefers being aggregated rather than being > spread across different LLCs. So with the spreading fix, the > hackbench might be put to different CPUs. Ack! But I cannot seem to come up with a theory on why it would be any worse than original. P.S. what does your SBM log in the dmesg look like? On my 3rd Generation EPYC machine (2 x 64C/128T) it looks like: CPU topo: SBM: shift(6) leafs(4) APIC(ff) Now, I suppose I get 4 leaves because I have 128CPUs per socket (2 x u64 per socket) but it is not super how it is achieved from doing: arch_sbm_shift = x86_topo_system.dom_shifts[TOPO_DIE_DOMAIN] - 1; that divides TOPO_DIE_DOMAIN into two but that should only be okay until 128CPUs per DIE. It is still not super clear to me how the logic deals with more than 128CPUs in a DIE domain because that'll need more than the u64 but sbm_find_next_bit() simply does: tmp = leaf->bitmap & mask; /* All are u64 */ expecting just the u64 bitmap to represent all the CPUs in the leaf. If we have, say 256 CPUs per DIE, we get shift(7) and arch_sbm_mask as 7f (127) which allows a leaf to more than 64 CPUs but we are using the "u64 bitmap" directly and not: find_next_bit(bitmap, arch_sbm_mask) Am I missing something here? AMD got the 0x80000026 for defining TOPO_DIE_DOMAIN as soon as we crossed 256CPUs per socket in 4th Generation EPYC so it'll have per CCD (up to 2LLCs) smb leaves but if I'm not mistaken, some of the SPR systems still advertised one large TILE / DIE domain. I'm curious if your test system exposed multiple DIE per PKG since 280 logical CPUs per socket based on the cover letter would still go beyond needing 64 bits if it is advertised as a single DIE. > Anyway, I'll run more > rounds of testing to check whether this is consistent or merely > due to run-to-run variance. And I'll try other workloads besides > hackbench. Or do you have suggestion on what workload we can try, > which is sensitive to nohz cpumask access(I chose hackbench because > I found Shrikanth was using hackbench for nohz evaluation in > commit 5d86d542f6) Most sensitive is schbench's tail latency when system is fully loaded (#workers = #CPUs) but that data point also has large run to run variation - I generally look for crazy jumps like the tail latency turning 5-8x consistently across multiple runs before actually concluding it is a regression. hackbench (/ sched-messaging) should be good enough from a throughput standpoint. -- Thanks and Regards, Prateek