From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11012002.outbound.protection.outlook.com [52.101.53.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 365F1279DAF for ; Mon, 23 Feb 2026 14:27:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.2 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771856878; cv=fail; b=duyX0fMJy7U92khsqAL535Rzu5MVIc24Nxv5mShjAvlXFgUlt88MHe/jeYxGhVd6hIi68PJeyn8NrEbdHe2DCTVvFBgXk7MC4LopO1JBp0+YsDqd99Fp6BNiRadMDMhRZI9zYPhU+XB6/VnfAQ3hDqcAwp+DfZZdYuD79I009zM= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771856878; c=relaxed/simple; bh=CTwIik+tSfMGpXeEM1q6ci/mY89gq25Br/5C93HRmec=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=Ue5tBj7LWoDUFfGOEXZjSlB3XIOYSgxI0M/o+tjjTRKNixD+n32i2dRKu7Z8lPmZJWQC51crJZQMN1W92u8AWbgGESEfumH2JHRkpw2JIXkhFvfTA15eBn/gHjaWkdPb8NU/CRbBKAedaiodWYIMlEHWTWKy2JZz2r00AaTG/TA= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=XW/hRZ5l; arc=fail smtp.client-ip=52.101.53.2 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="XW/hRZ5l" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=EsmRSKwMxMwb6YyZKlFkfaSUVamdJNlGd9lWUq5CWHiZGMjzQFE/MAeHmdi1Qgk1nNtPK4EJHdNHtsRHpL1kI1ihKDD3VCVp6kWNa+xynhd9phOKdA+R1qHLa8Bg9bKvIEpovhS49JQOoGxaoUL2WahDsyeG/iRLMBR7/uefj6x8cvV5Wdxmh0HI2c2xf8XwNW/DTSdhrWh1xH7Ak0NSpN03EAE3/8I4kep+Z+CIofRYGbeLwv+1l08epZp4pvJcZwL2uhuSN1bQGNemIvMO1Z8bwFBej7jxlfzwtIsbHAhYTPzq/X3yxI62Lem94pPzMA8Za3Piynw4nZSWDXayfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=D4M0bto+fSumVxsJQ0r0IFQ8RsFZIaHQ4rvRjEXNo64=; b=wPuOOnls6f3Op6tiCmvT4Kk43c3BPDSsYzp6RRZF60/yBtGclwuRmg6o+q17CkH4L8kq6b4UdNrxhH2KJSJ0tHHEViTgglNONVI38qrELfnZvltzIoT9Dv2uBnwFGOklYuG542PwUPnIEN7YZwyJ1ruVGBbdsxXqtWut64pgzP+iKsZqftZ4/h5h5Hc1jax1QtFvqtJztVaMljx0D6BqK7y3mopIUzfxVv7IsQOTyWOwCrMzJ2WskR35KSxWgy9uGvAubiIsmfK1IlgoEgcml+dHVHiQCGna4JHGHe6wFuloRs4hgaDdRfro7d9ZkcKSdh0rv5IJMjKhZDH8OoEG9A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D4M0bto+fSumVxsJQ0r0IFQ8RsFZIaHQ4rvRjEXNo64=; b=XW/hRZ5lgHpxXQ6KP243+iW4Row+xgHawZeWiAJ38eksuvN2JgcUqRd6bxefIkuVqaI267f3KAIeOCd2zgTHE4HV4TTTcy+nzLslwhEKlHWgFHn3vzYPc+tBV4krlR2DYFTTY1ZlIQB4W47qmLmRvkO/f9peRYZzsqsX+fgjxQ4= Received: from BY3PR05CA0044.namprd05.prod.outlook.com (2603:10b6:a03:39b::19) by DM6PR12MB4156.namprd12.prod.outlook.com (2603:10b6:5:218::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 14:27:50 +0000 Received: from SJ1PEPF000023D1.namprd02.prod.outlook.com (2603:10b6:a03:39b:cafe::79) by BY3PR05CA0044.outlook.office365.com (2603:10b6:a03:39b::19) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 14:27:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb08.amd.com; pr=C Received: from satlexmb08.amd.com (165.204.84.17) by SJ1PEPF000023D1.mail.protection.outlook.com (10.167.244.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 14:27:50 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.2562.17; Mon, 23 Feb 2026 08:27:49 -0600 Received: from satlexmb08.amd.com (10.181.42.217) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Mon, 23 Feb 2026 08:27:49 -0600 Received: from [10.252.192.21] (10.180.168.240) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Mon, 23 Feb 2026 08:27:40 -0600 Message-ID: Date: Mon, 23 Feb 2026 19:57:39 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH v5 00/10] mm: Hot page tracking and promotion infrastructure To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , References: <20260129144043.231636-1-bharata@amd.com> Content-Language: en-US From: Bharata B Rao In-Reply-To: <20260129144043.231636-1-bharata@amd.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Received-SPF: None (SATLEXMB03.amd.com: bharata@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF000023D1:EE_|DM6PR12MB4156:EE_ X-MS-Office365-Filtering-Correlation-Id: 3e2aa836-c8e0-45f8-0fa2-08de72e7b931 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|376014|36860700013|7416014; X-Microsoft-Antispam-Message-Info: =?utf-8?B?M05SdjdEZU1admhLOEhoVEh1T3hINzl6VTlkQ3pINUZuNkplM0htUTVtb1R5?= =?utf-8?B?VDFPVnZWU0huMFNOM0wwNWZyZVR1djhWbjNNaWNJcjZsUWg5QmJXc0l5MjhT?= =?utf-8?B?a2R2REowNW5VVjBSR1VEb3h2Z21FbVJUUjdoc3NlTDRCZlZSbU1iSDlrZVVN?= =?utf-8?B?TU5SWklmVEt4bk5MZUdyZHUrSUVhUEVYVVkzd09NQTZ6ZGt6WUpQK0d4bHFm?= =?utf-8?B?WWt1QlZ3K3ZrUU1BVjI3ZklrK2sxa3NFUXFVZFFqZ3VMSGNFQ01zNXowRWhx?= =?utf-8?B?a0hLUUZBVlhBbHFSYlZpREpIdFNQamRuaGN4TmpCRUovb0lQaFpmNFIxT1d4?= =?utf-8?B?d3RPRXNMY2htSkM5c01oaEhtK21sY3RiTE5DNk1EbUl3SFlhcEt0WXlBeVFU?= =?utf-8?B?VW1nOU9IMm5zdE5LbExtNWpoRHIrWmwvVEFNOGl6VjNZcVZJVGlHT2R0ZlF0?= =?utf-8?B?VzJGNlhWVTFlRlpaWWxkQWs3ZCtocjhGU2pmZjBxZG10bTdjNUZuMXFVWERq?= =?utf-8?B?bzgwUllRdmZoejRxSEdSaWhsMjFrZ3VEVXEybkx4U1ljVDZ4Z043SHlDSE9l?= =?utf-8?B?R0VqWVVNZ0FGOVF5QjQyNmRNUklJbzhZaEdLSmczSnM2MGlnK1lSR3VHZ0lJ?= =?utf-8?B?dkZiNFJNR1ZDNVAxYlZBRnY3M28vUDFBQnJ4d1MwRHZmT1MxV2RCN3JyK3lI?= =?utf-8?B?UjJSVjdxWUNIeWpvaUoyeTB3Q2IxZlFudFRVYWptdmp2NE0xaHV5bjB4eitZ?= =?utf-8?B?THpKa0xJVGt3SGxMQmszeGtOUEVNMkJPZkg5R2pnditZSlNNdGsvUHpuUDVt?= =?utf-8?B?WFJRM1oyNkRDQnZ6cC9ubkU3SnppQ3kzcWN0dTRUajJJcjk2V0tHa29qSzJU?= =?utf-8?B?U2Vud1BKWFdpdGhXbGhzOWQ4dWVmbStUZ1pKOXUrNHhiZkYrSDJxSEtwOUJi?= =?utf-8?B?S0NuZk4wdHJwRFlEbmNHekprVVpxRjU2bnZTZWxJNHBpQzc2engrM3ZWWG93?= =?utf-8?B?clhIdVM0a1FZV0E2MmxxUTZOWkovODVMVW5tUnRPbTMzREd4MWh4MHFwby9I?= =?utf-8?B?K1FyY0V1K2NuSkhqaVB4anBrZ2p2aEJKeUgzcXEwOWFqT1I4QW9zeW9HK3VN?= =?utf-8?B?SWRJQ3ljbmZWMlZzS3hGSW5ZOWt1MTJUekpOU2VlSWYwMGZublJhcVJMQWw4?= =?utf-8?B?dGtWcDFFdHJwWHQ1YjJvNE1kMGw5WjNDb1UrM0J1d3hrWlkza3F5akZtZmQ1?= =?utf-8?B?QVhmOU1oRDNCaVE2a1h2ZzNrYmd0R1ZZeEN3QzdORjlxNFRtMzBya3F2MUdC?= =?utf-8?B?ZFY4NSswZjNMbEpJeFZTZEJVSzAwQmg2S2ZUT3BjWHNnNHcyMXRuS0c1bU1G?= =?utf-8?B?aWdtTDNPaElhMnZQc0NLd2RmUVlXRnNnVlk5RUVBOEM4RUlUamJJbDROUDVh?= =?utf-8?B?WXA4YmtvbE45elpmZVBLeURoMjlGOUN5cEJ3bHltLzJtTEcyR2RxbktCWFZG?= =?utf-8?B?Tmw0bWpjYTlXZ2djNmlCZXpPeVFsb2RXRWwzM3dFZXhocWhyYWFaZi8xNlAw?= =?utf-8?B?MVFnVzREaWtCZmZqc2E5QzY5TFREbXJpQ2QrZVQ2WDhhU1FRRHJvZ2pDQXM5?= =?utf-8?B?NVl1MXBVUkx5UUZ5b3VzeGZ1dktscHNNeEIvVUY3U1FRd1lFcnd1VytqL0NF?= =?utf-8?B?NnJOVjM5SjhBUTg5RVBtZEdsZXA2bDZ2Z3g4ZkxKTE1JSU5TKzdrMVMwRTRq?= =?utf-8?B?bnd0eTk0dlBubVU3OHJEMkdHUWdKRjl2ZUozRkRGRk5RNWx4UnpIbTN4QVBK?= =?utf-8?B?T3RMTlp6RTlWK211ZE9lZVM3c0hWNkxFNFpSY0dmMkNUcjI0dVRTZmNvQTND?= =?utf-8?B?c21LcC9yYjYweFhqNVdOVVZmZjZuQlIzS1drR3Y0L21lY1VMWG1rM050UWlG?= =?utf-8?B?MkJNUmU4bXB0YmlKTFRnWWNGd0pxZnRmMkRBb3QwNlNTOXlrYzJjcm1OTE4y?= =?utf-8?B?dEd5V294YytxT3dlMGtmb0EvUEIzK0VFZnNVSGphdXJyR1o4dHlGbEJLRWhz?= =?utf-8?B?T1BmZ2cwdUtURUM2WGJ1dmpSU09HanhlYUhWaGt5QTgxNkdsVVhNd2NveEFS?= =?utf-8?B?VTBQRG05Zy96QTkzZU9SM0ExeVR3eVhFbEZNOHJpYUdHdXk0MEk5cm8rL25F?= =?utf-8?B?RU54RkFwbHoxNXJOdHZBN0lzbFcvZzNUOWp1T1pTUzQ2SHJWR2t2L0NDWEdj?= =?utf-8?B?Y2szUGVtZHpXNk8yd3RzTGx2N2p3PT0=?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb08.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(376014)(36860700013)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: emRfIroJVaoZ1aXL8Ob19SMYJ8q7+xGRsNYglDlgDO07VoWyBVE3AsqPoVRQWuwHbXwxG/YM1Qd6Kdsww8MkQ/wxFHoAmk2Vz97w1TrmCYCdB9hx5WuCR4f1HImDNnlEruSltoxkGpXB/vXTIjJJsIGAOMziVWaaMIRv7sBjfmAFwVUkdQU1+nhNQNOVEaTSSE9SKZcLwAkOV8YlqV2liailEvzXo9kZIJZ5ahHtq/njW5y3ORM23K8aUXClagd7ECrSOFkOkvEDBMjpbPdJZ30WcwedG5e84sfQN05gfgL/kp+s1yCnaQ1jPnI1yGt8b6itolbfNXatM+4XPUtm3fmNahaQGIma/E1IkeX6+NTyJsCd01JotEEL1L0CFaBXkEOp+L9jjD7ZDX6UQGY/wn7SoM7Vp2iUVc/POL8GavhNqfAK3cIPQoaI/141F6wl X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 14:27:50.2373 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3e2aa836-c8e0-45f8-0fa2-08de72e7b931 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF000023D1.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4156 On 29-Jan-26 8:10 PM, Bharata B Rao wrote: > > Results > ======= > TODO: Will post benchmark nubmers as reply to this patchset soon. > Here are some numbers from NAS Parallel Benchmark (NPB) with BT application: Test system details ------------------- 3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2) $ numactl -H available: 3 nodes (0-2) node 0 cpus: 0-95,192-287 node 0 size: 128460 MB node 1 cpus: 96-191,288-383 node 1 size: 128893 MB node 2 cpus: node 2 size: 257993 MB node distances: node 0 1 2 0: 10 32 50 1: 32 10 60 2: 255 255 10 Hotness sources --------------- NUMAB0 - Without NUMA Balancing in base case and with no source enabled in the pghot case. No migrations occur. NUMAB2 - Existing hot page promotion for the base case and use of hint faults as source in the pghot case. Both promotion and demotion are enabled in this case. Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1) NAS-BT details -------------- Command: mpirun -np 16 /usr/bin/numactl --cpunodebind=0,1 NPB3.4.4/NPB3.4-MPI/bin/bt.F.x While class D uses around 24G of memory (which is too less to show the benefit of promition), class E results in around 368G of memory which overflows my toptier. Hence I wanted something in between these classes. So I have modified class F to the problem size of 768 which results in around 160GB of memory. After the memory consumption stabilizes, all the rank PIDs are paused and their memory is moved to CXL node using migratepages command. This simulates the situation of memory residing on lower tier node and access by BT processes leading to promotion. Time in seconds - Lower is better Mop/s total - Higher is better ===================================================================================== Base Base pghot-default pghot-precise NUMAB0 NUMAB2 NUMAB2 NUMAB2 ===================================================================================== Time in seconds 7349.86 4422.50 6219.71 4113.56 Mop/s total 53247.66 88493.630 62923.030 95139.810 pgpromote_success 0 42181834 248503390 41955718 pgpromote_candidate 0 0 577086192 0 pgpromote_candidate_nrl 0 42181834 29410329 41956171 pgdemote_kswapd 0 0 216489010 0 numa_pte_updates 0 42252749 607470975 42037882 numa_hint_faults 0 42183772 606540729 41968150 ===================================================================================== - In the base case, the benchmark numbers improve significantly due to hot page promotion. - Though the benchmark runs for hundreds of minutes, the pages get promoted within the first few mins. - pghot-precise is able to match the base case numbers. - The benchmark suffers in pghot-default case due to promotion being limited to the default NID (0) only. This leads to excessive PTE updates, hint faults, demotion and promotion churn.