From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from invmail3.skhynix.com (exvmail3.skhynix.com [166.125.252.90]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 40A45258A for ; Fri, 24 Jan 2025 05:53:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.90 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737697997; cv=none; b=OCuk24HcIZAuMbgLECLMK8U1wFJ5+P2K7EkuCcJLkZPXyKEFEcrfieCuDaKibWwgQlZ+nNt/yz9+cLBvU4rQzRAAA0Z+m4M+WF98c8wH02Hruo7fGj0y9jRxjjCs7nF119hqbUGAbILMTz5kJGGtfJx4xtJ0E+qU96huPQW17vk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737697997; c=relaxed/simple; bh=ocRiUUEFLzEqC1rBLMEZ+AiQdwNg5Un5ys+cUKMbaog=; h=Message-ID:Date:MIME-Version:Cc:Subject:To:References:From: In-Reply-To:Content-Type; b=AZvKHJQU0ACT60ZoV2RxJQ1ecL9oUed6+bUOJRft2BZif0rBTK+UMDVsINbok+KFz28Rd0gMoHAfZ518ygobJU7ClGIim2XESyXjbNFUn7MumZ6JVhYM7Qapwb91FM7RfsIopsHdXyQ5eebxGWjY0LG61V+ckJupwisiojjF48E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.90 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc59-7a9ff700000194b3-2d-67932ac4f167 Message-ID: <0a9914f8-bec7-4e58-ab12-a87fe3876187@sk.com> Date: Fri, 24 Jan 2025 14:53:08 +0900 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Cc: kernel_team@skhynix.com, 42.hyeyoo@gmail.com, "gourry@gourry.net" , "nehagholkar@meta.com" , "abhishekd@meta.com" , "ying.huang@linux.alibaba.com" , "nphamcs@gmail.com" , "hannes@cmpxchg.org" , "feng.tang@intel.com" , "kbusch@meta.com" , "Hasan.Maruf@amd.com" , "sj@kernel.org" , "david@redhat.com" , "willy@infradead.org" , "k.shutemov@gmail.com" , "mgorman@techsingularity.net" , "vbabka@suse.cz" , "hughd@google.com" , "rientjes@google.com" , "shy828301@gmail.com" , "liam.howlett@oracle.com" , "peterz@infradead.org" , "mingo@redhat.com" , "nadav.amit@gmail.com" , "shivankg@amd.com" , "ziy@nvidia.com" , "jhubbard@nvidia.com" , "AneeshKumar.KizhakeVeetil@arm.com" , "linux-kernel@vger.kernel.org" , "jon.grimm@amd.com" , "santosh.shukla@amd.com" , "Michael.Day@amd.com" , "riel@surriel.com" , "weixugc@google.com" , "leesuyeon0506@gmail.com" , honggyu.kim@sk.com, "leillc@google.com" , "kmanaouil.dev@gmail.com" , "rppt@kernel.org" , "dave.hansen@intel.com" , yuanchu@google.com Subject: Re: [LSF/MM/BPF TOPIC] Overhauling hot page detection and promotion based on PTE A bit scanning To: Raghavendra K T , "linux-mm@kvack.org" , "akpm@linux-foundation.org" , "lsf-pc@lists.linux-foundation.org" , "bharata@amd.com" References: <20250123105721.424117-1-raghavendra.kt@amd.com> Content-Language: en-US From: Hyeonggon Yoo In-Reply-To: <20250123105721.424117-1-raghavendra.kt@amd.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa0iTcRTG+b93Z4vXpfW3qGhh1LrZhTiRSUTR/0uSRER+qZUv+tZcsqVp ENjNSLPmrXTa0syytjAnMe1i81JaaBdNaZJW5KKV5rTsZrFcEvjtx3Oe85znwxFolYudLsj6 A5JBr9WpOQWjGJhUuviBJjcu3JJJQ/bpcMj2dHJQXGnj4H2bk4by7M80DHveIBip/EXD2Xtv WfjZ08yD1b4ZThTe5MA9fIaBKvclHn65/yAoPPeVh9bOUgYGTA4Ed7vmw3CtiwZH2hseOm4X c9Br87FQ9+keDTXldRTUujpZaHeWUFA/UobA7D3Ng+PVeR6as5wUfMz7wUKX/RYL6ZerKGiq dFDQf0cPT2xfeChpGVtoHOmnoOjIGQSDXUM8jP4YO2G1xcP9nxUIih708usWkRPtfzhis9gQ aewfpEmtuYcnJfYkkpnewZPqCg0pu+uhiP36KY7Yh3N40lIwypDat6tJ7suriDx5nkKyjn3m yJC7m9kSEqOIiJV0crJkWBq5SxFfXv+USrROTTEdD0lDFlUGChCwuBL3drxn/3Npu5fxs1Jc jd2X8zk/M2IYHnrchsb1IPyosO+fJ0ScjV93F/AZSCHQYvVk3FZ2cmwgCFPEvfjC7e1+PVhM p3DOYy/tX1CJEbi7d/Qf0+I03N13kfL7OVGDa05JfjlAXIsbH15jxy2rcMatDDTOs7FjoJj2 Z2LRJ+CKPA8/XjoU11e8ZEwoyDyhn3nCCfOELPOErBLEXEfBsj45QSvrViwx7otP1cspS/bs T7CjsYe7cvj3jhr07dnWBiQKSD1JSVblxKlYbbIxNaEBYYFWByt9faY4lTJWm3pIMuzfaUjS ScYGNENg1NOUy78djFWJcdoD0j5JSpQM/6eUEDA9DSXecKjDDFHb5IKita9Njwb2Pt3oSZ+p mz9jedlcS4HVtbV13uTkLMonf98D6rDo3KRnzWtsXMdgKEse+sKd3i3rZ50PdLVHWgJb3D1N gS/kqK6I0Cm7lV5r7JyGTToq6oMm/1x+a0zT4WNy/ruFkbkbg6Ln5tVnxhw9vmGBc3udmjHG a5dpaINR+xelyn3xbAMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa0iTcRTG+b93R7PXpfliF2kU5KhZFHKELn4IfAmK6IvVh3LU27bUlZuZ BpZWGtm0aWVzXpqpteZIndTUzLxVJlmouFLKNJxg07y1sEzKJYHfHp7n+Z1zPhwGlzSSQYxa kyBoNYpYKSUiRMZNCZvbZDeVW/ry1kGOfgvkjPZSUFhpo2CkswmH8pxvOEyPDiLwVP7C4caz IRJ+fnpFQ4V9H6TnV1EwM+LGwTWdTUC16x4Nv1zzCPLzvtPwpreEgNai1ySMGxwIGpwbYbqu DwdH6iANPfWFFAzY/pDQ6H6GQ215IwZ1fb0kdDeZMWj2lCIwTeppcHy8Q8OrrCYMvt6aJcFp f0xCRlk1Bm2VDgzGnmrgrW2GBnP7AtDqGcOgIC0bwYRzioa52YUVFTYVPP9pQVDwYoCOkPPp 3fMUbyu2Ib51bALn60yfaN5sP8tfz+ih+RqLjC9tGMV4u/Uaxdunc2m+3ThH8HVD4fzNDw8Q /7Yric+6/I3ip1z9xIHAI6IdJ4RYdaKgDd0VLVKVN7/DzlSsTDJcCUhFxZJM5MNw7HaupHuS 8GoxG865ym5TXk2wG7ipjk606Ptxr/OH/3UC2GDuc7+RzkQiBmdrfLnO0qsLAcOsYE9xRfVR Xt+fzcC43I5J3AtI2B1c/8DcP42zgVz/8F3M26dYGVd7TfDaPuxOrvXlQ3KxEsZlPs5EizqY c4wX4gbka1pyhmnJJNMSxLQEMSPCivzVmsQ4hTo2TK6LUSVr1Eny46fj7Gjhq+6n/M6tRT96 IlsQyyDpMnFEaK5SQioSdclxLYhjcKm/+M+wQSkRn1Aknxe0p49pz8YKuha0iiGkgeK9UUK0 hFUqEoQYQTgjaP+nGOMTlIriSlIaPOtlFwqnLH5yy9oJZ3O7KlwZErZ89Xu/tEazc7xizbby 7LTfvRddu08GRsYbrW2sWX6RtPjt1e33P39YzxgO/Rjqts6PRoWQ8eyxvIjVs3s8j8RCltGM vUm/5D58tEqfE+3zJGDIfe6QZqRL/+WCVXGq60n9QevEiMQiJXQqxVYZrtUp/gIletfMUQMA AA== X-CFilter-Loop: Reflected On 1/23/2025 7:57 PM, Raghavendra K T wrote: > Bharata and I would like to propose the following topic for LSFMM. > > Topic: Overhauling hot page detection and promotion based on PTE A bit scanning. > > In the Linux kernel, hot page information can potentially be obtained from > multiple sources: > > a. PROT_NONE faults (NUMA balancing) > b. PTE Access bit (LRU scanning) > c. Hardware provided page hotness info (like AMD IBS) > > This information is further used to migrate (or promote) pages from slow memory > tier to top tier to increase performance. > > In the current hot page promotion mechanism, all the activities including the > process address space scanning, NUMA hint fault handling and page migration are > performed in the process context. i.e., scanning overhead is borne by the > applications. > > I had recently posted a patch [1] to improve this in the context of slow-tier > page promotion. Here, Scanning is done by a global kernel thread which routinely > scans all the processes' address spaces and checks for accesses by reading the > PTE A bit. The hot pages thus identified are maintained in list and subsequently> are promoted to a default top-tier node. Thus, the approach pushes overhead of > scanning, NUMA hint faults and migrations off from process context. > > The topic was presented in the MM alignment session hosted by David Rientjes [2]. > The topic also finds a mention in S J Park's LSFMM proposal [3]. > > Here is the list of potential discussion points: > 1. Other improvements and enhancements to PTE A bit scanning approach. Use of > multiple kernel threads, throttling improvements, promotion policies, per-process > opt-in via prctl, virtual vs physical address based scanning, tuning hot page > detection algorithm etc. Yuanchu's MGLRU periodic aging series [1] seems quite relevant here, you might want to look at it. adding Yuanchu to Cc. By the way, do you have any reason why you'd prefer opt-in prctl over per-memcg control? [1] https://lore.kernel.org/all/20221214225123.2770216-1-yuanchu@google.com/ > 2. Possibility of maintaining single source of truth for page hotness that would > maintain hot page information from multiple sources and let other sub-systems > use that info. > > 3. Discuss how hardware provided hotness info (like AMD IBS) can further aid > promotion. Bharata had posted an RFC [4] on this a while back. > > 4. Overlap with DAMON and potential reuse. > > Links: > > [1] https://lore.kernel.org/all/20241201153818.2633616-1-raghavendra.kt@amd.com/ > [2] https://lore.kernel.org/linux-mm/20241226012833.rmmbkws4wdhzdht6@ed.ac.uk/T/ > [3] https://lore.kernel.org/lkml/Z4XUoWlU-UgRik18@gourry-fedora-PF4VCD3F/T/ > [4] https://lore.kernel.org/lkml/20230208073533.715-2-bharata@amd.com/ > >