From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from outbound.mr.icloud.com (mr-2001j-snip4-4.eps.apple.com [57.103.68.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 05F4C22259F for ; Sun, 31 May 2026 04:27:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=57.103.68.117 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780201664; cv=none; b=tq66ZQ7Bk/u6FMIpk9CqNYjFBMOzc+cLzJx4wBwd8jVVf6c1LqK0VITliGT+9kOWYj+3STGqFKQsE/ccjL1nTI9lCBUtz3NYY9mxYY9jJ44ORFecXrBN7alp/PEJlw4WTi+YJNSNvy9uw+30R9O0rigicFkaKsDcS5pmnhNuTT0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780201664; c=relaxed/simple; bh=ITwVpGnFllNUNNmdzsS7+Wks5AE0w/8QyFWXf1B+9zk=; h=From:Subject:Date:Message-Id:MIME-Version:Content-Type:To:Cc; b=YsMPEYfamcErkdBe52ecQboRsBq67xJK0SvVPrYeCNs/JUZOmw8gSqFUUyq7cNOZHVy7iC01h10sC9Gqkdx5Hqt0IxzFt7t/wPquYMittIUyo0LYjuC8vJSqkG98b7wyBask6T0Y9VblmaWWhx9GQWTxk1qwRkeeM/Gq2BbGPWs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=icloud.com; spf=pass smtp.mailfrom=icloud.com; dkim=pass (2048-bit key) header.d=icloud.com header.i=@icloud.com header.b=DLpC1l0q; arc=none smtp.client-ip=57.103.68.117 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=icloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=icloud.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=icloud.com header.i=@icloud.com header.b="DLpC1l0q" Received: from outbound.mr.icloud.com (unknown [127.0.0.2]) by p00-icloudmta-asmtp-us-west-2a-100-percent-4 (Postfix) with ESMTPS id 5ADFD1800162; Sun, 31 May 2026 04:27:36 +0000 (UTC) X-ICL-Out-Info: HUtFAUMEWwJACUgBTUQeDx5WFlZNRAJCTQhJB0MFXwteDUAdVAVLVxQEFEYGVg1dE0wLcwRUB10FXVZQAlpLVBQEEVABWB5WXloXXk1FCA9CAVhbCFsEDx9MDFECQgVWXlQKHQRUB10FXVZQAlpLQgRLRWhcBVwcQBdIHV9qS1YUBBFQAVgeVl5aF15NWgJWTQVKA18BWwdDCFVHBUc0UR9VFFIdRA5tGFAWR0BBWh9AFEAFWwRYCxNdTFBfVitGFVcbVgNDRVEfVEYTGU4bV01QG18CQg8= Dkim-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1780201662; x=1782793662; bh=E8WdjNOls66S20g/c+qMGPBkpLZguCRKlXTYItbtX2Y=; h=From:Subject:Date:Message-Id:MIME-Version:Content-Type:To:x-icloud-hme; b=DLpC1l0qZo6SgFFF2QwSe0K+r9JH5iJpRYCvMt9qxkHZrTStE3UwR1IvKmTfBVVyCcxYaD3RU+i2rRUgBquxMrw/4r/Cfuzjm2uvmYnpuOtKlTDjVRkrOgE61Veru/rE61qxl0fzHWQW+K9ttlWFoSP+PBjNLDZEQptaM2qvXgW7wxNMB8EmZ7wQUCOF+txXQ/ER/8Ok3ljuwWSFtIy5vmpC70nQlOPSt/QxJuCdjYnlqOWprFXjKcz0nvbwXiwiIUL6aD+D83ZGW4wdaTx6NhjT6RjlFbaioCgQ1a9FvI+4HQ20MQR8KKpvHWgI/wHVcR3eS1YZt/W64P5yj4RDkg== Received: from [127.0.0.1] (unknown [17.57.152.38]) by p00-icloudmta-asmtp-us-west-2a-100-percent-4 (Postfix) with ESMTPSA id B23651800160; Sun, 31 May 2026 04:27:23 +0000 (UTC) From: Luka Bai Subject: [PATCH 0/5] mm/khugepaged: add collapse hint machanism for khugepaged and use in mglru Date: Sun, 31 May 2026 12:27:16 +0800 Message-Id: <20260531-thp_collapse_hint-v1-0-866339cd4c2a@tencent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-B4-Tracking: v=1; b=H4sIAAAAAAAC/yXMQQqDMBBA0avIrA3E2CrxKiKicTQjEkMmloJ49 8a6fIv/T2AMhAxNdkLADzHtLqHIMzB2cAsKmpJBSVXJdylFtL43+7YNnrG35KJAo9U46VdZ6xp S5wPO9P0/2+4xH+OKJt4juK4f7PekxnUAAAA= X-Change-ID: 20260530-thp_collapse_hint-ec92bd943797 To: linux-mm@kvack.org Cc: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Rik van Riel , Harry Yoo , Jann Horn , Johannes Weiner , linux-kernel@vger.kernel.org, Luka Bai X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1780201643; l=7268; i=lukabai@tencent.com; s=20260501; h=from:subject:message-id; bh=ITwVpGnFllNUNNmdzsS7+Wks5AE0w/8QyFWXf1B+9zk=; b=jbjCBcflz46h+SveS0b5ouH4UBW07H9+M17/6gIJ+jrmfhyoagdnzld1oX8o8otqY8BbzLMJ+ qTIAsOTQiTwBi98htoYg7YbLu8SlMOuEmEybmX58UKirjoktbH5ZPTk X-Developer-Key: i=lukabai@tencent.com; a=ed25519; pk=KeaVteSWd00GIAjFyWZnuFsKAKixjga1ZkLMcI66nPM= X-Proofpoint-GUID: DQtyHUDcmmKbaGmZ9Xfsh5uYLG7hUwB2 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNTMxMDA0NiBTYWx0ZWRfX+DsMixvDzfQd oEwGo5RWGtOad6yOXbT/BmZoF78yrfb15wLSeVfisTDcqMcrOvAsx8vERsak/NY/U18LyHYDABU osX5qLOU44T3dmcL4IygO/EXj1ImRK7jStgixkikVHBz316mXDGvAOjMipi/4QPvSNazscN+42A 1Q2dqc/zCky1cbYRA93np7ZLUXUcp2Jw/TWJsPpXpCRZxvqkLew0guIqFxpamAHAFl2X7MQ7zeb WGg0Cy2qusYHEnBRkVfY5N74NrjvgY6u+lDhdea2Ok86j5DXBPTIulgKR/7HHrbZYggv1VTYhog 3xyyASCao9DgORCCbDToWYOROCJ392niIsof40HpCKOyMTZmocs0datbb5928o= X-Proofpoint-ORIG-GUID: DQtyHUDcmmKbaGmZ9Xfsh5uYLG7hUwB2 X-Authority-Info-Out: v=2.4 cv=H57WAuYi c=1 sm=1 tr=0 ts=6a1bb8bc cx=c_apl:c_pps:t_out a=9OgfyREA4BUYbbCgc0Y0oA==:117 a=9OgfyREA4BUYbbCgc0Y0oA==:17 a=IkcTkHD0fZMA:10 a=NGcC8JguVDcA:10 a=x7bEGLp0ZPQA:10 a=UaoJkeuwEpQA:10 a=VkNPw1HP01LnGYTKEx00:22 a=GvQkQWPkAAAA:8 a=bbyK-OfsBG3tpf3ilmAA:9 a=QEXdDO2ut3YA:10 Khugepaged is a background daemon for collapsing feasible pages together into a transparent hugepage in all sorts of orders up to PMD_ORDER. However, it doesn't have any preference in its collapsing and just iterate through all the qualified mm_struct, and scan their page tables from the beginning to the end. It is quite inefficient especially for large address spaces considering how slow the khugepaged can be, and may waste many hugepage resources collapsing memory areas that are seldomly accessed. We would like to give khugepaged some preference hints when we found certain areas are good condidates for collapsing. For example, if some memory areas are frequently accessed, then we know that it's valuable to merge them into a bigger folio since it will reduce many tlb misses. For example, MGLRU has walk_mm() and lru_gen_look_around() that are used to scan frequently accessed areas to save some works on rmap walking and generation elevation. By the same time, they are able to find those hot memory areas, it should be valuable to merge these areas into folios. MADV_COLLAPSE can be used, but that will cost too much time and will harm the performance of reclaimation and slow down the process that may enter the slow path of memory allocation. So the better choice shoule be to tell khugepaged to asynchronously do it. We add a khugepaged collapse hint framework in this patchset. The caller can call khugepaged_add_collapse_hint() to add hints for khugepaged to make it prioritize collapsing these specific address we found before doing Round-Robin scanning. Each mm_slot which belongs to a mm_struct in the previous mm_slots_hash is now a khugepaged_mm_slot, it comprises the old mm_slot struct and a number of NR_KHUGEPAGED_PRIORITY_LEVEL struct khugepaged_collapse_requests. The request struct for each mm_struct will be put in the global struct khugepaged_priority_queue with respect to its priority when __khugepaged_enter() is called on this mm (we give each mm request structs for hint dispersion and balancing across all the mm_structs that will be added in the future patches), and all the hints will be put in these request structs. Each hint will have the target address and the target vma struct. An example of the framework is like below: global collapse hints queues: prio 0 ------()----------------------------------()--------------- mm_slot0(process A) mm_slot1(process B) | | hint0---hint1---hint2---hint3 hint4---hint5---hint6 prio 1 ------()----------------------------------()--------------- mm_slot0(process A) mm_slot1(process B) | | ------- hint7---hint8 The khugepaged will try to scan queues from highest priority (which is prio 0 in the graph above) to the lowest priority (which is prio 1 in the graph), then go through the list, and check out all the struct khugepaged_mm_slot (which are the mm_slot0 and mm_slot1 in the graph above), so it will start from mm_slot0 in queue of priority 0. Then khugepaged will scan all the hints listed in the slot (hint0 ~ hint3 in the above graph). After handling one hint (no mater success or fail on collapsing), the hint will be deleted. If one khugepaged_mm_slot doesn't have any hints in it, khugepaged will skip it and scan the next mm_slot in the same priority; if there is no hint in the queue of prio 0 anymore, khugepaged will scan the ones of prio 1; if there is no hint in any prio queues, it will fallback to do Round-Robin scanning like before. khugepaged_add_collapse_hint() is for adding hints, and it only gets called by walk_mm() and lru_gen_look_around() right now. In the future we may call it in more scenorios when we found hot memory areas. For example: in damon. We tested the performance by using valkey-server (based on redis) together with memtier_benchmark to simulate a gauss distribution on the get/set operations on a 160G, 64core x86 VM. The dataset is about 3G. After preloading db, the testing parameter was like below: memtier_benchmark -s 127.0.0.1 -p 6379 \ --ratio=1:1 \ --key-pattern=G:G \ --key-minimum=1 --key-maximum=3000000 \ --key-median=2000000 \ --key-stddev=150000 \ -d 1024 \ -t 1 -c 10 \ -n 2500000 \ --pipeline=32 \ --hide-histogram Since we wanted to see the influence of khugepaged collapse hints on the reduction of tlb misses, we made khugepaged do scanning every 1 second, and used the userspace interface to do walk_mm() for the cgroup which valkey-server was set into every 2 seconds. We made sure the server was all 4k pages before we run test, and only khugepaged could collapse them into large folios. We enable the anonymous THP of order 9, which is pmd size in most setup. We used perf stat to monitor the tlb misses statistics. After repeated tests, we could see dTLB-load-misses with a 13.50% reduction, and saw dTLB-store-misses with a 5% reduction compared to the setup without any collapse hint. The final throughput for the memtier_benchmark was about 2% to 5% improvement on average, which was not that obvious compared to the tlb miss reduction. We believed that was because there were too many factors to influence the final result of a random redis test, so the influence of tlb misses to the final throughput were compromised by other factors. Patch Details: ======== * Patch 1 is to add the basic khugepaged hint framework like we introduced above. Details can be seen in the commit itself and the comments in the codes. * Patch 2 is to add a slab_cache for khugepaged_collapse_hint which can improve the performance of allocating and freeing the hints. * Patch 3 is to add a deduplication machanism for the hints so that we will not add a hint that points to a repeated address. * Patch 4 is to add the accounting for successful collapses initiated by hint or non-hint. * Patch 5 is to add the collapse hint in lru_gen_look_around() and walk_mm() of mglru. Thanks for reading. Comments and suggestions are very welcome! Signed-off-by: Luka Bai --- Luka Bai (5): mm/khugepaged: add framework for khugepaged collapse hint mm/khugepaged: use slab cache instead of normal kmalloc mm/khugepaged: add deduplication when adding new collapse hint mm/khugepaged: add accounting for successful hint or non-hint collapse mm/khugepaged: add khugepaged collapse hint in mglru reference checking include/linux/huge_mm.h | 2 + include/linux/khugepaged.h | 20 ++ include/linux/mmzone.h | 17 +- mm/huge_memory.c | 4 + mm/khugepaged.c | 460 ++++++++++++++++++++++++++++++++++++++++++++- mm/rmap.c | 27 ++- mm/vmscan.c | 33 +++- 7 files changed, 549 insertions(+), 14 deletions(-) --- base-commit: e1af79f3291a268adf4e149e1faba3052743e898 change-id: 20260530-thp_collapse_hint-ec92bd943797 Best regards, -- Luka Bai