From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BN8PR05CU002.outbound.protection.outlook.com (mail-eastus2azon11011052.outbound.protection.outlook.com [52.101.57.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 039552561A7 for ; Sun, 12 Jul 2026 07:41:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.57.52 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842089; cv=fail; b=dN3BiTgB+3MT68+rMeb8pAlN6PPZaVqhlaRPAPXB7WRVJUNKjXvsReGh4AsFJ8IV4705LZY7VBxYMvz+JG2IbAAyGYXH7bp8hQ1N+PODbvIZL3Qm8QcR6JnczwOlOgTDeyq7PLtGGLmsC64p6hauN6XCFkXGk7P7xNO5bw0k0Nk= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842089; c=relaxed/simple; bh=KPzNifbI7PmPRQvQWc6sP+itQG02VGDINUXQGQWdZPQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=XVCr0aPF1GNK2J+nB1lDlq9YhkwwIMYX95xl1OZe1kWO6jmhOtxHq6+q7H0SNArZUZQUu6HY7nGFz+MKV8ImNmxihELoITM7nRy9R8pobjZt3qa0jsnjMGJtaJJoxWt+6ojp/qQabJiSU76nWzXxoRV+FDeWR0V6wPxmTjCexxI= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=O6//IY6q; arc=fail smtp.client-ip=52.101.57.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="O6//IY6q" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uewvZedIh8p6mNJyNNJpc7SthXJZ6Z0aBopGvRnzPgDLXBgn+2/LxEy/ptSJFIwPkzxKuczV9i/5Pb/dOT8uDm7mFRQjqbLtgw4rCSxsov2bWRTd1m/Lr4J0dGDn9y5da3AwL/Lwd9UarLELj0uKEIXW/bVVbI9FIl+MJIzK/IroxcH6Mld7ttW0yH+/gPe2WpKtixii5UVfb05q0atBOK/vHo7a5GKuxN1zdZaOxIk6aeWRkedfRheouxi2nYr7N6vyoMDQ2J2146gmeoCZlplTA/RXTKtxaiyFaO0m0L6WHPzfYvl2GhXcPgznzvhv8etEn6WBBa4E4GGmDZGkFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rnMC8aGBoA4+MFdtra1b+Fcp5uvcOnfcFWmUVebdKcw=; b=QEzXQhJp6ubB04Uk2yeT/bRsOpizB1y4mrxJId8pVGqu/r5ccisO0SMhX0kq1q1T7rtGIDUmG6za0OkT1N4QtnGfDD2DEG4WVQ4nKbLv2lBuEB6piNr7kCnmYlHNBx1iyaHCqfCTw+Zepgje+4pzSPIpaY3sjOuxoRjzvYpVaOOT3UGLVrmMWoQLjylVCFFQzKJc+NKI7TAci6J0erswpoeBcPPlSWPzZRc3d9Mnz6HzkW0t0Ps7NlbkJDoWwx9Et+t5fsR81dA3YlIwdIh7FNa76MIAyVRPbacAjTc+WhEB9nllZ6CNbIp3PNZDUNyROLOxM2wmVE+ur2jTcyUXBw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rnMC8aGBoA4+MFdtra1b+Fcp5uvcOnfcFWmUVebdKcw=; b=O6//IY6qs1wfgVlwcRiDts79vQfLPJ6skd4CzPnq0B7/3zEVxcp7w2oKE8CYBxG9ccPKasja3LruPHOHA/gUD7xbMBnOqCPo4g4SPrmcLc0oQ+B0qfx29AdS8qNSD409aDkxqoGZEpY/yjwzhAUSJt5SkslAO7czbaEmEiTLG2M= Received: from MN0PR04CA0011.namprd04.prod.outlook.com (2603:10b6:208:52d::7) by CY8PR12MB7193.namprd12.prod.outlook.com (2603:10b6:930:5b::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.22; Sun, 12 Jul 2026 07:41:22 +0000 Received: from BL02EPF0001A108.namprd05.prod.outlook.com (2603:10b6:208:52d:cafe::75) by MN0PR04CA0011.outlook.office365.com (2603:10b6:208:52d::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.202.18 via Frontend Transport; Sun, 12 Jul 2026 07:41:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF0001A108.mail.protection.outlook.com (10.167.241.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.9 via Frontend Transport; Sun, 12 Jul 2026 07:41:21 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Sun, 12 Jul 2026 02:41:13 -0500 From: Shivank Garg Date: Sun, 12 Jul 2026 07:38:50 +0000 Subject: [PATCH 4/6] mm/migrate: batch the restore-side migration rmap walk Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-ID: <20260712-migrate-rmap-batch-v1-4-872a734431d1@amd.com> References: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> In-Reply-To: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1783842044; l=4278; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=KPzNifbI7PmPRQvQWc6sP+itQG02VGDINUXQGQWdZPQ=; b=eXfk3cc3LyZQuBa6jNPVkKsajnJuqFzigyboOEqF8gJZP/Ihn8sotuuB4RsJ4NiJ0AFVLv/Ey CJ9K47gCWx3DHVNDTAJnD+fGIYEtBzWYfjzLFURsbXAO6hkoUG4lxsh X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A108:EE_|CY8PR12MB7193:EE_ X-MS-Office365-Filtering-Correlation-Id: a964e2bc-b374-4aaf-35df-08dedfe8f7f7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|82310400026|36860700016|23010399003|22082099003|18002099003|56012099006|11063799006|921020; X-Microsoft-Antispam-Message-Info: oOp9+EMJSIt9hYyBnWzErkQ1IzPsQHS3p+ydwSZTpZfSIAWrfkIutyO0ONFDP85u5dhyWmP94AXuEsw1m6luMv9IhMSTVl+Wg6ZUpboVN/zk+l+TgQ4JbfRHpJ9jJ+c53siA7T7LpawWuSQLlteRzv4LhMWc3pkchnPTbIIymcwIzI9gMN8D0oi/O3xRBjYfaCNVya/jqYpxT2DVwOg18uUF5mngZVfFKSisQglk4GHZA7ykvEQ/W1ml5EOR3XDTitlfaBFRSCaOw9P+IWDXvePGr/4c5/thNPwzNJqirlK5lOdgCruFrDht+GxQVlaUUQXQKcGR5oX0v8yKn2yWLsanugYTa9tt8aHK74QyupZm6Z8llDpeIi8p3CYirL0cN+4AqHlOvGW9dTn+6L+qtgS7hr8VcKqHvjm429YIGezboE/EbHQaZYInKZuyEhDS3x6HFp0HIsZ0wVcCCRhLdkTkqZHFn2D2k/1MMEJ2avHPhYfDV2ysYAJAprbCYlZMaT3plsGoKiLmceLqy5G43xlyEsCpNMRxtcyOrKGmhnf4g3DcC1AGeVDlcemDfc9Q7dsIIemYeBLGbBLiJznjGDjLad2Gu59o7mZkZx6BC7T8LEvBJu/NFrMungXHdkD9dg6JC9nSq0LXEbT3cfAYoSMthxQaWNJsh6ubHT7AtlrjP41ilvSY8lytYO6HxdeUV+mLg1F+g2HkIZmrBcAdP6yLyFDPRgieVBq0Jj0AVoGjFDXC2IX5/COaBK8P4k7f X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(82310400026)(36860700016)(23010399003)(22082099003)(18002099003)(56012099006)(11063799006)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: J8CZ9mRqfHETFMt8MwOxiBYMvgWWjFh4OcYmOVfKQ/bDsNce2n2WupoREONXauZrugy1JXh9QgmrMs2Yc7v7ETJRwN43P2W9owRxpyZOKp6dAp8J+ZdAKfP/fkGO8V4Il71ZSuVmyrHUEvzEsyhCUAsJ+g1D7Zm8D5P/s6Fkwf8bnh8w/EMzRG2aC95M968nXNAOy05X4d2wsQU+Pun9jB1/sVY278mEWGbwupxd45aPXODsCEjX2UIyEk2JpnahmWzV/XNNBB/uSP8YPs2hPs6IxEaGsIeZG+lzUidzTuoXaH97OhFmJiriVMPT1iF4ynP245olcVd9YLcRVzk566lIjj3fRnir6vGTnFIj8eahnAx1n+QJ/5m6FevA/o5m+vVV1wYF3eeVcnBr9hcy6CzzDjPtlRDvAe2SVNhOct3zpEP8LSbaCw5SwF3fZfuo X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2026 07:41:21.8419 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a964e2bc-b374-4aaf-35df-08dedfe8f7f7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A108.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7193 remove_migration_pte() restores migration entries one PTE at a time. For a PTE-mapped large folio, it repeats calls to building the PTE, add rmap, set_pte_at() and page_vma_mapped_walk() per base page (256 times for a 1M folio). Add migration_pte_batch() to detect batch of contiguous migration entries that map consecutive subpages, and have identical PTE bits. While at it, use folio_is_device_private() instead of is_device_private_page(). Signed-off-by: Shivank Garg --- mm/migrate.c | 49 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 7 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 8a098402b8c9..6d05d3f41095 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -345,6 +345,28 @@ struct rmap_walk_arg { bool map_unused_to_zeropage; }; +/* + * Detect a batch of contiguous migration entries: consecutive (non-present) PTEs + * containing migration entries with consecutive offsets and matching pte bits. + */ +static unsigned int migration_pte_batch(struct page_vma_mapped_walk *pvmw, + struct folio *folio, pte_t first_pte, unsigned long start_idx) +{ + struct vm_area_struct *vma = pvmw->vma; + unsigned long end_addr = pmd_addr_end(pvmw->address, vma->vm_end); + unsigned int folio_nr = folio_nr_pages(folio); + unsigned int max_nr; + + VM_WARN_ON(!softleaf_is_migration(softleaf_from_pte(first_pte))); + + /* Bound by VMA / PMD end and by the remaining subpages of the folio. */ + max_nr = min((end_addr - pvmw->address) >> PAGE_SHIFT, folio_nr - start_idx); + if (max_nr <= 1) + return 1; + + return softleaf_pte_batch(pvmw->pte, max_nr, first_pte); +} + static pte_t migration_softleaf_entry_to_pte(struct folio *folio, struct page *new, softleaf_t entry, pte_t old_pte, struct vm_area_struct *vma, rmap_t *rmap_flags) @@ -435,6 +457,7 @@ static bool remove_migration_pte(struct folio *folio, struct vm_area_struct *vma, unsigned long addr, void *arg) { struct rmap_walk_arg *rmap_walk_arg = arg; + bool is_devpriv = folio_is_device_private(folio); DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION); while (page_vma_mapped_walk(&pvmw)) { @@ -444,6 +467,7 @@ static bool remove_migration_pte(struct folio *folio, softleaf_t entry; struct page *new; unsigned long idx = 0; + unsigned int nr = 1; /* pgoff is invalid for ksm pages, but they are never large */ if (folio_test_large(folio)) @@ -465,11 +489,18 @@ static bool remove_migration_pte(struct folio *folio, entry = softleaf_from_pte(old_pte); - folio_get(folio); + /* + * Try to restore nr>1 contiguous PTEs in one shot. Falls back + * to original per-PTE path (nr=1) if batching is not possible. + */ + if (!rmap_walk_arg->map_unused_to_zeropage && likely(!is_devpriv)) + nr = migration_pte_batch(&pvmw, folio, old_pte, idx); + + folio_ref_add(folio, nr); pte = migration_softleaf_entry_to_pte(folio, new, entry, old_pte, vma, &rmap_flags); - if (unlikely(is_device_private_page(new))) { + if (unlikely(is_devpriv)) { if (pte_write(pte)) entry = make_writable_device_private_entry( page_to_pfn(new)); @@ -484,11 +515,11 @@ static bool remove_migration_pte(struct folio *folio, } if (folio_test_anon(folio)) - folio_add_anon_rmap_pte(folio, new, vma, - pvmw.address, rmap_flags); + folio_add_anon_rmap_ptes(folio, new, nr, vma, + pvmw.address, rmap_flags); else - folio_add_file_rmap_pte(folio, new, vma); - set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); + folio_add_file_rmap_ptes(folio, new, nr, vma); + set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); if (READ_ONCE(vma->vm_flags) & VM_LOCKED) mlock_drain_local(); @@ -496,7 +527,11 @@ static bool remove_migration_pte(struct folio *folio, compound_order(new)); /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, pvmw.address, pvmw.pte); + update_mmu_cache_range(NULL, vma, pvmw.address, pvmw.pte, nr); + + /* Skip the batched PTEs */ + pvmw.pte += nr - 1; + pvmw.address += (nr - 1) * PAGE_SIZE; } return true; -- 2.43.0