From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6DA2382F02 for ; Thu, 25 Jun 2026 01:51:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782352277; cv=none; b=r1J2w1uhjQgH7DkofFFMeSTj2NMT1ThyDC9LhDti5E98pRIfzKqhbFAR9m8NTnC4Jx8sUSYRDS8klfSD0d5o9q/alfDZMwSlFbY7RL3koiatC2pxcuUxkNEuUrQKtyQ8EiYhBVx3e45pCygbcoAH3febdi8mAU7UP/LGn4EAZE4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782352277; c=relaxed/simple; bh=4qkvjSSY36Rz9WgCbdpz06FfFgv6UxDWwbtxrXW9wXM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cZtNJsUwCqIrdVaNaeutAJbXKHVoCE4yGBK01HDGTJQvxvSNksmSeO4w4KH8f8BcPpA9YkJxAFVf/mgcqvPt5shT5XwsiKgrbYbasa5c8bkN3S9DnAjjPe2VPCAh/upUDPk/0L2NRrDG7QGeSUEtG8rM7fvqtmz/rpyw7CI2qwU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=IX33nwhF; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="IX33nwhF" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=xmq18u+aZ3R3zjSmFpi6jjGs08uIHYwgJWkswlXPCNo=; b=IX33nwhFWy4iz09trcl5hDMnfl /8BN8QKka2zqaH0dUWN5YKRmgbLBVdI6BuVNY6gdkluftzW9NZVmQKivNDDHM6mNU5C7Is8aKBQrv KBgmC5lLxAPk8dvekfjDWRTF2gQcFzRgstjxpM5tuz0vbaZTpc2GdiYRNeKeG3cFxxFWK1MiTTzEA 2VzGMn5fIMMAmi+mGFb1jNgT0Y7KCLd1r7PcaxEx/X/SLTwMG9msxIgGbo745gRa44YWii/JJr0Fu IRZOd34fYRk6ynHbCJuUWnqJUR52CYvPbVlwOahlN4OiL3QjQgp+u/2mzBLaPZJYJIowULK9hqjFU TtNc6AQw==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wcZF7-0000000043x-42H9; Wed, 24 Jun 2026 21:51:01 -0400 From: Rik van Riel To: linux-kernel@vger.kernel.org Cc: Rik van Riel , x86@kernel.org, linux-mm@kvack.org, "Thomas Gleixner" , "Ingo Molnar" , "Dmitry Ilvokhin" , "Borislav Petkov" , "Dave Hansen" , "Andrew Morton" , "David Hildenbrand" , "Lorenzo Stoakes" , "Liam R. Howlett" , "Vlastimil Babka" , "Suren Baghdasaryan" , kernel-team@meta.com Subject: [PATCH 2/3] mm/pagewalk: let folio_walk_start() run under the per-VMA lock Date: Wed, 24 Jun 2026 21:50:52 -0400 Message-ID: <20260625015053.2445008-3-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260625015053.2445008-1-riel@surriel.com> References: <20260625015053.2445008-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit folio_walk_start() asserts the mmap lock is held. For callers that only need to read a single, already-present page, the mmap lock is a heavy and often badly contended hammer. Such a caller can instead hold the per-VMA lock, which keeps the VMA itself stable. The per-VMA lock does not, however, keep the page tables walked below that VMA from being freed. A concurrent munmap() or THP collapse of an adjacent region in the same mm can free a shared upper-level table, and THP collapse (collapse_huge_page() -> retract_page_tables()) frees page tables of VMAs whose lock it does not hold. Page table freeing synchronizes against lockless walkers the way gup_fast relies on: tlb_remove_table_sync_one() sends an IPI and waits for every CPU to enable interrupts, so a walker that keeps interrupts disabled across the walk cannot be observing a table that is about to be freed. rcu_read_lock() is not sufficient -- it does not block that IPI -- so the caller must keep interrupts disabled, not merely hold an RCU read-side critical section. Add an FW_VMA_LOCKED flag. When passed, folio_walk_start() asserts the per-VMA lock and that interrupts are disabled, instead of asserting the mmap lock; it requires CONFIG_MMU_GATHER_RCU_TABLE_FREE and refuses hugetlb VMAs (PMD sharing maps page tables this VMA's lock does not cover). The caller must keep interrupts disabled until folio_walk_end(). No existing caller passes FW_VMA_LOCKED, so behaviour is unchanged. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Rik van Riel --- include/linux/pagewalk.h | 7 +++++++ mm/pagewalk.c | 29 +++++++++++++++++++++++++++-- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index b41d7265c01b..d0387470d732 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -150,6 +150,13 @@ typedef int __bitwise folio_walk_flags_t; /* Walk shared zeropages (small + huge) as well. */ #define FW_ZEROPAGE ((__force folio_walk_flags_t)BIT(0)) +/* + * The caller holds the per-VMA lock instead of the mmap lock, with interrupts + * disabled across the walk (until folio_walk_end()) to serialize against page + * table freeing, the same way gup_fast does. Only valid with RCU-freed page + * tables (CONFIG_MMU_GATHER_RCU_TABLE_FREE) and not for hugetlb. + */ +#define FW_VMA_LOCKED ((__force folio_walk_flags_t)BIT(1)) enum folio_walk_level { FW_LEVEL_PTE, diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 3ae2586ff45b..ab1e81983cb8 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -890,7 +890,10 @@ int walk_page_mapping(struct address_space *mapping, pgoff_t first_index, * huge_ptep_set_*, ...). Note that the page table entry stored in @fw might * not correspond to the first physical entry of a logical hugetlb entry. * - * The mmap lock must be held in read mode. + * The mmap lock must be held in read mode. Alternatively, if @FW_VMA_LOCKED is + * passed, the VMA's per-VMA lock must be held and interrupts must be disabled + * across the walk and until folio_walk_end() (only supported with RCU-freed page + * tables, i.e. CONFIG_MMU_GATHER_RCU_TABLE_FREE, and not for hugetlb). * * Return: folio pointer on success, otherwise NULL. */ @@ -908,7 +911,29 @@ struct folio *folio_walk_start(struct folio_walk *fw, pgd_t *pgdp; p4d_t *p4dp; - mmap_assert_locked(vma->vm_mm); + if (flags & FW_VMA_LOCKED) { + /* + * Lockless walk under the per-VMA lock instead of the mmap + * lock. The VMA lock keeps the VMA stable, but the page tables + * walked below it can still be freed concurrently: a munmap() or + * THP collapse of an adjacent region in the same mm can free a + * shared upper-level table, and collapse_huge_page() -> + * retract_page_tables() frees page tables of VMAs whose lock it + * does not hold. Page table freeing serializes against lockless + * walkers via tlb_remove_table_sync_one(), which IPIs and waits + * for every CPU to enable interrupts; an RCU read-side critical + * section does not block that IPI, so the caller must keep + * interrupts disabled across the whole walk, like gup_fast. + * Hugetlb (PMD sharing) maps page tables not covered by this + * VMA's lock and is not supported. + */ + VM_WARN_ON_ONCE(!IS_ENABLED(CONFIG_MMU_GATHER_RCU_TABLE_FREE)); + VM_WARN_ON_ONCE(is_vm_hugetlb_page(vma)); + lockdep_assert_irqs_disabled(); + vma_assert_locked(vma); + } else { + mmap_assert_locked(vma->vm_mm); + } vma_pgtable_walk_begin(vma); if (WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end)) -- 2.53.0-Meta