From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pz2-f1.google.com (mail-pz2-f1.google.com [74.125.228.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79CE529AB1A for ; Tue, 7 Jul 2026 13:00:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.228.1 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783429235; cv=none; b=aFx2EXfxrMUSWkwx9vrmzomFFU62KR7lFXPMSaPaLu3490AXWmC+IWGNZ+VDeQiHq+WjKiJFTiO1sISR1wcPbclEtB+MtRZoqr3lJOADXUn45t9YrhUfJK52Bma1eE+gijW8XoRwBcS0plnLoRv/4YMjUWY5unPUX2YcydPln18= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783429235; c=relaxed/simple; bh=RvXzUh+Z5rLxburgzcvo+g9ZGozfoPunlEYEQZcFbHU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Re6ho0W1GzQIFBAVS3beeehX+wUg1Zjbc4NLPW70R/OH62QOO8gg9T1A/J/wqr7KeEWyslwKzknK8tlb+RP3NYof1fA2kOsi/uj/Bwp8HVEDwoI+Yx+uhL5Kx9Kw0hIhd68xDFUrc9IRVMRmjt1txR+wNOCOvdge714bpZ7fmhI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=R/MXcVGo; arc=none smtp.client-ip=74.125.228.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="R/MXcVGo" Received: by mail-pz2-f1.google.com with SMTP id 41be03b00d2f7-c999d839661so796996a12.0 for ; Tue, 07 Jul 2026 06:00:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783429232; x=1784034032; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5W5DUkb1JUBfDdw5NFLWUB/50kxa7q0vorvfnztH0I4=; b=R/MXcVGovvHJ1FQtgUZyl77G4rx3dRgmO9eqWDl/DtoFC8heKGDf98n7fhQHF2osIi HcqsTdfIlZg3AmgMW8ou8/IgzOXNvkt8ewSyK1k35MeTxNagjDiSpaqwcUCOVgtedY0E CxF9fFw3FS00ffbnoHKe1c60FIykgWhoHw/ULe7pHGNpOABTO6585dBeyhsMymYJfM8S LL37JlaWYmV7+rjY54AoTuvKqojoP6BEtBptUQBiUp42jIclQPFoOgy2o8AMPBtTYjCB j2RB2XlrxUbqed93AICy33rqn0/bpJ0oVRiSMNKN1lpjaPQRe7KoJuiVOMv8b+ne/vF5 k0aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783429232; x=1784034032; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=5W5DUkb1JUBfDdw5NFLWUB/50kxa7q0vorvfnztH0I4=; b=N4zT49Kdc4fdrEGAKBNX3hqXi6NhdS0xUjh5rhT5hgtr7WRSIL0F501TNLX9FX7aT0 oVesIej6RjIB2BieVAMSIppx6LbtR+QVnKITC5EYQ0mlncbXbTeBJf6Ip8cBhfHvx4up PcZDtJAxv0Da5+FGWGwsL6SokBMj34OUDlbkD9UMVHJ0aeOa9W6aq9KBA93XxOI7/473 OaWFLpqwOAQ3J8vh6U4Oapr5eWak96gchqzJmeGfe2ksyJjtzXoRHh4axz/tQgp/63TP 2qC0yjP7Zoml/y9TOpBI9uu6Y8snviyi2AHjts25fQFBgQElvCn7ioVFLRChCEk3Q/Zn DUbQ== X-Forwarded-Encrypted: i=1; AHgh+Rr6mXEbUlOr7U3isxKAurURUCEe4ZRaJY7dsBwq5BIfzlJKcy+AMVOr8n87p0qTql6gjjKJtOKSMbgXGd4=@vger.kernel.org X-Gm-Message-State: AOJu0YwJnhClkbQgtQN9JyRivsF4i6MYIHs9AWngevMZRpcJQk7lOum9 2QCDON28l0XCqgjtsuu1h6zsb7iKFEfZ9ydfwM3nfu8JhOzcAFLwuvLm X-Gm-Gg: AfdE7cnnuHmAboZEeogxuYEikPqTTeb67qYSHnyJGdCfIPgQ2pF45TzKVu+IUF9EE3v p3raIdvB/t9AhK9exmmSACRcj6CtGJhgN7kI+WEeoG0SBlSiqGILVQnz+HH7EHKRH6XQQ8+Jn/1 Q5hl0/nubV+9Wzmp/RWxYxSplXJxhSKTL5HU7FSApoM18nYa0S6kP/PirelV1YW7U4INlYGdEwU Moj4HBRnBn3XkPewtJBP6+EDHoo1dY820ebtFhz/b0Ui/avhjbQh+BBZQASHMqy3dHLsh5Lh8+d Q1NmsJ1THyVPoTW9VYohoj1jaGAXI8f0JgLf39mVRoZHb5qyRXcI3KCnvV1faRnT2e99xxX96q/ JZkRDAPMIBkYSd83kUC+QWzcxJYpF1JGrn7uRQialesCUUu1jANvcN94kv/+6pDZ76Bs3kb1M6L lEgC/sWgdQa7qlxG8Lu+kW1PXprw== X-Received: by 2002:a05:6a00:1819:b0:847:8b1d:22fe with SMTP id d2e1a72fcca58-84826df12a7mr4541501b3a.50.1783429231350; Tue, 07 Jul 2026 06:00:31 -0700 (PDT) Received: from intel.company.local ([210.184.73.204]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-847f6b5e566sm5415983b3a.3.2026.07.07.06.00.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jul 2026 06:00:30 -0700 (PDT) From: Wandun Chen To: vbabka@kernel.org, david@kernel.org, rostedt@goodmis.org, mhiramat@kernel.org, Alexander.Krabler@kuka.com, hughd@google.com, fvdl@google.com, bigeasy@linutronix.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev Cc: akpm@linux-foundation.org, surenb@google.com, mhocko@suse.com, jackmanb@google.com, hannes@cmpxchg.org, ziy@nvidia.com, ljs@kernel.org, riel@surriel.com, liam@infradead.org, harry@kernel.org, jannh@google.com, lance.yang@linux.dev, mathieu.desnoyers@efficios.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, pfalcato@suse.de Subject: [PATCH v2 4/4] mm/mlock: migrate folios out of CMA when mlocking a range Date: Tue, 7 Jul 2026 20:59:25 +0800 Message-ID: <20260707125925.3725177-5-chenwandun1@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260707125925.3725177-1-chenwandun1@gmail.com> References: <20260707125925.3725177-1-chenwandun1@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Wandun Chen The region covered by mlock[all] may contain CMA pages. cma_alloc installs migration entries in the page table, if a memory access occurs at this point, it must wait for the migration to complete, which may cause latency spikes on the RT kernels. Try to move the migration cost into the mlock[all] caller, which is typically a setup path. So reduce the chance of latency spikes on RT kernels by migrating the currently mapped CMA pages out of CMA region. Suggested-by: Frank van der Linden Signed-off-by: Wandun Chen Link: https://lore.kernel.org/all/CAPTztWZpnX1j8-7yeppVUsxE=O9hbVeqricDjZt8_pnN7a-kBQ@mail.gmail.com/#t --- mm/mlock.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 118 insertions(+), 1 deletion(-) diff --git a/mm/mlock.c b/mm/mlock.c index ac65de40b22b..f56c685533f5 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include "internal.h" @@ -428,6 +429,119 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, return 0; } +#ifdef CONFIG_CMA +static int mlock_collect_migratable_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end, struct mm_walk *walk) +{ + struct vm_area_struct *vma = walk->vma; + struct list_head *folio_list = walk->private; + spinlock_t *ptl; + pte_t *start_pte, *pte; + pte_t ptent; + struct folio *folio; + unsigned int step = 1; + + if (!(vma->vm_flags & VM_LOCKED)) + return 0; + + ptl = pmd_trans_huge_lock(pmd, vma); + if (ptl) { + if (!pmd_present(*pmd)) { + if (unlikely(softleaf_is_migration(softleaf_from_pmd(*pmd)))) { + spin_unlock(ptl); + pmd_migration_entry_wait(vma->vm_mm, pmd); + walk->action = ACTION_AGAIN; + return 0; + } + goto out; + } + if (is_huge_zero_pmd(*pmd)) + goto out; + folio = pmd_folio(*pmd); + if (folio_is_zone_device(folio)) + goto out; + if (is_migrate_cma_page(&folio->page)) + isolate_folio_to_list(folio, folio_list); + goto out; + } + + start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!start_pte) { + walk->action = ACTION_AGAIN; + return 0; + } + + for (pte = start_pte; addr != end; pte += step, addr += step * PAGE_SIZE) { + step = 1; + ptent = ptep_get(pte); + if (!pte_present(ptent)) { + if (unlikely(softleaf_is_migration(softleaf_from_pte(ptent)))) { + pte_unmap_unlock(start_pte, ptl); + migration_entry_wait(vma->vm_mm, pmd, addr); + walk->action = ACTION_AGAIN; + return 0; + } + continue; + } + folio = vm_normal_folio(vma, addr, ptent); + if (!folio || folio_is_zone_device(folio)) + continue; + step = folio_mlock_step(folio, pte, addr, end); + if (is_migrate_cma_page(&folio->page)) + continue; + isolate_folio_to_list(folio, folio_list); + } + pte_unmap(start_pte); +out: + spin_unlock(ptl); + cond_resched(); + return 0; +} + +static const struct mm_walk_ops mlock_collect_migratable_ops = { + .pmd_entry = mlock_collect_migratable_pte_range, + .walk_lock = PGWALK_RDLOCK, +}; + +static void mlock_migrate_cma_range(unsigned long start, unsigned long len) +{ + struct mm_struct *mm = current->mm; + unsigned long end = start + len; + LIST_HEAD(folio_list); + struct migration_target_control mtc = { + .nid = NUMA_NO_NODE, + .gfp_mask = GFP_HIGHUSER | __GFP_NOWARN, + .reason = MR_SYSCALL, + }; + + if (compaction_allow_unevictable()) + return; + + lru_cache_disable(); + + if (mmap_read_lock_killable(mm)) + goto out; + + walk_page_range(mm, start, end, &mlock_collect_migratable_ops, + &folio_list); + mmap_read_unlock(mm); + + if (list_empty(&folio_list)) + goto out; + + if (migrate_pages(&folio_list, alloc_migration_target, NULL, + (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL)) + putback_movable_pages(&folio_list); +out: + lru_cache_enable(); +} +#else +static inline void mlock_migrate_cma_range(unsigned long start, + unsigned long len) +{ +} +#endif /* CONFIG_CMA */ + /* * mlock_vma_pages_range() - mlock any pages already in the range, * or munlock all pages in the range. @@ -678,6 +792,7 @@ static __must_check int do_mlock(unsigned long start, size_t len, vm_flags_t fla error = __mm_populate(start, len, 0); if (error) return __mlock_posix_error_return(error); + mlock_migrate_cma_range(start, len); return 0; } @@ -790,8 +905,10 @@ SYSCALL_DEFINE1(mlockall, int, flags) capable(CAP_IPC_LOCK)) ret = apply_mlockall_flags(flags); mmap_write_unlock(current->mm); - if (!ret && (flags & MCL_CURRENT)) + if (!ret && (flags & MCL_CURRENT)) { mm_populate(0, TASK_SIZE); + mlock_migrate_cma_range(0, TASK_SIZE); + } return ret; } -- 2.43.0