From: "Lorenzo Stoakes (ARM)" <ljs@kernel.org>
To: David Hildenbrand <david@kernel.org>, Zi Yan <ziy@nvidia.com>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
"Liam R. Howlett" <liam@infradead.org>,
Nico Pache <nico.pache@linux.dev>,
Ryan Roberts <ryan.roberts@arm.com>, Dev Jain <dev.jain@arm.com>,
Barry Song <baohua@kernel.org>,
Lance Yang <lance.yang@linux.dev>,
Usama Arif <usama.arif@linux.dev>,
Kiryl Shutsemau <kas@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>, Will Deacon <will@kernel.org>,
Boqun Feng <boqun@kernel.org>, Waiman Long <longman@redhat.com>,
Suren Baghdasaryan <surenb@google.com>,
Vlastimil Babka <vbabka@kernel.org>,
Shakeel Butt <shakeel.butt@linux.dev>,
Yonghong Song <yonghong.song@linux.dev>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>,
Eduard Zingerman <eddyz87@gmail.com>,
Kumar Kartikeya Dwivedi <memxor@gmail.com>,
Martin KaFai Lau <martin.lau@linux.dev>,
Song Liu <song@kernel.org>, Jiri Olsa <jolsa@kernel.org>,
Emil Tsalapatis <emil@etsalapatis.com>,
Ihor Solodrai <ihor.solodrai@linux.dev>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
bpf@vger.kernel.org, "Lorenzo Stoakes (ARM)" <ljs@kernel.org>
Subject: [PATCH 4/5] mm: add and use mmap_read_lock[_killable]_nosteal() on contention
Date: Tue, 03 Nov 2026 08:27:45 +0000 [thread overview]
Message-ID: <20261103-khugepaged-latency-fix-v1-4-a07ab16ac860@kernel.org> (raw)
In-Reply-To: <20261103-khugepaged-latency-fix-v1-0-a07ab16ac860@kernel.org>
With down_read[_killable]_nosteal() available, it becomes possible to
release the mmap read lock in such a way that a queued write lock always
takes precedence.
Expose this to mm through two new wrappers - mmap_read_lock_nosteal() and
mmap_read_lock_killable_nosteal().
Make use of this in scenarios where the mmap read lock is dropped
explicitly for the purposes of relieving mmap_lock contention - the BPF
task VMA iterator, and lock_next_vma_under_mmap_lock(), which is where
/proc/$pid/smaps_rollup re-acquires the mmap lock after dropping it on
contention and falling back from per-VMA locking.
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
---
include/linux/mmap_lock.h | 44 ++++++++++++++++++++++++++++++++++++++++++++
kernel/bpf/task_iter.c | 2 +-
mm/mmap_lock.c | 2 +-
3 files changed, 46 insertions(+), 2 deletions(-)
diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h
index 9ccbbcbd7aca..88906ecd6ca3 100644
--- a/include/linux/mmap_lock.h
+++ b/include/linux/mmap_lock.h
@@ -627,6 +627,31 @@ static inline int __must_check mmap_read_lock_killable(struct mm_struct *mm)
return ret;
}
+/**
+ * mmap_read_lock_killable_nosteal() - Acquire a killable mmap read lock, but
+ * yield to any queued write locks.
+ * @mm: The mm whose mmap read lock is required.
+ *
+ * If an mmap write lock is queued behind readers, but its handoff timeout has
+ * yet to expire, releasing then re-acquiring a read lock can steal the lock
+ * from it.
+ *
+ * mmap_read_lock_killable_nosteal() will queue in this case instead, which is
+ * useful when releasing the mmap read lock specifically to relieve lock
+ * contention.
+ *
+ * Returns: 0 on success, or an error otherwise.
+ */
+static inline int __must_check mmap_read_lock_killable_nosteal(struct mm_struct *mm)
+{
+ int ret;
+
+ __mmap_lock_trace_start_locking(mm, false);
+ ret = down_read_killable_nosteal(&mm->mmap_lock);
+ __mmap_lock_trace_acquire_returned(mm, false, ret == 0);
+ return ret;
+}
+
static inline bool __must_check mmap_read_trylock(struct mm_struct *mm)
{
bool ret;
@@ -637,6 +662,25 @@ static inline bool __must_check mmap_read_trylock(struct mm_struct *mm)
return ret;
}
+/**
+ * mmap_read_lock_nosteal() - Acquire the mmap read lock, but yield to
+ * any queued write locks.
+ * @mm: The mm whose mmap read lock is required.
+ *
+ * If an mmap write lock is queued behind readers, but its handoff timeout has
+ * yet to expire, releasing then re-acquiring a read lock can steal the lock
+ * from it.
+ *
+ * mmap_read_lock_nosteal() will queue in this case instead, which is useful when
+ * releasing the mmap read lock specifically to relieve lock contention.
+ */
+static inline void mmap_read_lock_nosteal(struct mm_struct *mm)
+{
+ __mmap_lock_trace_start_locking(mm, false);
+ down_read_nosteal(&mm->mmap_lock);
+ __mmap_lock_trace_acquire_returned(mm, false, true);
+}
+
static inline void mmap_read_unlock(struct mm_struct *mm)
{
__mmap_lock_trace_released(mm, false);
diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c
index 2ee7b3b3c10b..b455306a6b52 100644
--- a/kernel/bpf/task_iter.c
+++ b/kernel/bpf/task_iter.c
@@ -486,7 +486,7 @@ task_vma_seq_get_next(struct bpf_iter_seq_task_vma_info *info)
info->prev_vm_end = curr_vma->vm_end;
op = task_vma_iter_find_vma;
mmap_read_unlock(curr_mm);
- if (mmap_read_lock_killable(curr_mm)) {
+ if (mmap_read_lock_killable_nosteal(curr_mm)) {
mmput(curr_mm);
goto finish;
}
diff --git a/mm/mmap_lock.c b/mm/mmap_lock.c
index 834bbf86e004..41b11b843211 100644
--- a/mm/mmap_lock.c
+++ b/mm/mmap_lock.c
@@ -412,7 +412,7 @@ static struct vm_area_struct *lock_next_vma_under_mmap_lock(struct mm_struct *mm
struct vm_area_struct *vma;
int ret;
- ret = mmap_read_lock_killable(mm);
+ ret = mmap_read_lock_killable_nosteal(mm);
if (ret)
return ERR_PTR(ret);
--
2.56.0
next prev parent reply other threads:[~2026-11-03 8:28 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-03 8:27 [PATCH 0/5] mm/khugepaged: bound mmap_lock hold time when walking an mm's VMAs Lorenzo Stoakes (ARM)
2026-11-03 8:27 ` [PATCH 1/5] Revert "mm/khugepaged: remove definition of struct khugepaged_mm_slot" Lorenzo Stoakes (ARM)
2026-11-04 7:46 ` Lance Yang
2026-11-05 10:17 ` David Hildenbrand (Arm)
2026-11-03 8:27 ` [PATCH 2/5] mm/khugepaged: scan mm's more efficiently under mmap_lock contention Lorenzo Stoakes (ARM)
2026-11-04 8:43 ` Lance Yang
2026-11-05 10:23 ` David Hildenbrand (Arm)
2026-11-05 14:01 ` Lorenzo Stoakes (ARM)
2026-11-03 8:27 ` [PATCH 3/5] locking/rwsem: add down_read[_killable]_nosteal() Lorenzo Stoakes (ARM)
2026-11-03 11:51 ` Peter Zijlstra
2026-11-03 12:24 ` Lorenzo Stoakes (ARM)
2026-11-05 10:25 ` David Hildenbrand (Arm)
2026-11-05 10:29 ` Lorenzo Stoakes (ARM)
2026-11-03 8:27 ` Lorenzo Stoakes (ARM) [this message]
2026-11-05 10:16 ` [PATCH 4/5] mm: add and use mmap_read_lock[_killable]_nosteal() on contention David Hildenbrand (Arm)
2026-11-05 14:02 ` Lorenzo Stoakes (ARM)
2026-11-05 16:23 ` Alexei Starovoitov
2026-11-03 8:27 ` [PATCH 5/5] mm/khugepaged: drop/reacquire mmap read lock if skipped enough VMAs Lorenzo Stoakes (ARM)
2026-11-05 10:28 ` David Hildenbrand (Arm)
2026-11-05 14:46 ` Lorenzo Stoakes (ARM)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261103-khugepaged-latency-fix-v1-4-a07ab16ac860@kernel.org \
--to=ljs@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=baohua@kernel.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=boqun@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=david@kernel.org \
--cc=dev.jain@arm.com \
--cc=eddyz87@gmail.com \
--cc=emil@etsalapatis.com \
--cc=ihor.solodrai@linux.dev \
--cc=jolsa@kernel.org \
--cc=kas@kernel.org \
--cc=lance.yang@linux.dev \
--cc=liam@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=longman@redhat.com \
--cc=martin.lau@linux.dev \
--cc=memxor@gmail.com \
--cc=mingo@redhat.com \
--cc=nico.pache@linux.dev \
--cc=peterz@infradead.org \
--cc=ryan.roberts@arm.com \
--cc=shakeel.butt@linux.dev \
--cc=song@kernel.org \
--cc=surenb@google.com \
--cc=usama.arif@linux.dev \
--cc=vbabka@kernel.org \
--cc=will@kernel.org \
--cc=yonghong.song@linux.dev \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®