mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Lorenzo Stoakes (ARM)" <ljs@kernel.org>
To: David Hildenbrand <david@kernel.org>, Zi Yan <ziy@nvidia.com>,
	 Baolin Wang <baolin.wang@linux.alibaba.com>,
	 "Liam R. Howlett" <liam@infradead.org>,
	Nico Pache <nico.pache@linux.dev>,
	 Ryan Roberts <ryan.roberts@arm.com>, Dev Jain <dev.jain@arm.com>,
	 Barry Song <baohua@kernel.org>,
	Lance Yang <lance.yang@linux.dev>,
	 Usama Arif <usama.arif@linux.dev>,
	Kiryl Shutsemau <kas@kernel.org>,
	 Andrew Morton <akpm@linux-foundation.org>,
	 Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>,  Will Deacon <will@kernel.org>,
	Boqun Feng <boqun@kernel.org>,  Waiman Long <longman@redhat.com>,
	Suren Baghdasaryan <surenb@google.com>,
	 Vlastimil Babka <vbabka@kernel.org>,
	Shakeel Butt <shakeel.butt@linux.dev>,
	 Yonghong Song <yonghong.song@linux.dev>,
	 Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	 Andrii Nakryiko <andrii@kernel.org>,
	Eduard Zingerman <eddyz87@gmail.com>,
	 Kumar Kartikeya Dwivedi <memxor@gmail.com>,
	 Martin KaFai Lau <martin.lau@linux.dev>,
	Song Liu <song@kernel.org>,  Jiri Olsa <jolsa@kernel.org>,
	Emil Tsalapatis <emil@etsalapatis.com>,
	 Ihor Solodrai <ihor.solodrai@linux.dev>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
	bpf@vger.kernel.org,  "Lorenzo Stoakes (ARM)" <ljs@kernel.org>
Subject: [PATCH 4/5] mm: add and use mmap_read_lock[_killable]_nosteal() on contention
Date: Tue, 03 Nov 2026 08:27:45 +0000	[thread overview]
Message-ID: <20261103-khugepaged-latency-fix-v1-4-a07ab16ac860@kernel.org> (raw)
In-Reply-To: <20261103-khugepaged-latency-fix-v1-0-a07ab16ac860@kernel.org>

With down_read[_killable]_nosteal() available, it becomes possible to
release the mmap read lock in such a way that a queued write lock always
takes precedence.

Expose this to mm through two new wrappers - mmap_read_lock_nosteal() and
mmap_read_lock_killable_nosteal().

Make use of this in scenarios where the mmap read lock is dropped
explicitly for the purposes of relieving mmap_lock contention - the BPF
task VMA iterator, and lock_next_vma_under_mmap_lock(), which is where
/proc/$pid/smaps_rollup re-acquires the mmap lock after dropping it on
contention and falling back from per-VMA locking.

Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
---
 include/linux/mmap_lock.h | 44 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/bpf/task_iter.c    |  2 +-
 mm/mmap_lock.c            |  2 +-
 3 files changed, 46 insertions(+), 2 deletions(-)

diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h
index 9ccbbcbd7aca..88906ecd6ca3 100644
--- a/include/linux/mmap_lock.h
+++ b/include/linux/mmap_lock.h
@@ -627,6 +627,31 @@ static inline int __must_check mmap_read_lock_killable(struct mm_struct *mm)
 	return ret;
 }
 
+/**
+ * mmap_read_lock_killable_nosteal() - Acquire a killable mmap read lock, but
+ * yield to any queued write locks.
+ * @mm: The mm whose mmap read lock is required.
+ *
+ * If an mmap write lock is queued behind readers, but its handoff timeout has
+ * yet to expire, releasing then re-acquiring a read lock can steal the lock
+ * from it.
+ *
+ * mmap_read_lock_killable_nosteal() will queue in this case instead, which is
+ * useful when releasing the mmap read lock specifically to relieve lock
+ * contention.
+ *
+ * Returns: 0 on success, or an error otherwise.
+ */
+static inline int __must_check mmap_read_lock_killable_nosteal(struct mm_struct *mm)
+{
+	int ret;
+
+	__mmap_lock_trace_start_locking(mm, false);
+	ret = down_read_killable_nosteal(&mm->mmap_lock);
+	__mmap_lock_trace_acquire_returned(mm, false, ret == 0);
+	return ret;
+}
+
 static inline bool __must_check mmap_read_trylock(struct mm_struct *mm)
 {
 	bool ret;
@@ -637,6 +662,25 @@ static inline bool __must_check mmap_read_trylock(struct mm_struct *mm)
 	return ret;
 }
 
+/**
+ * mmap_read_lock_nosteal() - Acquire the mmap read lock, but yield to
+ * any queued write locks.
+ * @mm: The mm whose mmap read lock is required.
+ *
+ * If an mmap write lock is queued behind readers, but its handoff timeout has
+ * yet to expire, releasing then re-acquiring a read lock can steal the lock
+ * from it.
+ *
+ * mmap_read_lock_nosteal() will queue in this case instead, which is useful when
+ * releasing the mmap read lock specifically to relieve lock contention.
+ */
+static inline void mmap_read_lock_nosteal(struct mm_struct *mm)
+{
+	__mmap_lock_trace_start_locking(mm, false);
+	down_read_nosteal(&mm->mmap_lock);
+	__mmap_lock_trace_acquire_returned(mm, false, true);
+}
+
 static inline void mmap_read_unlock(struct mm_struct *mm)
 {
 	__mmap_lock_trace_released(mm, false);
diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c
index 2ee7b3b3c10b..b455306a6b52 100644
--- a/kernel/bpf/task_iter.c
+++ b/kernel/bpf/task_iter.c
@@ -486,7 +486,7 @@ task_vma_seq_get_next(struct bpf_iter_seq_task_vma_info *info)
 			info->prev_vm_end = curr_vma->vm_end;
 			op = task_vma_iter_find_vma;
 			mmap_read_unlock(curr_mm);
-			if (mmap_read_lock_killable(curr_mm)) {
+			if (mmap_read_lock_killable_nosteal(curr_mm)) {
 				mmput(curr_mm);
 				goto finish;
 			}
diff --git a/mm/mmap_lock.c b/mm/mmap_lock.c
index 834bbf86e004..41b11b843211 100644
--- a/mm/mmap_lock.c
+++ b/mm/mmap_lock.c
@@ -412,7 +412,7 @@ static struct vm_area_struct *lock_next_vma_under_mmap_lock(struct mm_struct *mm
 	struct vm_area_struct *vma;
 	int ret;
 
-	ret = mmap_read_lock_killable(mm);
+	ret = mmap_read_lock_killable_nosteal(mm);
 	if (ret)
 		return ERR_PTR(ret);
 

-- 
2.56.0


  parent reply	other threads:[~2026-11-03  8:28 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-03  8:27 [PATCH 0/5] mm/khugepaged: bound mmap_lock hold time when walking an mm's VMAs Lorenzo Stoakes (ARM)
2026-11-03  8:27 ` [PATCH 1/5] Revert "mm/khugepaged: remove definition of struct khugepaged_mm_slot" Lorenzo Stoakes (ARM)
2026-11-04  7:46   ` Lance Yang
2026-11-05 10:17   ` David Hildenbrand (Arm)
2026-11-03  8:27 ` [PATCH 2/5] mm/khugepaged: scan mm's more efficiently under mmap_lock contention Lorenzo Stoakes (ARM)
2026-11-04  8:43   ` Lance Yang
2026-11-05 10:23   ` David Hildenbrand (Arm)
2026-11-05 14:01     ` Lorenzo Stoakes (ARM)
2026-11-03  8:27 ` [PATCH 3/5] locking/rwsem: add down_read[_killable]_nosteal() Lorenzo Stoakes (ARM)
2026-11-03 11:51   ` Peter Zijlstra
2026-11-03 12:24     ` Lorenzo Stoakes (ARM)
2026-11-05 10:25       ` David Hildenbrand (Arm)
2026-11-05 10:29         ` Lorenzo Stoakes (ARM)
2026-11-03  8:27 ` Lorenzo Stoakes (ARM) [this message]
2026-11-05 10:16   ` [PATCH 4/5] mm: add and use mmap_read_lock[_killable]_nosteal() on contention David Hildenbrand (Arm)
2026-11-05 14:02     ` Lorenzo Stoakes (ARM)
2026-11-05 16:23     ` Alexei Starovoitov
2026-11-03  8:27 ` [PATCH 5/5] mm/khugepaged: drop/reacquire mmap read lock if skipped enough VMAs Lorenzo Stoakes (ARM)
2026-11-05 10:28   ` David Hildenbrand (Arm)
2026-11-05 14:46     ` Lorenzo Stoakes (ARM)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261103-khugepaged-latency-fix-v1-4-a07ab16ac860@kernel.org \
    --to=ljs@kernel.org \
    --cc=akpm@linux-foundation.org \
    --cc=andrii@kernel.org \
    --cc=ast@kernel.org \
    --cc=baohua@kernel.org \
    --cc=baolin.wang@linux.alibaba.com \
    --cc=boqun@kernel.org \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=david@kernel.org \
    --cc=dev.jain@arm.com \
    --cc=eddyz87@gmail.com \
    --cc=emil@etsalapatis.com \
    --cc=ihor.solodrai@linux.dev \
    --cc=jolsa@kernel.org \
    --cc=kas@kernel.org \
    --cc=lance.yang@linux.dev \
    --cc=liam@infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=longman@redhat.com \
    --cc=martin.lau@linux.dev \
    --cc=memxor@gmail.com \
    --cc=mingo@redhat.com \
    --cc=nico.pache@linux.dev \
    --cc=peterz@infradead.org \
    --cc=ryan.roberts@arm.com \
    --cc=shakeel.butt@linux.dev \
    --cc=song@kernel.org \
    --cc=surenb@google.com \
    --cc=usama.arif@linux.dev \
    --cc=vbabka@kernel.org \
    --cc=will@kernel.org \
    --cc=yonghong.song@linux.dev \
    --cc=ziy@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®