[PATCH v4 0/6] vdso/datastore: Allow prefaulting by mlockall()

[PATCH v4 0/6] vdso/datastore: Allow prefaulting by mlockall()

[Thomas Weißschuh]

While mlockall() is meant to lock page *memory*, effectively it will
also create and lock the corresponding page table entries.
Latency-sensitive applications expect not to experience any pagefaults
after calling mlockall(). However mlockall() ignores VM_IO mappings,
which is used by the generic vDSO datastore.
While the fault handler itself is very fast, going through the full
pagefault exception handling is much slower, on the order of 20us in a
test machine.

Since the memory behind the datastore mappings is always present and
accessible it is not necessary to use VM_IO for them.

---
Changes in v4:
- Rebase on v7.2-rc1
- Extend commit messages
- Link to v3: https://patch.msgid.link/20260518-vdso-mlockall-v3-0-bf4f2af2b41f@linutronix.de

Changes in v3:
- Rebase on v7.1-rc1
- Stop using zero page, which can not be mapped in this way
- Link to v2: https://lore.kernel.org/r/20250901-vdso-mlockall-v2-0-68f5a6f03345@linutronix.de

Changes in v2:
- Stop using nth_page() which is being removed
- Link to v1: https://lore.kernel.org/r/20250812-vdso-mlockall-v1-0-2f49ba7cf819@linutronix.de

To: Anna-Maria Behnsen <anna-maria@linutronix.de>
To: Frederic Weisbecker <frederic@kernel.org>
To: Thomas Gleixner <tglx@kernel.org>
To: Andy Lutomirski <luto@kernel.org>
To: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Nam Cao <namcao@linutronix.de>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: linux-mm@kvack.org
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <Liam.Howlett@oracle.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>

---
Thomas Weißschuh (6):
      vdso/datastore: Rename data pages variable
      vdso/datastore: Map pages in terms of the faults pgoff
      vdso/datastore: Map zeroed pages for unavailable data
      vdso/datastore: Explicitly prevent remote access to timens vvar page
      vdso/datastore: Allow prefaulting by mlockall()
      vdso/datastore: Simplify the mapping logic for VDSO_TIME_PAGE_OFFSET

 kernel/time/namespace_vdso.c |  7 ++---
 lib/vdso/datastore.c         | 65 ++++++++++++++++++++------------------------
 2 files changed, 32 insertions(+), 40 deletions(-)
---
base-commit: 0f4c4aeffbe38fb638f9a978bfb64caa582c2681
change-id: 20250721-vdso-mlockall-461bb33205b1

Best regards,
--  
Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>

[PATCH v4 1/6] vdso/datastore: Rename data pages variable

[Thomas Weißschuh]

An upcoming change will make this a file-scoped variable, for which it
should have a clearer name.

Rename the variable to prepare for that.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Tested-by: Nam Cao <namcao@linutronix.de>
---
 lib/vdso/datastore.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index 17d37b82ebc6..7f78a32656aa 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -32,7 +32,7 @@ struct vdso_arch_data *vdso_k_arch_data __ro_after_init =
 void __init vdso_setup_data_pages(void)
 {
 	unsigned int order = get_order(VDSO_NR_PAGES * PAGE_SIZE);
-	struct page *pages;
+	struct page *vdso_data_pages;
 
 	/*
 	 * Allocate the data pages dynamically. SPARC does not support mapping
@@ -42,24 +42,24 @@ void __init vdso_setup_data_pages(void)
 	 * Do not use folios. In time namespaces the pages are mapped in a different order
 	 * to userspace, which is not handled by the folio optimizations in finish_fault().
 	 */
-	pages = alloc_pages(GFP_KERNEL, order);
-	if (!pages)
+	vdso_data_pages = alloc_pages(GFP_KERNEL, order);
+	if (!vdso_data_pages)
 		panic("Unable to allocate VDSO storage pages");
 
 	/* The pages are mapped one-by-one into userspace and each one needs to be refcounted. */
-	split_page(pages, order);
+	split_page(vdso_data_pages, order);
 
 	/* Move the data already written by other subsystems to the new pages */
-	memcpy(page_address(pages), vdso_initdata, VDSO_NR_PAGES * PAGE_SIZE);
+	memcpy(page_address(vdso_data_pages), vdso_initdata, VDSO_NR_PAGES * PAGE_SIZE);
 
 	if (IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY))
-		vdso_k_time_data = page_address(pages + VDSO_TIME_PAGE_OFFSET);
+		vdso_k_time_data = page_address(vdso_data_pages + VDSO_TIME_PAGE_OFFSET);
 
 	if (IS_ENABLED(CONFIG_VDSO_GETRANDOM))
-		vdso_k_rng_data = page_address(pages + VDSO_RNG_PAGE_OFFSET);
+		vdso_k_rng_data = page_address(vdso_data_pages + VDSO_RNG_PAGE_OFFSET);
 
 	if (IS_ENABLED(CONFIG_ARCH_HAS_VDSO_ARCH_DATA))
-		vdso_k_arch_data = page_address(pages + VDSO_ARCH_PAGES_START);
+		vdso_k_arch_data = page_address(vdso_data_pages + VDSO_ARCH_PAGES_START);
 }
 
 static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,

-- 
2.55.0

[tip: timers/vdso] vdso/datastore: Rename data pages variable

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the timers/vdso branch of tip:

Commit-ID:     ff868f43eb8faf77498efcb206b721e406714e3e
Gitweb:        https://git.kernel.org/tip/ff868f43eb8faf77498efcb206b721e406714e3e
Author:        Thomas Weißschuh <thomas.weissschuh@linutronix.de>
AuthorDate:    Tue, 30 Jun 2026 09:31:55 +02:00
Committer:     Thomas Gleixner <tglx@kernel.org>
CommitterDate: Tue, 07 Jul 2026 23:52:52 +02:00

vdso/datastore: Rename data pages variable

An upcoming change will make this a file-scoped variable, for which it
should have a clearer name.

Rename the variable to prepare for that.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Nam Cao <namcao@linutronix.de>
Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-1-6c93708ce723@linutronix.de
---
 lib/vdso/datastore.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index 17d37b8..7f78a32 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -32,7 +32,7 @@ struct vdso_arch_data *vdso_k_arch_data __ro_after_init =
 void __init vdso_setup_data_pages(void)
 {
 	unsigned int order = get_order(VDSO_NR_PAGES * PAGE_SIZE);
-	struct page *pages;
+	struct page *vdso_data_pages;
 
 	/*
 	 * Allocate the data pages dynamically. SPARC does not support mapping
@@ -42,24 +42,24 @@ void __init vdso_setup_data_pages(void)
 	 * Do not use folios. In time namespaces the pages are mapped in a different order
 	 * to userspace, which is not handled by the folio optimizations in finish_fault().
 	 */
-	pages = alloc_pages(GFP_KERNEL, order);
-	if (!pages)
+	vdso_data_pages = alloc_pages(GFP_KERNEL, order);
+	if (!vdso_data_pages)
 		panic("Unable to allocate VDSO storage pages");
 
 	/* The pages are mapped one-by-one into userspace and each one needs to be refcounted. */
-	split_page(pages, order);
+	split_page(vdso_data_pages, order);
 
 	/* Move the data already written by other subsystems to the new pages */
-	memcpy(page_address(pages), vdso_initdata, VDSO_NR_PAGES * PAGE_SIZE);
+	memcpy(page_address(vdso_data_pages), vdso_initdata, VDSO_NR_PAGES * PAGE_SIZE);
 
 	if (IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY))
-		vdso_k_time_data = page_address(pages + VDSO_TIME_PAGE_OFFSET);
+		vdso_k_time_data = page_address(vdso_data_pages + VDSO_TIME_PAGE_OFFSET);
 
 	if (IS_ENABLED(CONFIG_VDSO_GETRANDOM))
-		vdso_k_rng_data = page_address(pages + VDSO_RNG_PAGE_OFFSET);
+		vdso_k_rng_data = page_address(vdso_data_pages + VDSO_RNG_PAGE_OFFSET);
 
 	if (IS_ENABLED(CONFIG_ARCH_HAS_VDSO_ARCH_DATA))
-		vdso_k_arch_data = page_address(pages + VDSO_ARCH_PAGES_START);
+		vdso_k_arch_data = page_address(vdso_data_pages + VDSO_ARCH_PAGES_START);
 }
 
 static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,

[PATCH v4 2/6] vdso/datastore: Map pages in terms of the faults pgoff

[Thomas Weißschuh]

To support mlockall() on the datapages the VMA can have no holes where
inner pages return VM_FAULT_SIGBUS. An upcoming change will avoid these
holes by mapping a zeroed pages into these holes. That logic will be
simpler when the mapping logic is based on vmf->pgoff instead of the
vdso_k_ symbols.

Switch to the equivalent vmf->pgoff logic.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Tested-by: Nam Cao <namcao@linutronix.de>
---
 lib/vdso/datastore.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index 7f78a32656aa..e01db9607534 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -29,10 +29,11 @@ struct vdso_arch_data *vdso_k_arch_data __ro_after_init =
 	(void *)&vdso_initdata[VDSO_ARCH_PAGES_START * PAGE_SIZE];
 #endif /* CONFIG_ARCH_HAS_VDSO_ARCH_DATA */
 
+static struct page *vdso_data_pages __ro_after_init;
+
 void __init vdso_setup_data_pages(void)
 {
 	unsigned int order = get_order(VDSO_NR_PAGES * PAGE_SIZE);
-	struct page *vdso_data_pages;
 
 	/*
 	 * Allocate the data pages dynamically. SPARC does not support mapping
@@ -67,13 +68,13 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 {
 	struct page *page, *timens_page;
 
+	page = vdso_data_pages + vmf->pgoff;
 	timens_page = find_timens_vvar_page(vma);
 
 	switch (vmf->pgoff) {
 	case VDSO_TIME_PAGE_OFFSET:
 		if (!IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY))
 			return VM_FAULT_SIGBUS;
-		page = virt_to_page(vdso_k_time_data);
 		if (timens_page) {
 			/*
 			 * Fault in VVAR page too, since it will be accessed
@@ -99,17 +100,15 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 		 */
 		if (!IS_ENABLED(CONFIG_TIME_NS) || !timens_page)
 			return VM_FAULT_SIGBUS;
-		page = virt_to_page(vdso_k_time_data);
+		page = vdso_data_pages + VDSO_TIME_PAGE_OFFSET;
 		break;
 	case VDSO_RNG_PAGE_OFFSET:
 		if (!IS_ENABLED(CONFIG_VDSO_GETRANDOM))
 			return VM_FAULT_SIGBUS;
-		page = virt_to_page(vdso_k_rng_data);
 		break;
 	case VDSO_ARCH_PAGES_START ... VDSO_ARCH_PAGES_END:
 		if (!IS_ENABLED(CONFIG_ARCH_HAS_VDSO_ARCH_DATA))
 			return VM_FAULT_SIGBUS;
-		page = virt_to_page(vdso_k_arch_data) + vmf->pgoff - VDSO_ARCH_PAGES_START;
 		break;
 	default:
 		return VM_FAULT_SIGBUS;

-- 
2.55.0

[tip: timers/vdso] vdso/datastore: Map pages in terms of the faults pgoff

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the timers/vdso branch of tip:

Commit-ID:     7557273419dd618df6d83ed37449b0e5918fd501
Gitweb:        https://git.kernel.org/tip/7557273419dd618df6d83ed37449b0e5918fd501
Author:        Thomas Weißschuh <thomas.weissschuh@linutronix.de>
AuthorDate:    Tue, 30 Jun 2026 09:31:56 +02:00
Committer:     Thomas Gleixner <tglx@kernel.org>
CommitterDate: Tue, 07 Jul 2026 23:52:52 +02:00

vdso/datastore: Map pages in terms of the faults pgoff

To support mlockall() on the datapages the VMA can have no holes where
inner pages return VM_FAULT_SIGBUS. An upcoming change will avoid these
holes by mapping a zeroed pages into these holes. That logic will be
simpler when the mapping logic is based on vmf->pgoff instead of the
vdso_k_ symbols.

Switch to the equivalent vmf->pgoff logic.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Nam Cao <namcao@linutronix.de>
Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-2-6c93708ce723@linutronix.de
---
 lib/vdso/datastore.c |  9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index 7f78a32..e01db96 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -29,10 +29,11 @@ struct vdso_arch_data *vdso_k_arch_data __ro_after_init =
 	(void *)&vdso_initdata[VDSO_ARCH_PAGES_START * PAGE_SIZE];
 #endif /* CONFIG_ARCH_HAS_VDSO_ARCH_DATA */
 
+static struct page *vdso_data_pages __ro_after_init;
+
 void __init vdso_setup_data_pages(void)
 {
 	unsigned int order = get_order(VDSO_NR_PAGES * PAGE_SIZE);
-	struct page *vdso_data_pages;
 
 	/*
 	 * Allocate the data pages dynamically. SPARC does not support mapping
@@ -67,13 +68,13 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 {
 	struct page *page, *timens_page;
 
+	page = vdso_data_pages + vmf->pgoff;
 	timens_page = find_timens_vvar_page(vma);
 
 	switch (vmf->pgoff) {
 	case VDSO_TIME_PAGE_OFFSET:
 		if (!IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY))
 			return VM_FAULT_SIGBUS;
-		page = virt_to_page(vdso_k_time_data);
 		if (timens_page) {
 			/*
 			 * Fault in VVAR page too, since it will be accessed
@@ -99,17 +100,15 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 		 */
 		if (!IS_ENABLED(CONFIG_TIME_NS) || !timens_page)
 			return VM_FAULT_SIGBUS;
-		page = virt_to_page(vdso_k_time_data);
+		page = vdso_data_pages + VDSO_TIME_PAGE_OFFSET;
 		break;
 	case VDSO_RNG_PAGE_OFFSET:
 		if (!IS_ENABLED(CONFIG_VDSO_GETRANDOM))
 			return VM_FAULT_SIGBUS;
-		page = virt_to_page(vdso_k_rng_data);
 		break;
 	case VDSO_ARCH_PAGES_START ... VDSO_ARCH_PAGES_END:
 		if (!IS_ENABLED(CONFIG_ARCH_HAS_VDSO_ARCH_DATA))
 			return VM_FAULT_SIGBUS;
-		page = virt_to_page(vdso_k_arch_data) + vmf->pgoff - VDSO_ARCH_PAGES_START;
 		break;
 	default:
 		return VM_FAULT_SIGBUS;

[PATCH v4 3/6] vdso/datastore: Map zeroed pages for unavailable data

[Thomas Weißschuh]

mlockall() stops if a page in a VMA is unmappable. As the datastore VMA
can contain holes, mlockall() would not process all data pages.

Replace the mapping error VM_FAULT_SIGBUS by just mapping the underlying
unused and zeroed-out data page. The vDSO will not access these pages in
any case and for other userspace these pages have undefined contents.

This will allow mlockall() to process all pages within the VMA as soon
as VM_IO is removed from the VMA.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Tested-by: Nam Cao <namcao@linutronix.de>
---
 lib/vdso/datastore.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index e01db9607534..cf7aa80355cf 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -74,7 +74,7 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 	switch (vmf->pgoff) {
 	case VDSO_TIME_PAGE_OFFSET:
 		if (!IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY))
-			return VM_FAULT_SIGBUS;
+			break;
 		if (timens_page) {
 			/*
 			 * Fault in VVAR page too, since it will be accessed
@@ -99,16 +99,11 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 		 * See also the comment near timens_setup_vdso_data().
 		 */
 		if (!IS_ENABLED(CONFIG_TIME_NS) || !timens_page)
-			return VM_FAULT_SIGBUS;
+			break;
 		page = vdso_data_pages + VDSO_TIME_PAGE_OFFSET;
 		break;
 	case VDSO_RNG_PAGE_OFFSET:
-		if (!IS_ENABLED(CONFIG_VDSO_GETRANDOM))
-			return VM_FAULT_SIGBUS;
-		break;
 	case VDSO_ARCH_PAGES_START ... VDSO_ARCH_PAGES_END:
-		if (!IS_ENABLED(CONFIG_ARCH_HAS_VDSO_ARCH_DATA))
-			return VM_FAULT_SIGBUS;
 		break;
 	default:
 		return VM_FAULT_SIGBUS;

-- 
2.55.0

[tip: timers/vdso] vdso/datastore: Map zeroed pages for unavailable data

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the timers/vdso branch of tip:

Commit-ID:     43648f9f3a67125ba649027a5be00bc005065482
Gitweb:        https://git.kernel.org/tip/43648f9f3a67125ba649027a5be00bc005065482
Author:        Thomas Weißschuh <thomas.weissschuh@linutronix.de>
AuthorDate:    Tue, 30 Jun 2026 09:31:57 +02:00
Committer:     Thomas Gleixner <tglx@kernel.org>
CommitterDate: Tue, 07 Jul 2026 23:52:52 +02:00

vdso/datastore: Map zeroed pages for unavailable data

mlockall() stops if a page in a VMA is unmappable. As the datastore VMA
can contain holes, mlockall() would not process all data pages.

Replace the mapping error VM_FAULT_SIGBUS by just mapping the underlying
unused and zeroed-out data page. The vDSO will not access these pages in
any case and for other userspace these pages have undefined contents.

This will allow mlockall() to process all pages within the VMA as soon
as VM_IO is removed from the VMA.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Nam Cao <namcao@linutronix.de>
Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-3-6c93708ce723@linutronix.de
---
 lib/vdso/datastore.c |  9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index e01db96..cf7aa80 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -74,7 +74,7 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 	switch (vmf->pgoff) {
 	case VDSO_TIME_PAGE_OFFSET:
 		if (!IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY))
-			return VM_FAULT_SIGBUS;
+			break;
 		if (timens_page) {
 			/*
 			 * Fault in VVAR page too, since it will be accessed
@@ -99,16 +99,11 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 		 * See also the comment near timens_setup_vdso_data().
 		 */
 		if (!IS_ENABLED(CONFIG_TIME_NS) || !timens_page)
-			return VM_FAULT_SIGBUS;
+			break;
 		page = vdso_data_pages + VDSO_TIME_PAGE_OFFSET;
 		break;
 	case VDSO_RNG_PAGE_OFFSET:
-		if (!IS_ENABLED(CONFIG_VDSO_GETRANDOM))
-			return VM_FAULT_SIGBUS;
-		break;
 	case VDSO_ARCH_PAGES_START ... VDSO_ARCH_PAGES_END:
-		if (!IS_ENABLED(CONFIG_ARCH_HAS_VDSO_ARCH_DATA))
-			return VM_FAULT_SIGBUS;
 		break;
 	default:
 		return VM_FAULT_SIGBUS;

[PATCH v4 4/6] vdso/datastore: Explicitly prevent remote access to timens vvar page

[Thomas Weißschuh]

The fault handler for the timens page does not have access to the target
task and therefore can not be invoked remotely.
Currently the handler relies on the fact that the vvar mapping is marked as
VM_IO and VM_PFNMAP for which the mm core always prevents remote access.
However the VM_IO and VM_PFNMAP flags are going to be removed.

Add an explicit check to prevent remote access to the mapping.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Tested-by: Nam Cao <namcao@linutronix.de>
---
 kernel/time/namespace_vdso.c | 7 ++-----
 lib/vdso/datastore.c         | 3 +++
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/kernel/time/namespace_vdso.c b/kernel/time/namespace_vdso.c
index 0d74d160eec9..5ac7b6a6d3a8 100644
--- a/kernel/time/namespace_vdso.c
+++ b/kernel/time/namespace_vdso.c
@@ -74,11 +74,8 @@ struct page *find_timens_vvar_page(struct vm_area_struct *vma)
 		return current->nsproxy->time_ns->vvar_page;
 
 	/*
-	 * VM_PFNMAP | VM_IO protect .fault() handler from being called
-	 * through interfaces like /proc/$pid/mem or
-	 * process_vm_{readv,writev}() as long as there's no .access()
-	 * in special_mapping_vmops().
-	 * For more details check_vma_flags() and __access_remote_vm()
+	 * vvar_fault() protects this from being called through remote interfaces like
+	 * /proc/$pid/mem or process_vm_{readv,writev}().
 	 */
 
 	WARN(1, "vvar_page accessed remotely");
diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index cf7aa80355cf..0ca43965a8a9 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -68,6 +68,9 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 {
 	struct page *page, *timens_page;
 
+	if (unlikely(vmf->flags & FAULT_FLAG_REMOTE))
+		return VM_FAULT_SIGBUS;
+
 	page = vdso_data_pages + vmf->pgoff;
 	timens_page = find_timens_vvar_page(vma);
 

-- 
2.55.0

[tip: timers/vdso] vdso/datastore: Explicitly prevent remote access to timens vvar page

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the timers/vdso branch of tip:

Commit-ID:     9ab500d47f5f1b8c463a4fd8f345a8756da625ab
Gitweb:        https://git.kernel.org/tip/9ab500d47f5f1b8c463a4fd8f345a8756da625ab
Author:        Thomas Weißschuh <thomas.weissschuh@linutronix.de>
AuthorDate:    Tue, 30 Jun 2026 09:31:58 +02:00
Committer:     Thomas Gleixner <tglx@kernel.org>
CommitterDate: Tue, 07 Jul 2026 23:52:52 +02:00

vdso/datastore: Explicitly prevent remote access to timens vvar page

The fault handler for the timens page does not have access to the target
task and therefore can not be invoked remotely.
Currently the handler relies on the fact that the vvar mapping is marked as
VM_IO and VM_PFNMAP for which the mm core always prevents remote access.
However the VM_IO and VM_PFNMAP flags are going to be removed.

Add an explicit check to prevent remote access to the mapping.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Nam Cao <namcao@linutronix.de>
Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-4-6c93708ce723@linutronix.de
---
 kernel/time/namespace_vdso.c | 7 ++-----
 lib/vdso/datastore.c         | 3 +++
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/kernel/time/namespace_vdso.c b/kernel/time/namespace_vdso.c
index 0d74d16..5ac7b6a 100644
--- a/kernel/time/namespace_vdso.c
+++ b/kernel/time/namespace_vdso.c
@@ -74,11 +74,8 @@ struct page *find_timens_vvar_page(struct vm_area_struct *vma)
 		return current->nsproxy->time_ns->vvar_page;
 
 	/*
-	 * VM_PFNMAP | VM_IO protect .fault() handler from being called
-	 * through interfaces like /proc/$pid/mem or
-	 * process_vm_{readv,writev}() as long as there's no .access()
-	 * in special_mapping_vmops().
-	 * For more details check_vma_flags() and __access_remote_vm()
+	 * vvar_fault() protects this from being called through remote interfaces like
+	 * /proc/$pid/mem or process_vm_{readv,writev}().
 	 */
 
 	WARN(1, "vvar_page accessed remotely");
diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index cf7aa80..0ca4396 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -68,6 +68,9 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 {
 	struct page *page, *timens_page;
 
+	if (unlikely(vmf->flags & FAULT_FLAG_REMOTE))
+		return VM_FAULT_SIGBUS;
+
 	page = vdso_data_pages + vmf->pgoff;
 	timens_page = find_timens_vvar_page(vma);
 

[PATCH v4 5/6] vdso/datastore: Allow prefaulting by mlockall()

[Thomas Weißschuh]

While mlockall() is meant to lock page *memory*, effectively it will
also create and lock the corresponding page table entries.
Latency-sensitive applications expect not to experience any pagefaults
after calling mlockall(). However mlockall() ignores VM_IO mappings,
which is used by the generic vDSO datastore.
While the fault handler itself is very fast, going through the full
pagefault exception handling is much slower, on the order of 20us in a
test machine.

Since the memory behind the datastore mappings is always present and
accessible it is not necessary to use VM_IO for them.

The data page mapping is now also aligned with the architecture-specific
code pages. Some architecture-specific data pages, like the x86 VCLOCK
pages, continue to use VM_IO as they are not always mappable. They will
require their own special handling later when the general approach has
been agreed upon.

As a side-effect this will allow GUP on these pages and allow more ways
to access the data in them. This is fine, as all data in this mapping
is globally visible anyways. Either because it is mapped into all tasks,
or in the case of the time namespace pages, can be read from procfs.

Regular mlock() would also work, but userspace does not know the boundaries
of the vDSO.

Reported-by: Nam Cao <namcao@linutronix.de>
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Tested-by: Nam Cao <namcao@linutronix.de>
---
 lib/vdso/datastore.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index 0ca43965a8a9..d7ab077cd604 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -125,7 +125,7 @@ const struct vm_special_mapping vdso_vvar_mapping = {
 struct vm_area_struct *vdso_install_vvar_mapping(struct mm_struct *mm, unsigned long addr)
 {
 	return _install_special_mapping(mm, addr, VDSO_NR_PAGES * PAGE_SIZE,
-					VM_READ | VM_MAYREAD | VM_IO | VM_DONTDUMP |
+					VM_READ | VM_MAYREAD | VM_DONTDUMP |
 					VM_MIXEDMAP | VM_SEALED_SYSMAP,
 					&vdso_vvar_mapping);
 }

-- 
2.55.0

[tip: timers/vdso] vdso/datastore: Allow prefaulting by mlockall()

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the timers/vdso branch of tip:

Commit-ID:     c27e727c9a60e6303088a956eea45b6b5c02cef3
Gitweb:        https://git.kernel.org/tip/c27e727c9a60e6303088a956eea45b6b5c02cef3
Author:        Thomas Weißschuh <thomas.weissschuh@linutronix.de>
AuthorDate:    Tue, 30 Jun 2026 09:31:59 +02:00
Committer:     Thomas Gleixner <tglx@kernel.org>
CommitterDate: Tue, 07 Jul 2026 23:52:52 +02:00

vdso/datastore: Allow prefaulting by mlockall()

While mlockall() is meant to lock page *memory*, effectively it will
also create and lock the corresponding page table entries.
Latency-sensitive applications expect not to experience any pagefaults
after calling mlockall(). However mlockall() ignores VM_IO mappings,
which is used by the generic vDSO datastore.
While the fault handler itself is very fast, going through the full
pagefault exception handling is much slower, on the order of 20us in a
test machine.

Since the memory behind the datastore mappings is always present and
accessible it is not necessary to use VM_IO for them.

The data page mapping is now also aligned with the architecture-specific
code pages. Some architecture-specific data pages, like the x86 VCLOCK
pages, continue to use VM_IO as they are not always mappable. They will
require their own special handling later when the general approach has
been agreed upon.

As a side-effect this will allow GUP on these pages and allow more ways
to access the data in them. This is fine, as all data in this mapping
is globally visible anyways. Either because it is mapped into all tasks,
or in the case of the time namespace pages, can be read from procfs.

Regular mlock() would also work, but userspace does not know the boundaries
of the vDSO.

Reported-by: Nam Cao <namcao@linutronix.de>
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Nam Cao <namcao@linutronix.de>
Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-5-6c93708ce723@linutronix.de
---
 lib/vdso/datastore.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index 0ca4396..d7ab077 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -125,7 +125,7 @@ const struct vm_special_mapping vdso_vvar_mapping = {
 struct vm_area_struct *vdso_install_vvar_mapping(struct mm_struct *mm, unsigned long addr)
 {
 	return _install_special_mapping(mm, addr, VDSO_NR_PAGES * PAGE_SIZE,
-					VM_READ | VM_MAYREAD | VM_IO | VM_DONTDUMP |
+					VM_READ | VM_MAYREAD | VM_DONTDUMP |
 					VM_MIXEDMAP | VM_SEALED_SYSMAP,
 					&vdso_vvar_mapping);
 }

[PATCH v4 6/6] vdso/datastore: Simplify the mapping logic for VDSO_TIME_PAGE_OFFSET

[Thomas Weißschuh]

The logic for CONFIG_GENERIC_GETTIMEOFDAY=n and !timens_page is
identical now.

Use this to simplify the logic a bit.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
 lib/vdso/datastore.c | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index d7ab077cd604..1426bf4e0c12 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -76,22 +76,20 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 
 	switch (vmf->pgoff) {
 	case VDSO_TIME_PAGE_OFFSET:
-		if (!IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY))
+		if (!IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY) || !timens_page)
 			break;
-		if (timens_page) {
-			/*
-			 * Fault in VVAR page too, since it will be accessed
-			 * to get clock data anyway.
-			 */
-			unsigned long addr;
-			vm_fault_t err;
-
-			addr = vmf->address + VDSO_TIMENS_PAGE_OFFSET * PAGE_SIZE;
-			err = vmf_insert_page(vma, addr, page);
-			if (unlikely(err & VM_FAULT_ERROR))
-				return err;
-			page = timens_page;
-		}
+		/*
+		 * Fault in VVAR page too, since it will be accessed
+		 * to get clock data anyway.
+		 */
+		unsigned long addr;
+		vm_fault_t err;
+
+		addr = vmf->address + VDSO_TIMENS_PAGE_OFFSET * PAGE_SIZE;
+		err = vmf_insert_page(vma, addr, page);
+		if (unlikely(err & VM_FAULT_ERROR))
+			return err;
+		page = timens_page;
 		break;
 	case VDSO_TIMENS_PAGE_OFFSET:
 		/*

-- 
2.55.0

[tip: timers/vdso] vdso/datastore: Simplify the mapping logic for VDSO_TIME_PAGE_OFFSET

[tip-bot2 for Thomas Weißschuh]

The following commit has been merged into the timers/vdso branch of tip:

Commit-ID:     0b50763e8439b171d8fe2c6df0d27705fa89f21a
Gitweb:        https://git.kernel.org/tip/0b50763e8439b171d8fe2c6df0d27705fa89f21a
Author:        Thomas Weißschuh <thomas.weissschuh@linutronix.de>
AuthorDate:    Tue, 30 Jun 2026 09:32:00 +02:00
Committer:     Thomas Gleixner <tglx@kernel.org>
CommitterDate: Tue, 07 Jul 2026 23:52:53 +02:00

vdso/datastore: Simplify the mapping logic for VDSO_TIME_PAGE_OFFSET

The logic for CONFIG_GENERIC_GETTIMEOFDAY=n and !timens_page is
identical now.

Use this to simplify the logic a bit.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-6-6c93708ce723@linutronix.de
---
 lib/vdso/datastore.c | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c
index d7ab077..1426bf4 100644
--- a/lib/vdso/datastore.c
+++ b/lib/vdso/datastore.c
@@ -76,22 +76,20 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
 
 	switch (vmf->pgoff) {
 	case VDSO_TIME_PAGE_OFFSET:
-		if (!IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY))
+		if (!IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY) || !timens_page)
 			break;
-		if (timens_page) {
-			/*
-			 * Fault in VVAR page too, since it will be accessed
-			 * to get clock data anyway.
-			 */
-			unsigned long addr;
-			vm_fault_t err;
-
-			addr = vmf->address + VDSO_TIMENS_PAGE_OFFSET * PAGE_SIZE;
-			err = vmf_insert_page(vma, addr, page);
-			if (unlikely(err & VM_FAULT_ERROR))
-				return err;
-			page = timens_page;
-		}
+		/*
+		 * Fault in VVAR page too, since it will be accessed
+		 * to get clock data anyway.
+		 */
+		unsigned long addr;
+		vm_fault_t err;
+
+		addr = vmf->address + VDSO_TIMENS_PAGE_OFFSET * PAGE_SIZE;
+		err = vmf_insert_page(vma, addr, page);
+		if (unlikely(err & VM_FAULT_ERROR))
+			return err;
+		page = timens_page;
 		break;
 	case VDSO_TIMENS_PAGE_OFFSET:
 		/*