* [PATCH 00/11] rust: page: improve access methods for racy memory regions
@ 2026-11-20 10:26 Andreas Hindborg
2026-11-20 10:27 ` [PATCH 01/11] rust: page: tighten safety requirements of `SysMem` copy methods Andreas Hindborg
` (10 more replies)
0 siblings, 11 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:26 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
`Page` has a number of methods for manipulating the memory of the page.
These use `core::ptr::{copy_nonoverlapping,write_bytes}`, which are not
valid for use when one end of the operation may be subject to concurrent
reads or writes.
With recent patches that allow pages to be exposed as `SysMem` regions, we
can use the `SysMem` infrastructure to do the operations as byte-wise
atomic operations, thus making the operations well defined.
This series builds on the atomic memcpy series [1], the `Ownable` series
[2] and the `Page` refactor series [3] to prepare for handling the pages of
a `Bio` in a well defined way. As part of this work:
Remove the `core::ptr` memory operations from page.
Add a safe method to get a `SysMem` view of a page and use that when
copying into pages from `UserSliceReader`. Remove code paths in
`UserSliceReader` that would make a `&mut [MaybeUninit<u8>]` to page
contents.
As `SysMem` is `Send` and `Sync` but pages are mapped with `kmap_local`,
add a `SysMemLocal` variant that is neither `Send` nor `Sync` and use this
for the views of page data.
Finally, remove `Page::copy_from_user_slice_raw` as it is no longer used.
The three dependency series do not compose without conflicts, so I have
provided my b4 tree with all of them applied [4].
[1] https://lore.kernel.org/all/20261113-page-volatile-io-v6-0-5603e527c919@kernel.org
[2] https://lore.kernel.org/all/20261116-unique-ref-v22-0-12029383206d@kernel.org
[3] https://lore.kernel.org/all/20261118-page-ownable-v22-0-d63ff73f4075@kernel.org
[4] https://github.com/metaspace/linux/tree/page-shared
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
Andreas Hindborg (11):
rust: page: tighten safety requirements of `SysMem` copy methods
rust: sync: atomic: add memset
rust: io: add `SysMem::fill_zero`
rust: page: use byte-wise atomic operations in raw accessors
rust: uaccess: take a raw pointer in `UserSliceReader::read_raw`
rust: uaccess: add `UserSliceReader::read_sysmem`
rust: page: add `Shared` page kind
rust: io: add `SysMemLocal`
rust: page: add `Page::with_page_as_sysmem_local`
rust_binder: access page data through `SysMemLocal`
rust: page: remove `Page::copy_from_user_slice_raw`
drivers/android/binder/page_range.rs | 35 ++--
rust/kernel/io.rs | 151 +++++++++++++--
rust/kernel/page.rs | 355 +++++++++++++++++++++++++----------
rust/kernel/sync/atomic.rs | 25 +++
rust/kernel/uaccess.rs | 79 ++++++--
5 files changed, 496 insertions(+), 149 deletions(-)
---
base-commit: fd9230005e85472b81c1c7c16afe93dd878996b1
change-id: 20261119-page-shared-9b748d499e3e
prerequisite-change-id: 20260130-page-volatile-io-05ff595507d3:v6
prerequisite-patch-id: 112f7ee8c101c766ba33128a08c053ff7dd1bd6f
prerequisite-patch-id: 4b725c8c4d3313096ea722ac2866bd5e36227b88
prerequisite-patch-id: bf25326b450044d86f2cc7392f67d17d674afe8a
prerequisite-patch-id: acbb5d7f331c45b8da915ea32bef91447ac0b234
prerequisite-change-id: 20250305-unique-ref-29fcd675f9e9:v22
prerequisite-patch-id: 4e3c7a1a2077aaa0ece2d557fa73486355903ae5
prerequisite-patch-id: d1073c5dcd6287fbcba7848a5d182108930230c1
prerequisite-patch-id: 3244def11bbdf2eabaa6e94e15ce06540b2f5814
prerequisite-patch-id: cef14c5bef5ba519ceaf0a0e7c9e93ef974db7b3
prerequisite-patch-id: 56410da35ed8dbe7b210adae0985af705cc0b5b0
prerequisite-change-id: 20261112-page-ownable-2e97941a83b2:v22
prerequisite-patch-id: 4e3c7a1a2077aaa0ece2d557fa73486355903ae5
prerequisite-patch-id: d1073c5dcd6287fbcba7848a5d182108930230c1
prerequisite-patch-id: 3244def11bbdf2eabaa6e94e15ce06540b2f5814
prerequisite-patch-id: cef14c5bef5ba519ceaf0a0e7c9e93ef974db7b3
prerequisite-patch-id: 56410da35ed8dbe7b210adae0985af705cc0b5b0
prerequisite-patch-id: 7454d72e9d18045c4b45c3be614c51288c7c437a
prerequisite-patch-id: edcc308e1ba9596e7939859cb50230ca56f0b265
prerequisite-patch-id: 90fcb80e26232818fea9472fa83871b6577bbddd
prerequisite-patch-id: 10e8ae82a9750cd76b42754393d3e39b9e849474
prerequisite-patch-id: 881055f06785a8df0ec9f8107065428900ca78d9
prerequisite-patch-id: e0759f2b41824d562a594c9e2452bf41ba514cbd
Best regards,
--
Andreas Hindborg <a.hindborg@kernel.org>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 01/11] rust: page: tighten safety requirements of `SysMem` copy methods
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 02/11] rust: sync: atomic: add memset Andreas Hindborg
` (9 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
`Page::read_sysmem` and `Page::write_sysmem` are available for all page
kinds, so they can be called on a `Page<Exclusive>`, whose data can be
referenced as a slice. Add the requirements that the raw accessors of
`Page` have for this case: no mutable reference to the data that is read
may exist, no reference to the data that is written may exist, and data
written to a `Page<Exclusive>` must be initialized.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/page.rs | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index 4ea7368b4699..3c48ecde4d14 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -374,6 +374,8 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
/// Callers must ensure that:
///
/// - This call does not race with a write to the source page that overlaps with this read.
+ /// - No mutable reference to the data of the page that overlaps with this read exists during
+ /// this call.
/// - The memory region of `dst` does not overlap with the memory of the page.
///
/// # Examples
@@ -391,7 +393,8 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
/// // way while `dst` is alive.
/// let dst = unsafe { SysMem::new(&raw mut buf as *mut [u8]) };
///
- /// // SAFETY: `page` is not written to during this call, and `buf` is not part of the page.
+ /// // SAFETY: `page` is not written to during this call, no reference to the data of `page`
+ /// // exists, and `buf` is not part of the page.
/// unsafe { page.read_sysmem(dst, 0)? };
/// assert_eq!(buf, [0u8; 16]);
/// # Ok::<(), Error>(())
@@ -453,6 +456,10 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res
///
/// - This call does not race with a read or write to the destination page that overlaps with
/// this write.
+ /// - No reference to the data of the page that overlaps with this write exists during this
+ /// call. Such references can be obtained from a [`Page<Exclusive>`], see
+ /// [`Page::as_slice`].
+ /// - If the page is a [`Page<Exclusive>`], the bytes of `src` are initialized.
/// - The memory region of `src` does not overlap with the memory of the page.
///
/// # Examples
@@ -476,8 +483,8 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res
/// )
/// };
///
- /// // SAFETY: `page` is not accessed in any other way during these calls, and neither `data`
- /// // nor `buf` is part of the page.
+ /// // SAFETY: `page` is not accessed in any other way during these calls, no reference to the
+ /// // data of `page` exists, and neither `data` nor `buf` is part of the page.
/// unsafe {
/// page.write_sysmem(src, 0)?;
/// page.read_sysmem(dst, 0)?;
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 02/11] rust: sync: atomic: add memset
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
2026-11-20 10:27 ` [PATCH 01/11] rust: page: tighten safety requirements of `SysMem` copy methods Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 03/11] rust: io: add `SysMem::fill_zero` Andreas Hindborg
` (8 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
Add `atomic::memset()`, the byte-wise atomic counterpart of
`core::ptr::write_bytes()`. Just as `atomic::memcpy()` is implemented in
terms of the C `memcpy()` function, `atomic::memset()` is implemented in
terms of the C `memset()` function.
This is used to zero the data of a page that is subject to concurrent
accesses.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/sync/atomic.rs | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/rust/kernel/sync/atomic.rs b/rust/kernel/sync/atomic.rs
index d25627fa6557..ca4ea8cadae1 100644
--- a/rust/kernel/sync/atomic.rs
+++ b/rust/kernel/sync/atomic.rs
@@ -884,3 +884,28 @@ pub unsafe fn memcpy(src: *const u8, dst: *mut u8, len: usize) {
)
};
}
+
+/// Set `len` bytes at `dst` to `value` using byte-wise atomic operations.
+///
+/// This is the concurrent-safe counterpart of `core::ptr::write_bytes()`. Because of the atomicity
+/// at byte level, when racing with another concurrent atomic access, or with an external access
+/// (from DMA or userspace), the behavior of this function is defined: the memory is written at (at
+/// least) byte granularity.
+///
+/// Implementation note: this is currently implemented by the kernel's `memset()`, which is
+/// implemented in a way such that byte-wise atomic memory store instructions are used.
+///
+/// # Safety
+///
+/// Callers must ensure that `dst` is valid for atomic writes for `len` bytes for the duration of
+/// the call.
+#[inline]
+pub unsafe fn memset(dst: *mut u8, value: u8, len: usize) {
+ // SAFETY: By the safety requirements of this function, the following operation will not:
+ // - Trap.
+ // - Invalidate any reference invariants.
+ // - Race with any operation by the Rust AM, as `bindings::memset` is a byte-wise atomic
+ // operation and all operations by the Rust AM to the involved memory area use byte-wise
+ // atomic semantics.
+ unsafe { bindings::memset(dst.cast::<kernel::ffi::c_void>(), value.into(), len) };
+}
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 03/11] rust: io: add `SysMem::fill_zero`
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
2026-11-20 10:27 ` [PATCH 01/11] rust: page: tighten safety requirements of `SysMem` copy methods Andreas Hindborg
2026-11-20 10:27 ` [PATCH 02/11] rust: sync: atomic: add memset Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 04/11] rust: page: use byte-wise atomic operations in raw accessors Andreas Hindborg
` (7 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
Add a method for zeroing the memory of a `SysMem` byte view. The method
uses `atomic::memset()`, so the memory can be subject to concurrent
accesses.
This is used to zero the data of a page that is mapped to user space or
shared with a device.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/io.rs | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/rust/kernel/io.rs b/rust/kernel/io.rs
index 76904c63eb0a..33cb5a9fb3a8 100644
--- a/rust/kernel/io.rs
+++ b/rust/kernel/io.rs
@@ -1586,6 +1586,17 @@ unsafe impl<T: ?Sized + Sync> Send for SysMem<'_, T> {}
// SAFETY: `SysMem<'_, T>` is conceptually `&T`.
unsafe impl<T: ?Sized + Sync> Sync for SysMem<'_, T> {}
+impl SysMem<'_, [u8]> {
+ /// Writes zeroes to all bytes of the view using byte-wise atomic operations.
+ #[inline]
+ pub fn fill_zero(self) {
+ // SAFETY: Per type invariant, `self.ptr` is a valid kernel accessible memory region of
+ // `self.ptr.len()` bytes, and no plain access races with this write, so it is valid for
+ // atomic writes.
+ unsafe { atomic::memset(self.ptr.cast(), 0, self.ptr.len()) };
+ }
+}
+
impl<'a, T: ?Sized> SysMem<'a, T> {
/// Create a `SysMem` from a raw pointer.
///
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 04/11] rust: page: use byte-wise atomic operations in raw accessors
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
` (2 preceding siblings ...)
2026-11-20 10:27 ` [PATCH 03/11] rust: io: add `SysMem::fill_zero` Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 05/11] rust: uaccess: take a raw pointer in `UserSliceReader::read_raw` Andreas Hindborg
` (6 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
The raw accessors of `Page` are used to access pages that may be mapped to
user space or shared with devices. A plain access to such a page is a data
race when the other party writes to the page.
Add the private method `Page::with_page_as_sysmem()`, which maps a range
of a page and passes it to a closure as a `SysMem` view. Access the data
of the page through this view in `Page::read_raw()`, `Page::write_raw()`
and `Page::fill_zero_raw()`, so that all accesses are byte-wise atomic.
Zero the data with `SysMem::fill_zero()`.
Narrow the safety requirements of these methods, and of
`Page::read_sysmem()` and `Page::write_sysmem()`, accordingly: a call must
not race with a plain access to the same data, but it is allowed to race
with other non-plain accesses.
`Page::copy_from_user_slice_raw()` still writes to the page with plain
accesses. State that in the safety requirements.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/page.rs | 153 ++++++++++++++++++++++++++++++++++++----------------
1 file changed, 107 insertions(+), 46 deletions(-)
diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index 3c48ecde4d14..ff191bfcd3f9 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -334,28 +334,74 @@ fn with_pointer_into_page<T>(
}
}
+ /// Runs a piece of code with a range of this page mapped as a [`SysMem`] view.
+ ///
+ /// If `f` is called, then it is called with a view of the `len` bytes of the page that start
+ /// at `offset`. Accesses through the view are volatile or byte-wise atomic, so the view is
+ /// suitable for a page whose data is subject to concurrent writes by hardware or user space.
+ /// The view is valid for the duration of the call to `f`.
+ ///
+ /// If `offset` and `len` refer to a region outside of this page, then this method returns
+ /// [`EINVAL`] and does not call `f`.
+ ///
+ /// # Safety
+ ///
+ /// * Callers must ensure that no plain access to the mapped range races with an access
+ /// performed through the view, see [`SysMem`].
+ /// * Callers must ensure that no mutable reference to the data of the page that overlaps with
+ /// the mapped range exists during this call. If `f` writes through the view, no reference to
+ /// that data may exist during this call. Such references can be obtained from a
+ /// [`Page<Exclusive>`], see [`Page::as_slice`].
+ /// * If the page is a [`Page<Exclusive>`], `f` must only write initialized bytes through the
+ /// view.
+ /// * Callers must ensure that the view is only used on the current task, as this method may
+ /// use a local mapping.
+ unsafe fn with_page_as_sysmem<T>(
+ &self,
+ offset: usize,
+ len: usize,
+ f: impl FnOnce(SysMem<'_, [u8]>) -> Result<T>,
+ ) -> Result<T> {
+ self.with_pointer_into_page(offset, len, move |ptr| {
+ // SAFETY:
+ // - If `with_pointer_into_page` calls into this closure, then it has performed a
+ // bounds check and guarantees that `ptr` is valid for `len` bytes for the duration
+ // of the closure. A byte slice has an alignment of 1.
+ // - By the safety requirements of this function, no plain access to the range races
+ // with an access through the view.
+ f(unsafe { SysMem::new(ptr::slice_from_raw_parts_mut(ptr, len)) })
+ })
+ }
+
/// Maps the page and reads from it into the given buffer.
///
/// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
/// outside of the page, then this call returns [`EINVAL`].
///
+ /// The copy is byte-wise atomic, so it is well-defined even if the data of the page is
+ /// concurrently written by hardware, by user space, or by other kernel code using atomic or
+ /// volatile accesses. In that case, the data that is read may be arbitrary.
+ ///
/// # Safety
///
- /// * Callers must ensure that `dst` is valid for writing `len` bytes.
- /// * Callers must ensure that this call does not race with a write to the same page that
- /// overlaps with this read.
+ /// * Callers must ensure that `dst` is valid for writing `len` bytes, and that this memory
+ /// does not overlap with the data of the page.
+ /// * Callers must ensure that this call does not race with a plain write to the same page that
+ /// overlaps with this read, see [`SysMem`].
/// * Callers must ensure that no mutable reference to the data of the page that overlaps with
/// this read exists during this call.
pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
- self.with_pointer_into_page(offset, len, move |src| {
- // SAFETY: If `with_pointer_into_page` calls into this closure, then
- // it has performed a bounds check and guarantees that `src` is
- // valid for `len` bytes.
- //
- // There caller guarantees that there is no data race.
- unsafe { ptr::copy_nonoverlapping(src, dst, len) };
+ let copy = |src: SysMem<'_, [u8]>| {
+ // SAFETY: By the safety requirements of this function, `dst` is valid for writing
+ // `len` bytes, which is equal to `src.size()`, and it does not overlap with `src`.
+ unsafe { SysMemBackend::copy_from_io(src, dst) };
Ok(())
- })
+ };
+
+ // SAFETY: By the safety requirements of this function, no plain write races with the read
+ // through the view, and no mutable reference to the data that is read exists. `copy` does
+ // not write through the view, and it only uses the view on the current task.
+ unsafe { self.with_page_as_sysmem(offset, len, copy) }
}
/// Maps the page and reads from it into the given system memory region using byte-wise atomic
@@ -366,14 +412,14 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
///
/// The writes to `dst` are byte-wise atomic, so they are well-defined even if `dst` is
/// concurrently accessed by hardware, by user space, or by other kernel code using atomic or
- /// volatile accesses. The reads from the page are plain accesses, so the page must not be
- /// written to concurrently.
+ /// volatile accesses. The reads from the page are byte-wise atomic as well.
///
/// # Safety
///
/// Callers must ensure that:
///
- /// - This call does not race with a write to the source page that overlaps with this read.
+ /// - This call does not race with a plain write to the source page that overlaps with this
+ /// read, see [`SysMem`].
/// - No mutable reference to the data of the page that overlaps with this read exists during
/// this call.
/// - The memory region of `dst` does not overlap with the memory of the page.
@@ -393,8 +439,8 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
/// // way while `dst` is alive.
/// let dst = unsafe { SysMem::new(&raw mut buf as *mut [u8]) };
///
- /// // SAFETY: `page` is not written to during this call, no reference to the data of `page`
- /// // exists, and `buf` is not part of the page.
+ /// // SAFETY: No plain write to `page` happens during this call, no reference to the data of
+ /// // `page` exists, and `buf` is not part of the page.
/// unsafe { page.read_sysmem(dst, 0)? };
/// assert_eq!(buf, [0u8; 16]);
/// # Ok::<(), Error>(())
@@ -405,8 +451,8 @@ pub unsafe fn read_sysmem(&self, dst: SysMem<'_, [u8]>, offset: usize) -> Result
// - If `with_pointer_into_page` calls into this closure, then it has performed a
// bounds check and guarantees that `src` is valid for `dst.len()` bytes, which is
// equal to `dst.size()`.
- // - By the safety requirements of this function, the page is not written to during
- // this call, so `src` is valid for volatile reads.
+ // - By the safety requirements of this function, no plain write to the page races
+ // with this call, so `src` is valid for volatile reads.
// - By the safety requirements of this function, `src` does not overlap with `dst`.
unsafe { SysMemBackend::copy_to_io(dst, src) };
Ok(())
@@ -418,25 +464,34 @@ pub unsafe fn read_sysmem(&self, dst: SysMem<'_, [u8]>, offset: usize) -> Result
/// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
/// outside of the page, then this call returns [`EINVAL`].
///
+ /// The copy is byte-wise atomic, so it is well-defined even if the data of the page is
+ /// concurrently accessed by hardware, by user space, or by other kernel code using atomic or
+ /// volatile accesses.
+ ///
/// # Safety
///
- /// * Callers must ensure that `src` is valid for reading `len` bytes.
- /// * Callers must ensure that this call does not race with a read or write to the same page
- /// that overlaps with this write.
+ /// * Callers must ensure that `src` is valid for reading `len` bytes, and that this memory
+ /// does not overlap with the data of the page.
+ /// * Callers must ensure that this call does not race with a plain read or write to the same
+ /// page that overlaps with this write, see [`SysMem`].
/// * Callers must ensure that no reference to the data of the page that overlaps with this
/// write exists during this call. Such references can be obtained from a
/// [`Page<Exclusive>`], see [`Page::as_slice`].
/// * If the page is a [`Page<Exclusive>`], callers must ensure that the `len` bytes at `src`
/// are initialized.
pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Result {
- self.with_pointer_into_page(offset, len, move |dst| {
- // SAFETY: If `with_pointer_into_page` calls into this closure, then it has performed a
- // bounds check and guarantees that `dst` is valid for `len` bytes.
- //
- // There caller guarantees that there is no data race.
- unsafe { ptr::copy_nonoverlapping(src, dst, len) };
+ let copy = |dst: SysMem<'_, [u8]>| {
+ // SAFETY: By the safety requirements of this function, `src` is valid for reading
+ // `len` bytes, which is equal to `dst.size()`, and it does not overlap with `dst`.
+ unsafe { SysMemBackend::copy_to_io(dst, src) };
Ok(())
- })
+ };
+
+ // SAFETY: By the safety requirements of this function, no plain access races with the
+ // write through the view, no reference to the data that is written exists, and the bytes
+ // written are initialized if the page requires it. `copy` only uses the view on the
+ // current task.
+ unsafe { self.with_page_as_sysmem(offset, len, copy) }
}
/// Maps the page and writes into it from the given system memory region using byte-wise atomic
@@ -447,15 +502,14 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res
///
/// The reads from `src` are byte-wise atomic, so they are well-defined even if `src` is
/// concurrently accessed by hardware, by user space, or by other kernel code using atomic or
- /// volatile accesses. The writes to the page are plain accesses, so the page must not be read
- /// from or written to concurrently.
+ /// volatile accesses. The writes to the page are byte-wise atomic as well.
///
/// # Safety
///
/// Callers must ensure that:
///
- /// - This call does not race with a read or write to the destination page that overlaps with
- /// this write.
+ /// - This call does not race with a plain read or write to the destination page that overlaps
+ /// with this write, see [`SysMem`].
/// - No reference to the data of the page that overlaps with this write exists during this
/// call. Such references can be obtained from a [`Page<Exclusive>`], see
/// [`Page::as_slice`].
@@ -483,8 +537,9 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res
/// )
/// };
///
- /// // SAFETY: `page` is not accessed in any other way during these calls, no reference to the
- /// // data of `page` exists, and neither `data` nor `buf` is part of the page.
+ /// // SAFETY: No plain access to `page` happens during these calls, no reference to the data
+ /// // of `page` exists, `data` is initialized, and neither `data` nor `buf` is part of the
+ /// // page.
/// unsafe {
/// page.write_sysmem(src, 0)?;
/// page.read_sysmem(dst, 0)?;
@@ -498,8 +553,8 @@ pub unsafe fn write_sysmem(&self, src: SysMem<'_, [u8]>, offset: usize) -> Resul
// - If `with_pointer_into_page` calls into this closure, then it has performed a
// bounds check and guarantees that `dst` is valid for `src.len()` bytes, which is
// equal to `src.size()`.
- // - By the safety requirements of this function, the page is not read from or written
- // to during this call, so `dst` is valid for volatile writes.
+ // - By the safety requirements of this function, no plain access to the page races
+ // with this call, so `dst` is valid for volatile writes.
// - By the safety requirements of this function, `dst` does not overlap with `src`.
unsafe { SysMemBackend::copy_from_io(src, dst) };
Ok(())
@@ -511,22 +566,27 @@ pub unsafe fn write_sysmem(&self, src: SysMem<'_, [u8]>, offset: usize) -> Resul
/// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
/// outside of the page, then this call returns [`EINVAL`].
///
+ /// The writes are byte-wise atomic, so they are well-defined even if the data of the page is
+ /// concurrently accessed by hardware, by user space, or by other kernel code using atomic or
+ /// volatile accesses.
+ ///
/// # Safety
///
- /// * Callers must ensure that this call does not race with a read or write to the same page
- /// that overlaps with this write.
+ /// * Callers must ensure that this call does not race with a plain read or write to the same
+ /// page that overlaps with this write, see [`SysMem`].
/// * Callers must ensure that no reference to the data of the page that overlaps with this
/// write exists during this call. Such references can be obtained from a
/// [`Page<Exclusive>`], see [`Page::as_slice`].
pub unsafe fn fill_zero_raw(&self, offset: usize, len: usize) -> Result {
- self.with_pointer_into_page(offset, len, move |dst| {
- // SAFETY: If `with_pointer_into_page` calls into this closure, then it has performed a
- // bounds check and guarantees that `dst` is valid for `len` bytes.
- //
- // There caller guarantees that there is no data race.
- unsafe { ptr::write_bytes(dst, 0u8, len) };
+ let fill = |dst: SysMem<'_, [u8]>| {
+ dst.fill_zero();
Ok(())
- })
+ };
+
+ // SAFETY: By the safety requirements of this function, no plain access races with the
+ // write through the view, and no reference to the data that is written exists. The bytes
+ // written are initialized. `fill` only uses the view on the current task.
+ unsafe { self.with_page_as_sysmem(offset, len, fill) }
}
/// Copies data from userspace into this page.
@@ -540,7 +600,8 @@ pub unsafe fn fill_zero_raw(&self, offset: usize, len: usize) -> Result {
/// # Safety
///
/// * Callers must ensure that this call does not race with a read or write to the same page
- /// that overlaps with this write.
+ /// that overlaps with this write. This includes byte-wise atomic accesses, as this method
+ /// writes to the page with plain accesses.
/// * Callers must ensure that no reference to the data of the page that overlaps with this
/// write exists during this call. Such references can be obtained from a
/// [`Page<Exclusive>`], see [`Page::as_slice`].
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 05/11] rust: uaccess: take a raw pointer in `UserSliceReader::read_raw`
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
` (3 preceding siblings ...)
2026-11-20 10:27 ` [PATCH 04/11] rust: page: use byte-wise atomic operations in raw accessors Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 06/11] rust: uaccess: add `UserSliceReader::read_sysmem` Andreas Hindborg
` (5 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
`UserSliceReader::read_raw()` takes the destination as a mutable slice.
Thus, it cannot be used to copy into memory that is subject to concurrent
accesses, such as a page that is mapped to user space, since a mutable
reference to such memory must not exist.
Change `UserSliceReader::read_raw()` to take the destination as a raw
pointer and a length, and make it unsafe. The copy is done by
`copy_from_user()`, which uses byte-wise atomic store instructions for the
kernel memory, so the destination only has to be valid for atomic writes.
Update the callers. `UserSliceReader::read_slice()` remains as the safe
method for copying into a slice.
`Page::copy_from_user_slice_raw()` no longer creates a mutable slice of
the data of the page, and thus no longer writes to the page with plain
accesses. Remove the remark about this from its safety requirements.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/page.rs | 9 ++++-----
rust/kernel/uaccess.rs | 47 ++++++++++++++++++++++++++++++++---------------
2 files changed, 36 insertions(+), 20 deletions(-)
diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index ff191bfcd3f9..643af7cfbef4 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -600,8 +600,7 @@ pub unsafe fn fill_zero_raw(&self, offset: usize, len: usize) -> Result {
/// # Safety
///
/// * Callers must ensure that this call does not race with a read or write to the same page
- /// that overlaps with this write. This includes byte-wise atomic accesses, as this method
- /// writes to the page with plain accesses.
+ /// that overlaps with this write.
/// * Callers must ensure that no reference to the data of the page that overlaps with this
/// write exists during this call. Such references can be obtained from a
/// [`Page<Exclusive>`], see [`Page::as_slice`].
@@ -613,9 +612,9 @@ pub unsafe fn copy_from_user_slice_raw(
) -> Result {
self.with_pointer_into_page(offset, len, move |dst| {
// SAFETY: If `with_pointer_into_page` calls into this closure, then it has performed a
- // bounds check and guarantees that `dst` is valid for `len` bytes. Furthermore, we have
- // exclusive access to the slice since the caller guarantees that there are no races.
- reader.read_raw(unsafe { core::slice::from_raw_parts_mut(dst.cast(), len) })
+ // bounds check and guarantees that `dst` is valid for `len` bytes. The caller
+ // guarantees that there are no races, so `dst` is valid for atomic writes.
+ unsafe { reader.read_raw(dst, len) }
})
}
}
diff --git a/rust/kernel/uaccess.rs b/rust/kernel/uaccess.rs
index f1aecda7a90c..b1df79500b98 100644
--- a/rust/kernel/uaccess.rs
+++ b/rust/kernel/uaccess.rs
@@ -252,25 +252,38 @@ pub fn is_empty(&self) -> bool {
self.length == 0
}
- /// Reads raw data from the user slice into a kernel buffer.
+ /// Reads raw data from the user slice into the kernel memory at `out`.
///
- /// For a version that uses `&mut [u8]`, please see [`UserSliceReader::read_slice`].
+ /// The writes to `out` are byte-wise atomic, so they are well-defined even if the memory at
+ /// `out` is concurrently accessed by hardware, by user space, or by other kernel code using
+ /// atomic or volatile accesses. For a safe version that uses `&mut [u8]`, please see
+ /// [`UserSliceReader::read_slice`].
///
/// Fails with [`EFAULT`] if the read happens on a bad address, or if the read goes out of
- /// bounds of this [`UserSliceReader`]. This call may modify `out` even if it returns an error.
+ /// bounds of this [`UserSliceReader`]. This call may modify the memory at `out` even if it
+ /// returns an error.
+ ///
+ /// Implementation note: this is implemented by the kernel's `copy_from_user()`, which is
+ /// implemented in a way such that byte-wise atomic memory store instructions are used.
///
/// # Guarantees
///
- /// After a successful call to this method, all bytes in `out` are initialized.
- pub fn read_raw(&mut self, out: &mut [MaybeUninit<u8>]) -> Result {
- let len = out.len();
- let out_ptr = out.as_mut_ptr().cast::<c_void>();
+ /// This method only writes initialized bytes to `out`. After a successful call to this method,
+ /// all `len` bytes at `out` are initialized.
+ ///
+ /// # Safety
+ ///
+ /// Callers must ensure that `out` is valid for atomic writes for `len` bytes for the duration
+ /// of the call.
+ pub unsafe fn read_raw(&mut self, out: *mut u8, len: usize) -> Result {
if len > self.length {
return Err(EFAULT);
}
- // SAFETY: `out_ptr` points into a mutable slice of length `len`, so we may write
- // that many bytes to it.
- let res = unsafe { bindings::copy_from_user(out_ptr, self.ptr.as_const_ptr(), len) };
+ // SAFETY: By the safety requirements of this function, `out` is valid for atomic writes
+ // for `len` bytes. `copy_from_user()` is a byte-wise atomic operation on the kernel
+ // memory.
+ let res =
+ unsafe { bindings::copy_from_user(out.cast::<c_void>(), self.ptr.as_const_ptr(), len) };
if res != 0 {
return Err(EFAULT);
}
@@ -284,10 +297,10 @@ pub fn read_raw(&mut self, out: &mut [MaybeUninit<u8>]) -> Result {
/// Fails with [`EFAULT`] if the read happens on a bad address, or if the read goes out of
/// bounds of this [`UserSliceReader`]. This call may modify `out` even if it returns an error.
pub fn read_slice(&mut self, out: &mut [u8]) -> Result {
- // SAFETY: The types are compatible and `read_raw` doesn't write uninitialized bytes to
- // `out`.
- let out = unsafe { &mut *(core::ptr::from_mut(out) as *mut [MaybeUninit<u8>]) };
- self.read_raw(out)
+ // SAFETY: `out` is a mutable slice, so it is valid for writing `out.len()` bytes, and
+ // nothing else accesses it during this call. `read_raw` doesn't write uninitialized bytes
+ // to `out`.
+ unsafe { self.read_raw(out.as_mut_ptr(), out.len()) }
}
/// Reads raw data from the user slice into a kernel buffer partially.
@@ -372,7 +385,11 @@ pub fn read_all<A: Allocator>(mut self, buf: &mut Vec<u8, A>, flags: Flags) -> R
buf.reserve(len, flags)?;
// The call to `reserve` was successful, so the spare capacity is at least `len` bytes long.
- self.read_raw(&mut buf.spare_capacity_mut()[..len])?;
+ let spare = &mut buf.spare_capacity_mut()[..len];
+
+ // SAFETY: `spare` is a mutable slice, so it is valid for writing `spare.len()` bytes, and
+ // nothing else accesses it during this call.
+ unsafe { self.read_raw(spare.as_mut_ptr().cast(), spare.len()) }?;
// SAFETY: Since the call to `read_raw` was successful, so the next `len` bytes of the
// vector have been initialized.
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 06/11] rust: uaccess: add `UserSliceReader::read_sysmem`
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
` (4 preceding siblings ...)
2026-11-20 10:27 ` [PATCH 05/11] rust: uaccess: take a raw pointer in `UserSliceReader::read_raw` Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 07/11] rust: page: add `Shared` page kind Andreas Hindborg
` (4 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
Add a safe method for copying data from user space to a system memory
region that is represented by a `SysMem` view. The copy is byte-wise
atomic, so the destination can be subject to concurrent accesses, such as
the data of a page that is mapped to user space.
The method builds on `UserSliceReader::read_raw()`.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/uaccess.rs | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/rust/kernel/uaccess.rs b/rust/kernel/uaccess.rs
index b1df79500b98..313479d80834 100644
--- a/rust/kernel/uaccess.rs
+++ b/rust/kernel/uaccess.rs
@@ -11,6 +11,7 @@
error::Result,
ffi::{c_char, c_void},
fs::file,
+ io::SysMem,
prelude::*,
ptr::KnownSize,
transmute::{AsBytes, FromBytes},
@@ -292,6 +293,37 @@ pub unsafe fn read_raw(&mut self, out: *mut u8, len: usize) -> Result {
Ok(())
}
+ /// Reads raw data from the user slice into the system memory region `out`.
+ ///
+ /// Reads `out.len()` bytes. The writes to `out` are byte-wise atomic, so they are well-defined
+ /// even if the memory of `out` is concurrently accessed by hardware, by user space, or by
+ /// other kernel code using atomic or volatile accesses.
+ ///
+ /// Fails with [`EFAULT`] if the read happens on a bad address, or if the read goes out of
+ /// bounds of this [`UserSliceReader`]. This call may modify the memory of `out` even if it
+ /// returns an error.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use kernel::{
+ /// io::SysMem,
+ /// uaccess::UserSliceReader, //
+ /// };
+ ///
+ /// fn fill_from_user(reader: &mut UserSliceReader, mem: SysMem<'_, [u8]>) -> Result {
+ /// reader.read_sysmem(mem)
+ /// }
+ /// ```
+ pub fn read_sysmem(&mut self, out: SysMem<'_, [u8]>) -> Result {
+ let out = out.as_ptr();
+
+ // SAFETY: By the type invariant of `SysMem`, the memory of `out` is valid for writing
+ // `out.len()` bytes, and no plain access races with this write, so it is valid for atomic
+ // writes.
+ unsafe { self.read_raw(out.cast(), out.len()) }
+ }
+
/// Reads raw data from the user slice into a kernel buffer.
///
/// Fails with [`EFAULT`] if the read happens on a bad address, or if the read goes out of
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 07/11] rust: page: add `Shared` page kind
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
` (5 preceding siblings ...)
2026-11-20 10:27 ` [PATCH 06/11] rust: uaccess: add `UserSliceReader::read_sysmem` Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 08/11] rust: io: add `SysMemLocal` Andreas Hindborg
` (3 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
A `Page<Exclusive>` hands out Rust references to the data of the page, and
it dereferences to a `Page<Base>`. Thus, no safe method on `Page<Base>`
can access the data of a page, since the access could alias a slice
obtained from a `Page<Exclusive>`.
Add a page kind `Shared` for a page whose data is initialized and is not
referenced by Rust references. This allows safe accessors that use
byte-wise atomic operations to be added for this kind.
Let `Page<RefCounted>` and `Page<Frozen>` dereference to `Page<Shared>`,
and let `Page<Shared>` dereference to `Page<Base>`. `Page<Exclusive>` keeps
dereferencing to `Page<Base>`, so it does not reach the methods of
`Page<Shared>`.
Force `__GFP_ZERO` in `Page::alloc_page()` and `Page::alloc_frozen()`, so
that the data of the allocated pages is initialized.
Require the source bytes of `Page::write_raw()` and `Page::write_sysmem()`
to be initialized for all pages that are referenced as another kind than
`Base`.
Add a TODO in binder, which writes objects that are not bounded by
`IntoBytes` to its pages.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
drivers/android/binder/page_range.rs | 6 +-
rust/kernel/page.rs | 109 ++++++++++++++++++++++++-----------
2 files changed, 78 insertions(+), 37 deletions(-)
diff --git a/drivers/android/binder/page_range.rs b/drivers/android/binder/page_range.rs
index db0853751993..bddc4538745e 100644
--- a/drivers/android/binder/page_range.rs
+++ b/drivers/android/binder/page_range.rs
@@ -588,8 +588,10 @@ pub(crate) unsafe fn write<T: ?Sized>(&self, offset: usize, obj: &T) -> Result {
let obj_ptr = ptr::from_ref(obj).cast::<u8>().add(obj_offset);
// SAFETY: We have a reference to the object, so the pointer is valid. The pages
// of this range are `Page<RefCounted>` pages, and this driver never creates a
- // reference to their data, so no reference to the data that is written exists. As
- // the page is not a `Page<Exclusive>`, the object does not have to be initialized.
+ // reference to their data, so no reference to the data that is written exists.
+ // TODO: `write_raw()` requires the bytes of the object to be initialized, as the
+ // data of a `Page<RefCounted>` is initialized. `T` is not bounded by `IntoBytes`,
+ // so this is not ensured for a `T` with padding.
page.write_raw(obj_ptr, offset, to_copy)?;
obj_offset += to_copy;
Ok(())
diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index 643af7cfbef4..85c710fa3dfd 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -104,11 +104,13 @@ pub trait Sealed {}
/// Marker trait for the kinds of a [`Page`].
///
/// A page kind describes what is known about a `struct page`, and thereby what a [`Page`] of that
-/// kind can be used for. See [`Base`], [`RefCounted`], [`Frozen`] and [`Exclusive`].
+/// kind can be used for. See [`Base`], [`Shared`], [`RefCounted`], [`Frozen`] and [`Exclusive`].
///
/// A [`Page`] of any other kind dereferences to a [`Page`] of the base kind [`Base`]. The
/// purpose of the dereference is coercion: a reference to a page of any kind can be passed to code
-/// that takes a `&Page<Base>`.
+/// that takes a `&Page<Base>`. A [`Page<RefCounted>`] and a [`Page<Frozen>`] dereference to a
+/// [`Page<Shared>`], which in turn dereferences to a [`Page<Base>`]. A [`Page<Exclusive>`]
+/// dereferences directly to a [`Page<Base>`], as Rust references to its data can exist.
pub trait PageKind: private::Sealed {}
/// The base [`PageKind`]: a valid page with no further guarantees.
@@ -126,6 +128,29 @@ pub trait PageKind: private::Sealed {}
impl private::Sealed for Base {}
impl PageKind for Base {}
+/// A [`PageKind`] for a page whose data is not subject to the Rust aliasing rules.
+///
+/// The data of a [`Page<Shared>`] can be accessed concurrently by other parties, such as user
+/// space, devices or other kernel code. Thus, no Rust reference to the data of the page exists.
+///
+/// A [`Page<RefCounted>`] and a [`Page<Frozen>`] dereference to a [`Page<Shared>`]. A page that
+/// is managed by other means can be referenced as a [`Page<Shared>`] with [`Page::from_raw`].
+///
+/// # Invariants
+///
+/// For a [`Page<Shared>`]:
+///
+/// - No Rust reference to the data of the page exists, except for the duration of an unsafe
+/// operation whose caller rules out all other accesses to that data. In particular, the page is
+/// not a [`Page<Exclusive>`].
+/// - The data of the page is initialized.
+/// - No plain access to the data of the page races with another access to that data. A plain
+/// access is a non-atomic, non-volatile access performed by kernel code.
+pub struct Shared;
+
+impl private::Sealed for Shared {}
+impl PageKind for Shared {}
+
/// The default [`PageKind`]: a page that is managed by the page reference count.
///
/// A [`Page<RefCounted>`] is accessed through a shared reference or through an owning
@@ -138,6 +163,7 @@ impl PageKind for Base {}
/// - The allocation of the page is managed by the page reference count, and the reference count is
/// non-zero.
/// - It is permitted to take additional references to the page with `get_page()`.
+/// - The invariants of [`Shared`] hold for the page.
pub struct RefCounted;
impl private::Sealed for RefCounted {}
@@ -148,13 +174,14 @@ impl PageKind for RefCounted {}
/// A `Page` is generic over a [`PageKind`], which describes what is known about the page:
///
/// - `Page<Base>` is a page for which nothing is known beyond its validity.
+/// - `Page<Shared>` is a page whose data is initialized and not referenced by Rust references.
/// - `Page<RefCounted>`, or just `Page`, is a page that is managed by the page reference count.
/// - `Page<Frozen>` is a uniquely owned page that references cannot be taken to.
/// - `Page<Exclusive>` is a uniquely owned page whose data follows standard Rust aliasing rules.
///
/// Operations that are valid for any page are available for all page kinds. All other page kinds
/// dereference to [`Page<Base>`], so that a reference to a page of any kind coerces to a
-/// `&Page<Base>`.
+/// `&Page<Base>`. See [`PageKind`] for details.
///
/// # Invariants
///
@@ -174,7 +201,7 @@ unsafe impl<Kind: PageKind> Send for Page<Kind> {}
// them concurrently is safe.
unsafe impl<Kind: PageKind> Sync for Page<Kind> {}
-impl Deref for Page<RefCounted> {
+impl Deref for Page<Shared> {
type Target = Page<Base>;
#[inline]
@@ -186,12 +213,24 @@ fn deref(&self) -> &Self::Target {
}
}
+impl Deref for Page<RefCounted> {
+ type Target = Page<Shared>;
+
+ #[inline]
+ fn deref(&self) -> &Self::Target {
+ // SAFETY: `Page` is transparent over `struct page` for all page kinds, so the pointer cast
+ // is valid. By the type invariant of `Page<RefCounted>`, the invariants of `Shared` hold
+ // for `self`.
+ unsafe { &*ptr::from_ref(self).cast() }
+ }
+}
+
impl Page<RefCounted> {
/// Allocates a new page.
///
- /// # Examples
+ /// The page is always zeroed. That is, `__GFP_ZERO` is added to `flags`.
///
- /// Allocate memory for a page.
+ /// # Examples
///
/// ```
/// use kernel::page::Page;
@@ -199,21 +238,17 @@ impl Page<RefCounted> {
/// let page = Page::alloc_page(GFP_KERNEL)?;
/// # Ok::<(), kernel::alloc::AllocError>(())
/// ```
- ///
- /// Allocate memory for a page and zero its contents.
- ///
- /// ```
- /// use kernel::page::Page;
- ///
- /// let page = Page::alloc_page(GFP_KERNEL | __GFP_ZERO)?;
- /// # Ok::<(), kernel::alloc::AllocError>(())
- /// ```
#[inline]
pub fn alloc_page(flags: Flags) -> Result<ARef<Self>, AllocError> {
+ let flags = flags | __GFP_ZERO;
+
// SAFETY: Depending on the value of `gfp_flags`, this call may sleep. Other than that, it
// is always safe to call this method.
let page = unsafe { bindings::alloc_pages(flags.as_raw(), 0) };
let page = NonNull::new(page).ok_or(AllocError)?;
+ // INVARIANT: The page was just allocated with `alloc_pages()`, so it is managed by the
+ // page reference count. We allocated with `__GFP_ZERO`, so the data of the page is
+ // initialized. The page is newly allocated, so nothing references or accesses its data.
// SAFETY: We just successfully allocated a page, so we own the reference count returned
// by `alloc_pages`, and we transfer that reference to the new `ARef<Page>`. Since `Page`
// is transparent, we can cast the pointer directly.
@@ -246,8 +281,10 @@ pub fn nid(&self) -> i32 {
/// the [`PageKind`] types.
///
/// Note that the invariants of [`RefCounted`] do not hold for all pages. For instance, pages
- /// owned by the slab allocator are not managed by the page reference count. Use [`Base`] for
- /// pages where nothing is known beyond their validity.
+ /// owned by the slab allocator are not managed by the page reference count. Likewise, the
+ /// invariants of [`Shared`] do not hold for a page whose data is referenced by Rust
+ /// references, or whose data is not initialized. Use [`Base`] for pages where nothing is
+ /// known beyond their validity.
#[inline]
pub unsafe fn from_raw<'a>(ptr: *const bindings::page) -> &'a Self {
// INVARIANT: By the function safety requirements, `ptr` refers to a valid `struct page`
@@ -352,8 +389,8 @@ fn with_pointer_into_page<T>(
/// the mapped range exists during this call. If `f` writes through the view, no reference to
/// that data may exist during this call. Such references can be obtained from a
/// [`Page<Exclusive>`], see [`Page::as_slice`].
- /// * If the page is a [`Page<Exclusive>`], `f` must only write initialized bytes through the
- /// view.
+ /// * If the page is referenced as a [`Page`] of another kind than [`Base`], `f` must only
+ /// write initialized bytes through the view.
/// * Callers must ensure that the view is only used on the current task, as this method may
/// use a local mapping.
unsafe fn with_page_as_sysmem<T>(
@@ -477,8 +514,9 @@ pub unsafe fn read_sysmem(&self, dst: SysMem<'_, [u8]>, offset: usize) -> Result
/// * Callers must ensure that no reference to the data of the page that overlaps with this
/// write exists during this call. Such references can be obtained from a
/// [`Page<Exclusive>`], see [`Page::as_slice`].
- /// * If the page is a [`Page<Exclusive>`], callers must ensure that the `len` bytes at `src`
- /// are initialized.
+ /// * If the page is referenced as a [`Page`] of another kind than [`Base`], callers must
+ /// ensure that the `len` bytes at `src` are initialized, as the invariants of those page
+ /// kinds require the data of the page to be initialized.
pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Result {
let copy = |dst: SysMem<'_, [u8]>| {
// SAFETY: By the safety requirements of this function, `src` is valid for reading
@@ -513,7 +551,8 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res
/// - No reference to the data of the page that overlaps with this write exists during this
/// call. Such references can be obtained from a [`Page<Exclusive>`], see
/// [`Page::as_slice`].
- /// - If the page is a [`Page<Exclusive>`], the bytes of `src` are initialized.
+ /// - If the page is referenced as a [`Page`] of another kind than [`Base`], the bytes of `src`
+ /// are initialized.
/// - The memory region of `src` does not overlap with the memory of the page.
///
/// # Examples
@@ -650,7 +689,8 @@ unsafe impl AlwaysRefCounted for Page {}
///
/// Unlike for a [`Page<Exclusive>`], nothing is known about who accesses the data of a
/// [`Page<Frozen>`]. For instance, the owner can share the data of the page with a device. Thus,
-/// the data is accessed through the unsafe raw accessors of [`Page`] only.
+/// no Rust reference to the data of the page exists, and a [`Page<Frozen>`] dereferences to a
+/// [`Page<Shared>`].
///
/// # Invariants
///
@@ -660,18 +700,19 @@ unsafe impl AlwaysRefCounted for Page {}
/// reference count is zero for as long as the [`Page<Frozen>`] exists.
/// - The page is owned by a unique [`Owned<Page<Frozen>>`]. No references to the page exist
/// together with the [`Owned<Page<Frozen>>`], other than the ones borrowed from it.
+/// - The invariants of [`Shared`] hold for the page.
pub struct Frozen;
impl private::Sealed for Frozen {}
impl PageKind for Frozen {}
impl Deref for Page<Frozen> {
- type Target = Page<Base>;
+ type Target = Page<Shared>;
#[inline]
fn deref(&self) -> &Self::Target {
// SAFETY: `Page` is transparent over `struct page` for all page kinds, so the pointer cast
- // is valid. `Base` has no invariants in addition to the ones of `Page`, which hold for
+ // is valid. By the type invariant of `Page<Frozen>`, the invariants of `Shared` hold for
// `self`.
unsafe { &*ptr::from_ref(self).cast() }
}
@@ -680,30 +721,28 @@ fn deref(&self) -> &Self::Target {
impl Page<Frozen> {
/// Allocates a new frozen page.
///
+ /// The page is always zeroed. That is, `__GFP_ZERO` is added to `flags`.
+ ///
/// # Examples
///
/// ```
- /// use kernel::page::{
- /// Page,
- /// PAGE_SIZE, //
- /// };
+ /// use kernel::page::Page;
///
/// let page = Page::alloc_frozen(GFP_KERNEL)?;
- ///
- /// // SAFETY: The page was just allocated and is not shared, so there are no concurrent reads
- /// // or writes to the page, and no reference to the data of the page exists.
- /// unsafe { page.fill_zero_raw(0, PAGE_SIZE)? };
- /// # Ok::<(), Error>(())
+ /// # Ok::<(), kernel::alloc::AllocError>(())
/// ```
#[inline]
pub fn alloc_frozen(flags: Flags) -> Result<Owned<Self>, AllocError> {
+ let flags = flags | __GFP_ZERO;
+
// SAFETY: Depending on the value of `gfp_flags`, this call may sleep. Other than that, it
// is always safe to call this method.
let page = unsafe { bindings::alloc_frozen_pages(flags.as_raw(), 0) };
let page = NonNull::new(page).ok_or(AllocError)?;
// INVARIANT: The page was just allocated with `alloc_frozen_pages()`, so it is frozen and
- // we own it.
+ // we own it. We allocated with `__GFP_ZERO`, so the data of the page is initialized. The
+ // page is newly allocated, so nothing references or accesses its data.
// SAFETY:
// - We just successfully allocated the page, so `page` points to a valid `struct page`.
// Since `Page` is transparent over `struct page`, we can cast the pointer directly.
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 08/11] rust: io: add `SysMemLocal`
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
` (6 preceding siblings ...)
2026-11-20 10:27 ` [PATCH 07/11] rust: page: add `Shared` page kind Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 09/11] rust: page: add `Page::with_page_as_sysmem_local` Andreas Hindborg
` (2 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
A `SysMem` view is `Send` and `Sync`. Thus, it cannot safely represent
memory that is only accessible to the current task, such as a page that is
mapped with a mapping that is local to the current task.
Make `SysMem` and `SysMemBackend` generic over a new marker trait
`SysMemScope`, with the scopes `AnyTask` and `CurrentTask`. `AnyTask` is
the default, so existing users are not affected. Only a `SysMem` of the
scope `AnyTask` is `Send` and `Sync`. All other functionality is shared
between the scopes.
Add the alias `SysMemLocal` for a `SysMem` of the scope `CurrentTask`,
together with the constructor `SysMemLocal::new_local()` and a conversion
from a `SysMem` of the default scope.
Make `SysMem::fill_zero()` and `UserSliceReader::read_sysmem()` available
for a byte view of any scope.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/io.rs | 142 ++++++++++++++++++++++++++++++++++++++++++-------
rust/kernel/uaccess.rs | 4 +-
2 files changed, 125 insertions(+), 21 deletions(-)
diff --git a/rust/kernel/io.rs b/rust/kernel/io.rs
index 33cb5a9fb3a8..1496735c08a2 100644
--- a/rust/kernel/io.rs
+++ b/rust/kernel/io.rs
@@ -1436,11 +1436,41 @@ pub fn relaxed(self) -> RelaxedMmio<'a, T> {
#[cfg(CONFIG_64BIT)]
impl_mmio_io_capable!(RelaxedMmioBackend, u64, readq_relaxed, writeq_relaxed);
+mod private {
+ pub trait Sealed {}
+}
+
+/// Marker trait for the scopes of a [`SysMem`].
+///
+/// The scope of a [`SysMem`] describes which tasks the view can be used on. See [`AnyTask`] and
+/// [`CurrentTask`].
+pub trait SysMemScope: private::Sealed {}
+
+/// The default [`SysMemScope`]: the view can be used on any task.
+///
+/// A [`SysMem`] of this scope is [`Send`] and [`Sync`] when its target is [`Sync`].
+pub enum AnyTask {}
+
+impl private::Sealed for AnyTask {}
+impl SysMemScope for AnyTask {}
+
+/// A [`SysMemScope`] for a view that can only be used on the task that created it.
+///
+/// This is the scope of a view whose memory is only accessible to the current task, such as a
+/// view that is backed by a mapping that is local to the current task. A [`SysMem`] of this scope,
+/// that is, a [`SysMemLocal`], is neither [`Send`] nor [`Sync`].
+pub enum CurrentTask {}
+
+impl private::Sealed for CurrentTask {}
+impl SysMemScope for CurrentTask {}
+
/// I/O Backend for system memory.
-pub struct SysMemBackend;
+///
+/// The backend is generic over the [`SysMemScope`] of its views.
+pub struct SysMemBackend<S: SysMemScope = AnyTask>(PhantomData<S>);
-impl IoBackend for SysMemBackend {
- type View<'a, T: ?Sized + KnownSize> = SysMem<'a, T>;
+impl<S: SysMemScope> IoBackend for SysMemBackend<S> {
+ type View<'a, T: ?Sized + KnownSize> = SysMem<'a, T, S>;
#[inline]
fn as_ptr<'a, T: ?Sized + KnownSize>(view: Self::View<'a, T>) -> *mut T {
@@ -1466,9 +1496,9 @@ unsafe fn project_view<'a, T: ?Sized + KnownSize, U: ?Sized + KnownSize>(
/// `write_volatile`.
macro_rules! impl_sysmem_io_capable {
($ty:ty) => {
- impl IoCapable<$ty> for SysMemBackend {
+ impl<S: SysMemScope> IoCapable<$ty> for SysMemBackend<S> {
#[inline]
- fn io_read(view: SysMem<'_, $ty>) -> $ty {
+ fn io_read(view: SysMem<'_, $ty, S>) -> $ty {
// SAFETY:
// - Per type invariant, `ptr` is valid and aligned.
// - Using read_volatile() here so that race with hardware is well-defined.
@@ -1480,7 +1510,7 @@ fn io_read(view: SysMem<'_, $ty>) -> $ty {
}
#[inline]
- fn io_write(view: SysMem<'_, $ty>, value: $ty) {
+ fn io_write(view: SysMem<'_, $ty, S>, value: $ty) {
// SAFETY:
// - Per type invariant, `ptr` is valid and aligned.
// - Using write_volatile() here so that race with hardware is well-defined.
@@ -1499,7 +1529,7 @@ fn io_write(view: SysMem<'_, $ty>, value: $ty) {
#[cfg(CONFIG_64BIT)]
impl_sysmem_io_capable!(u64);
-impl IoCopyable for SysMemBackend {
+impl<S: SysMemScope> IoCopyable for SysMemBackend<S> {
#[inline]
unsafe fn copy_from_io(view: Self::View<'_, [u8]>, buffer: *mut u8) {
// SAFETY:
@@ -1562,31 +1592,103 @@ fn copy_write<T: IntoBytes>(view: Self::View<'_, T>, value: T) {
/// Accesses performed by hardware or by user space are not plain accesses in this sense. Racing
/// with them is well-defined, but the data observed may be arbitrary.
///
+/// A `SysMem` is generic over a [`SysMemScope`], which describes the tasks that the view can be
+/// used on. A view of the default scope [`AnyTask`] can be sent to and shared with other tasks. A
+/// view of the scope [`CurrentTask`], a [`SysMemLocal`], cannot leave the task that created it.
+/// Apart from that, the two are identical.
+///
/// # Invariants
///
/// - `self.ptr.addr() .. self.ptr.addr() + KnownSize::size(self.ptr)` is valid and aligned kernel
-/// accessible memory region for the lifetime `'a`.
+/// accessible memory region for the lifetime `'a`. For a view of the scope [`CurrentTask`],
+/// this holds on the task that created the view.
/// - No plain access to that memory region races with an access performed through this view.
-pub struct SysMem<'a, T: ?Sized> {
+pub struct SysMem<'a, T: ?Sized, S: SysMemScope = AnyTask> {
ptr: *mut T,
- phantom: PhantomData<&'a ()>,
+ phantom: PhantomData<(&'a (), S)>,
}
-impl<T: ?Sized> Copy for SysMem<'_, T> {}
-impl<T: ?Sized> Clone for SysMem<'_, T> {
+/// A view of a system memory region that can only be used on the task that created it.
+///
+/// This is a [`SysMem`] that is neither [`Send`] nor [`Sync`]. It is used for memory that is only
+/// accessible to the current task, such as a page that is mapped with a mapping that is local to
+/// the current task.
+///
+/// A [`SysMem`] of the default scope converts to a `SysMemLocal` with [`From`].
+///
+/// # Examples
+///
+/// ```
+/// use kernel::io::{
+/// Io,
+/// SysMem,
+/// SysMemLocal, //
+/// };
+///
+/// let mut data = [0u8; 4];
+///
+/// // SAFETY: `data` is valid for the lifetime of `mem`, and it is not accessed in any other way
+/// // while `mem` is alive.
+/// let mem = unsafe { SysMem::new(&raw mut data as *mut [u8]) };
+/// let local = SysMemLocal::from(mem);
+///
+/// local.copy_from_slice(&[1, 2, 3, 4]);
+/// local.fill_zero();
+///
+/// let mut buf = [0xffu8; 4];
+/// local.copy_to_slice(&mut buf);
+/// assert_eq!(buf, [0u8; 4]);
+/// ```
+pub type SysMemLocal<'a, T> = SysMem<'a, T, CurrentTask>;
+
+impl<T: ?Sized, S: SysMemScope> Copy for SysMem<'_, T, S> {}
+impl<T: ?Sized, S: SysMemScope> Clone for SysMem<'_, T, S> {
#[inline]
fn clone(&self) -> Self {
*self
}
}
-// SAFETY: `SysMem<'_, T>` is conceptually `&T`.
-unsafe impl<T: ?Sized + Sync> Send for SysMem<'_, T> {}
+// SAFETY: `SysMem<'_, T>` is conceptually `&T`. A view of the scope `AnyTask` can be used on any
+// task.
+unsafe impl<T: ?Sized + Sync> Send for SysMem<'_, T, AnyTask> {}
-// SAFETY: `SysMem<'_, T>` is conceptually `&T`.
-unsafe impl<T: ?Sized + Sync> Sync for SysMem<'_, T> {}
+// SAFETY: `SysMem<'_, T>` is conceptually `&T`. A view of the scope `AnyTask` can be used on any
+// task.
+unsafe impl<T: ?Sized + Sync> Sync for SysMem<'_, T, AnyTask> {}
-impl SysMem<'_, [u8]> {
+impl<'a, T: ?Sized> From<SysMem<'a, T>> for SysMemLocal<'a, T> {
+ #[inline]
+ fn from(value: SysMem<'a, T>) -> Self {
+ // INVARIANT: By the type invariant of `value`, the memory region is valid on all tasks,
+ // and no plain access races with an access through a view of it.
+ Self {
+ ptr: value.ptr,
+ phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ?Sized> SysMemLocal<'a, T> {
+ /// Create a `SysMemLocal` from a raw pointer.
+ ///
+ /// # Safety
+ ///
+ /// - `ptr.addr() .. ptr.addr() + KnownSize::size(ptr)` must be valid and aligned kernel
+ /// accessible memory region on the current task for the lifetime `'a`.
+ /// - No plain access (see [`SysMem`]) to that memory region may race with an access performed
+ /// through the returned view.
+ #[inline]
+ pub unsafe fn new_local(ptr: *mut T) -> Self {
+ // INVARIANT: Per safety requirement.
+ Self {
+ ptr,
+ phantom: PhantomData,
+ }
+ }
+}
+
+impl<S: SysMemScope> SysMem<'_, [u8], S> {
/// Writes zeroes to all bytes of the view using byte-wise atomic operations.
#[inline]
pub fn fill_zero(self) {
@@ -1614,7 +1716,9 @@ pub unsafe fn new(ptr: *mut T) -> Self {
phantom: PhantomData,
}
}
+}
+impl<T: ?Sized, S: SysMemScope> SysMem<'_, T, S> {
/// Obtain the raw pointer to the memory.
#[inline]
pub fn as_ptr(self) -> *mut T {
@@ -1622,8 +1726,8 @@ pub fn as_ptr(self) -> *mut T {
}
}
-impl<'a, T: ?Sized + KnownSize> IoBase<'a> for SysMem<'a, T> {
- type Backend = SysMemBackend;
+impl<'a, T: ?Sized + KnownSize, S: SysMemScope> IoBase<'a> for SysMem<'a, T, S> {
+ type Backend = SysMemBackend<S>;
type Target = T;
#[inline]
diff --git a/rust/kernel/uaccess.rs b/rust/kernel/uaccess.rs
index 313479d80834..6956a7cc519a 100644
--- a/rust/kernel/uaccess.rs
+++ b/rust/kernel/uaccess.rs
@@ -11,7 +11,7 @@
error::Result,
ffi::{c_char, c_void},
fs::file,
- io::SysMem,
+ io::{SysMem, SysMemScope},
prelude::*,
ptr::KnownSize,
transmute::{AsBytes, FromBytes},
@@ -315,7 +315,7 @@ pub unsafe fn read_raw(&mut self, out: *mut u8, len: usize) -> Result {
/// reader.read_sysmem(mem)
/// }
/// ```
- pub fn read_sysmem(&mut self, out: SysMem<'_, [u8]>) -> Result {
+ pub fn read_sysmem<S: SysMemScope>(&mut self, out: SysMem<'_, [u8], S>) -> Result {
let out = out.as_ptr();
// SAFETY: By the type invariant of `SysMem`, the memory of `out` is valid for writing
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 09/11] rust: page: add `Page::with_page_as_sysmem_local`
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
` (7 preceding siblings ...)
2026-11-20 10:27 ` [PATCH 08/11] rust: io: add `SysMemLocal` Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 10/11] rust_binder: access page data through `SysMemLocal` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 11/11] rust: page: remove `Page::copy_from_user_slice_raw` Andreas Hindborg
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
The data of a `Page<Shared>` is initialized and is not referenced by Rust
references, so it can be accessed safely with byte-wise atomic operations.
The existing accessors of `Page` are available for all page kinds and are
thus unsafe: a write can alias a slice obtained from a `Page<Exclusive>`.
Add the safe method `Page::with_page_as_sysmem_local()` for
`Page<Shared>`. It maps a range of the page and passes it to a closure as
a `SysMemLocal` view. The view is not `Send`, since the mapping that backs
it can be local to the current task.
A `Page<Exclusive>` does not dereference to a `Page<Shared>`, so the
method is not available for it.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/page.rs | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 85 insertions(+), 4 deletions(-)
diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index 85c710fa3dfd..7fa037f5938b 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -20,7 +20,8 @@
Io,
IoCopyable,
SysMem,
- SysMemBackend, //
+ SysMemBackend,
+ SysMemLocal, //
},
sync::aref::{
self,
@@ -132,6 +133,8 @@ impl PageKind for Base {}
///
/// The data of a [`Page<Shared>`] can be accessed concurrently by other parties, such as user
/// space, devices or other kernel code. Thus, no Rust reference to the data of the page exists.
+/// The data is accessed with volatile or byte-wise atomic operations through a [`SysMemLocal`],
+/// see [`Page::with_page_as_sysmem_local`].
///
/// A [`Page<RefCounted>`] and a [`Page<Frozen>`] dereference to a [`Page<Shared>`]. A page that
/// is managed by other means can be referenced as a [`Page<Shared>`] with [`Page::from_raw`].
@@ -374,13 +377,14 @@ fn with_pointer_into_page<T>(
/// Runs a piece of code with a range of this page mapped as a [`SysMem`] view.
///
/// If `f` is called, then it is called with a view of the `len` bytes of the page that start
- /// at `offset`. Accesses through the view are volatile or byte-wise atomic, so the view is
- /// suitable for a page whose data is subject to concurrent writes by hardware or user space.
- /// The view is valid for the duration of the call to `f`.
+ /// at `offset`. The view is valid for the duration of the call to `f`.
///
/// If `offset` and `len` refer to a region outside of this page, then this method returns
/// [`EINVAL`] and does not call `f`.
///
+ /// This is the building block for the accessors of [`Page`]. Use
+ /// [`Page::with_page_as_sysmem_local`] for safe access to the data of a [`Page<Shared>`].
+ ///
/// # Safety
///
/// * Callers must ensure that no plain access to the mapped range races with an access
@@ -658,6 +662,83 @@ pub unsafe fn copy_from_user_slice_raw(
}
}
+impl Page<Shared> {
+ /// Runs a piece of code with a range of this page mapped as a [`SysMemLocal`] view.
+ ///
+ /// If `f` is called, then it is called with a view of the `len` bytes of the page that start
+ /// at `offset`. Accesses through the view are volatile or byte-wise atomic, so the view is
+ /// suitable for a page whose data is subject to concurrent accesses by hardware, by user space
+ /// or by other kernel code. When an access races with a write, the data observed may be
+ /// arbitrary. The view is valid for the duration of the call to `f`.
+ ///
+ /// The view is not [`Send`], as it can be backed by a mapping that is local to the current
+ /// task.
+ ///
+ /// If `offset` and `len` refer to a region outside of this page, then this method returns
+ /// [`EINVAL`] and does not call `f`.
+ ///
+ /// This method is not available for a [`Page<Exclusive>`], as writing through the view would
+ /// violate the aliasing rules for the slices returned by [`Page::as_slice`] and
+ /// [`Page::as_mut_slice`].
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use kernel::{
+ /// io::Io,
+ /// page::Page, //
+ /// };
+ ///
+ /// let page = Page::alloc_page(GFP_KERNEL)?;
+ /// let mut buf = [0xffu8; 4];
+ ///
+ /// page.with_page_as_sysmem_local(8, 4, |view| {
+ /// view.copy_to_slice(&mut buf);
+ /// assert_eq!(buf, [0u8; 4]);
+ ///
+ /// view.copy_from_slice(&[1, 2, 3, 4]);
+ /// view.copy_to_slice(&mut buf);
+ /// assert_eq!(buf, [1, 2, 3, 4]);
+ ///
+ /// view.fill_zero();
+ /// view.copy_to_slice(&mut buf);
+ /// assert_eq!(buf, [0u8; 4]);
+ /// Ok(())
+ /// })?;
+ /// # Ok::<(), Error>(())
+ /// ```
+ ///
+ /// Copy data from user space to a page:
+ ///
+ /// ```
+ /// use kernel::{
+ /// page::Page,
+ /// uaccess::UserSliceReader, //
+ /// };
+ ///
+ /// fn copy_to_page(page: &Page, reader: &mut UserSliceReader, len: usize) -> Result {
+ /// page.with_page_as_sysmem_local(0, len, |view| reader.read_sysmem(view))
+ /// }
+ /// ```
+ pub fn with_page_as_sysmem_local<T>(
+ &self,
+ offset: usize,
+ len: usize,
+ f: impl FnOnce(SysMemLocal<'_, [u8]>) -> Result<T>,
+ ) -> Result<T> {
+ let f = move |mem: SysMem<'_, [u8]>| f(SysMemLocal::from(mem));
+
+ // SAFETY:
+ // - By the type invariant of `Page<Shared>`, no plain access to the data of the page races
+ // with another access, and no Rust reference to the data of the page exists.
+ // - Safe code can only write initialized bytes through a `SysMemLocal`.
+ // - `f` converts the view to a `SysMemLocal` before it is used. A `SysMemLocal` is not
+ // `Send` and it cannot outlive the call to `f`, so the view is only used on the current
+ // task.
+ unsafe { self.with_page_as_sysmem(offset, len, f) }
+ }
+}
+
// SAFETY: By the type invariant, a `Page<RefCounted>` is managed by the page reference count.
// Incrementing it via `get_page()` keeps the page alive at least until a matching `put_page()`,
// which frees the page when the count reaches zero.
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 10/11] rust_binder: access page data through `SysMemLocal`
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
` (8 preceding siblings ...)
2026-11-20 10:27 ` [PATCH 09/11] rust: page: add `Page::with_page_as_sysmem_local` Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 11/11] rust: page: remove `Page::copy_from_user_slice_raw` Andreas Hindborg
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
The pages of a binder page range are `Page<RefCounted>` pages, which
dereference to `Page<Shared>`. Thus, their data can be accessed through
the safe `Page::with_page_as_sysmem_local()`.
Use the view in `ShrinkablePageRange::copy_from_user_slice()` and
`ShrinkablePageRange::fill_zero()`. This removes the reliance on the
safety requirements of `Page::copy_from_user_slice_raw()` and
`Page::fill_zero_raw()` in these functions.
`ShrinkablePageRange::read()` and `ShrinkablePageRange::write()` copy to
and from raw pointers, so they keep using the raw accessors.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
drivers/android/binder/page_range.rs | 29 ++++++++++++++---------------
1 file changed, 14 insertions(+), 15 deletions(-)
diff --git a/drivers/android/binder/page_range.rs b/drivers/android/binder/page_range.rs
index bddc4538745e..2d360c32202e 100644
--- a/drivers/android/binder/page_range.rs
+++ b/drivers/android/binder/page_range.rs
@@ -538,14 +538,12 @@ pub(crate) unsafe fn copy_from_user_slice(
offset: usize,
size: usize,
) -> Result {
- // SAFETY: `self.iterate` has the same safety requirements as `copy_from_user_slice`. The
- // pages of this range are `Page<RefCounted>` pages, and this driver never creates a
- // reference to their data, so no reference to the data that is written exists.
- unsafe {
- self.iterate(offset, size, |page, offset, to_copy| {
- page.copy_from_user_slice_raw(reader, offset, to_copy)
- })
- }
+ let copy = |page: &Page, offset: usize, to_copy: usize| {
+ page.with_page_as_sysmem_local(offset, to_copy, |view| reader.read_sysmem(view))
+ };
+
+ // SAFETY: `self.iterate` has the same safety requirements as `copy_from_user_slice`.
+ unsafe { self.iterate(offset, size, copy) }
}
/// Copy from this page range into kernel space.
@@ -605,14 +603,15 @@ pub(crate) unsafe fn write<T: ?Sized>(&self, offset: usize, obj: &T) -> Result {
///
/// All pages touched by this operation must be in use for the duration of this call.
pub(crate) unsafe fn fill_zero(&self, offset: usize, size: usize) -> Result {
- // SAFETY: `self.iterate` has the same safety requirements as `fill_zero`. The pages of
- // this range are `Page<RefCounted>` pages, and this driver never creates a reference to
- // their data, so no reference to the data that is written exists.
- unsafe {
- self.iterate(offset, size, |page, offset, len| {
- page.fill_zero_raw(offset, len)
+ let fill = |page: &Page, offset: usize, len: usize| {
+ page.with_page_as_sysmem_local(offset, len, |view| {
+ view.fill_zero();
+ Ok(())
})
- }
+ };
+
+ // SAFETY: `self.iterate` has the same safety requirements as `fill_zero`.
+ unsafe { self.iterate(offset, size, fill) }
}
}
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 11/11] rust: page: remove `Page::copy_from_user_slice_raw`
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
` (9 preceding siblings ...)
2026-11-20 10:27 ` [PATCH 10/11] rust_binder: access page data through `SysMemLocal` Andreas Hindborg
@ 2026-11-20 10:27 ` Andreas Hindborg
10 siblings, 0 replies; 12+ messages in thread
From: Andreas Hindborg @ 2026-11-20 10:27 UTC (permalink / raw)
To: Alice Ryhl, Lorenzo Stoakes, Liam R. Howlett, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Will Deacon, Peter Zijlstra,
Mark Rutland, Lyude Paul, Greg Kroah-Hartman,
Arve Hjønnevåg, Todd Kjos, Christian Brauner,
Carlos Llamas, Zi Yan
Cc: linux-mm, rust-for-linux, linux-kernel, driver-core, Andreas Hindborg
The last user of `Page::copy_from_user_slice_raw()` now copies from user
space through a `SysMemLocal` view, see `UserSliceReader::read_sysmem()`.
Remove the unused method.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/page.rs | 32 +-------------------------------
1 file changed, 1 insertion(+), 31 deletions(-)
diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index 7fa037f5938b..adb131fb94dc 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -32,8 +32,7 @@
Opaque,
Ownable,
Owned, //
- },
- uaccess::UserSliceReader, //
+ }, //
};
use core::{
marker::PhantomData,
@@ -631,35 +630,6 @@ pub unsafe fn fill_zero_raw(&self, offset: usize, len: usize) -> Result {
// written are initialized. `fill` only uses the view on the current task.
unsafe { self.with_page_as_sysmem(offset, len, fill) }
}
-
- /// Copies data from userspace into this page.
- ///
- /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
- /// outside of the page, then this call returns [`EINVAL`].
- ///
- /// Like the other `UserSliceReader` methods, data races are allowed on the userspace address.
- /// However, they are not allowed on the page you are copying into.
- ///
- /// # Safety
- ///
- /// * Callers must ensure that this call does not race with a read or write to the same page
- /// that overlaps with this write.
- /// * Callers must ensure that no reference to the data of the page that overlaps with this
- /// write exists during this call. Such references can be obtained from a
- /// [`Page<Exclusive>`], see [`Page::as_slice`].
- pub unsafe fn copy_from_user_slice_raw(
- &self,
- reader: &mut UserSliceReader,
- offset: usize,
- len: usize,
- ) -> Result {
- self.with_pointer_into_page(offset, len, move |dst| {
- // SAFETY: If `with_pointer_into_page` calls into this closure, then it has performed a
- // bounds check and guarantees that `dst` is valid for `len` bytes. The caller
- // guarantees that there are no races, so `dst` is valid for atomic writes.
- unsafe { reader.read_raw(dst, len) }
- })
- }
}
impl Page<Shared> {
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2026-11-20 10:28 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-20 10:26 [PATCH 00/11] rust: page: improve access methods for racy memory regions Andreas Hindborg
2026-11-20 10:27 ` [PATCH 01/11] rust: page: tighten safety requirements of `SysMem` copy methods Andreas Hindborg
2026-11-20 10:27 ` [PATCH 02/11] rust: sync: atomic: add memset Andreas Hindborg
2026-11-20 10:27 ` [PATCH 03/11] rust: io: add `SysMem::fill_zero` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 04/11] rust: page: use byte-wise atomic operations in raw accessors Andreas Hindborg
2026-11-20 10:27 ` [PATCH 05/11] rust: uaccess: take a raw pointer in `UserSliceReader::read_raw` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 06/11] rust: uaccess: add `UserSliceReader::read_sysmem` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 07/11] rust: page: add `Shared` page kind Andreas Hindborg
2026-11-20 10:27 ` [PATCH 08/11] rust: io: add `SysMemLocal` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 09/11] rust: page: add `Page::with_page_as_sysmem_local` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 10/11] rust_binder: access page data through `SysMemLocal` Andreas Hindborg
2026-11-20 10:27 ` [PATCH 11/11] rust: page: remove `Page::copy_from_user_slice_raw` Andreas Hindborg
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®