* [PATCH v4 1/2] rust: sync: atomic: add atomic_per_byte_memcpy
2026-06-05 10:34 [PATCH v4 0/2] (no cover subject) Andreas Hindborg
@ 2026-06-05 10:34 ` Andreas Hindborg
2026-06-05 10:34 ` [PATCH v4 2/2] rust: page: add byte-wise atomic memory copy methods Andreas Hindborg
1 sibling, 0 replies; 3+ messages in thread
From: Andreas Hindborg @ 2026-06-05 10:34 UTC (permalink / raw)
To: Alice Ryhl, Miguel Ojeda, Gary Guo, Björn Roy Baron,
Benno Lossin, Trevor Gross, Danilo Krummrich, Will Deacon,
Peter Zijlstra, Mark Rutland, Boqun Feng, Lorenzo Stoakes,
Liam R. Howlett, Lorenzo Stoakes, Liam R. Howlett, Boqun Feng
Cc: linux-mm, rust-for-linux, linux-kernel, Andreas Hindborg
Add a helper to copy `len` bytes from `src` to `dst` using byte-wise
atomic memory operations. This is the concurrent-safe counterpart of
`core::ptr::copy()` (the equivalent of standard C's `memcpy()`).
Because of the atomicity at byte level, when the copy races with
another concurrent atomic access (or when a normal read races with
an atomic read), or with an external access (from DMA or userspace),
the behavior of this function is defined: the memory is copied at
(at least) byte granularity.
The helper is the building block for higher-level byte-wise atomic
copy methods (e.g., on `Page`) that need to remain defined under
racing accesses, such as when copying to or from a buffer that may
be concurrently accessed by userspace or DMA.
The implementation forwards to the kernel's `memcpy()`, which is
implemented in a way that byte-wise atomic memory load/store
instructions are used.
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/sync/atomic.rs | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/rust/kernel/sync/atomic.rs b/rust/kernel/sync/atomic.rs
index 9cd009d57e35..3c76f1a14b53 100644
--- a/rust/kernel/sync/atomic.rs
+++ b/rust/kernel/sync/atomic.rs
@@ -848,3 +848,38 @@ pub unsafe fn cmpxchg<T: AtomicType, Ordering: ordering::Ordering>(
// per LKMM.
unsafe { Atomic::from_ptr(ptr) }.cmpxchg(old, new, o)
}
+
+/// Copy `len` bytes from `src` to `dst` using byte-wise atomic operations.
+///
+/// This is the concurrent-safe counterpart of `core::ptr::copy()` (the equivalent of standard
+/// C's `memcpy()`). Because of the atomicity at byte level, when racing with another concurrent
+/// atomic access (or when a normal read races with an atomic read), or with an external access
+/// (from DMA or userspace), the behavior of this function is defined: the memory is copied at
+/// (at least) byte granularity.
+///
+/// Implementation note: this is currently implemented by the kernel's `memcpy()`, which is
+/// implemented in a way such that byte-wise atomic memory load/store instructions are used.
+///
+/// This copy operation is volatile.
+///
+/// # Safety
+///
+/// Callers must ensure that:
+///
+/// - `src` is valid for atomic reads for `len` bytes for the duration of the call.
+/// - `dst` is valid for atomic writes for `len` bytes for the duration of the call.
+pub unsafe fn atomic_per_byte_memcpy(src: *const u8, dst: *mut u8, len: usize) {
+ // SAFETY: By the safety requirements of this function, the following operation will not:
+ // - Trap.
+ // - Invalidate any reference invariants.
+ // - Race with any operation by the Rust AM, as `bindings::memcpy` is a byte-wise atomic
+ // operation and all operations by the Rust AM to the involved memory areas use byte-wise
+ // atomic semantics.
+ unsafe {
+ bindings::memcpy(
+ dst.cast::<kernel::ffi::c_void>(),
+ src.cast::<kernel::ffi::c_void>(),
+ len,
+ )
+ };
+}
--
2.51.2
^ permalink raw reply [flat|nested] 3+ messages in thread* [PATCH v4 2/2] rust: page: add byte-wise atomic memory copy methods
2026-06-05 10:34 [PATCH v4 0/2] (no cover subject) Andreas Hindborg
2026-06-05 10:34 ` [PATCH v4 1/2] rust: sync: atomic: add atomic_per_byte_memcpy Andreas Hindborg
@ 2026-06-05 10:34 ` Andreas Hindborg
1 sibling, 0 replies; 3+ messages in thread
From: Andreas Hindborg @ 2026-06-05 10:34 UTC (permalink / raw)
To: Alice Ryhl, Miguel Ojeda, Gary Guo, Björn Roy Baron,
Benno Lossin, Trevor Gross, Danilo Krummrich, Will Deacon,
Peter Zijlstra, Mark Rutland, Boqun Feng, Lorenzo Stoakes,
Liam R. Howlett, Lorenzo Stoakes, Liam R. Howlett, Boqun Feng
Cc: linux-mm, rust-for-linux, linux-kernel, Andreas Hindborg
When copying data from buffers that are mapped to user space, it is
impossible to guarantee absence of concurrent memory operations on
those buffers. Copying data to/from `Page` from/to these buffers
would be undefined behavior if no special considerations are made.
Add `Page::{read,write}_bytewise_atomic` to read from / write to a
page using byte-wise atomic operations layered on
`atomic_per_byte_memcpy`. The methods are asymmetric: the parameter
buffer must support byte-wise atomic accesses, while the page side
held through `&self` only requires the absence of concurrent writes
(or of concurrent reads or writes for the write variant). This
follows the intended usage where the page is private to the caller
and the parameter buffer may be shared (e.g., with userspace).
Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
rust/kernel/page.rs | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)
diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index adecb200c654..7bb201442679 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -296,6 +296,38 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
})
}
+ /// Maps the page and reads from it into the given memory region using byte-wise atomic memory
+ /// operations.
+ ///
+ /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
+ /// outside of the page, then this call returns [`EINVAL`].
+ ///
+ /// This function is guaranteed to perform byte-wise atomic memory writes to `dst`, but it may
+ /// perform only normal (non-atomic) memory reads from the [`Page`] `self`. Accordingly, the
+ /// safety requirements below ask for byte-wise atomic discipline on `dst` and only for absence
+ /// of concurrent writes on the source page.
+ ///
+ /// # Safety
+ ///
+ /// Callers must ensure that:
+ ///
+ /// - `dst` is valid for atomic writes for `len` bytes for the duration of the call.
+ /// - This call does not race with a write to the source page that overlaps with this read.
+ pub unsafe fn read_bytewise_atomic(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
+ self.with_pointer_into_page(offset, len, move |src| {
+ // SAFETY:
+ // - If `with_pointer_into_page` calls into this closure, then it has performed a
+ // bounds check and guarantees that `src` is valid for `len` bytes.
+ // - By function safety requirements `dst` is valid for writes for `len` bytes.
+ // - By function safety requirements there are no other writes to `src` during this
+ // call.
+ // - By function safety requirements all other access to `dst` during this call are
+ // atomic.
+ unsafe { kernel::sync::atomic::atomic_per_byte_memcpy(src, dst, len) };
+ Ok(())
+ })
+ }
+
/// Maps the page and writes into it from the given buffer.
///
/// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
@@ -317,6 +349,44 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res
})
}
+ /// Maps the page and writes into it from the given memory region using byte-wise atomic memory
+ /// operations.
+ ///
+ /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
+ /// outside of the page, then this call returns [`EINVAL`].
+ ///
+ /// This function is guaranteed to perform byte-wise atomic memory reads from `src`, but it may
+ /// perform only normal (non-atomic) memory writes to the [`Page`] `self`. Accordingly, the
+ /// safety requirements below ask for byte-wise atomic discipline on `src` and only for absence
+ /// of concurrent reads or writes on the destination page.
+ ///
+ /// # Safety
+ ///
+ /// Callers must ensure that:
+ ///
+ /// - `src` is valid for atomic reads for `len` bytes for the duration of the call.
+ /// - This call does not race with a read or write to the destination page that overlaps with
+ /// this write.
+ pub unsafe fn write_bytewise_atomic(
+ &self,
+ src: *const u8,
+ offset: usize,
+ len: usize,
+ ) -> Result {
+ self.with_pointer_into_page(offset, len, move |dst| {
+ // SAFETY:
+ // - By function safety requirements `src` is valid for writes for `len` bytes.
+ // - If `with_pointer_into_page` calls into this closure, then it has performed a
+ // bounds check and guarantees that `dst` is valid for `len` bytes.
+ // - By function safety requirements there are no other writes to `dst` during this
+ // call.
+ // - By function safety requirements all other access to `src` during this call are
+ // atomic.
+ unsafe { kernel::sync::atomic::atomic_per_byte_memcpy(src, dst, len) };
+ Ok(())
+ })
+ }
+
/// Maps the page and zeroes the given slice.
///
/// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
--
2.51.2
^ permalink raw reply [flat|nested] 3+ messages in thread