* [PATCH RFC 1/3] x86/uaccess: Split the STAC/CLAC out of copy_user_generic()
2026-11-13 11:51 [PATCH RFC 0/3] fs/pipe: copy a run of pipe buffers in one user access section Breno Leitao
@ 2026-11-13 11:51 ` Breno Leitao
2026-11-13 11:51 ` [PATCH RFC 2/3] iov_iter: add copy_bvecs_to_iter_full() Breno Leitao
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Breno Leitao @ 2026-11-13 11:51 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Alexander Viro, Christian Brauner, Jan Kara,
Andrew Morton
Cc: linux-kernel, linux-fsdevel, linux-block, Breno Leitao, kernel-team
copy_user_generic() brackets every copy with STAC/CLAC. A caller that
copies several kernel buffers to or from one user range pays for a pair
per buffer, even when one user access section could cover them all.
The x86 unsafe_copy_to_user() works inside such a section, but it moves
8 bytes per store, which is the wrong tool for page sized copies. x86
has no unsafe_copy_from_user() yet, and the one posted with the sframe
unwinder [1] is the same loop for reads.
Move the body of copy_user_generic() into unsafe_copy_user_generic(),
which leaves STAC/CLAC to the caller's user access section, and turn
copy_user_generic() into a STAC/CLAC wrapper around it. The existing
callers get the same code.
Link: https://lore.kernel.org/all/20260821195259.2688377-19-jremus@linux.ibm.com/ [1]
Signed-off-by: Breno Leitao <leitao@debian.org>
---
arch/x86/include/asm/uaccess_64.h | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h
index 20de34cc9aa6e7..b2bd40d3768d4c 100644
--- a/arch/x86/include/asm/uaccess_64.h
+++ b/arch/x86/include/asm/uaccess_64.h
@@ -115,10 +115,13 @@ static inline bool __access_ok(const void __user *ptr, unsigned long size)
__must_check unsigned long
rep_movs_alternative(void *to, const void *from, unsigned len);
+/*
+ * copy_user_generic() without the STAC/CLAC, for several copies inside one
+ * user access section. Returns the number of bytes not copied.
+ */
static __always_inline __must_check unsigned long
-copy_user_generic(void *to, const void *from, unsigned long len)
+unsafe_copy_user_generic(void *to, const void *from, unsigned long len)
{
- stac();
/*
* If CPU has FSRM feature, use 'rep movs'.
* Otherwise, use rep_movs_alternative.
@@ -131,6 +134,19 @@ copy_user_generic(void *to, const void *from, unsigned long len)
_ASM_EXTABLE_UA(1b, 2b)
:"+c" (len), "+D" (to), "+S" (from), ASM_CALL_CONSTRAINT
: : "memory", "rax");
+ return len;
+}
+
+/*
+ * Hack: lets generic code test for this helper.
+ */
+#define unsafe_copy_user_generic unsafe_copy_user_generic
+
+static __always_inline __must_check unsigned long
+copy_user_generic(void *to, const void *from, unsigned long len)
+{
+ stac();
+ len = unsafe_copy_user_generic(to, from, len);
clac();
return len;
}
--
2.53.1-Meta
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH RFC 2/3] iov_iter: add copy_bvecs_to_iter_full()
2026-11-13 11:51 [PATCH RFC 0/3] fs/pipe: copy a run of pipe buffers in one user access section Breno Leitao
2026-11-13 11:51 ` [PATCH RFC 1/3] x86/uaccess: Split the STAC/CLAC out of copy_user_generic() Breno Leitao
@ 2026-11-13 11:51 ` Breno Leitao
2026-11-13 11:51 ` [PATCH RFC 3/3] fs/pipe: copy a run of anonymous buffers in one user access section Breno Leitao
2026-11-17 9:04 ` [PATCH RFC 0/3] fs/pipe: copy a run of pipe " Chris Down
3 siblings, 0 replies; 6+ messages in thread
From: Breno Leitao @ 2026-11-13 11:51 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Alexander Viro, Christian Brauner, Jan Kara,
Andrew Morton
Cc: linux-kernel, linux-fsdevel, linux-block, Breno Leitao, kernel-team
copy_page_to_iter() opens and closes a user access section for every
page it copies to a user buffer. On x86 that is a STAC/CLAC pair per
page, even when the caller holds a batch of pages, as anon_pipe_read()
does, that one section could cover.
Add copy_bvecs_to_iter_full(), which copies an array of pages to an
iov_iter like a copy_page_to_iter() loop, but all or nothing: after a
short copy the iterator is reverted, as copy_from_iter_full() does.
For a plain user buffer (ITER_UBUF) on architectures that provide
unsafe_copy_user_generic(), all the copies run inside one scoped user
write access. It keeps the usercopy fault injection and the
instrumentation that copy_to_user_iter() does for every copy. Every
other destination gets the copy_page_to_iter() loop.
Signed-off-by: Breno Leitao <leitao@debian.org>
---
include/linux/uio.h | 2 ++
lib/iov_iter.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 63 insertions(+)
diff --git a/include/linux/uio.h b/include/linux/uio.h
index 76e4103b9ec423..7ab25080b0edea 100644
--- a/include/linux/uio.h
+++ b/include/linux/uio.h
@@ -197,6 +197,8 @@ size_t fault_in_iov_iter_writeable(const struct iov_iter *i, size_t bytes);
size_t iov_iter_single_seg_count(const struct iov_iter *i);
size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes,
struct iov_iter *i);
+bool copy_bvecs_to_iter_full(const struct bio_vec *bv, unsigned int nr,
+ struct iov_iter *i);
size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes,
struct iov_iter *i);
size_t copy_folio_from_iter_atomic(struct folio *folio, size_t offset,
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index de7a48205e0117..2df0539228be84 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -387,6 +387,67 @@ size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes,
}
EXPORT_SYMBOL(copy_page_to_iter);
+#ifdef unsafe_copy_user_generic
+static bool copy_bvecs_to_ubuf_full(const struct bio_vec *bv, unsigned int nr,
+ struct iov_iter *i)
+{
+ char __user *dst = i->ubuf + i->iov_offset;
+ size_t len = 0, off = 0;
+ unsigned int k;
+
+ for (k = 0; k < nr; k++)
+ len += bv[k].bv_len;
+ if (len > iov_iter_count(i) || should_fail_usercopy())
+ return false;
+
+ might_fault();
+ scoped_user_write_access_size(dst, len, efault) {
+ for (k = 0; k < nr; off += bv[k].bv_len, k++) {
+ void *src = page_address(bv[k].bv_page) +
+ bv[k].bv_offset;
+
+ instrument_copy_to_user(dst + off, src, bv[k].bv_len);
+ if (unsafe_copy_user_generic((__force void *)dst + off,
+ src, bv[k].bv_len))
+ goto efault;
+ }
+ }
+ iov_iter_advance(i, len);
+ return true;
+
+efault:
+ return false;
+}
+#endif
+
+/*
+ * Copy the pages in @bv to @i one after the other, like copy_page_to_iter(),
+ * but all or nothing: after a short copy @i is reverted. For a plain user
+ * buffer the copies share one user access section where the architecture
+ * allows it.
+ */
+bool copy_bvecs_to_iter_full(const struct bio_vec *bv, unsigned int nr,
+ struct iov_iter *i)
+{
+ size_t copied = 0, n;
+ unsigned int k;
+
+#ifdef unsafe_copy_user_generic
+ if (iter_is_ubuf(i))
+ return copy_bvecs_to_ubuf_full(bv, nr, i);
+#endif
+ for (k = 0; k < nr; k++) {
+ n = copy_page_to_iter(bv[k].bv_page, bv[k].bv_offset,
+ bv[k].bv_len, i);
+ copied += n;
+ if (n != bv[k].bv_len) {
+ iov_iter_revert(i, copied);
+ return false;
+ }
+ }
+ return true;
+}
+
size_t copy_page_to_iter_nofault(struct page *page, unsigned offset, size_t bytes,
struct iov_iter *i)
{
--
2.53.1-Meta
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH RFC 3/3] fs/pipe: copy a run of anonymous buffers in one user access section
2026-11-13 11:51 [PATCH RFC 0/3] fs/pipe: copy a run of pipe buffers in one user access section Breno Leitao
2026-11-13 11:51 ` [PATCH RFC 1/3] x86/uaccess: Split the STAC/CLAC out of copy_user_generic() Breno Leitao
2026-11-13 11:51 ` [PATCH RFC 2/3] iov_iter: add copy_bvecs_to_iter_full() Breno Leitao
@ 2026-11-13 11:51 ` Breno Leitao
2026-11-17 9:04 ` Chris Down
2026-11-17 9:04 ` [PATCH RFC 0/3] fs/pipe: copy a run of pipe " Chris Down
3 siblings, 1 reply; 6+ messages in thread
From: Breno Leitao @ 2026-11-13 11:51 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Alexander Viro, Christian Brauner, Jan Kara,
Andrew Morton
Cc: linux-kernel, linux-fsdevel, linux-block, Breno Leitao,
kernel-team, Mateusz Guzik
anon_pipe_read() copies one pipe_buffer per copy_page_to_iter() call.
On x86 every call opens and closes its own user access section, so a
64 KiB read that drains a full default pipe executes 16 STAC/CLAC pairs.
Hand up to PIPE_DEF_BUFFERS consecutive anonymous buffers that the read
consumes whole to copy_bvecs_to_iter_full(), then release them. That
includes buffers that splice() or tee() took from another pipe, which
keep anon_pipe_buf_ops.
A last buffer that the read takes only part of, packet buffers, and
buffers with other pipe_buf_operations, such as watch queue notes, page
cache pages and vmsplice()d pages, keep the buffer by buffer loop.
A fault inside the batch consumes nothing and leaves the buffers to the
existing loop, so short reads and -EFAULT behave as before. An injected
usercopy fault takes the same way out.
What this buys depends on what SMAP costs the CPU. With the buffer by
buffer loop, booting with clearcpuid=smap speeds pipe_bench up by 4-14%
on a Sapphire Rapids Xeon Gold 6450C.
pipe_bench from the pipe selftests runs 1, 2 or 5 writers against 1, 5
or 10 readers on one 1 MiB pipe, with 64 KiB writes and reads.
The numbers below come from an earlier version of this patch, which did
the copies in fs/pipe.c and left runs of a single buffer to the old
loop. Throughput change, median of 2 boots x 5 runs, SMAP on:
writers readers Sapphire Rapids
1 1 +5.2%
1 5 +9.7%
1 10 +8.7%
2 1 +15.3%
2 5 +8.4%
2 10 +8.6%
5 1 +11.9%
5 5 +9.8%
5 10 +8.0%
Suggested-by: Mateusz Guzik <mjguzik@gmail.com>
Link: https://lore.kernel.org/all/CAGudoHF3de_55YvdztPs7QiYs25u6nPEV=uMOzfr=bWmwof69g@mail.gmail.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
---
fs/pipe.c | 44 +++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 43 insertions(+), 1 deletion(-)
diff --git a/fs/pipe.c b/fs/pipe.c
index 29ae6c4a6c642c..a0d4c8b2f685bb 100644
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -297,6 +297,40 @@ static inline unsigned int pipe_update_tail(struct pipe_inode_info *pipe,
return tail;
}
+/*
+ * Read the anonymous buffers at @tail that the read consumes whole with one
+ * copy_bvecs_to_iter_full() call, which can copy them to a user buffer in one
+ * user access section. Returns the bytes read, or 0 to go buffer by buffer.
+ */
+static size_t anon_pipe_read_batch(struct pipe_inode_info *pipe,
+ struct iov_iter *to, size_t total_len,
+ unsigned int head, unsigned int *tailp,
+ bool *wake_writer)
+{
+ struct bio_vec bv[PIPE_DEF_BUFFERS];
+ unsigned int tail = *tailp, n;
+ size_t len = 0;
+
+ for (n = 0; n < PIPE_DEF_BUFFERS && !pipe_empty(head, tail + n); n++) {
+ struct pipe_buffer *buf = pipe_buf(pipe, tail + n);
+
+ if (buf->ops != &anon_pipe_buf_ops ||
+ (buf->flags & PIPE_BUF_FLAG_PACKET) ||
+ buf->len > total_len - len)
+ break;
+ bvec_set_page(&bv[n], buf->page, buf->len, buf->offset);
+ len += buf->len;
+ }
+ if (!n || !copy_bvecs_to_iter_full(bv, n, to))
+ return 0;
+
+ *wake_writer |= pipe_full(head, tail, pipe->max_usage);
+ while (n--)
+ tail = pipe_update_tail(pipe, pipe_buf(pipe, tail), tail);
+ *tailp = tail;
+ return len;
+}
+
static ssize_t
anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
{
@@ -304,7 +338,7 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
struct file *filp = iocb->ki_filp;
struct pipe_inode_info *pipe = filp->private_data;
bool wake_writer = false, wake_next_reader = false;
- ssize_t ret;
+ ssize_t ret, batched;
/* Null read succeeds. */
if (unlikely(total_len == 0))
@@ -350,6 +384,14 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
}
#endif
+ batched = anon_pipe_read_batch(pipe, to, total_len, head,
+ &tail, &wake_writer);
+ if (batched) {
+ ret += batched;
+ total_len -= batched;
+ if (!total_len)
+ break; /* common path: read succeeded */
+ }
if (!pipe_empty(head, tail)) {
struct pipe_buffer *buf = pipe_buf(pipe, tail);
size_t chars = buf->len;
--
2.53.1-Meta
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH RFC 3/3] fs/pipe: copy a run of anonymous buffers in one user access section
2026-11-13 11:51 ` [PATCH RFC 3/3] fs/pipe: copy a run of anonymous buffers in one user access section Breno Leitao
@ 2026-11-17 9:04 ` Chris Down
0 siblings, 0 replies; 6+ messages in thread
From: Chris Down @ 2026-11-17 9:04 UTC (permalink / raw)
To: Breno Leitao
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Alexander Viro, Christian Brauner, Jan Kara,
Andrew Morton, linux-kernel, linux-fsdevel, linux-block,
kernel-team, Mateusz Guzik
Hi Breno,
Breno Leitao writes:
>The numbers below come from an earlier version of this patch, which did
>the copies in fs/pipe.c and left runs of a single buffer to the old
>loop. Throughput change, median of 2 boots x 5 runs, SMAP on:
I timed a single read() from a pipe that was already full of anonymous
buffers, on an i9-14900HX in a 2 CPU KVM guest:
read size base this base, nosmap this, nosmap
4 KiB 175ns 181ns 152ns 155ns
64 KiB 2123ns 2061ns 1971ns 2070ns
1 MiB 49.6us 49.0us 46.4us 48.3us
(nosmap here being clearcpuid=smap.)
So on this CPU the 16 STAC/CLAC pairs in a 64 KiB read cost about 150ns,
and this patch gets back about 60ns of that. But with SMAP off this patch
is about 100ns slower for the same read. So the batched path seems to
cost more than the old loop, and only comes out ahead where STAC/CLAC
are expensive. I haven't looked into where that comes from.
On CPUs where they're cheap, like the later Zen parts Linus mentioned in
[1], this may well be a loss.
It would be good for the changelog to have numbers with some
clearcpuid=smap runs to compare with, and preferably on a few machine
types.
[1] https://lore.kernel.org/all/CAHk-=wiTN=1Sm5DVnXDtQ-tSiPuz-7QVpgP2vt6LgaEnhbubfw@mail.gmail.com/
Thanks!
Chris
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH RFC 0/3] fs/pipe: copy a run of pipe buffers in one user access section
2026-11-13 11:51 [PATCH RFC 0/3] fs/pipe: copy a run of pipe buffers in one user access section Breno Leitao
` (2 preceding siblings ...)
2026-11-13 11:51 ` [PATCH RFC 3/3] fs/pipe: copy a run of anonymous buffers in one user access section Breno Leitao
@ 2026-11-17 9:04 ` Chris Down
3 siblings, 0 replies; 6+ messages in thread
From: Chris Down @ 2026-11-17 9:04 UTC (permalink / raw)
To: Breno Leitao
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Alexander Viro, Christian Brauner, Jan Kara,
Andrew Morton, linux-kernel, linux-fsdevel, linux-block,
kernel-team, Mateusz Guzik
Hi Breno,
Breno Leitao writes:
>fs/pipe.c into lib/iov_iter.c. The main question is whether that helper
>is welcome there, or, make the party a bit uglier.
Hmm, I think the part more likely to get pushback is patch 1, rather
than where the helper lives. :-) It adds an x86-only user copy primitive,
and then lib/iov_iter.c has to look for it with an #ifdef on its name,
which the patch (rightly) acknowledges as a bit of a Macgyver.
When Christophe proposed another user copy variant in April, Linus asked
for the existing unsafe_copy_*() helpers to be extended instead of
growing a new interface [1]. And in a related thread he called x86's
unsafe_copy_to_user() "almost entirely unoptimized" [2], which is the
same reason patch 1 gives for not using it.
Based on that discussion, I'd think that the better way would be to make
x86's unsafe_copy_to_user() use the rep movs body from patch 1 for
larger copies. Then copy_bvecs_to_iter_full() can be written with
scoped_user_write_access_size() and unsafe_copy_to_user() for every
architecture, without the #ifdef.
[1] https://lore.kernel.org/all/CAHk-=whC1DZojwdMB1=sJWG2=dsCdfyU8N6tDE1qx50HRZ-WJQ@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHk-=wi1rrcijcD0i7V7JD6bLL-yKHUX-hcxtLx=BUd34phdug@mail.gmail.com/
Thanks!
Chris
^ permalink raw reply [flat|nested] 6+ messages in thread