mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Breno Leitao <leitao@debian.org>
To: Thomas Gleixner <tglx@kernel.org>, Ingo Molnar <mingo@redhat.com>,
	 Borislav Petkov <bp@alien8.de>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	 x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
	 Alexander Viro <viro@zeniv.linux.org.uk>,
	 Christian Brauner <brauner@kernel.org>, Jan Kara <jack@suse.cz>,
	 Andrew Morton <akpm@linux-foundation.org>
Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	 linux-block@vger.kernel.org, Breno Leitao <leitao@debian.org>,
	 kernel-team@meta.com, Mateusz Guzik <mjguzik@gmail.com>
Subject: [PATCH RFC 3/3] fs/pipe: copy a run of anonymous buffers in one user access section
Date: Fri, 13 Nov 2026 03:51:46 -0800	[thread overview]
Message-ID: <20261113-pipe-pool-regression-smap-v1-3-b0dc69558ab3@debian.org> (raw)
In-Reply-To: <20261113-pipe-pool-regression-smap-v1-0-b0dc69558ab3@debian.org>

anon_pipe_read() copies one pipe_buffer per copy_page_to_iter() call.
On x86 every call opens and closes its own user access section, so a
64 KiB read that drains a full default pipe executes 16 STAC/CLAC pairs.

Hand up to PIPE_DEF_BUFFERS consecutive anonymous buffers that the read
consumes whole to copy_bvecs_to_iter_full(), then release them. That
includes buffers that splice() or tee() took from another pipe, which
keep anon_pipe_buf_ops.

A last buffer that the read takes only part of, packet buffers, and
buffers with other pipe_buf_operations, such as watch queue notes, page
cache pages and vmsplice()d pages, keep the buffer by buffer loop.

A fault inside the batch consumes nothing and leaves the buffers to the
existing loop, so short reads and -EFAULT behave as before. An injected
usercopy fault takes the same way out.

What this buys depends on what SMAP costs the CPU. With the buffer by
buffer loop, booting with clearcpuid=smap speeds pipe_bench up by 4-14%
on a Sapphire Rapids Xeon Gold 6450C.

pipe_bench from the pipe selftests runs 1, 2 or 5 writers against 1, 5
or 10 readers on one 1 MiB pipe, with 64 KiB writes and reads.

The numbers below come from an earlier version of this patch, which did
the copies in fs/pipe.c and left runs of a single buffer to the old
loop. Throughput change, median of 2 boots x 5 runs, SMAP on:

  writers  readers  Sapphire Rapids
        1        1           +5.2%
        1        5           +9.7%
        1       10           +8.7%
        2        1          +15.3%
        2        5           +8.4%
        2       10           +8.6%
        5        1          +11.9%
        5        5           +9.8%
        5       10           +8.0%

Suggested-by: Mateusz Guzik <mjguzik@gmail.com>
Link: https://lore.kernel.org/all/CAGudoHF3de_55YvdztPs7QiYs25u6nPEV=uMOzfr=bWmwof69g@mail.gmail.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
---
 fs/pipe.c | 44 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)

diff --git a/fs/pipe.c b/fs/pipe.c
index 29ae6c4a6c642c..a0d4c8b2f685bb 100644
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -297,6 +297,40 @@ static inline unsigned int pipe_update_tail(struct pipe_inode_info *pipe,
 	return tail;
 }
 
+/*
+ * Read the anonymous buffers at @tail that the read consumes whole with one
+ * copy_bvecs_to_iter_full() call, which can copy them to a user buffer in one
+ * user access section. Returns the bytes read, or 0 to go buffer by buffer.
+ */
+static size_t anon_pipe_read_batch(struct pipe_inode_info *pipe,
+				   struct iov_iter *to, size_t total_len,
+				   unsigned int head, unsigned int *tailp,
+				   bool *wake_writer)
+{
+	struct bio_vec bv[PIPE_DEF_BUFFERS];
+	unsigned int tail = *tailp, n;
+	size_t len = 0;
+
+	for (n = 0; n < PIPE_DEF_BUFFERS && !pipe_empty(head, tail + n); n++) {
+		struct pipe_buffer *buf = pipe_buf(pipe, tail + n);
+
+		if (buf->ops != &anon_pipe_buf_ops ||
+		    (buf->flags & PIPE_BUF_FLAG_PACKET) ||
+		    buf->len > total_len - len)
+			break;
+		bvec_set_page(&bv[n], buf->page, buf->len, buf->offset);
+		len += buf->len;
+	}
+	if (!n || !copy_bvecs_to_iter_full(bv, n, to))
+		return 0;
+
+	*wake_writer |= pipe_full(head, tail, pipe->max_usage);
+	while (n--)
+		tail = pipe_update_tail(pipe, pipe_buf(pipe, tail), tail);
+	*tailp = tail;
+	return len;
+}
+
 static ssize_t
 anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 {
@@ -304,7 +338,7 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 	struct file *filp = iocb->ki_filp;
 	struct pipe_inode_info *pipe = filp->private_data;
 	bool wake_writer = false, wake_next_reader = false;
-	ssize_t ret;
+	ssize_t ret, batched;
 
 	/* Null read succeeds. */
 	if (unlikely(total_len == 0))
@@ -350,6 +384,14 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		}
 #endif
 
+		batched = anon_pipe_read_batch(pipe, to, total_len, head,
+					       &tail, &wake_writer);
+		if (batched) {
+			ret += batched;
+			total_len -= batched;
+			if (!total_len)
+				break;	/* common path: read succeeded */
+		}
 		if (!pipe_empty(head, tail)) {
 			struct pipe_buffer *buf = pipe_buf(pipe, tail);
 			size_t chars = buf->len;

-- 
2.53.1-Meta


  parent reply	other threads:[~2026-11-13 11:54 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13 11:51 [PATCH RFC 0/3] fs/pipe: copy a run of pipe " Breno Leitao
2026-11-13 11:51 ` [PATCH RFC 1/3] x86/uaccess: Split the STAC/CLAC out of copy_user_generic() Breno Leitao
2026-11-13 11:51 ` [PATCH RFC 2/3] iov_iter: add copy_bvecs_to_iter_full() Breno Leitao
2026-11-13 11:51 ` Breno Leitao [this message]
2026-11-17  9:04   ` [PATCH RFC 3/3] fs/pipe: copy a run of anonymous buffers in one user access section Chris Down
2026-11-17  9:04 ` [PATCH RFC 0/3] fs/pipe: copy a run of pipe " Chris Down

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261113-pipe-pool-regression-smap-v1-3-b0dc69558ab3@debian.org \
    --to=leitao@debian.org \
    --cc=akpm@linux-foundation.org \
    --cc=bp@alien8.de \
    --cc=brauner@kernel.org \
    --cc=dave.hansen@linux.intel.com \
    --cc=hpa@zytor.com \
    --cc=jack@suse.cz \
    --cc=kernel-team@meta.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=mjguzik@gmail.com \
    --cc=tglx@kernel.org \
    --cc=viro@zeniv.linux.org.uk \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®