From: Breno Leitao <leitao@debian.org>
To: Alexander Viro <viro@zeniv.linux.org.uk>,
Christian Brauner <brauner@kernel.org>, Jan Kara <jack@suse.cz>,
Shuah Khan <shuah@kernel.org>
Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-kselftest@vger.kernel.org,
Breno Leitao <leitao@debian.org>,
usama.arif@linux.dev, kernel-team@meta.com
Subject: [PATCH 1/2] fs/pipe: bulk pre-allocate pages outside pipe->mutex in anon_pipe_write
Date: Fri, 15 May 2026 03:28:44 -0700 [thread overview]
Message-ID: <20260515-fix_pipe-v1-1-b14c840c7555@debian.org> (raw)
In-Reply-To: <20260515-fix_pipe-v1-0-b14c840c7555@debian.org>
anon_pipe_write() takes pipe->mutex and then, from the per-iteration
anon_pipe_get_page() helper, used to call alloc_page(GFP_HIGHUSER |
__GFP_ACCOUNT) once per page while still holding it. That allocation
can sleep doing direct reclaim and/or runs memcg charging, which extends
the critical section and stalls a concurrent reader on the very same
mutex.
Bulk pre-allocate DIV_ROUND_UP(total_len, PAGE_SIZE) pages (up to
PIPE_PREALLOC_MAX (8)) pages outside the mutex when total_len >=
PAGE_SIZE, using alloc_pages_bulk(). (Under memcg, alloc_pages_bulk()
with __GFP_ACCOUNT might return less pages than requested, but, this is
still a win, given some pages allocation is moved outside of the
lock).
Pass the array into anon_pipe_get_page(), which now consumes from
tmp_page[] first, then from the prealloc array, and only as a last
resort falls back to alloc_page() under the mutex (reached only for
writes larger than 8 pages, where the prealloc cap is exhausted).
Doing this in one bulk call before the lock keeps the fast path's
mutex held for a single, write-bounded critical section -- no extra
mutex_unlock/_lock cycles -- so it avoids the per-page lock-handoff
overhead that a per-page drop-and-retake design would introduce, while
still moving the typical multi-page allocation off the critical
section. Unused prealloc pages are pushed to the pipe's tmp_page[]
cache (or freed) before unlock, so a subsequent write to the same
pipe gets a hot cached page rather than paying for an alloc again.
Sub-PAGE_SIZE writes are unchanged: the merge path handles them
without ever needing a fresh page, so it is not worth speculatively
allocating for them.
This can improve the pipe throughput up to 48% and reduce the latency in
33%.
Signed-off-by: Breno Leitao <leitao@debian.org>
---
fs/pipe.c | 40 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 38 insertions(+), 2 deletions(-)
diff --git a/fs/pipe.c b/fs/pipe.c
index 9841648c9cf3e..7a1517d15107a 100644
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -111,7 +111,11 @@ void pipe_double_lock(struct pipe_inode_info *pipe1,
pipe_lock(pipe2);
}
-static struct page *anon_pipe_get_page(struct pipe_inode_info *pipe)
+#define PIPE_PREALLOC_MAX 8
+
+static struct page *anon_pipe_get_page(struct pipe_inode_info *pipe,
+ struct page **prealloc,
+ unsigned int *prealloc_n)
{
for (int i = 0; i < ARRAY_SIZE(pipe->tmp_page); i++) {
if (pipe->tmp_page[i]) {
@@ -121,6 +125,14 @@ static struct page *anon_pipe_get_page(struct pipe_inode_info *pipe)
}
}
+ if (*prealloc_n) {
+ unsigned int idx = --(*prealloc_n);
+ struct page *page = prealloc[idx];
+
+ prealloc[idx] = NULL;
+ return page;
+ }
+
return alloc_page(GFP_HIGHUSER | __GFP_ACCOUNT);
}
@@ -438,6 +450,8 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
ssize_t chars;
bool was_empty = false;
bool wake_next_writer = false;
+ struct page *prealloc[PIPE_PREALLOC_MAX] = { NULL };
+ unsigned int prealloc_n = 0;
/*
* Reject writing to watch queue pipes before the point where we lock
@@ -455,6 +469,26 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
if (unlikely(total_len == 0))
return 0;
+ /*
+ * Bulk pre-allocate up to PIPE_PREALLOC_MAX pages outside pipe->mutex
+ * for writes that span at least one full page. alloc_page() with
+ * GFP_HIGHUSER may sleep doing reclaim and runs memcg charging, so
+ * doing it under the mutex extends the critical section and stalls
+ * the reader. The merge path handles sub-PAGE_SIZE writes without
+ * needing a fresh page; for writes larger than PIPE_PREALLOC_MAX
+ * pages, anon_pipe_get_page() falls back to a single alloc_page()
+ * under the mutex for the remainder. Unused prealloc pages are
+ * returned to the pipe's tmp_page[] cache (or freed) before unlock.
+ */
+ if (total_len >= PAGE_SIZE) {
+ unsigned int want = min_t(unsigned int,
+ DIV_ROUND_UP(total_len, PAGE_SIZE),
+ PIPE_PREALLOC_MAX);
+
+ prealloc_n = alloc_pages_bulk(GFP_HIGHUSER | __GFP_ACCOUNT,
+ want, prealloc);
+ }
+
mutex_lock(&pipe->mutex);
if (!pipe->readers) {
@@ -512,7 +546,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
struct page *page;
int copied;
- page = anon_pipe_get_page(pipe);
+ page = anon_pipe_get_page(pipe, prealloc, &prealloc_n);
if (unlikely(!page)) {
if (!ret)
ret = -ENOMEM;
@@ -576,6 +610,8 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
wake_next_writer = true;
}
out:
+ while (prealloc_n)
+ anon_pipe_put_page(pipe, prealloc[--prealloc_n]);
if (pipe_is_full(pipe))
wake_next_writer = false;
mutex_unlock(&pipe->mutex);
--
2.53.0-Meta
next prev parent reply other threads:[~2026-05-15 10:29 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-15 10:28 [PATCH 0/2] fs/pipe: reduce pipe->mutex contention by bulk-allocating outside the lock Breno Leitao
2026-05-15 10:28 ` Breno Leitao [this message]
2026-05-15 20:18 ` [PATCH 1/2] fs/pipe: bulk pre-allocate pages outside pipe->mutex in anon_pipe_write Mateusz Guzik
2026-05-20 17:09 ` Breno Leitao
2026-05-21 12:58 ` Mateusz Guzik
2026-05-21 15:38 ` Breno Leitao
2026-05-21 17:26 ` Mateusz Guzik
2026-05-22 12:10 ` Breno Leitao
2026-05-15 10:28 ` [PATCH 2/2] selftests/pipe: add pipe_bench microbenchmark Breno Leitao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260515-fix_pipe-v1-1-b14c840c7555@debian.org \
--to=leitao@debian.org \
--cc=brauner@kernel.org \
--cc=jack@suse.cz \
--cc=kernel-team@meta.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=shuah@kernel.org \
--cc=usama.arif@linux.dev \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®