From: David Howells <dhowells@redhat.com>
To: Christian Brauner <christian@brauner.io>
Cc: David Howells <dhowells@redhat.com>,
Paulo Alcantara <pc@manguebit.org>,
Matthew Wilcox <willy@infradead.org>,
Namjae Jeon <linkinjeon@kernel.org>,
Marc Dionne <marc.dionne@auristor.com>,
Stefan Metzmacher <metze@samba.org>,
Eric Van Hensbergen <ericvh@kernel.org>,
Dominique Martinet <asmadeus@codewreck.org>,
Ilya Dryomov <idryomov@gmail.com>,
netfs@lists.linux.dev, linux-afs@lists.infradead.org,
linux-cifs@vger.kernel.org, linux-nfs@vger.kernel.org,
ceph-devel@vger.kernel.org, v9fs@lists.linux.dev,
linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
Christoph Hellwig <hch@infradead.org>
Subject: [PATCH v18 04/17] netfs: Add some tools for managing bvecq chains
Date: Tue, 17 Nov 2026 17:19:02 +0000 [thread overview]
Message-ID: <20261117171958.2982801-5-dhowells@redhat.com> (raw)
In-Reply-To: <20261117171958.2982801-1-dhowells@redhat.com>
Provide a selection of tools for managing bvec queue chains and a memory
pool for writeback to draw upon if needed. The tools include the
allocation, prepopulation and refcounting of bvecqs and bvecq chains. When
a chain is being cleaned up, the memory segments will be appropriately
disposed off according to the flags on each bvecq in that chain, allowing
for mixed memory cleanup types.
A pair of memory pools is provided for emergency allocation under low
memory conditions. The two pools are intended for use in different
circumstances with different usage characteristics. The request-level pool
will be used by netfslib requests to build a rolling buffer as it goes
along, whereas the subrequest-level pool will be used by subrequests
(e.g. cifs can use it to allocate an encryption buffer). Separate pools
are provided to avoid deadlocking between levels.
The main use for this is to replace the use of both folio_queues and
kvmalloc'd bio_vec arrays in netfslib and creating a structure that allows
network filesystems to easily build an entire message by gluing bits of
protocol on either side of the bvecq chains and also glue chains together
for sparse writes and compound operations. This allows TCP-based
transports to pass a message in a single sendmsg() call and avoid corking
as this makes things more efficient.
This will also be used to do things like creating an encryption buffer in
cifs or a directory content buffer in afs.
Note that these tools are placed in netfslib for the moment, but can be
moved to lib/ later if any non-netfslib code starts to use them. The
iov_iter bits don't depend on these tools and must live in the core.
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/Makefile | 1 +
fs/netfs/bvecq.c | 414 ++++++++++++++++++++++++++++++++++++++
fs/netfs/internal.h | 5 +
fs/netfs/main.c | 46 ++++-
fs/netfs/objects.c | 1 -
fs/netfs/rolling_buffer.c | 1 -
fs/netfs/stats.c | 6 +-
include/linux/bvecq.h | 145 +++++++++++++
include/linux/netfs.h | 1 +
9 files changed, 616 insertions(+), 4 deletions(-)
create mode 100644 fs/netfs/bvecq.c
diff --git a/fs/netfs/Makefile b/fs/netfs/Makefile
index 54834cde7e56..b1ea4439c1bb 100644
--- a/fs/netfs/Makefile
+++ b/fs/netfs/Makefile
@@ -3,6 +3,7 @@
netfs-y := \
buffered_read.o \
buffered_write.o \
+ bvecq.o \
direct_read.o \
direct_write.o \
iterator.o \
diff --git a/fs/netfs/bvecq.c b/fs/netfs/bvecq.c
new file mode 100644
index 000000000000..8ac2dc235258
--- /dev/null
+++ b/fs/netfs/bvecq.c
@@ -0,0 +1,414 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Buffering helpers for bvec queues
+ *
+ * Copyright (C) 2026 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+
+#include <linux/bvecq.h>
+#include <linux/semaphore.h>
+#include "internal.h"
+
+void bvecq_dump(const struct bvecq *bq)
+{
+ int b = 0;
+
+ for (; bq; bq = bvecq_next(bq), b++) {
+ pr_notice("BQ[%u] %u/%u\n", b, bq->nr_slots, bq->max_slots);
+ for (int s = 0; s < bq->nr_slots; s++) {
+ const struct bio_vec *bv = &bq->bv[s];
+
+ if (!IS_ERR_OR_NULL(bv->bv_page))
+ pr_notice("BQ[%u:%02u] %10lx %04x %04x %u\n",
+ b, s,
+ page_to_pfn(bv->bv_page),
+ bv->bv_offset, bv->bv_len,
+ page_count(bv->bv_page));
+ else
+ pr_notice("BQ[%u:%02u] - %04x %04x\n",
+ b, s, bv->bv_offset, bv->bv_len);
+ }
+ }
+}
+EXPORT_SYMBOL(bvecq_dump);
+
+/**
+ * bvecq_alloc_one - Allocate a single bvecq node with unpopulated slots
+ * @nr_slots: Number of slots to allocate
+ * @gfp: The allocation constraints.
+ * @flags: Pointer to the allocation flags
+ *
+ * Allocate a single bvecq node and initialise the header. The allocation is
+ * rounded up to the size of the smallest slab granule that will accommodate it
+ * and all the remaining space is set as an inline slot array with max_slots
+ * set to the number of slots. The slot array is not initialised.
+ *
+ * Note that the number of slots allocated is limited to USHRT_MAX. If more
+ * than this are desired, a number of bvecqs should be chained.
+ *
+ * If *@flags includes %BVECQ_USE_REQ_POOL or %BVECQ_USE_SUBREQ_POOL, then an
+ * emergency mempool may be used for allocation. If it does allocate from such
+ * a pool, the maximum number of slots available will be BVECQ_POOL_SLOTS which
+ * may be less than requested. Also, if once of these flags is set, the
+ * function will not fail - though it may have to wait (though if
+ * %BVECQ_SHORT_OKAY is set, it may return early but will have at least one
+ * bveq in the allocation).
+ *
+ * Further, in order to access an emergency pool, the function must take a
+ * count from the semaphore that governs access to that pool. When it has
+ * access to the pool, %BVECQ_HOLD_POOL_SEM is set in *@flags and the sem is
+ * held - and access allowed - until explicitly dropped. The pool has more
+ * objects than the sem has counts, which means that there are always more
+ * objects in the pool than processes trying to allocate from it - guaranteeing
+ * that at least some progress can be made.
+ *
+ * Return: The node pointer or NULL on allocation failure. BVECQ_HIT_OOM is
+ * set in *flags if an allocation failure occurs.
+ */
+struct bvecq *bvecq_alloc_one(size_t nr_slots, gfp_t gfp, unsigned int *flags)
+{
+ enum bvecq_pool from_pool = *flags & BVECQ__USE_POOL;
+ struct bvecq *bq;
+ size_t size;
+
+ /* Don't overrun ->max_slots or ->nr_slots. */
+ if (nr_slots > BVECQ_MAX_INLINE_SLOTS)
+ nr_slots = BVECQ_MAX_INLINE_SLOTS;
+
+ size = kmalloc_size_roundup(struct_size(bq, __bv, nr_slots));
+ gfp &= ~(GFP_ZONEMASK | __GFP_THISNODE);
+
+ if (from_pool != BVECQ_USE_NO_POOL) {
+ struct semaphore *sem = &netfs_bvecq_pool_sem[from_pool];
+ struct mempool *pool = &netfs_bvecq_pool[from_pool];
+ gfp_t gfp_temp = gfp;
+
+ gfp_temp |= __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN;
+ gfp_temp &= ~(__GFP_DIRECT_RECLAIM | __GFP_IO);
+
+ if (size != BVECQ_POOL_SIZE) {
+ bq = kmalloc(size, gfp_temp);
+ if (bq) {
+ from_pool = BVECQ_USE_NO_POOL;
+ goto success;
+ }
+ } else {
+ bq = mempool_alloc_noreserve(pool, gfp_temp);
+ if (bq) {
+ goto success;
+ }
+ }
+ *flags |= BVECQ_HIT_OOM;
+
+ if (!(*flags & BVECQ_HOLD_POOL_SEM)) {
+ down(sem);
+ *flags |= BVECQ_HOLD_POOL_SEM;
+ }
+
+ bq = mempool_alloc(pool, gfp);
+ if (!bq)
+ return bq;
+ size = BVECQ_POOL_SIZE;
+ } else {
+ bq = kmalloc(size, gfp);
+ if (!bq) {
+ *flags |= BVECQ_HIT_OOM;
+ return bq;
+ }
+ }
+
+success:
+ *bq = (struct bvecq) {
+ .ref = REFCOUNT_INIT(1),
+ .bv = bq->__bv,
+ .inline_bv = true,
+ .max_slots = min((size - sizeof(*bq)) / sizeof(bq->__bv[0]),
+ BVECQ_MAX_INLINE_SLOTS),
+ .from_pool = from_pool,
+ };
+ netfs_stat(&netfs_n_bvecq);
+ return bq;
+}
+EXPORT_SYMBOL(bvecq_alloc_one);
+
+/**
+ * bvecq_alloc_chain - Allocate an unpopulated bvecq chain
+ * @nr_slots: Number of slots to allocate
+ * @gfp: The allocation constraints.
+ * @flags: Pointer to the allocation flags
+ *
+ * Allocate a chain of bvecq nodes providing at least the requested cumulative
+ * number of slots. Each node is a maximum of 4KiB in size. If
+ * %BVECQ_SHORT_OKAY is set in *%flags, then the chain may be shortened on OOM.
+ *
+ * Return: The first node pointer or NULL on allocation failure. BVECQ_HIT_OOM
+ * is set in *flags if an allocation failure occurs.
+ */
+struct bvecq *bvecq_alloc_chain(size_t nr_slots, gfp_t gfp, unsigned int *flags)
+{
+ struct bvecq *head = NULL, *tail = NULL;
+
+ _enter("%zu", nr_slots);
+ *flags &= ~BVECQ_HIT_OOM;
+
+ for (;;) {
+ struct bvecq *bq;
+
+ bq = bvecq_alloc_one(min(nr_slots, BVECQ_4KB_SLOTS), gfp, flags);
+ if (!bq)
+ goto oom;
+
+ if (tail)
+ bvecq_append(tail, bq);
+ else
+ head = bq;
+ tail = bq;
+ if (unlikely((*flags & BVECQ_HIT_OOM) && (*flags & BVECQ_SHORT_OKAY)) ||
+ tail->max_slots >= nr_slots)
+ break;
+ nr_slots -= tail->max_slots;
+ }
+
+ return head;
+oom:
+ bvecq_put(head);
+ return NULL;
+}
+EXPORT_SYMBOL(bvecq_alloc_chain);
+
+/**
+ * bvecq_alloc_buffer2 - Allocate a bvecq chain and populate with buffers
+ * @size: Target size of the buffer (can be 0 for an empty buffer)
+ * @pre_slots: Number of preamble slots to set aside
+ * @gfp: The allocation constraints
+ * @flags: Pointer to the allocation flags
+ *
+ * Allocate a chain of bvecq nodes and populate the slots with sufficient pages
+ * to provide at least the requested amount of space, leaving the first
+ * @pre_slots slots unset. The pre-slots must all fit into the first bvecq.
+ * Note that any memory segments set in the pre_slots will be freed as
+ * specified by BVECQ_MEM_ALLOCED.
+ *
+ * The caller should assume that the pages allocated may be compound pages
+ * larger than PAGE_SIZE and thus occupy fewer slots. The pages have their
+ * refcounts set to 1 and can be passed to MSG_SPLICE_PAGES.
+ *
+ * Return: The first node pointer or NULL on allocation failure. BVECQ_HIT_OOM
+ * is set in *flags if a bvecq chain allocation failure occurs.
+ */
+struct bvecq *bvecq_alloc_buffer2(size_t size, unsigned int pre_slots, gfp_t gfp,
+ unsigned int *flags)
+{
+ struct bvecq *head = NULL, *tail = NULL;
+ size_t count = pre_slots + DIV_ROUND_UP(size, PAGE_SIZE);
+
+ _enter("%zx,%zx,%u", size, count, pre_slots);
+
+ *flags &= ~BVECQ_HIT_OOM;
+ if (WARN_ON_ONCE(pre_slots > BVECQ_POOL_SLOTS))
+ return NULL;
+
+ do {
+ struct page **pages;
+ size_t unused, want, got, slot;
+
+ if (!tail || bvecq_is_full(tail)) {
+ struct bvecq *bq;
+
+ bq = bvecq_alloc_one(min(count, BVECQ_4KB_SLOTS), gfp, flags);
+ if (!bq)
+ goto oom;
+
+ if (tail)
+ bvecq_append(tail, bq);
+ else
+ head = bq;
+ tail = bq;
+
+ /* Need to clear pre slots and pages[], so just clear all. */
+ memset(tail->bv, 0, tail->max_slots * sizeof(tail->bv[0]));
+ tail->mem_type = BVECQ_MEM_ALLOCED;
+ tail->nr_slots = pre_slots;
+ count -= pre_slots;
+ pre_slots = 0;
+ }
+
+ if (!count)
+ break;
+
+ unused = tail->max_slots - tail->nr_slots;
+
+ /* We can put pages[] at the tail of bv[] as the elements are
+ * smaller and we won't overrun if the slots are filled in
+ * ascending order.
+ */
+ pages = (struct page **)&tail->bv[tail->max_slots];
+ pages -= unused;
+
+ want = min(count, unused);
+ got = alloc_pages_bulk(gfp, want, pages);
+ if (!got) {
+ *flags |= BVECQ_HIT_OOM;
+ goto oom;
+ }
+
+ slot = tail->nr_slots;
+ for (int i = 0; i < got; i++)
+ bvec_set_page(&tail->bv[slot++], pages[i], PAGE_SIZE, 0);
+
+ bvecq_filled_to(tail, slot);
+ count -= got;
+
+ if (unlikely(*flags & BVECQ_HIT_OOM) && (*flags & BVECQ_SHORT_OKAY))
+ break;
+ } while (count > 0);
+
+out:
+ return head;
+oom:
+ if (*flags & BVECQ_SHORT_OKAY)
+ goto out;
+ bvecq_put(head);
+ return NULL;
+}
+EXPORT_SYMBOL(bvecq_alloc_buffer2);
+
+/*
+ * Free the page pointed to by a slot as necessary.
+ */
+static void bvecq_free_slot(struct bvecq *bq, unsigned int slot)
+{
+ struct page *page = bq->bv[slot].bv_page;
+
+ if (!page)
+ return;
+
+ switch (bq->mem_type) {
+ case BVECQ_MEM_EXTERNAL:
+ break;
+ case BVECQ_MEM_PAGECACHE:
+ put_page(page);
+ break;
+ case BVECQ_MEM_GUP:
+ unpin_user_page(page);
+ break;
+ case BVECQ_MEM_ALLOCED:
+ __free_pages(page, compound_order(page));
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
+}
+
+/**
+ * bvecq_put - Put a ref on a bvec queue
+ * @bq: The start of the folio queue to free
+ *
+ * Put the ref(s) on the nodes in a bvec queue, freeing up the node and the
+ * page fragments it points to as the refcounts become zero.
+ */
+void bvecq_put(struct bvecq *bq)
+{
+ struct bvecq *next;
+
+ for (; bq; bq = next) {
+ if (!refcount_dec_and_test(&bq->ref))
+ break;
+ for (int slot = 0; slot < bq->nr_slots; slot++)
+ bvecq_free_slot(bq, slot);
+ next = bq->next;
+ netfs_stat_d(&netfs_n_bvecq);
+
+ if (bq->from_pool != BVECQ_USE_NO_POOL)
+ mempool_free(bq, &netfs_bvecq_pool[bq->from_pool]);
+ else
+ kfree(bq);
+ }
+}
+EXPORT_SYMBOL(bvecq_put);
+
+/**
+ * bvecq_pool_unlock - Clean up emergency pool access
+ * @flags: Pointer to current allocation flag state
+ *
+ * Clean up the emergency pool access semaphore state as obtained by the
+ * bvecq_alloc_* functions.
+ */
+void bvecq_pool_unlock(unsigned int *flags)
+{
+ if (*flags & BVECQ_HOLD_POOL_SEM) {
+ up(&netfs_bvecq_pool_sem[*flags & BVECQ__USE_POOL]);
+ *flags &= ~BVECQ_HOLD_POOL_SEM;
+ }
+}
+EXPORT_SYMBOL(bvecq_pool_unlock);
+
+/**
+ * bvecq_expand_buffer - Allocate buffer space into a bvec queue
+ * @_buffer: Pointer to the bvecq chain to expand (may point to a NULL; updated).
+ * @_cur_size: Current size of the buffer (updated).
+ * @size: Target size of the buffer.
+ * @gfp: The allocation constraints.
+ *
+ * Append extra pages to a buffer to increase its capacity to the @size
+ * specified. If the current tail has space, but is not of the
+ * BVECQ_MEM_ALLOCED memory type, a separate bvecq will be allocated to hold
+ * the new memory.
+ *
+ * Return: 0 on success or -ENOMEM on failure. Note that in the case of
+ * failure, the buffer may have been partially expanded. In any event,
+ * *@_cur_size will be updated to the new buffer size.
+ */
+int bvecq_expand_buffer(struct bvecq **_buffer, size_t *_cur_size, size_t size, gfp_t gfp)
+{
+ struct bvecq *tail = *_buffer;
+ unsigned int alloc_flags = 0;
+
+ size = round_up(size, PAGE_SIZE);
+ if (tail)
+ while (tail->next)
+ tail = tail->next;
+
+ while (*_cur_size < size) {
+ struct page *page;
+ size_t need = size - *_cur_size;
+ gfp_t gfp_add;
+ int order = 0;
+
+ if (!tail || bvecq_is_full(tail) || tail->mem_type != BVECQ_MEM_ALLOCED) {
+ struct bvecq *p;
+
+ p = bvecq_alloc_one(BVECQ_POOL_SLOTS, gfp, &alloc_flags);
+ if (!p)
+ return -ENOMEM;
+ if (tail)
+ bvecq_append(tail, p);
+ else
+ *_buffer = p;
+ tail = p;
+ p->mem_type = BVECQ_MEM_ALLOCED;
+ }
+
+ if (need > PAGE_SIZE)
+ order = umin(ilog2(need) - PAGE_SHIFT, MAX_PAGECACHE_ORDER);
+
+ gfp_add = 0;
+ if (order > 0)
+ gfp_add |= __GFP_NORETRY | __GFP_NOWARN;
+ page = alloc_pages(gfp | __GFP_COMP | gfp_add, order);
+ if (!page && order > 0) {
+ page = alloc_pages(gfp | __GFP_COMP, 0);
+ order = 0;
+ }
+ if (!page)
+ return -ENOMEM;
+
+ bvec_set_page(&tail->bv[tail->nr_slots], page, PAGE_SIZE << order, 0);
+ *_cur_size += PAGE_SIZE << order;
+ bvecq_filled_to(tail, tail->nr_slots + 1);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(bvecq_expand_buffer);
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 6b583152132f..10688314660f 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -11,6 +11,8 @@
#include <linux/netfs.h>
#include <linux/fscache.h>
#include <linux/fscache-cache.h>
+#include <linux/mempool.h>
+#include <linux/semaphore.h>
#include <trace/events/netfs.h>
#include <trace/events/fscache.h>
@@ -46,6 +48,8 @@ extern spinlock_t netfs_proc_lock;
extern mempool_t netfs_request_pool;
extern mempool_t netfs_subrequest_pool;
extern mempool_t netfs_folioq_pool;
+extern mempool_t netfs_bvecq_pool[4];
+extern struct semaphore netfs_bvecq_pool_sem[4];
#ifdef CONFIG_PROC_FS
static inline void netfs_proc_add_rreq(struct netfs_io_request *rreq)
@@ -205,6 +209,7 @@ extern atomic_t netfs_n_wh_retry_write_subreq;
extern atomic_t netfs_n_wb_lock_skip;
extern atomic_t netfs_n_wb_lock_wait;
extern atomic_t netfs_n_folioq;
+extern atomic_t netfs_n_bvecq;
int netfs_stats_show(struct seq_file *m, void *v);
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index 609e22e8f76a..a185419a4039 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -7,7 +7,6 @@
#include <linux/module.h>
#include <linux/export.h>
-#include <linux/mempool.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
#include "internal.h"
@@ -30,6 +29,9 @@ mempool_t netfs_request_pool;
mempool_t netfs_subrequest_pool;
mempool_t netfs_folioq_pool;
+mempool_t netfs_bvecq_pool[4]; /* unused (NO_POOL), REQ_POOL, SUBREQ_POOL, unused */
+struct semaphore netfs_bvecq_pool_sem[4];
+
#ifdef CONFIG_PROC_FS
LIST_HEAD(netfs_io_requests);
DEFINE_SPINLOCK(netfs_proc_lock);
@@ -111,6 +113,42 @@ static int __init netfs_init(void)
if (mempool_init_kmalloc_pool(&netfs_folioq_pool, 100, sizeof(struct folio_queue)) < 0)
goto error_folioq_pool;
+ /* We set up multiple bvecq emergency pools: one for requests and one
+ * for subrequests, though there may also need to be one for
+ * reconnection. This is needed to avoid deadlock as a process will
+ * allocate for request and then for subrequest whilst still holding
+ * the first alloc (and may go through reconnection whilst setting up a
+ * subrequest).
+ *
+ * For the request pool, we allocate 100 bvecqs in the pool, but only
+ * allow half that number of processes to access the pool at any one
+ * time - which means at least one process will be able to access two
+ * bvecqs. Combined with accelerated issuing of subreqs be a process
+ * using the pool, the pool should refill at a reasonable pace.
+ *
+ */
+ int nr_per_proc = 2;
+ int nr_procs = 50;
+
+ if (mempool_init_kmalloc_pool(&netfs_bvecq_pool[1], nr_per_proc * nr_procs,
+ BVECQ_POOL_SIZE) < 0)
+ goto error_bvecq_pool_1;
+ sema_init(&netfs_bvecq_pool_sem[1], nr_procs);
+
+ /* For the subrequest pool, the allocation is expected to be set up in
+ * one go and torn down in one go (e.g. allocating a cifs encryption
+ * buffer) and an allocation may be quite large. Support up to 16MiB of
+ * bufferage in a single chain, and allow two concurrent chain
+ * allocations.
+ */
+ nr_per_proc = DIV_ROUND_UP((16 << 20) / PAGE_SIZE, BVECQ_POOL_SLOTS);
+ nr_procs = 2;
+
+ if (mempool_init_kmalloc_pool(&netfs_bvecq_pool[2], nr_per_proc * nr_procs,
+ BVECQ_POOL_SIZE) < 0)
+ goto error_bvecq_pool_2;
+ sema_init(&netfs_bvecq_pool_sem[2], nr_procs);
+
netfs_request_slab = kmem_cache_create("netfs_request",
sizeof(struct netfs_io_request), 0,
SLAB_HWCACHE_ALIGN | SLAB_ACCOUNT,
@@ -163,6 +201,10 @@ static int __init netfs_init(void)
error_reqpool:
kmem_cache_destroy(netfs_request_slab);
error_req:
+ mempool_exit(&netfs_bvecq_pool[2]);
+error_bvecq_pool_2:
+ mempool_exit(&netfs_bvecq_pool[1]);
+error_bvecq_pool_1:
mempool_exit(&netfs_folioq_pool);
error_folioq_pool:
return ret;
@@ -177,6 +219,8 @@ static void __exit netfs_exit(void)
kmem_cache_destroy(netfs_subrequest_slab);
mempool_exit(&netfs_request_pool);
kmem_cache_destroy(netfs_request_slab);
+ mempool_exit(&netfs_bvecq_pool[2]);
+ mempool_exit(&netfs_bvecq_pool[1]);
mempool_exit(&netfs_folioq_pool);
}
module_exit(netfs_exit);
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index 4b8d20559b0e..6d7839c0498d 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -6,7 +6,6 @@
*/
#include <linux/slab.h>
-#include <linux/mempool.h>
#include "internal.h"
static void netfs_free_request(struct work_struct *work);
diff --git a/fs/netfs/rolling_buffer.c b/fs/netfs/rolling_buffer.c
index d30d5ef6d86e..4347c99452d8 100644
--- a/fs/netfs/rolling_buffer.c
+++ b/fs/netfs/rolling_buffer.c
@@ -6,7 +6,6 @@
*/
#include <linux/bitops.h>
-#include <linux/mempool.h>
#include <linux/pagemap.h>
#include <linux/rolling_buffer.h>
#include <linux/slab.h>
diff --git a/fs/netfs/stats.c b/fs/netfs/stats.c
index 9a607c4e62dd..07c5f9165768 100644
--- a/fs/netfs/stats.c
+++ b/fs/netfs/stats.c
@@ -47,6 +47,7 @@ atomic_t netfs_n_wh_retry_write_subreq;
atomic_t netfs_n_wb_lock_skip;
atomic_t netfs_n_wb_lock_wait;
atomic_t netfs_n_folioq;
+atomic_t netfs_n_bvecq;
int netfs_stats_show(struct seq_file *m, void *v)
{
@@ -88,10 +89,13 @@ int netfs_stats_show(struct seq_file *m, void *v)
atomic_read(&netfs_n_rh_retry_read_subreq),
atomic_read(&netfs_n_wh_retry_write_req),
atomic_read(&netfs_n_wh_retry_write_subreq));
- seq_printf(m, "Objs : rr=%u sr=%u foq=%u wsc=%u\n",
+ seq_printf(m, "Objs : rr=%u sr=%u bq=%u foq=%u bqs=%d,%d wsc=%u\n",
atomic_read(&netfs_n_rh_rreq),
atomic_read(&netfs_n_rh_sreq),
+ atomic_read(&netfs_n_bvecq),
atomic_read(&netfs_n_folioq),
+ READ_ONCE(netfs_bvecq_pool_sem[1].count),
+ READ_ONCE(netfs_bvecq_pool_sem[2].count),
atomic_read(&netfs_n_wh_wstream_conflict));
seq_printf(m, "WbLock : skip=%u wait=%u\n",
atomic_read(&netfs_n_wb_lock_skip),
diff --git a/include/linux/bvecq.h b/include/linux/bvecq.h
index 77fd07852c33..676cccb6ce2c 100644
--- a/include/linux/bvecq.h
+++ b/include/linux/bvecq.h
@@ -21,6 +21,20 @@ enum bvecq_mem {
BVECQ_MEM_ALLOCED, /* Memory alloc'd by bvecq - can be freed/pooled */
} __mode(byte);
+/*
+ * There are two emergency pools from which bvecqs can be allocated under OOM
+ * conditions. These are intended for use in writeback where allocation is not
+ * typically allowed to fail. One pool is for request-level allocs and one is
+ * for subreq-level allocs - having two pools prevents the two levels from
+ * blocking each other; further, the two pools can have different ratios and
+ * capacities.
+ */
+enum bvecq_pool {
+ BVECQ_USE_NO_POOL = 0, /* Don't use a pool */
+ BVECQ_USE_REQ_POOL = 1, /* Use the request-level pool */
+ BVECQ_USE_SUBREQ_POOL = 2, /* Use the subrequest-level pool */
+};
+
/*
* Segmented bio_vec queue.
*
@@ -43,8 +57,139 @@ struct bvecq {
u16 max_slots; /* Number of elements allocated in bv[] */
enum bvecq_mem mem_type:3; /* What sort of memory and how to free it */
bool inline_bv:1; /* T if __bv[] is being used */
+ enum bvecq_pool from_pool; /* Which pool the bvecq can be returned to (if any) */
struct bio_vec *bv; /* Pointer to array of page fragments */
struct bio_vec __bv[]; /* Default array (if ->inline_bv) */
};
+/* Number of slots in a 512-byte mempool-backed bvecq. */
+#define BVECQ_POOL_SIZE 512
+#define BVECQ_POOL_SLOTS ((BVECQ_POOL_SIZE - sizeof(struct bvecq)) / sizeof(struct bio_vec))
+
+/* Number of slots in a 4K bvecq. */
+#define BVECQ_4KB_SLOTS ((4096 - sizeof(struct bvecq)) / sizeof(struct bio_vec))
+
+/* Maximum number of slots in an inline allocation */
+#define BVECQ_MAX_INLINE_SLOTS USHRT_MAX
+
+/* Allocation flags */
+#define BVECQ__USE_POOL 0x03 /* As enum bvecq_pool */
+#define BVECQ_SHORT_OKAY 0x04 /* A short alloc due to OOM is okay */
+#define BVECQ_HIT_OOM 0x08 /* OOM was hit during allocation (set by alloc func) */
+#define BVECQ_HOLD_POOL_SEM 0x10 /* The caller now holds a sem on the emergency pool */
+
+void bvecq_dump(const struct bvecq *bq);
+struct bvecq *bvecq_alloc_one(size_t nr_slots, gfp_t gfp, unsigned int *flags);
+struct bvecq *bvecq_alloc_chain(size_t nr_slots, gfp_t gfp, unsigned int *flags);
+struct bvecq *bvecq_alloc_buffer2(size_t size, unsigned int pre_slots, gfp_t gfp,
+ unsigned int *flags);
+void bvecq_put(struct bvecq *bq);
+int bvecq_expand_buffer(struct bvecq **_buffer, size_t *_cur_size, size_t size, gfp_t gfp);
+void bvecq_pool_unlock(unsigned int *flags);
+
+/**
+ * bvecq_alloc_buffer - Allocate a bvecq chain and populate with buffers
+ * @size: Target size of the buffer (can be 0 for an empty buffer)
+ * @gfp: The allocation constraints
+ * @flags: Pointer to the allocation flags
+ *
+ * Wrapper around ``bvecq_alloc_buffer2()``.
+ *
+ * Return: The first node pointer or NULL on allocation failure.
+ */
+static inline struct bvecq *bvecq_alloc_buffer(size_t size, gfp_t gfp, unsigned int *flags)
+{
+ return bvecq_alloc_buffer2(size, 0, gfp, flags);
+}
+
+/**
+ * bvecq_get - Get a ref on a bvecq
+ * @bq: The bvecq to get a ref on
+ *
+ * Return: @bq.
+ */
+static inline struct bvecq *bvecq_get(struct bvecq *bq)
+{
+ refcount_inc(&bq->ref);
+ return bq;
+}
+
+/**
+ * bvecq_is_full - Determine if a bvecq is full
+ * @bvecq: The object to query
+ *
+ * Return: true if full; false if not.
+ */
+static inline bool bvecq_is_full(const struct bvecq *bvecq)
+{
+ return bvecq->nr_slots >= bvecq->max_slots;
+}
+
+/**
+ * bvecq_filled_to - Release filled slots with release barrier
+ * @bvecq: The object modified
+ * @to: The latest slot filled + 1
+ */
+static inline void bvecq_filled_to(struct bvecq *bvecq, unsigned int to)
+{
+ /* Set the slot counter after filling the slot */
+ smp_store_release(&bvecq->nr_slots, to);
+}
+
+/**
+ * bvecq_nr_slots_acquire - Get the number of filled slots with acquire barrier
+ * @bvecq: The object to query
+ *
+ * Return: The number of filled slots
+ */
+static inline unsigned int bvecq_nr_slots_acquire(const struct bvecq *bvecq)
+{
+ /* Read the slot counter before looking at the slot */
+ return smp_load_acquire(&bvecq->nr_slots);
+}
+
+/**
+ * bvecq_acquire_slot - Determine if a slot is valid with acquire barrier
+ * @bvecq: The object to query
+ * @slot: The next slot
+ *
+ * Return: true if valid; false if might not be valid
+ */
+static inline bool bvecq_acquire_slot(const struct bvecq *bvecq, unsigned int slot)
+{
+ /* Read the slot counter before looking at the slot */
+ return slot < bvecq_nr_slots_acquire(bvecq);
+}
+
+/**
+ * bvecq_append - Get the next bvecq with appropriate barrier
+ * @to: The bvecq to append to
+ * @add: The bvecq to append
+ *
+ * Attach a new bvecq to a chain using an appropriate barrier to protect the
+ * write.
+ *
+ * [!] Note that this function transfers the caller's ref to the chain.
+ */
+static inline void bvecq_append(struct bvecq *to, struct bvecq *add)
+{
+ add->prev = to;
+
+ /* Make sure the initialisation is stored before the next pointer. */
+ smp_store_release(&to->next, add);
+}
+
+/**
+ * bvecq_next - Get the next bvecq with appropriate barrier
+ * @bq: The bvecq to start from
+ *
+ * Return: The next bvecq in a chain, using an appropriate barrier to protect
+ * the access.
+ */
+static inline struct bvecq *bvecq_next(const struct bvecq *bq)
+{
+ /* Read the contents of the next node after the pointer to it. */
+ return smp_load_acquire(&bq->next);
+}
+
#endif /* _LINUX_BVECQ_H */
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index e5961450f3f1..2cf1163aafa6 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -17,6 +17,7 @@
#include <linux/workqueue.h>
#include <linux/fs.h>
#include <linux/pagemap.h>
+#include <linux/bvecq.h>
#include <linux/uio.h>
#include <linux/rolling_buffer.h>
next prev parent reply other threads:[~2026-11-17 17:20 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 17:18 [PATCH v18 00/17] netfs, iov_iter: Use a chain of bio_vec arrays instead of folio_queue David Howells
2026-11-17 17:18 ` [PATCH v18 01/17] iov_iter: Fix kunit test to remove __init and __refdata marks David Howells
2026-11-17 17:19 ` [PATCH v18 02/17] Add a function to kmap one page of a multipage bio_vec David Howells
2026-11-17 17:19 ` [PATCH v18 03/17] iov_iter: Add a segmented queue of bio_vec[] David Howells
2026-11-17 17:19 ` David Howells [this message]
2026-11-17 17:19 ` [PATCH v18 05/17] iov_iter: Add an API document for the bvecq structure David Howells
2026-11-17 17:19 ` [PATCH v18 06/17] iov_iter: Add a bvecq kunit test David Howells
2026-11-17 17:19 ` [PATCH v18 07/17] afs: Fix an inconsistency in the symlink code David Howells
2026-11-17 17:19 ` [PATCH v18 08/17] netfs: Explicitly pass tail padding to netfs_writeback_single() David Howells
2026-11-17 17:19 ` [PATCH v18 09/17] netfs: Change netfs_writeback_single() to not look inside the iterator David Howells
2026-11-17 17:19 ` [PATCH v18 10/17] netfs: Pass the write length to request creation where appropriate David Howells
2026-11-17 17:19 ` [PATCH v18 11/17] afs: Use a bvecq to hold dir content rather than folioq David Howells
2026-11-17 17:19 ` [PATCH v18 12/17] cifs: Use a bvecq for buffering instead of a folioq David Howells
2026-11-17 17:19 ` [PATCH v18 13/17] smbdirect: Support ITER_BVECQ in smbdirect_map_sges_from_iter() David Howells
2026-11-17 17:19 ` [PATCH v18 14/17] netfs: Switch folioq to bvecq David Howells
2026-11-17 17:19 ` [PATCH v18 15/17] smbdirect: Remove support for ITER_FOLIOQ from smbdirect_map_sges_from_iter() David Howells
2026-11-17 17:19 ` [PATCH v18 16/17] iov_iter: Remove ITER_FOLIOQ David Howells
2026-11-17 17:19 ` [PATCH v18 17/17] netfs: Remove folio_queue David Howells
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117171958.2982801-5-dhowells@redhat.com \
--to=dhowells@redhat.com \
--cc=asmadeus@codewreck.org \
--cc=ceph-devel@vger.kernel.org \
--cc=christian@brauner.io \
--cc=ericvh@kernel.org \
--cc=hch@infradead.org \
--cc=idryomov@gmail.com \
--cc=linkinjeon@kernel.org \
--cc=linux-afs@lists.infradead.org \
--cc=linux-cifs@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=marc.dionne@auristor.com \
--cc=metze@samba.org \
--cc=netfs@lists.linux.dev \
--cc=pc@manguebit.org \
--cc=v9fs@lists.linux.dev \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®