* [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain
@ 2026-08-24 14:40 David Howells
2026-08-24 14:40 ` [PATCH v10 01/35] netfs: Fix uninitialized return value in netfs_unbuffered_write() David Howells
` (35 more replies)
0 siblings, 36 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:40 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Hi Paulo,
Could you look these patches over please?
The patches get rid of folio_queue, rolling_buffer and ITER_FOLIOQ,
replacing the folio queue construct used to manage buffers in netfslib with
one based around a segmented chain of bio_vec arrays instead. There are
three main aims here:
(1) The kernel file I/O subsystem seems to be moving towards consolidating
on the use of bio_vec arrays, so embrace this by moving netfslib to
keep track of its buffers for buffered I/O in bio_vec[] form.
(2) Netfslib already uses a bio_vec[] to handle unbuffered/DIO, so the
number of different buffering schemes used can be reduced to just a
single one.
(3) Always send an entire filesystem RPC request message to a TCP socket
with single kernel_sendmsg() call as this is faster, more efficient
and doesn't require the use of corking as it puts the entire
transmission loop inside of a single tcp_sendmsg().
For the replacement of folio_queue, a segmented chain of bio_vec arrays
rather than a single monolithic array is provided:
struct bvecq {
struct bvecq *next;
struct bvecq *prev;
refcount_t ref;
u32 priv;
u16 nr_slots;
u16 max_slots;
enum bvecq_mem mem_type:2;
bool inline_bv:1;
bool from_pool:1;
struct bio_vec *bv;
struct bio_vec __bv[];
};
The fields are:
(1) next, prev - Link segments together in a list. I want this to be
NULL-terminated linear rather than circular to make it possible to
arbitrarily glue bits on the front.
(2) ref - Refcount. Each bvecq keeps a ref on the next. I'm not sure
this is entirely necessary, but it makes sharing slices easier.
(3) priv - Private data for the owner. Dispensible; currently only used
for storing a debug ID for tracing in a patch not included here.
(4) max_slots, nr_slots. The size of bv[] and the number of slots used.
I've assumed a maximum of 65535 bio_vecs in the array (which would
represent a ~1MiB allocation).
(5) bv, __bv, inline_bv. bv points to the bio_vec[] array handled by
this segment. This may begin at __bv and if it does inline_bv should
be set (otherwise it's impossible to distinguish a separately
allocated bio_vec[] that follows immediately by coincidence).
(6) mem_type. Indicates how the memory attached to the bio_vecs should be
disposed of when the bvecq is destroyed. It can be one of:
BVECQ_MEM_EXTERNAL - Externally tracked ref; don't put
BVECQ_MEM_PAGECACHE - Pagecache; must be put
BVECQ_MEM_GUP - Pinned by from GUP; needs unpin
BVECQ_MEM_ALLOCED - Plain alloc'd pages; can be mempooled
[!] I'm not sure that this is a good name for this member or for the
enum values.
(7) from_pool. Set if the bvecq is allocated from netfslib's mempool and
should be freed to it.
I've also defined an iov_iter iterator type ITER_BVECQ to walk this sort of
construct so that it can be passed directly to sendmsg() or block-based DIO
(as cachefiles does).
This series makes the following changes to netfslib:
(1) Remove the writethrough code as the locking is really tricky to get
right and it looks like it could deadlock with Ceph if snapshots are
used.
(2) The folio_queue chain used to hold folios for buffered I/O is replaced
with a bvecq chain. Each bio_vec then holds (a portion of) one folio.
Each bvecq holds a contiguous sequence of folios, but adjacent bvecqs
in a chain may be discontiguous.
(3) For unbuffered/DIO, the source iov_iter is extracted into a bvecq
chain.
(4) An abstract position representation ('bvecq_pos') is created that can
used to hold a position in a bvecq chain. For the moment, this takes
a ref on the bvecq it points to, but that may be excessive.
(5) Buffer tracking is managed with three cursors: The load_cursor, at
which new folios are added as we go; the dispatch_cursor, at which new
subrequests' buffers start when they're created; and the
collect_cursor, the point at which folios are being unlocked.
Not all cursors are necessarily needed in all situations and during
buffered writeback, we need a dispatch cursor per stream (one for the
network filesystem and one for the cache).
(6) ->prepare_read(), buffer setting up and ->issue_read() are merged, as
are the write variants, with the filesystem calling back up to
netfslib to prepare its buffer. This simplifies the process of
setting up a subrequest. It may even make sense to have the
filesystem allocate the subrequest.
(7) Retry dispatch tracking is added to netfs_io_request so that the
buffer preparation functions can find it. Retry requires an
additional buffer cursor.
(8) Netfslib dispatches I/O by accumulating enough bufferage to dispatch
at least one subrequest, then looping to generate as many as the
filesystem wants to (they may be limited by other constraints,
e.g. max RDMA segment count or negotiated max size). This loop could
be moved down into the filesystem. A new method is provided by which
netfslib can ask the filesystem to provide an estimate of the data
that should be accumulated before dispatch begins.
(9) Reading from the cache is now managed by querying the cache to provide
a list of the next two data extents within the cache.
(10) AFS directories are switched to using a bvecq rather than a
folio_queue to hold their contents.
(11) CIFS is switch to using a bvecq rather than a folio_queue for holding
a temporary encryption buffer.
(12) CIFS RDMA is given the ability to extract ITER_BVECQ and support for
extracting ITER_FOLIOQ is removed.
(13) All the folio_queue and rolling_buffer code is removed.
Cachefiles is also modified:
(1) On-demand mode for Erofs is removed.
(2) The object type in the cachefiles file xattr is now correctly set to
CACHEFILES_CONTENT_{SINGLE,ALL,BACKFS_MAP} rather than just being 0,
to indicate whether we have a single monolithic blob, all the data up
to cache i_size with no holes or a sparse file with the data mapped by
the backing file system (as currently upstream).
(3) For "ALL" type files, the cache's i_size is used to track how much
data is saved in the cache and no longer bears any relation to the
netfs i_size. The actual object size is stored in the xattr.
(4) For most typical files which are contiguous and written progressively,
the object type is now set to "ALL". For anything else, cachefiles
uses SEEK_DATA/HOLE to find extent outlines at before (this is the
current behaviour and needs to be fixed, but in a separate set of
patches as it's not trivial).
(5) Preset the xattr on a cachefile to try to avoid having to deal with
ENOSPC from setxattr when committing the object.
Two further things that I'm working on (but not in this branch) are:
(1) Make it so that a filesystem can be given a copy of a subchain which
it can then tack header and trailer protocol elements upon to form a
single message (I have this working for cifs) and even join copies
together with intervening protocol elements to form compounds.
(2) Make it so that a filesystem can 'splice' out the contents of the TCP
receive queue into a bvecq chain. This allows the socket lock to be
dropped much more quickly and the copying of data read to the
destination buffers to happen without the lock. I have this working
for cifs too. Kernel recvmsg() doesn't then block kernel sendmsg()
for anywhere near as long.
There are also some things I want to consider for the future:
(1) Create one or more batched iteration functions to 'unlock' all the
folios in a bio_vec[], where 'unlock' is the appropriate action for
ending a read or a write. Batching should hopefully also improve the
efficiency of wrangling the marks on the xarray. Very often these
marks are going to be represented by contiguous bits, so there may be
a way to change them in bulk.
(2) Rather than walking the bvecq chain to get each individual folio out
via bv_page, use the file position stored on the bvecq and the sum of
bv_len to iterate over the appropriate range in i_pages.
(3) Change iov_iter to store the initial starting point and for
iov_iter_revert() to reset to that and advance. This would (a) help
prevent over-reversion and (b) dispense with the need for a prev
pointer.
(4) Use bvecq to replace scatterlist. One problem with replacing
scatterlist is that crypto drivers like to glue bits on the front of
the scatterlists they're given (something trivial with that API) - and
this is one way to achieve it.
The patches can also be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=netfs-next
Thanks,
David
Changes
=======
ver #10)
- Rebased on v7.2
- Pushed the cachefiles ondemand sunset patch separately.
- Pushed the patch to fix read progress handling separately.
- Changed some umin() to min().
- Changed netfs writeback to hold the list of regions that need unlocking
separately from the bvecq holding the folios to write.
- Added a patch to convert unsigned long long to uoff_t throughout
netfslib, fscache and cachefiles and to convert loff_t where appropriate.
- Split some bits out of the larger patches into smaller patches,
including:
- Made some minor adjustments to tracepoints.
- Provided helper functions to handle the barriering around setting
NETFS_RREQ_ALL_QUEUED.
- Made the use of the PG_private_2 flag optional, requiring anything
wanting to use it (NFS, Ceph) to select it in Kconfig.
- Added a new function to estimate the amount to write.
- Fixed a number of issues reported by Sashiko[8]:
- Fixed initialisation of ->progress_at to be 0, not ULONG_MAX, when
request is allocated otherwise netfs_prepare_read_iterator() won't set
it.
- Pass gfp into rolling_buffer_bulk_load_from_ra().
- Fixes iter_get_kernel_pages() to use folio_test_large_kmalloc() as well
as folio_test_slab() when deciding to take a ref.
- Fixed iter_count_bvecq_pages() and iov_iter_extract_bvecq_pages() to
assume that bio_vecs with bv_page==NULL and bv_len>0 are not allowed.
- Fixed iov_iter_extract_bvecq_pages() to check if the result of
iter_count_bvecq_pages() is zero, not less than zero.
- Fixed bvecq_pos_advance(), bvecq_zero(), bvecq_slice(),
bvecq_extract(), bvecq_pos_step() and bvecq_pos_nudge() to recheck the
slot count of the current bvecq after reading the next pointer in order
to advance to the next bvecq.
- Made bvecq_delete_spent() retain spent->next and instead take a ref on
what spend->next points to (so that putting spent doesn't delete the
entire chain) thereby avoiding breaking the chain.
- Fixed bvecq_alloc_one() to strip off __GFP_THISNODE as well as
GFP_ZONEMASK when turning a gfp for allocating folios into a gfp for
allocating bvecq objects.
- Change cachefiles_collect_write() to set CACHEFILES_CONTENT_NO_DATA if
a cache write fails and the cache content is truncated away.
- Fixed stream->cancelled boolean to be a 3-state enum instead for
tracking gap, data or cancelled writes to the cache.
- Fixed netfs_writeback_single() to take two lengths: one for the whole
buffer and one for the data content in that buffer to allow DIO
rounding to use extra buffer.
- Fixed netfs_collect_read_results() to set -EIO on the request when
abandoning it due to subreq over-read
- Fixed netfs_writeback_single() to pre-expand the buffer to include all
available buffer space, even beyond the data, and fixed
netfs_prepare_write_single_buffer() to do no rounding.
- Fixed netfs_alloc_read_subrequest() to not set stream->collected_to as
that's set when the object is initialised and subreq->start isn't set
yet.
- Fixed netfs_unbuffered_write() to not skip cleanup if it fails to
allocate a subreq.
- Fixed netfs_unbuffered_write() to fail immediately if ->issue_write()
returns an error rather than going for a retry.
- Changed netfs_unbuffered_write() to not get subreq->error if
netfs_wait_for_in_progress_subreq() fails as it returns that.
- Added a comment to netfs_writeback_single() to indicate that it doesn't
normally write to the server.
- Split the cachefiles_trace_write_nospace to distinguish the two users.
- Fixed the cachefiles file xattr set/get functions to make sure that the
storage for the cachefiles_xattr::data is at least 8 bytes and that
it's initialised to 0 so that the tracepoint doesn't cause a KASAN
error.
- Fixed missing export of the netfs_rreq tracepoint.
ver #9)
- Fixed a number of issues reported by Sashiko[7]:
- Fixed netfs_unlock_abandoned_read_pages() to make sure the refs are
dropped before unlocking.
- Fixed iov_iter_extract_bvecq_pages() to immediately return 0 if
there are no pages to be extracted from the iterator (e.g. if maxpages
is 0).
- Pass a flag into bvecq_alloc_one() to indicate if we can access
writeback memory pools rather than trying to divine this from gfp.
- Fixed bvecq_alloc_one() to use the same gfp adjustments as
mempool_alloc() when prospectively allocating what the caller asked for
during writeback.
- Simplified the bvecq-based rolling buffer in writeback by always adding
whole folios and then setting the offset appropriately. The offset and
length of the part added can then be aligned at that point when
tracking the new folio in the cache stream rather than copying and
rounding out later.
- Changed the writeback loop to ignore ENOMEM. There are now backup
memory pools for writeback.
- Simplified the writeback loop to update stream tracking as folios are
added and to flush at that point too if needed.
- Removed the warning given by netfs_extract_iter() if max_len or
max_pages is 0.
- Made netfs_extract_iter() set the fpos on the blank bvecq it allocates
in the alloc_empty path.
- Fixed netfs_writeback_single() to test NETFS_RREQ_UPLOAD_TO_SERVER
rather than test-and-setting it (nothing currently wants to
writeback-single to the server anyway, only to the cache).
- Removed cachefiles ondemand pincount.
- Fixed netfs_collect_write_results() to get the cancel/data flags to
netfs_collect_write_results() the right way round.
- Fixed netfs_writeback_unlock_folios() to just warn and oops in the
unlikely event that it sees a NULL (ie. already unlocked) folio
pointer.
- Moved some common initialisation of fields to rreq->start to
netfs_alloc_request().
- Fixed bvecq_load_from_ra() to set bq->fpos.
- Fixed read progress reporting to use an offset from rreq->start
(unchanging) rather than trying to use rreq->clean_to (tearable) as a
base.
ver #8)
- Removed the fix patches to their own branches and posted separately.
- Rebased on v7.2-rc6 as a number of fixes went in.
- Fixed cachefiles_collect_write() to handle cres->cache_priv2 (file) being
NULL due to failure to create an object (e.g. due to ENOSPC).
- Added a patch to cachefiles to preset the xattr on a cachefiles to avoid
dealing with ENOSPC from setxattr when committing an object.
- Fixed a number of bugs reported by Sashiko[6]:
- Fixed bvecq_alloc_buffer2() to break out of the loop if pre_slots > 0
and size == 0.
- Made netfs_extract_iter() limit max_pages to protect iov_iter_npages().
- Remove include/uapi/linux/cachefiles.h when ondemand mode is removed.
- Remove more ondemand-related trace bits.
- Remove ondemand-related BPF bits.
- Fixed a missing ENOMEM check in netfs_pgpriv2_begin_copy_to_cache().
- Fixed netfs_retry_read_subrequests() to unset dispatch_cursor on the
"abandon_after" error path.
- Fixed netfs_issue_write() to clear stream->construct if a subreq is
terminated due to ENOMEM.
- Fixed netfs_collect_read_results() to unpause the app if it decides to
abandon the read.
- Fixed cachefiles_issue_read() to not wait for the object state to match
FSCACHE_WANT_READ as cachefiles_query_occupancy() should have done that
already.
- Fixed netfs_retry_write_stream() to handle superfluous subreqs
correctly (borrowing from read retry).
ver #7)
- Rebased on v7.2-rc4.
- Added three fix patches to the front from Yichong Chen that conflict with
other patches in the series.
- Fixed a number of bugs reported by Sashiko[5].
- Fixed double page put in iov_kunit_destroy_bvecq().
- Fixed bvecq_alloc_buffer2() to handle pre_slots>0 and size==0
correctly.
- Fixes bvecq_expand_buffer() to use a barrer to set tail->nr_slots.
- Added functions to insert barriers when reading or setting bvecq::next
and fixed some places to use them.
- Fixed more meta block mapping in afs_edit_dir_remove().
- Added a patch to fix missing unmap in afs_dir_search_bucket().
- Removed the writethrough stat counter as the code gets removed.
- Cleaned up the cachefiles-ondemand removal patch:
- Fix removal of code that's conditional on !ONDEMAND.
- Removed msg_id_next field.
- Removed ondemand trace elements.
- Removed documentation.
- Fixed cachefiles_resize_cookie() to only consider a resize a shrink if
the new size is less than the object size.
- Fixed cachefiles_resize_cookie() to update object->object_size on
expansion.
- Add a comment on exclusion in cachefiles_collect_write().
- Fixed netfs_single_dispatch_read() to allow for the cache to have
rounded out the stored data to DIO size.
- Fixed netfs_prepare_read_iterator() to clamp subreq len to sreq_max_len
(obsoleted by last patch).
- Changed write subreq collection to never retry cache writes. Shouldn't
see them in the retry path either then.
- Fixed netfs_pgpriv2_unlock_copied_folios() to return made_progress
instead of false if we hit the end of the bvecq chain.
- Modified buffered read collection to abandon the rest of a read request
if a subreq over-reads as the buffer may have been corrupted. The read
termination function no longer winds subreq->transferred back.
- Fixed netfs_issue_read() to call the right termination func in the fail
path.
- Fixed netfs_dispatch_unbuffered_reads(), netfs_single_dispatch_read()
and netfs_retry_read_subrequests() to handle failed ->issue_read().
- Fixed afs_issue_read() to return after calling afs_end_read() as this
op is then released.
- Fixed cifs_issue_write() to always release credits on failure.
- Fixed cifs_issue_read() to not release credits twice.
- Fixed netfs_issue_writes() to advance stream->issued_to and
stream->dispatch_cursor on early failure of ->issue_write().
- Fixed netfs_issue_read() to only call netfs_all_subreqs_queued() if
stream->buffered is zero.
- Fixed ceph_netfs_issue_op_inline() to do a retry if file got uninlined
by someone else.
- Fixed ceph_netfs_issue_op_inline() to not return an error directly
once it's got a prepared buffer.
- Fixed nfs_netfs_issue_read() to return an int.
ver #6)
- Rebased on v7.2-rc3.
- Added a patch to remove erofs on-demand support from cachefiles.
- Added a patch to remove the writethrough code as it's really tricky to
get the locking right and it looks like can deadlock itself with Ceph.
- Fixed a number of bugs reported by Sashiko[4].
- Fixed iov_iter_single_seg_count(), iov_iter_alignment_bvecq() and
iov_npages_bvecq() to handle zero-length segments and segments with no
stuff remaining.
- Fixed bvecq_extract() to use GFP_NOFS rather than GFP_KERNEL.
- Fixed bvecq_alloc_buffer2()'s use of alloc_pages_bulk().
- Fixed netfs_extract_iter() to break out of the inner loop if max_pages
hits zero.
- Added a patch to fix meta block mapping in afs_edit_dir_remove().
- Fixed afs_init_new_symlink() to clear the tail of the symlink page.
- Fixed cachefiles_begin_operation() to deal with a NULL file.
- Fixed slot check in netfs_read_gaps() to be >=, not >.
- Fixed netfs_pgpriv2_copy_folio() to update creq->last_end.
- Fixed netfs_read_single() to return -EIO if the buffer isn't big
enough.
- Add a comment to netfs_write_folio() to indicate that truncate must
not run concurrently with writeback and must exclude/wait for it.
- In netfs_read_subreq_terminated(), move the netfs_sreq_trace_too_much
trace line earlier before ->transferred is clobbered.
- In netfs_read_subreq_terminated(), make the transfer-too-long case
avoid retrying and just end with EIO (in case data further along in the
buffer got corrupted).
- Fixed netfs_collect_read_results() if it sees front->transferred >
front->len, to just use front->len instead.
- Fixed ceph_netfs_issue_op_inline() to prep the buffer before setting
NETFS_SREQ_HIT_EOF so that subreq->content is set.
- Alter netfs_read_to_pagecache() to not round up reading from
zero-containing cache granules as no I/O is required.
- Made ->issue_write() return an error so that errors before any buffer
is sliced off don't result in an infinite loop in netfs_issue_writes().
- Made ->issue_read() return an error also for similiar reasons.
- In netfs_writepages(), handle -ENOMEM from bvecq prealloc before
caling netfs_queue_wb_folio() rather than doing it on the combined
error return of that and netfs_issue_streams() which can result in a
double unlock.
- Fix netfs_writeback_single() to write nothing if len == 0.
ver #5)
- Rebased on v7.2-rc2 as that has a bunch of outstanding netfs and afs
bugfixes included.
ver #4)
- Fixed a number of bugs reported by Sashiko[3].
- Added a patch to fix an underflow in iov_iter_extract_xarray_pages().
- Added a patch to fix alloc failure in iov_iter_extract_bvec_pages().
- Added a patch to remove an unused var in kunit code.
- Added a patch to fix the folio offset in extract_xarray_to_sg().
- Added a patch to fix the exclusion over writeback to make it cover
collection too.
- Fixed double fput() in cachefiles.
- Fixed the collection of cache writes to handle cancellation better.
- Fixed iterate_bvecq() to skip bvecq structs with nr_slots==0.
- Add a comment into iterate_bvecq() that a slot with bv_len>0 must have a
valid bv_page.
- Fixed iov_iter_bvecq_advance(), iov_iter_bvecq_revert(),
iter_count_bvecq_pages(), iov_iter_extract_bvecq_pages() and
extract_bvecq_to_sg() to correctly handle empty bvecqs.
- Fixed extract_bvecq_to_sg() to be limited by iter->count.
- Fixed bvecq_expand_buffer() to take an unsigned size param.
- Fixed bvecq_expand_buffer() to not mix memory types in alloc'd bvecqs.
- Fixed bvecq_shorten_buffer() occasional retention of zero-length slots.
- Fixed slot validity check polarity in bvecq_pos_advance(); also don't use
inner loop otherwise break then exits the wrong loop.
- Fixed bvecq_zero(), bvecq_slice() and bvecq_extract to use a barrier when
checking bq->nr_slots.
- Restructured bvecq_zero() to be similar to bvecq_pos_advance().
- Fixed an off-by-one error in bvecq_pos_step() and added a missing slot
reset.
- Fixed a break in netfs_extract_iter() that should have been a goto.
- Fixed netfs_extract_iter() to limit number of pages extracted to remnant
of max_pages.
- Fixed an uninit var in afs_do_read_symlink().
- Fixed netfs_read_gaps() to fill a multipart bvecq chain correctly.
- Fixed netfs_dispatch_unbuffered_reads() to initialise collect_cursor as
netfs_rreq_assess_dio() uses it to flush the data read.
- Fixed netfs_extract_iter() to init the slot counter outside the extract
loop to avoid overwriting already loaded slots.
- Fixed callers of bvecq_delete_spent() to update bvecq_pos::slot before
calling.
- Fixed netfs_reissue_write() to make sure subreq->content is unset before
setting.
- Altered netfs_extract_iter() to free any allocated bvecq chain if no pages
were extracted and an error occurred (and to initialise the return
pointer to NULL). Also, made it return an empty bvecq if nothing was
extracted, but no error occurred.
- Fixed ceph_netfs_issue_read() to just return if
ceph_netfs_issue_op_inline() returns anything other than 1 to avoid a
double termination.
- Fixed ceph_netfs_issue_read() to do the size calculation in the right
order to avoid the op expanding to larger than the buffer.
- Fixed netfs_issue_read(), in the NETFS_FILL_WITH_ZEROES case, to deduct
subreq->len from stream->buffered rather than just setting it to 0.
- Fixed netfs_perform_write() to put the folio if netfs_advance_writethrough()
fails.
- Restored the old ->prepare_write op specifically for
fscache_write_to_cache() which is still used by Ceph.
- Fixed undefined return in netfs_pgpriv2_issue_stream().
- Fixed netfs_collect_write_results() to try to make sure a request isn't
left paused if there are no further server-bound subreqs.
- Fixed netfs_queue_wb_folio() to redirty the folio before unlocking it if
it can't allocate a bvecq.
- Fixed netfs_writepages() to cancel the pagecache iteration after ENOMEM.
- Fixed netfs_advance/end_writethrough() to advance the dispatch cursor.
- Fixed netfs_retry_read_subrequests() to use barriers when walking
stream->subrequests as the app may add another subreq before pausing.
- Fixed netfs_prepare_write_retry_buffer() to use ->retry_start and
->retry_buffered rather than ->issue_from and ->buffered.
- Fixed netfs_retry_write_stream() to use barriers when walking
stream->subrequests as the app may add another subreq before pausing.
- Fixed netfs_retry_write_stream() to check the correct length when adding
additional subreqs.
- Fixed nfs_netfs_issue_read() to set -ENOMEM, not 0, on alloc failure.
- Fixed nfs_netfs_issue_read() to only terminate the subreq once.
- Fixed cifs_issue_read() to release the credits if cifs_reopen_file()
fails.
- Rebased on v7.1.
ver #3)
- Rebased to -rc7 as the patches wouldn't apply for Christian.
- Prepended a fix for a warning from generic/464 (the problem also exists
upstream, just not the warning).
- Renamed kmap_local_bvec() to bvec_kmap_partial() as requested by
Christoph.
- Adjusted smbdirect patch descriptions as requested by Stefan Metzmacher.
ver #2)
- Fixed a number of bugs reported by Sashiko[1].
- Split a bunch of fixes out and posted them separately[2].
[1] https://sashiko.dev/#/patchset/20260326104544.509518-1-dhowells%40redhat.com
[2] https://lore.kernel.org/linux-fsdevel/20260512-infozentrum-becher-7f86c47c96c8@brauner/T/#t
[3] https://sashiko.dev/#/patchset/20260608145432.681865-1-dhowells%40redhat.com
[4] https://sashiko.dev/#/patchset/20260706153408.1231650-1-dhowells%40redhat.com
[5] https://sashiko.dev/#/patchset/20260716103030.3065561-1-dhowells%40redhat.com
[6] https://sashiko.dev/#/patchset/20260722130218.78958-1-dhowells%40redhat.com
[7] https://sashiko.dev/#/patchset/20260804100224.2748935-1-dhowells%40redhat.com
[8] https://sashiko.dev/#/patchset/20260810144746.574036-1-dhowells%40redhat.com
David Howells (33):
netfs: Fix read progress reporting
cachefiles: Fix potential UAF/KASAN warning
netfs: Use uoff_t instead of unsigned long long and loff_t
mm: Make readahead store folio count in readahead_control
netfs: Bulk load the readahead-provided folios up front
Add a function to kmap one page of a multipage bio_vec
iov_iter: Make iov_iter_get_pages*() wrap iov_iter_extract_pages()
iov_iter: Add a segmented queue of bio_vec[]
netfs: Add some tools for managing bvecq chains
netfs: Make mempool available for bvecq
netfs: Add a function to extract from an iter into a bvecq
afs: Use a bvecq to hold dir content rather than folioq
cifs: Use a bvecq for buffering instead of a folioq
smbdirect: Support ITER_BVECQ in smbdirect_map_sges_from_iter()
netfs: Remove the writethrough code
netfs: trace: Change the "clear" folio traces to "endwb"
netfs: trace: Rejig a couple of the tracepoints
netfs: Add some functions to wrap the all-queued handling
netfs: Make deprecated PG_private_2 support optional
cachefiles: Don't rely on backing fs storage map for most use cases
netfs: Add the cache object ID to netfs_read/write tracepoints
netfs: Switch to using bvecq rather than folio_queue and
rolling_buffer
smbdirect: Remove support for ITER_FOLIOQ from
smbdirect_map_sges_from_iter()
netfs: Remove netfs_alloc/free_folioq_buffer()
netfs: Remove netfs_extract_user_iter()
iov_iter: Remove ITER_FOLIOQ
netfs: Remove folio_queue and rolling_buffer
netfs: Simplify read abandonment
netfs: Check for too much data being read
netfs: Add a method to get an estimate of the amount that can be
written
netfs: Rework writeback to use a separate list of regions to be
unlocked
netfs: Combine prepare and issue ops and grab the buffers on request
netfs: Clean up now-unused code
Gao Xiang (1):
cachefiles,netfs: sunset ondemand mode
Karl Mehltretter (1):
netfs: Fix uninitialized return value in netfs_unbuffered_write()
Documentation/core-api/folio_queue.rst | 209 ----
Documentation/core-api/index.rst | 1 -
.../filesystems/caching/cachefiles.rst | 179 ---
Documentation/filesystems/netfs_library.rst | 2 +-
fs/9p/vfs_addr.c | 61 +-
fs/afs/dir.c | 38 +-
fs/afs/dir_edit.c | 43 +-
fs/afs/dir_search.c | 33 +-
fs/afs/file.c | 35 +-
fs/afs/fsclient.c | 8 +-
fs/afs/inode.c | 2 +-
fs/afs/internal.h | 14 +-
fs/afs/symlink.c | 36 +-
fs/afs/write.c | 36 +-
fs/afs/yfsclient.c | 6 +-
fs/cachefiles/Kconfig | 12 -
fs/cachefiles/Makefile | 1 -
fs/cachefiles/daemon.c | 96 +-
fs/cachefiles/interface.c | 104 +-
fs/cachefiles/internal.h | 151 +--
fs/cachefiles/io.c | 603 +++++++---
fs/cachefiles/namei.c | 27 +-
fs/cachefiles/ondemand.c | 761 ------------
fs/cachefiles/xattr.c | 35 +-
fs/ceph/Kconfig | 1 +
fs/ceph/addr.c | 127 +-
fs/netfs/Kconfig | 3 +
fs/netfs/Makefile | 4 +-
fs/netfs/buffered_read.c | 555 +++++----
fs/netfs/buffered_write.c | 69 +-
fs/netfs/bvecq.c | 857 ++++++++++++++
fs/netfs/direct_read.c | 115 +-
fs/netfs/direct_write.c | 178 +--
fs/netfs/fscache_cookie.c | 8 +-
fs/netfs/fscache_io.c | 12 +-
fs/netfs/internal.h | 119 +-
fs/netfs/iterator.c | 387 ++-----
fs/netfs/main.c | 21 +-
fs/netfs/misc.c | 176 +--
fs/netfs/objects.c | 63 +-
fs/netfs/read_collect.c | 417 ++++---
fs/netfs/read_pgpriv2.c | 211 ++--
fs/netfs/read_retry.c | 261 ++---
fs/netfs/read_single.c | 184 ++-
fs/netfs/rolling_buffer.c | 226 ----
fs/netfs/stats.c | 10 +-
fs/netfs/write_collect.c | 345 ++++--
fs/netfs/write_issue.c | 1024 ++++++++---------
fs/netfs/write_retry.c | 189 +--
fs/nfs/Kconfig | 1 +
fs/nfs/fscache.c | 24 +-
fs/smb/client/cifsglob.h | 2 +-
fs/smb/client/cifssmb.c | 13 +-
fs/smb/client/file.c | 148 +--
fs/smb/client/smb2ops.c | 84 +-
fs/smb/client/smb2pdu.c | 28 +-
fs/smb/client/transport.c | 15 +-
fs/smb/smbdirect/connection.c | 135 ++-
include/linux/bvec.h | 18 +
include/linux/bvecq.h | 361 ++++++
include/linux/folio_queue.h | 282 -----
include/linux/fscache-cache.h | 2 +-
include/linux/fscache.h | 53 +-
include/linux/iov_iter.h | 87 +-
include/linux/netfs.h | 261 +++--
include/linux/pagemap.h | 10 +
include/linux/rolling_buffer.h | 61 -
include/linux/uio.h | 17 +-
include/trace/events/cachefiles.h | 254 +---
include/trace/events/fscache.h | 10 +-
include/trace/events/netfs.h | 296 +++--
include/uapi/linux/cachefiles.h | 68 --
kernel/bpf/btf.c | 9 -
lib/iov_iter.c | 556 ++++-----
lib/scatterlist.c | 82 +-
lib/tests/kunit_iov_iter.c | 131 +--
mm/readahead.c | 5 +
net/9p/client.c | 8 +-
78 files changed, 5519 insertions(+), 5557 deletions(-)
delete mode 100644 Documentation/core-api/folio_queue.rst
delete mode 100644 fs/cachefiles/ondemand.c
create mode 100644 fs/netfs/bvecq.c
delete mode 100644 fs/netfs/rolling_buffer.c
create mode 100644 include/linux/bvecq.h
delete mode 100644 include/linux/folio_queue.h
delete mode 100644 include/linux/rolling_buffer.h
delete mode 100644 include/uapi/linux/cachefiles.h
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 01/35] netfs: Fix uninitialized return value in netfs_unbuffered_write()
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
@ 2026-08-24 14:40 ` David Howells
2026-08-24 14:40 ` [PATCH v10 02/35] netfs: Fix read progress reporting David Howells
` (34 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:40 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, Karl Mehltretter, stable
From: Karl Mehltretter <kmehltretter@gmail.com>
If preparation of the first subrequest fails,
netfs_unbuffered_write() exits its loop before ret is initialized. The
empty-iterator check can do the same.
For synchronous writes, netfs_unbuffered_write_iter_locked() may then
return an unrelated error instead of wreq->error. This is reachable
through CIFS if cifs_prepare_write() fails to reopen the file or obtain
credits.
Initialize ret to 0 so the caller returns wreq->error if no data was
written, or the number of bytes already written otherwise.
Found with Clang's -Wconditional-uninitialized.
Fixes: a0b4c7a49137e ("netfs: Fix unbuffered/DIO writes to dispatch subrequests in strict sequence")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-fable-5
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Signed-off-by: David Howells <dhowells@redhat.com>
---
fs/netfs/direct_write.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fs/netfs/direct_write.c b/fs/netfs/direct_write.c
index c16fbad286a1..b04019097ab8 100644
--- a/fs/netfs/direct_write.c
+++ b/fs/netfs/direct_write.c
@@ -95,7 +95,7 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
{
struct netfs_io_subrequest *subreq = NULL;
struct netfs_io_stream *stream = &wreq->io_streams[0];
- int ret;
+ int ret = 0;
_enter("%llx", wreq->len);
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 02/35] netfs: Fix read progress reporting
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
2026-08-24 14:40 ` [PATCH v10 01/35] netfs: Fix uninitialized return value in netfs_unbuffered_write() David Howells
@ 2026-08-24 14:40 ` David Howells
2026-08-24 14:40 ` [PATCH v10 03/35] cachefiles,netfs: sunset ondemand mode David Howells
` (33 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:40 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
For really big read RPC ops that span multiple folios, netfslib allows the
filesystem to give progress notifications to wake up the collector thread
to do a collection of folios that have now been fetched, even if the RPC is
still ongoing, thereby allowing the application to make progress.
This works by taking the current rreq->cleaned_to value (which indicates
which folios have been unlocked) and adding the stashed size of the next
folio to it. cleaned_to, however, is subject to 64-bit tearing on a 32-bit
arch.
Fix this by stashing the next progress notification point as a size_t
(which won't tear) to be added to rreq->start (which won't change), with
the collector thread calculating that from cleaned_to plus the next folio
size.
Further, however, if the folios are small, the collector thread gets
constantly woken up - which has a negative performance impact on the
system.
Fix that too by setting a minimum trigger of 256KiB or the size of the
folio at the front of the queue, whichever is larger.
Also, make sure rreq->cleaned_to is initialised up front, along with
rreq->collected_to and stream->collected_to.
Fixes: e2d46f2ec332 ("netfs: Change the read result collector to only use one work item")
Link: https://sashiko.dev/#/patchset/20260804100224.2748935-1-dhowells%40redhat.com
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/buffered_read.c | 4 +++
fs/netfs/internal.h | 1 +
fs/netfs/objects.c | 32 ++++++++++++--------
fs/netfs/read_collect.c | 58 +++++++++++++++++++++++++++---------
fs/netfs/read_single.c | 2 ++
include/linux/netfs.h | 2 +-
include/trace/events/netfs.h | 21 +++++++++++++
7 files changed, 93 insertions(+), 27 deletions(-)
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 7fdfa4f27e34..3c32ef41a27f 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -106,6 +106,9 @@ static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq,
folio_batch_release(&put_batch);
return added;
}
+
+ if (!rreq->progress_at)
+ netfs_read_set_unlock_at(rreq);
rreq->submitted += added;
}
folio_batch_release(&put_batch);
@@ -387,6 +390,7 @@ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct fo
if (added < 0)
return added;
rreq->submitted = rreq->start + added;
+ rreq->progress_at = added;
return 0;
}
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 420ee7b26580..f92281a611de 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -109,6 +109,7 @@ static inline void netfs_see_subrequest(struct netfs_io_subrequest *subreq,
/*
* read_collect.c
*/
+void netfs_read_set_unlock_at(struct netfs_io_request *rreq);
bool netfs_read_collection(struct netfs_io_request *rreq);
void netfs_read_collection_worker(struct work_struct *work);
void netfs_cancel_read(struct netfs_io_subrequest *subreq, int error);
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index 01461a74642d..7f6a3e912602 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -41,24 +41,32 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
memset(rreq, 0, kmem_cache_size(cache));
INIT_WORK(&rreq->cleanup_work, netfs_free_request);
- rreq->gfp = gfp;
- rreq->start = start;
- rreq->len = len;
- rreq->origin = origin;
- rreq->netfs_ops = ctx->ops;
- rreq->mapping = mapping;
- rreq->inode = inode;
- rreq->i_size = i_size_read(inode);
- rreq->debug_id = atomic_inc_return(&debug_ids);
- rreq->wsize = INT_MAX;
+ rreq->gfp = gfp;
+ rreq->start = start;
+ rreq->collected_to = start;
+ rreq->cleaned_to = start;
+ rreq->len = len;
+ rreq->progress_at = 0;
+ rreq->origin = origin;
+ rreq->netfs_ops = ctx->ops;
+ rreq->mapping = mapping;
+ rreq->inode = inode;
+ rreq->i_size = i_size_read(inode);
+ rreq->debug_id = atomic_inc_return(&debug_ids);
+ rreq->wsize = INT_MAX;
rreq->io_streams[0].sreq_max_len = ULONG_MAX;
rreq->io_streams[0].sreq_max_segs = 0;
spin_lock_init(&rreq->lock);
- INIT_LIST_HEAD(&rreq->io_streams[0].subrequests);
- INIT_LIST_HEAD(&rreq->io_streams[1].subrequests);
init_waitqueue_head(&rreq->waitq);
refcount_set(&rreq->ref, 2);
+ for (int s = 0; s < NR_IO_STREAMS; s++) {
+ struct netfs_io_stream *stream = &rreq->io_streams[s];
+
+ INIT_LIST_HEAD(&stream->subrequests);
+ stream->collected_to = rreq->start;
+ }
+
if (origin == NETFS_READAHEAD ||
origin == NETFS_READPAGE ||
origin == NETFS_READ_GAPS ||
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index 23660a590124..723b479ef606 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -94,6 +94,35 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
folioq_clear(folioq, slot);
}
+/*
+ * Determine how much to gather before unlocking more folios.
+ */
+void netfs_read_set_unlock_at(struct netfs_io_request *rreq)
+{
+ struct folio_queue *folioq = rreq->buffer.tail;
+ unsigned int slot = rreq->buffer.first_tail_slot;
+ size_t cleaned_to = rreq->cleaned_to - rreq->start;
+ size_t progress_at = cleaned_to;
+ size_t minimum = 256 * 1024;
+
+ while (progress_at < rreq->len) {
+ if (slot >= folioq_nr_slots(folioq)) {
+ folioq = folioq->next;
+ if (!folioq)
+ break;
+ slot = 0;
+ }
+
+ progress_at += folioq_folio_size(folioq, slot);
+ if (progress_at - cleaned_to >= minimum)
+ break;
+ slot++;
+ }
+
+ WRITE_ONCE(rreq->progress_at, progress_at);
+ trace_netfs_read_progress_at(rreq);
+}
+
/*
* Unlock any folios we've finished with.
*/
@@ -112,7 +141,7 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
if (slot >= folioq_nr_slots(folioq)) {
folioq = rolling_buffer_delete_spent(&rreq->buffer);
if (!folioq) {
- rreq->front_folio_order = 0;
+ WRITE_ONCE(rreq->progress_at, ULONG_MAX);
return;
}
slot = 0;
@@ -120,8 +149,7 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
for (;;) {
struct folio *folio;
- unsigned long long fpos, fend;
- unsigned int order;
+ unsigned long long fpos = rreq->cleaned_to, fend;
size_t fsize;
if (*notes & COPY_TO_CACHE)
@@ -133,9 +161,7 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
rreq->debug_id, folio->index))
trace_netfs_folio(folio, netfs_folio_trace_not_locked);
- order = folioq_folio_order(folioq, slot);
- rreq->front_folio_order = order;
- fsize = PAGE_SIZE << order;
+ fsize = folioq_folio_size(folioq, slot);
fpos = folio_pos(folio);
fend = fpos + fsize;
@@ -146,7 +172,7 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
break;
netfs_unlock_read_folio(rreq, folioq, slot);
- WRITE_ONCE(rreq->cleaned_to, fpos + fsize);
+ WRITE_ONCE(rreq->cleaned_to, fend);
*notes |= MADE_PROGRESS;
clear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags);
@@ -172,6 +198,8 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
rreq->buffer.tail = folioq;
done:
rreq->buffer.first_tail_slot = slot;
+
+ netfs_read_set_unlock_at(rreq);
}
/*
@@ -232,7 +260,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
* subreqs.
*/
if (notes & BUFFERED) {
- size_t fsize = PAGE_SIZE << rreq->front_folio_order;
+ uoff_t unlock_at = rreq->start + rreq->progress_at;
/* Clear the tail of a short read. */
if (!(notes & HIT_PENDING) &&
@@ -257,7 +285,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
transferred = front->len;
trace_netfs_rreq(rreq, netfs_rreq_trace_set_abandon);
}
- if (front->start + transferred >= rreq->cleaned_to + fsize ||
+ if (front->start + transferred >= unlock_at ||
test_bit(NETFS_SREQ_HIT_EOF, &front->flags))
netfs_read_unlock_folios(rreq, ¬es);
} else {
@@ -477,20 +505,22 @@ void netfs_read_collection_worker(struct work_struct *work)
void netfs_read_subreq_progress(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
- struct netfs_io_stream *stream = &rreq->io_streams[0];
- size_t fsize = PAGE_SIZE << rreq->front_folio_order;
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_progress);
+ struct netfs_io_stream *stream = &rreq->io_streams[subreq->stream_nr];
+ size_t progress_at = READ_ONCE(rreq->progress_at);
+ uoff_t update_at = rreq->start + progress_at;
+ uoff_t transferred_to = subreq->start + subreq->transferred;
/* If we are at the head of the queue, wake up the collector,
* getting a ref to it if we were the ones to do so.
*/
- if (subreq->start + subreq->transferred > rreq->cleaned_to + fsize &&
+ if (progress_at != ULONG_MAX &&
+ transferred_to >= update_at &&
(rreq->origin == NETFS_READAHEAD ||
rreq->origin == NETFS_READPAGE ||
rreq->origin == NETFS_READ_FOR_WRITE) &&
list_is_first(&subreq->rreq_link, &stream->subrequests)
) {
+ trace_netfs_sreq(subreq, netfs_sreq_trace_progress);
__set_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
netfs_wake_collector(rreq);
}
diff --git a/fs/netfs/read_single.c b/fs/netfs/read_single.c
index 8833550d2eb6..de67ac41548d 100644
--- a/fs/netfs/read_single.c
+++ b/fs/netfs/read_single.c
@@ -170,6 +170,8 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite
if (IS_ERR(rreq))
return PTR_ERR(rreq);
+ rreq->progress_at = rreq->len;
+
ret = netfs_single_begin_cache_read(rreq, ictx);
if (ret == -ENOMEM || ret == -EINTR || ret == -ERESTARTSYS)
goto cleanup_free;
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index d0b62d53eea9..cc8d6500d059 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -247,6 +247,7 @@ struct netfs_io_request {
unsigned long long submitted; /* Amount submitted for I/O so far */
unsigned long long len; /* Length of the request */
size_t transferred; /* Amount to be indicated as transferred */
+ size_t progress_at; /* Report read progress when hit this much read */
long error; /* 0 or error that occurred */
unsigned long long i_size; /* Size of the file */
unsigned long long start; /* Start position */
@@ -263,7 +264,6 @@ struct netfs_io_request {
atomic_t subreq_counter; /* Next subreq->debug_index */
unsigned int nr_group_rel; /* Number of refs to release on ->group */
spinlock_t lock; /* Lock for queuing subreqs */
- unsigned char front_folio_order; /* Order (size) of front folio */
enum netfs_io_origin origin; /* Origin of the request */
bool direct_bv_unpin; /* T if direct_bv[] must be unpinned */
refcount_t ref;
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 082cb03c6131..8ec10c076875 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -786,6 +786,27 @@ TRACE_EVENT(netfs_folioq,
__print_symbolic(__entry->trace, netfs_folioq_traces))
);
+TRACE_EVENT(netfs_read_progress_at,
+ TP_PROTO(const struct netfs_io_request *rreq),
+
+ TP_ARGS(rreq),
+
+ TP_STRUCT__entry(
+ __field(unsigned int, rreq)
+ __field(size_t, progress_at)
+ __field(size_t, cleaned_to)
+ ),
+
+ TP_fast_assign(
+ __entry->rreq = rreq->debug_id;
+ __entry->cleaned_to = rreq->cleaned_to - rreq->start;
+ __entry->progress_at = rreq->progress_at;
+ ),
+
+ TP_printk("R=%08x cln=%zx prg=%zx",
+ __entry->rreq, __entry->cleaned_to, __entry->progress_at)
+ );
+
#undef EM
#undef E_
#endif /* _TRACE_NETFS_H */
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 03/35] cachefiles,netfs: sunset ondemand mode
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
2026-08-24 14:40 ` [PATCH v10 01/35] netfs: Fix uninitialized return value in netfs_unbuffered_write() David Howells
2026-08-24 14:40 ` [PATCH v10 02/35] netfs: Fix read progress reporting David Howells
@ 2026-08-24 14:40 ` David Howells
2026-08-24 14:40 ` [PATCH v10 04/35] cachefiles: Fix potential UAF/KASAN warning David Howells
` (32 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:40 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, Gao Xiang, bpf
From: Gao Xiang <xiang@kernel.org>
It was an effort to enhance fscache as a kernel cache for lazy
pulling (at least according to previous Incremental FS discussion [1])
and EROFS over fscache was the in-tree user of this mode.
fscache has since evolved to be netfslib-oriented, serving network
filesystem inodes via the netfs library, but EROFS never acts as a
network filesystem and we need to cache golden filesystem images rather
than individual EROFS inodes.
Since EROFS over fscache is now removed, clean up netfs/fscache/
cachefiles upstream too.
[1] https://lore.kernel.org/r/CAOQ4uxi4dzxArY24YO=+kBCK2gGoq3Ptb8WkzCqSogPgU_R3dQ@mail.gmail.com
[dh] Fixed up comments on:
https://sashiko.dev/#/patchset/20260716103030.3065561-1-dhowells%40redhat.com
https://sashiko.dev/#/patchset/20260722130218.78958-1-dhowells%40redhat.com
Signed-off-by: Gao Xiang <xiang@kernel.org>
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-erofs@lists.ozlabs.org
cc: bpf@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
---
.../filesystems/caching/cachefiles.rst | 179 ----
fs/cachefiles/Kconfig | 12 -
fs/cachefiles/Makefile | 1 -
fs/cachefiles/daemon.c | 96 +--
fs/cachefiles/interface.c | 8 -
fs/cachefiles/internal.h | 134 ---
fs/cachefiles/io.c | 26 +-
fs/cachefiles/namei.c | 8 -
fs/cachefiles/ondemand.c | 761 ------------------
include/linux/netfs.h | 9 -
include/trace/events/cachefiles.h | 184 +----
include/uapi/linux/cachefiles.h | 68 --
kernel/bpf/btf.c | 7 -
13 files changed, 9 insertions(+), 1484 deletions(-)
delete mode 100644 fs/cachefiles/ondemand.c
delete mode 100644 include/uapi/linux/cachefiles.h
diff --git a/Documentation/filesystems/caching/cachefiles.rst b/Documentation/filesystems/caching/cachefiles.rst
index b3ccc782cb3b..adfb7d079027 100644
--- a/Documentation/filesystems/caching/cachefiles.rst
+++ b/Documentation/filesystems/caching/cachefiles.rst
@@ -28,8 +28,6 @@ Cache on Already Mounted Filesystem
(*) Debugging.
- (*) On-demand Read.
-
Overview
========
@@ -483,180 +481,3 @@ the control file. For example::
echo $((1|4|8)) >/sys/module/cachefiles/parameters/debug
will turn on all function entry debugging.
-
-
-On-demand Read
-==============
-
-When working in its original mode, CacheFiles serves as a local cache for a
-remote networking fs - while in on-demand read mode, CacheFiles can boost the
-scenario where on-demand read semantics are needed, e.g. container image
-distribution.
-
-The essential difference between these two modes is seen when a cache miss
-occurs: In the original mode, the netfs will fetch the data from the remote
-server and then write it to the cache file; in on-demand read mode, fetching
-the data and writing it into the cache is delegated to a user daemon.
-
-``CONFIG_CACHEFILES_ONDEMAND`` should be enabled to support on-demand read mode.
-
-
-Protocol Communication
-----------------------
-
-The on-demand read mode uses a simple protocol for communication between kernel
-and user daemon. The protocol can be modeled as::
-
- kernel --[request]--> user daemon --[reply]--> kernel
-
-CacheFiles will send requests to the user daemon when needed. The user daemon
-should poll the devnode ('/dev/cachefiles') to check if there's a pending
-request to be processed. A POLLIN event will be returned when there's a pending
-request.
-
-The user daemon then reads the devnode to fetch a request to process. It should
-be noted that each read only gets one request. When it has finished processing
-the request, the user daemon should write the reply to the devnode.
-
-Each request starts with a message header of the form::
-
- struct cachefiles_msg {
- __u32 msg_id;
- __u32 opcode;
- __u32 len;
- __u32 object_id;
- __u8 data[];
- };
-
-where:
-
- * ``msg_id`` is a unique ID identifying this request among all pending
- requests.
-
- * ``opcode`` indicates the type of this request.
-
- * ``object_id`` is a unique ID identifying the cache file operated on.
-
- * ``data`` indicates the payload of this request.
-
- * ``len`` indicates the whole length of this request, including the
- header and following type-specific payload.
-
-
-Turning on On-demand Mode
--------------------------
-
-An optional parameter becomes available to the "bind" command::
-
- bind [ondemand]
-
-When the "bind" command is given no argument, it defaults to the original mode.
-When it is given the "ondemand" argument, i.e. "bind ondemand", on-demand read
-mode will be enabled.
-
-
-The OPEN Request
-----------------
-
-When the netfs opens a cache file for the first time, a request with the
-CACHEFILES_OP_OPEN opcode, a.k.a an OPEN request will be sent to the user
-daemon. The payload format is of the form::
-
- struct cachefiles_open {
- __u32 volume_key_size;
- __u32 cookie_key_size;
- __u32 fd;
- __u32 flags;
- __u8 data[];
- };
-
-where:
-
- * ``data`` contains the volume_key followed directly by the cookie_key.
- The volume key is a NUL-terminated string; the cookie key is binary
- data.
-
- * ``volume_key_size`` indicates the size of the volume key in bytes.
-
- * ``cookie_key_size`` indicates the size of the cookie key in bytes.
-
- * ``fd`` indicates an anonymous fd referring to the cache file, through
- which the user daemon can perform write/llseek file operations on the
- cache file.
-
-
-The user daemon can use the given (volume_key, cookie_key) pair to distinguish
-the requested cache file. With the given anonymous fd, the user daemon can
-fetch the data and write it to the cache file in the background, even when
-kernel has not triggered a cache miss yet.
-
-Be noted that each cache file has a unique object_id, while it may have multiple
-anonymous fds. The user daemon may duplicate anonymous fds from the initial
-anonymous fd indicated by the @fd field through dup(). Thus each object_id can
-be mapped to multiple anonymous fds, while the usr daemon itself needs to
-maintain the mapping.
-
-When implementing a user daemon, please be careful of RLIMIT_NOFILE,
-``/proc/sys/fs/nr_open`` and ``/proc/sys/fs/file-max``. Typically these needn't
-be huge since they're related to the number of open device blobs rather than
-open files of each individual filesystem.
-
-The user daemon should reply the OPEN request by issuing a "copen" (complete
-open) command on the devnode::
-
- copen <msg_id>,<cache_size>
-
-where:
-
- * ``msg_id`` must match the msg_id field of the OPEN request.
-
- * When >= 0, ``cache_size`` indicates the size of the cache file;
- when < 0, ``cache_size`` indicates any error code encountered by the
- user daemon.
-
-
-The CLOSE Request
------------------
-
-When a cookie withdrawn, a CLOSE request (opcode CACHEFILES_OP_CLOSE) will be
-sent to the user daemon. This tells the user daemon to close all anonymous fds
-associated with the given object_id. The CLOSE request has no extra payload,
-and shouldn't be replied.
-
-
-The READ Request
-----------------
-
-When a cache miss is encountered in on-demand read mode, CacheFiles will send a
-READ request (opcode CACHEFILES_OP_READ) to the user daemon. This tells the user
-daemon to fetch the contents of the requested file range. The payload is of the
-form::
-
- struct cachefiles_read {
- __u64 off;
- __u64 len;
- };
-
-where:
-
- * ``off`` indicates the starting offset of the requested file range.
-
- * ``len`` indicates the length of the requested file range.
-
-
-When it receives a READ request, the user daemon should fetch the requested data
-and write it to the cache file identified by object_id.
-
-When it has finished processing the READ request, the user daemon should reply
-by using the CACHEFILES_IOC_READ_COMPLETE ioctl on one of the anonymous fds
-associated with the object_id given in the READ request. The ioctl is of the
-form::
-
- ioctl(fd, CACHEFILES_IOC_READ_COMPLETE, msg_id);
-
-where:
-
- * ``fd`` is one of the anonymous fds associated with the object_id
- given.
-
- * ``msg_id`` must match the msg_id field of the READ request.
diff --git a/fs/cachefiles/Kconfig b/fs/cachefiles/Kconfig
index c5a070550ee3..afb25b6af5aa 100644
--- a/fs/cachefiles/Kconfig
+++ b/fs/cachefiles/Kconfig
@@ -26,15 +26,3 @@ config CACHEFILES_ERROR_INJECTION
help
This permits error injection to be enabled in cachefiles whilst a
cache is in service.
-
-config CACHEFILES_ONDEMAND
- bool "Support for on-demand read"
- depends on CACHEFILES
- default n
- help
- This permits userspace to enable the cachefiles on-demand read mode.
- In this mode, when a cache miss occurs, responsibility for fetching
- the data lies with the cachefiles backend instead of with the netfs
- and is delegated to userspace.
-
- If unsure, say N.
diff --git a/fs/cachefiles/Makefile b/fs/cachefiles/Makefile
index c37a7a9af10b..16d811f1a2fa 100644
--- a/fs/cachefiles/Makefile
+++ b/fs/cachefiles/Makefile
@@ -16,6 +16,5 @@ cachefiles-y := \
xattr.o
cachefiles-$(CONFIG_CACHEFILES_ERROR_INJECTION) += error_inject.o
-cachefiles-$(CONFIG_CACHEFILES_ONDEMAND) += ondemand.o
obj-$(CONFIG_CACHEFILES) := cachefiles.o
diff --git a/fs/cachefiles/daemon.c b/fs/cachefiles/daemon.c
index 4117b145ac94..1a66e0af2837 100644
--- a/fs/cachefiles/daemon.c
+++ b/fs/cachefiles/daemon.c
@@ -76,10 +76,6 @@ static const struct cachefiles_daemon_cmd cachefiles_daemon_cmds[] = {
{ "inuse", cachefiles_daemon_inuse },
{ "secctx", cachefiles_daemon_secctx },
{ "tag", cachefiles_daemon_tag },
-#ifdef CONFIG_CACHEFILES_ONDEMAND
- { "copen", cachefiles_ondemand_copen },
- { "restore", cachefiles_ondemand_restore },
-#endif
{ "", NULL }
};
@@ -113,9 +109,6 @@ static int cachefiles_daemon_open(struct inode *inode, struct file *file)
INIT_LIST_HEAD(&cache->volumes);
INIT_LIST_HEAD(&cache->object_list);
spin_lock_init(&cache->object_list_lock);
- refcount_set(&cache->unbind_pincount, 1);
- xa_init_flags(&cache->reqs, XA_FLAGS_ALLOC);
- xa_init_flags(&cache->ondemand_ids, XA_FLAGS_ALLOC1);
/* set default caching limits
* - limit at 1% free space and/or free files
@@ -134,54 +127,6 @@ static int cachefiles_daemon_open(struct inode *inode, struct file *file)
return 0;
}
-void cachefiles_flush_reqs(struct cachefiles_cache *cache)
-{
- struct xarray *xa = &cache->reqs;
- struct cachefiles_req *req;
- unsigned long index;
-
- /*
- * Make sure the following two operations won't be reordered.
- * 1) set CACHEFILES_DEAD bit
- * 2) flush requests in the xarray
- * Otherwise the request may be enqueued after xarray has been
- * flushed, leaving the orphan request never being completed.
- *
- * CPU 1 CPU 2
- * ===== =====
- * flush requests in the xarray
- * test CACHEFILES_DEAD bit
- * enqueue the request
- * set CACHEFILES_DEAD bit
- */
- smp_mb();
-
- xa_lock(xa);
- xa_for_each(xa, index, req) {
- req->error = -EIO;
- complete(&req->done);
- __xa_erase(xa, index);
- }
- xa_unlock(xa);
-
- xa_destroy(&cache->reqs);
- xa_destroy(&cache->ondemand_ids);
-}
-
-void cachefiles_put_unbind_pincount(struct cachefiles_cache *cache)
-{
- if (refcount_dec_and_test(&cache->unbind_pincount)) {
- cachefiles_daemon_unbind(cache);
- cachefiles_open = 0;
- kfree(cache);
- }
-}
-
-void cachefiles_get_unbind_pincount(struct cachefiles_cache *cache)
-{
- refcount_inc(&cache->unbind_pincount);
-}
-
/*
* Release a cache.
*/
@@ -194,16 +139,14 @@ static int cachefiles_daemon_release(struct inode *inode, struct file *file)
ASSERT(cache);
set_bit(CACHEFILES_DEAD, &cache->flags);
-
- if (cachefiles_in_ondemand_mode(cache))
- cachefiles_flush_reqs(cache);
+ cachefiles_daemon_unbind(cache);
/* clean up the control file interface */
cache->cachefilesd = NULL;
file->private_data = NULL;
+ cachefiles_open = 0;
- cachefiles_put_unbind_pincount(cache);
-
+ kfree(cache);
_leave("");
return 0;
}
@@ -266,10 +209,7 @@ static ssize_t cachefiles_daemon_read(struct file *file, char __user *_buffer,
if (!test_bit(CACHEFILES_READY, &cache->flags))
return 0;
- if (cachefiles_in_ondemand_mode(cache))
- return cachefiles_ondemand_daemon_read(cache, _buffer, buflen);
- else
- return cachefiles_do_daemon_read(cache, _buffer, buflen);
+ return cachefiles_do_daemon_read(cache, _buffer, buflen);
}
/*
@@ -358,28 +298,13 @@ static __poll_t cachefiles_daemon_poll(struct file *file,
struct poll_table_struct *poll)
{
struct cachefiles_cache *cache = file->private_data;
- XA_STATE(xas, &cache->reqs, 0);
- struct cachefiles_req *req;
__poll_t mask;
poll_wait(file, &cache->daemon_pollwq, poll);
mask = 0;
- if (cachefiles_in_ondemand_mode(cache)) {
- if (!xa_empty(&cache->reqs)) {
- xas_lock(&xas);
- xas_for_each_marked(&xas, req, ULONG_MAX, CACHEFILES_REQ_NEW) {
- if (!cachefiles_ondemand_is_reopening_read(req)) {
- mask |= EPOLLIN;
- break;
- }
- }
- xas_unlock(&xas);
- }
- } else {
- if (test_bit(CACHEFILES_STATE_CHANGED, &cache->flags))
- mask |= EPOLLIN;
- }
+ if (test_bit(CACHEFILES_STATE_CHANGED, &cache->flags))
+ mask |= EPOLLIN;
if (test_bit(CACHEFILES_CULLING, &cache->flags))
mask |= EPOLLOUT;
@@ -779,14 +704,7 @@ static int cachefiles_daemon_bind(struct cachefiles_cache *cache, char *args)
return -EBUSY;
}
- if (IS_ENABLED(CONFIG_CACHEFILES_ONDEMAND)) {
- if (!strcmp(args, "ondemand")) {
- set_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags);
- } else if (*args) {
- pr_err("Invalid argument to the 'bind' command\n");
- return -EINVAL;
- }
- } else if (*args) {
+ if (*args) {
pr_err("'bind' command doesn't take an argument\n");
return -EINVAL;
}
diff --git a/fs/cachefiles/interface.c b/fs/cachefiles/interface.c
index a08250d244ea..50a000310a8c 100644
--- a/fs/cachefiles/interface.c
+++ b/fs/cachefiles/interface.c
@@ -32,11 +32,6 @@ struct cachefiles_object *cachefiles_alloc_object(struct fscache_cookie *cookie)
if (!object)
return NULL;
- if (cachefiles_ondemand_init_obj_info(object, volume)) {
- kmem_cache_free(cachefiles_object_jar, object);
- return NULL;
- }
-
refcount_set(&object->ref, 1);
spin_lock_init(&object->lock);
@@ -94,7 +89,6 @@ void cachefiles_put_object(struct cachefiles_object *object,
ASSERTCMP(object->file, ==, NULL);
kfree(object->d_name);
- cachefiles_ondemand_deinit_obj_info(object);
cache = object->volume->cache->cache;
fscache_put_cookie(object->cookie, fscache_cookie_put_object);
object->cookie = NULL;
@@ -374,8 +368,6 @@ static void cachefiles_withdraw_cookie(struct fscache_cookie *cookie)
spin_unlock(&cache->object_list_lock);
}
- cachefiles_ondemand_clean_object(object);
-
if (object->file) {
cachefiles_begin_secure(cache, &saved_cred);
cachefiles_clean_up_object(object, cache);
diff --git a/fs/cachefiles/internal.h b/fs/cachefiles/internal.h
index b62cd3e9a18e..c93324e0f98c 100644
--- a/fs/cachefiles/internal.h
+++ b/fs/cachefiles/internal.h
@@ -15,8 +15,6 @@
#include <linux/fscache-cache.h>
#include <linux/cred.h>
#include <linux/security.h>
-#include <linux/xarray.h>
-#include <linux/cachefiles.h>
#define CACHEFILES_DIO_BLOCK_SIZE 4096
@@ -44,21 +42,6 @@ struct cachefiles_volume {
struct dentry *fanout[256]; /* Fanout subdirs */
};
-enum cachefiles_object_state {
- CACHEFILES_ONDEMAND_OBJSTATE_CLOSE, /* Anonymous fd closed by daemon or initial state */
- CACHEFILES_ONDEMAND_OBJSTATE_OPEN, /* Anonymous fd associated with object is available */
- CACHEFILES_ONDEMAND_OBJSTATE_REOPENING, /* Object that was closed and is being reopened. */
- CACHEFILES_ONDEMAND_OBJSTATE_DROPPING, /* Object is being dropped. */
-};
-
-struct cachefiles_ondemand_info {
- struct work_struct ondemand_work;
- int ondemand_id;
- enum cachefiles_object_state state;
- struct cachefiles_object *object;
- spinlock_t lock;
-};
-
/*
* Backing file state.
*/
@@ -74,13 +57,8 @@ struct cachefiles_object {
enum cachefiles_content content_info:8; /* Info about content presence */
unsigned long flags;
#define CACHEFILES_OBJECT_USING_TMPFILE 0 /* Have an unlinked tmpfile */
-#ifdef CONFIG_CACHEFILES_ONDEMAND
- struct cachefiles_ondemand_info *ondemand;
-#endif
};
-#define CACHEFILES_ONDEMAND_ID_CLOSED -1
-
/*
* Cache files cache definition
*/
@@ -119,35 +97,12 @@ struct cachefiles_cache {
#define CACHEFILES_DEAD 1 /* T if cache dead */
#define CACHEFILES_CULLING 2 /* T if cull engaged */
#define CACHEFILES_STATE_CHANGED 3 /* T if state changed (poll trigger) */
-#define CACHEFILES_ONDEMAND_MODE 4 /* T if in on-demand read mode */
char *rootdirname; /* name of cache root directory */
char *tag; /* cache binding tag */
- refcount_t unbind_pincount;/* refcount to do daemon unbind */
- struct xarray reqs; /* xarray of pending on-demand requests */
- unsigned long req_id_next;
- struct xarray ondemand_ids; /* xarray for ondemand_id allocation */
- u32 ondemand_id_next;
- u32 msg_id_next;
u32 secid; /* LSM security id */
bool have_secid; /* whether "secid" was set */
};
-static inline bool cachefiles_in_ondemand_mode(struct cachefiles_cache *cache)
-{
- return IS_ENABLED(CONFIG_CACHEFILES_ONDEMAND) &&
- test_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags);
-}
-
-struct cachefiles_req {
- struct cachefiles_object *object;
- struct completion done;
- refcount_t ref;
- int error;
- struct cachefiles_msg msg;
-};
-
-#define CACHEFILES_REQ_NEW XA_MARK_1
-
#include <trace/events/cachefiles.h>
static inline
@@ -190,9 +145,6 @@ extern int cachefiles_has_space(struct cachefiles_cache *cache,
* daemon.c
*/
extern const struct file_operations cachefiles_daemon_fops;
-extern void cachefiles_flush_reqs(struct cachefiles_cache *cache);
-extern void cachefiles_get_unbind_pincount(struct cachefiles_cache *cache);
-extern void cachefiles_put_unbind_pincount(struct cachefiles_cache *cache);
/*
* error_inject.c
@@ -298,90 +250,6 @@ extern struct file *cachefiles_create_tmpfile(struct cachefiles_object *object);
extern bool cachefiles_commit_tmpfile(struct cachefiles_cache *cache,
struct cachefiles_object *object);
-/*
- * ondemand.c
- */
-#ifdef CONFIG_CACHEFILES_ONDEMAND
-extern ssize_t cachefiles_ondemand_daemon_read(struct cachefiles_cache *cache,
- char __user *_buffer, size_t buflen);
-
-extern int cachefiles_ondemand_copen(struct cachefiles_cache *cache,
- char *args);
-
-extern int cachefiles_ondemand_restore(struct cachefiles_cache *cache,
- char *args);
-
-extern int cachefiles_ondemand_init_object(struct cachefiles_object *object);
-extern void cachefiles_ondemand_clean_object(struct cachefiles_object *object);
-
-extern int cachefiles_ondemand_read(struct cachefiles_object *object,
- loff_t pos, size_t len);
-
-extern int cachefiles_ondemand_init_obj_info(struct cachefiles_object *obj,
- struct cachefiles_volume *volume);
-extern void cachefiles_ondemand_deinit_obj_info(struct cachefiles_object *obj);
-
-#define CACHEFILES_OBJECT_STATE_FUNCS(_state, _STATE) \
-static inline bool \
-cachefiles_ondemand_object_is_##_state(const struct cachefiles_object *object) \
-{ \
- return object->ondemand->state == CACHEFILES_ONDEMAND_OBJSTATE_##_STATE; \
-} \
- \
-static inline void \
-cachefiles_ondemand_set_object_##_state(struct cachefiles_object *object) \
-{ \
- object->ondemand->state = CACHEFILES_ONDEMAND_OBJSTATE_##_STATE; \
-}
-
-CACHEFILES_OBJECT_STATE_FUNCS(open, OPEN);
-CACHEFILES_OBJECT_STATE_FUNCS(close, CLOSE);
-CACHEFILES_OBJECT_STATE_FUNCS(reopening, REOPENING);
-CACHEFILES_OBJECT_STATE_FUNCS(dropping, DROPPING);
-
-static inline bool cachefiles_ondemand_is_reopening_read(struct cachefiles_req *req)
-{
- return cachefiles_ondemand_object_is_reopening(req->object) &&
- req->msg.opcode == CACHEFILES_OP_READ;
-}
-
-#else
-static inline ssize_t cachefiles_ondemand_daemon_read(struct cachefiles_cache *cache,
- char __user *_buffer, size_t buflen)
-{
- return -EOPNOTSUPP;
-}
-
-static inline int cachefiles_ondemand_init_object(struct cachefiles_object *object)
-{
- return 0;
-}
-
-static inline void cachefiles_ondemand_clean_object(struct cachefiles_object *object)
-{
-}
-
-static inline int cachefiles_ondemand_read(struct cachefiles_object *object,
- loff_t pos, size_t len)
-{
- return -EOPNOTSUPP;
-}
-
-static inline int cachefiles_ondemand_init_obj_info(struct cachefiles_object *obj,
- struct cachefiles_volume *volume)
-{
- return 0;
-}
-static inline void cachefiles_ondemand_deinit_obj_info(struct cachefiles_object *obj)
-{
-}
-
-static inline bool cachefiles_ondemand_is_reopening_read(struct cachefiles_req *req)
-{
- return false;
-}
-#endif
-
/*
* security.c
*/
@@ -430,8 +298,6 @@ do { \
pr_err("I/O Error: " FMT"\n", ##__VA_ARGS__); \
fscache_io_error((___cache)->cache); \
set_bit(CACHEFILES_DEAD, &(___cache)->flags); \
- if (cachefiles_in_ondemand_mode(___cache)) \
- cachefiles_flush_reqs(___cache); \
} while (0)
#define cachefiles_io_error_obj(object, FMT, ...) \
diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c
index d879b80a0bed..9540ec25b3cb 100644
--- a/fs/cachefiles/io.c
+++ b/fs/cachefiles/io.c
@@ -390,7 +390,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
size_t len = *_len;
loff_t off, to;
ino_t ino = file ? file_inode(file)->i_ino : 0;
- int rc;
_enter("%zx @%llx/%llx", len, start, i_size);
@@ -403,8 +402,7 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
if (test_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, &cookie->flags)) {
__set_bit(NETFS_SREQ_COPY_TO_CACHE, _flags);
why = cachefiles_trace_read_no_data;
- if (!test_bit(NETFS_SREQ_ONDEMAND, _flags))
- goto out_no_object;
+ goto out_no_object;
}
/* The object and the file may be being created in the background. */
@@ -421,7 +419,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
object = cachefiles_cres_object(cres);
cache = object->volume->cache;
cachefiles_begin_secure(cache, &saved_cred);
-retry:
off = cachefiles_inject_read_error();
if (off == 0)
off = vfs_llseek(file, start, SEEK_DATA);
@@ -474,14 +471,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
download_and_store:
__set_bit(NETFS_SREQ_COPY_TO_CACHE, _flags);
- if (test_bit(NETFS_SREQ_ONDEMAND, _flags)) {
- rc = cachefiles_ondemand_read(object, start, len);
- if (!rc) {
- __clear_bit(NETFS_SREQ_ONDEMAND, _flags);
- goto retry;
- }
- ret = NETFS_INVALID_READ;
- }
out:
cachefiles_end_secure(cache, saved_cred);
out_no_object:
@@ -501,18 +490,6 @@ static enum netfs_io_source cachefiles_prepare_read(struct netfs_io_subrequest *
&subreq->flags, subreq->rreq->inode->i_ino);
}
-/*
- * Prepare an on-demand read operation, shortening it to a cached/uncached
- * boundary as appropriate.
- */
-static enum netfs_io_source
-cachefiles_prepare_ondemand_read(struct netfs_cache_resources *cres,
- loff_t start, size_t *_len, loff_t i_size,
- unsigned long *_flags, ino_t ino)
-{
- return cachefiles_do_prepare_read(cres, start, _len, i_size, _flags, ino);
-}
-
/*
* Prepare for a write to occur.
*/
@@ -731,7 +708,6 @@ static const struct netfs_cache_ops cachefiles_netfs_cache_ops = {
.prepare_read = cachefiles_prepare_read,
.prepare_write = cachefiles_prepare_write,
.prepare_write_subreq = cachefiles_prepare_write_subreq,
- .prepare_ondemand_read = cachefiles_prepare_ondemand_read,
.query_occupancy = cachefiles_query_occupancy,
};
diff --git a/fs/cachefiles/namei.c b/fs/cachefiles/namei.c
index 8a9f6be15828..88955249a1a6 100644
--- a/fs/cachefiles/namei.c
+++ b/fs/cachefiles/namei.c
@@ -442,10 +442,6 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)
if (!cachefiles_mark_inode_in_use(object, file_inode(file)))
WARN_ON(1);
- ret = cachefiles_ondemand_init_object(object);
- if (ret < 0)
- goto err_unuse;
-
ni_size = object->cookie->object_size;
ni_size = round_up(ni_size, CACHEFILES_DIO_BLOCK_SIZE);
@@ -546,10 +542,6 @@ static bool cachefiles_open_file(struct cachefiles_object *object,
}
_debug("file -> %pd positive", dentry);
- ret = cachefiles_ondemand_init_object(object);
- if (ret < 0)
- goto error_fput;
-
ret = cachefiles_check_auxdata(object, file);
if (ret < 0)
goto check_failed;
diff --git a/fs/cachefiles/ondemand.c b/fs/cachefiles/ondemand.c
deleted file mode 100644
index 0849eaf583cd..000000000000
--- a/fs/cachefiles/ondemand.c
+++ /dev/null
@@ -1,761 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-#include <linux/anon_inodes.h>
-#include <linux/uio.h>
-#include "internal.h"
-
-struct ondemand_anon_file {
- struct file *file;
- int fd;
-};
-
-static inline void cachefiles_req_put(struct cachefiles_req *req)
-{
- if (refcount_dec_and_test(&req->ref))
- kfree(req);
-}
-
-static int cachefiles_ondemand_fd_release(struct inode *inode,
- struct file *file)
-{
- struct cachefiles_object *object = file->private_data;
- struct cachefiles_cache *cache;
- struct cachefiles_ondemand_info *info;
- int object_id;
- struct cachefiles_req *req;
- XA_STATE(xas, NULL, 0);
-
- if (!object)
- return 0;
-
- info = object->ondemand;
- cache = object->volume->cache;
- xas.xa = &cache->reqs;
-
- xa_lock(&cache->reqs);
- spin_lock(&info->lock);
- object_id = info->ondemand_id;
- info->ondemand_id = CACHEFILES_ONDEMAND_ID_CLOSED;
- cachefiles_ondemand_set_object_close(object);
- spin_unlock(&info->lock);
-
- /* Only flush CACHEFILES_REQ_NEW marked req to avoid race with daemon_read */
- xas_for_each_marked(&xas, req, ULONG_MAX, CACHEFILES_REQ_NEW) {
- if (req->msg.object_id == object_id &&
- req->msg.opcode == CACHEFILES_OP_CLOSE) {
- complete(&req->done);
- xas_store(&xas, NULL);
- }
- }
- xa_unlock(&cache->reqs);
-
- xa_erase(&cache->ondemand_ids, object_id);
- trace_cachefiles_ondemand_fd_release(object, object_id);
- cachefiles_put_object(object, cachefiles_obj_put_ondemand_fd);
- cachefiles_put_unbind_pincount(cache);
- return 0;
-}
-
-static ssize_t cachefiles_ondemand_fd_write_iter(struct kiocb *kiocb,
- struct iov_iter *iter)
-{
- struct cachefiles_object *object = kiocb->ki_filp->private_data;
- struct cachefiles_cache *cache = object->volume->cache;
- struct file *file;
- size_t len = iter->count, aligned_len = len;
- loff_t pos = kiocb->ki_pos;
- const struct cred *saved_cred;
- int ret;
-
- spin_lock(&object->lock);
- file = object->file;
- if (!file) {
- spin_unlock(&object->lock);
- return -ENOBUFS;
- }
- get_file(file);
- spin_unlock(&object->lock);
-
- cachefiles_begin_secure(cache, &saved_cred);
- ret = __cachefiles_prepare_write(object, file, &pos, &aligned_len, len, true);
- cachefiles_end_secure(cache, saved_cred);
- if (ret < 0)
- goto out;
-
- trace_cachefiles_ondemand_fd_write(object, file_inode(file), pos, len);
- ret = __cachefiles_write(object, file, pos, iter, NULL, NULL);
- if (ret > 0)
- kiocb->ki_pos += ret;
-
-out:
- fput(file);
- return ret;
-}
-
-static loff_t cachefiles_ondemand_fd_llseek(struct file *filp, loff_t pos,
- int whence)
-{
- struct cachefiles_object *object = filp->private_data;
- struct file *file;
- loff_t ret;
-
- spin_lock(&object->lock);
- file = object->file;
- if (!file) {
- spin_unlock(&object->lock);
- return -ENOBUFS;
- }
- get_file(file);
- spin_unlock(&object->lock);
-
- ret = vfs_llseek(file, pos, whence);
- fput(file);
-
- return ret;
-}
-
-static long cachefiles_ondemand_fd_ioctl(struct file *filp, unsigned int ioctl,
- unsigned long id)
-{
- struct cachefiles_object *object = filp->private_data;
- struct cachefiles_cache *cache = object->volume->cache;
- struct cachefiles_req *req;
- XA_STATE(xas, &cache->reqs, id);
-
- if (ioctl != CACHEFILES_IOC_READ_COMPLETE)
- return -EINVAL;
-
- if (!test_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags))
- return -EOPNOTSUPP;
-
- xa_lock(&cache->reqs);
- req = xas_load(&xas);
- if (!req || req->msg.opcode != CACHEFILES_OP_READ ||
- req->object != object) {
- xa_unlock(&cache->reqs);
- return -EINVAL;
- }
- xas_store(&xas, NULL);
- xa_unlock(&cache->reqs);
-
- trace_cachefiles_ondemand_cread(object, id);
- complete(&req->done);
- return 0;
-}
-
-static const struct file_operations cachefiles_ondemand_fd_fops = {
- .owner = THIS_MODULE,
- .release = cachefiles_ondemand_fd_release,
- .write_iter = cachefiles_ondemand_fd_write_iter,
- .llseek = cachefiles_ondemand_fd_llseek,
- .unlocked_ioctl = cachefiles_ondemand_fd_ioctl,
-};
-
-/*
- * OPEN request Completion (copen)
- * - command: "copen <id>,<cache_size>"
- * <cache_size> indicates the object size if >=0, error code if negative
- */
-int cachefiles_ondemand_copen(struct cachefiles_cache *cache, char *args)
-{
- struct cachefiles_req *req;
- struct fscache_cookie *cookie;
- struct cachefiles_ondemand_info *info;
- char *pid, *psize;
- unsigned long id;
- long size;
- int ret;
- XA_STATE(xas, &cache->reqs, 0);
-
- if (!test_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags))
- return -EOPNOTSUPP;
-
- if (!*args) {
- pr_err("Empty id specified\n");
- return -EINVAL;
- }
-
- pid = args;
- psize = strchr(args, ',');
- if (!psize) {
- pr_err("Cache size is not specified\n");
- return -EINVAL;
- }
-
- *psize = 0;
- psize++;
-
- ret = kstrtoul(pid, 0, &id);
- if (ret)
- return ret;
-
- xa_lock(&cache->reqs);
- xas.xa_index = id;
- req = xas_load(&xas);
- if (!req || req->msg.opcode != CACHEFILES_OP_OPEN ||
- !req->object->ondemand->ondemand_id) {
- xa_unlock(&cache->reqs);
- return -EINVAL;
- }
- xas_store(&xas, NULL);
- xa_unlock(&cache->reqs);
-
- info = req->object->ondemand;
- /* fail OPEN request if copen format is invalid */
- ret = kstrtol(psize, 0, &size);
- if (ret) {
- req->error = ret;
- goto out;
- }
-
- /* fail OPEN request if daemon reports an error */
- if (size < 0) {
- if (!IS_ERR_VALUE(size)) {
- req->error = -EINVAL;
- ret = -EINVAL;
- } else {
- req->error = size;
- ret = 0;
- }
- goto out;
- }
-
- spin_lock(&info->lock);
- /*
- * The anonymous fd was closed before copen ? Fail the request.
- *
- * t1 | t2
- * ---------------------------------------------------------
- * cachefiles_ondemand_copen
- * req = xa_erase(&cache->reqs, id)
- * // Anon fd is maliciously closed.
- * cachefiles_ondemand_fd_release
- * xa_lock(&cache->reqs)
- * cachefiles_ondemand_set_object_close(object)
- * xa_unlock(&cache->reqs)
- * cachefiles_ondemand_set_object_open
- * // No one will ever close it again.
- * cachefiles_ondemand_daemon_read
- * cachefiles_ondemand_select_req
- *
- * Get a read req but its fd is already closed. The daemon can't
- * issue a cread ioctl with an closed fd, then hung.
- */
- if (info->ondemand_id == CACHEFILES_ONDEMAND_ID_CLOSED) {
- spin_unlock(&info->lock);
- req->error = -EBADFD;
- goto out;
- }
- cookie = req->object->cookie;
- cookie->object_size = size;
- if (size)
- clear_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, &cookie->flags);
- else
- set_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, &cookie->flags);
- trace_cachefiles_ondemand_copen(req->object, id, size);
-
- cachefiles_ondemand_set_object_open(req->object);
- spin_unlock(&info->lock);
- wake_up_all(&cache->daemon_pollwq);
-
-out:
- spin_lock(&info->lock);
- /* Need to set object close to avoid reopen status continuing */
- if (info->ondemand_id == CACHEFILES_ONDEMAND_ID_CLOSED)
- cachefiles_ondemand_set_object_close(req->object);
- spin_unlock(&info->lock);
- complete(&req->done);
- return ret;
-}
-
-int cachefiles_ondemand_restore(struct cachefiles_cache *cache, char *args)
-{
- struct cachefiles_req *req;
-
- XA_STATE(xas, &cache->reqs, 0);
-
- if (!test_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags))
- return -EOPNOTSUPP;
-
- /*
- * Reset the requests to CACHEFILES_REQ_NEW state, so that the
- * requests have been processed halfway before the crash of the
- * user daemon could be reprocessed after the recovery.
- */
- xas_lock(&xas);
- xas_for_each(&xas, req, ULONG_MAX)
- xas_set_mark(&xas, CACHEFILES_REQ_NEW);
- xas_unlock(&xas);
-
- wake_up_all(&cache->daemon_pollwq);
- return 0;
-}
-
-static int cachefiles_ondemand_get_fd(struct cachefiles_req *req,
- struct ondemand_anon_file *anon_file)
-{
- struct cachefiles_object *object;
- struct cachefiles_cache *cache;
- struct cachefiles_open *load;
- u32 object_id;
- int ret;
-
- object = cachefiles_grab_object(req->object,
- cachefiles_obj_get_ondemand_fd);
- cache = object->volume->cache;
-
- ret = xa_alloc_cyclic(&cache->ondemand_ids, &object_id, NULL,
- XA_LIMIT(1, INT_MAX),
- &cache->ondemand_id_next, GFP_KERNEL);
- if (ret < 0)
- goto err;
-
- anon_file->fd = get_unused_fd_flags(O_WRONLY);
- if (anon_file->fd < 0) {
- ret = anon_file->fd;
- goto err_free_id;
- }
-
- anon_file->file = anon_inode_getfile_fmode("[cachefiles]",
- &cachefiles_ondemand_fd_fops, object,
- O_WRONLY, FMODE_PWRITE | FMODE_LSEEK);
- if (IS_ERR(anon_file->file)) {
- ret = PTR_ERR(anon_file->file);
- goto err_put_fd;
- }
-
- spin_lock(&object->ondemand->lock);
- if (object->ondemand->ondemand_id > 0) {
- spin_unlock(&object->ondemand->lock);
- /* Pair with check in cachefiles_ondemand_fd_release(). */
- anon_file->file->private_data = NULL;
- ret = -EEXIST;
- goto err_put_file;
- }
-
- load = (void *)req->msg.data;
- load->fd = anon_file->fd;
- object->ondemand->ondemand_id = object_id;
- spin_unlock(&object->ondemand->lock);
-
- cachefiles_get_unbind_pincount(cache);
- trace_cachefiles_ondemand_open(object, &req->msg, load);
- return 0;
-
-err_put_file:
- fput(anon_file->file);
- anon_file->file = NULL;
-err_put_fd:
- put_unused_fd(anon_file->fd);
- anon_file->fd = ret;
-err_free_id:
- xa_erase(&cache->ondemand_ids, object_id);
-err:
- spin_lock(&object->ondemand->lock);
- /* Avoid marking an opened object as closed. */
- if (object->ondemand->ondemand_id <= 0)
- cachefiles_ondemand_set_object_close(object);
- spin_unlock(&object->ondemand->lock);
- cachefiles_put_object(object, cachefiles_obj_put_ondemand_fd);
- return ret;
-}
-
-static void ondemand_object_worker(struct work_struct *work)
-{
- struct cachefiles_ondemand_info *info =
- container_of(work, struct cachefiles_ondemand_info, ondemand_work);
-
- cachefiles_ondemand_init_object(info->object);
-}
-
-/*
- * If there are any inflight or subsequent READ requests on the
- * closed object, reopen it.
- * Skip read requests whose related object is reopening.
- */
-static struct cachefiles_req *cachefiles_ondemand_select_req(struct xa_state *xas,
- unsigned long xa_max)
-{
- struct cachefiles_req *req;
- struct cachefiles_object *object;
- struct cachefiles_ondemand_info *info;
-
- xas_for_each_marked(xas, req, xa_max, CACHEFILES_REQ_NEW) {
- if (req->msg.opcode != CACHEFILES_OP_READ)
- return req;
- object = req->object;
- info = object->ondemand;
- if (cachefiles_ondemand_object_is_close(object)) {
- cachefiles_ondemand_set_object_reopening(object);
- queue_work(fscache_wq, &info->ondemand_work);
- continue;
- }
- if (cachefiles_ondemand_object_is_reopening(object))
- continue;
- return req;
- }
- return NULL;
-}
-
-static inline bool cachefiles_ondemand_finish_req(struct cachefiles_req *req,
- struct xa_state *xas, int err)
-{
- if (unlikely(!xas || !req))
- return false;
-
- if (xa_cmpxchg(xas->xa, xas->xa_index, req, NULL, 0) != req)
- return false;
-
- req->error = err;
- complete(&req->done);
- return true;
-}
-
-ssize_t cachefiles_ondemand_daemon_read(struct cachefiles_cache *cache,
- char __user *_buffer, size_t buflen)
-{
- struct cachefiles_req *req;
- struct cachefiles_msg *msg;
- size_t n;
- int ret = 0;
- struct ondemand_anon_file anon_file;
- XA_STATE(xas, &cache->reqs, cache->req_id_next);
-
- xa_lock(&cache->reqs);
- /*
- * Cyclically search for a request that has not ever been processed,
- * to prevent requests from being processed repeatedly, and make
- * request distribution fair.
- */
- req = cachefiles_ondemand_select_req(&xas, ULONG_MAX);
- if (!req && cache->req_id_next > 0) {
- xas_set(&xas, 0);
- req = cachefiles_ondemand_select_req(&xas, cache->req_id_next - 1);
- }
- if (!req) {
- xa_unlock(&cache->reqs);
- return 0;
- }
-
- msg = &req->msg;
- n = msg->len;
-
- if (n > buflen) {
- xa_unlock(&cache->reqs);
- return -EMSGSIZE;
- }
-
- xas_clear_mark(&xas, CACHEFILES_REQ_NEW);
- cache->req_id_next = xas.xa_index + 1;
- refcount_inc(&req->ref);
- cachefiles_grab_object(req->object, cachefiles_obj_get_read_req);
- xa_unlock(&cache->reqs);
-
- if (msg->opcode == CACHEFILES_OP_OPEN) {
- ret = cachefiles_ondemand_get_fd(req, &anon_file);
- if (ret)
- goto out;
- }
-
- msg->msg_id = xas.xa_index;
- msg->object_id = req->object->ondemand->ondemand_id;
-
- if (copy_to_user(_buffer, msg, n) != 0)
- ret = -EFAULT;
-
- if (msg->opcode == CACHEFILES_OP_OPEN) {
- if (ret < 0) {
- fput(anon_file.file);
- put_unused_fd(anon_file.fd);
- goto out;
- }
- fd_install(anon_file.fd, anon_file.file);
- }
-out:
- cachefiles_put_object(req->object, cachefiles_obj_put_read_req);
- /* Remove error request and CLOSE request has no reply */
- if (ret || msg->opcode == CACHEFILES_OP_CLOSE)
- cachefiles_ondemand_finish_req(req, &xas, ret);
- cachefiles_req_put(req);
- return ret ? ret : n;
-}
-
-typedef int (*init_req_fn)(struct cachefiles_req *req, void *private);
-
-static int cachefiles_ondemand_send_req(struct cachefiles_object *object,
- enum cachefiles_opcode opcode,
- size_t data_len,
- init_req_fn init_req,
- void *private)
-{
- struct cachefiles_cache *cache = object->volume->cache;
- struct cachefiles_req *req = NULL;
- XA_STATE(xas, &cache->reqs, 0);
- int ret;
-
- if (!test_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags))
- return 0;
-
- if (test_bit(CACHEFILES_DEAD, &cache->flags)) {
- ret = -EIO;
- goto out;
- }
-
- req = kzalloc(sizeof(*req) + data_len, GFP_KERNEL);
- if (!req) {
- ret = -ENOMEM;
- goto out;
- }
-
- refcount_set(&req->ref, 1);
- req->object = object;
- init_completion(&req->done);
- req->msg.opcode = opcode;
- req->msg.len = sizeof(struct cachefiles_msg) + data_len;
-
- ret = init_req(req, private);
- if (ret)
- goto out;
-
- do {
- /*
- * Stop enqueuing the request when daemon is dying. The
- * following two operations need to be atomic as a whole.
- * 1) check cache state, and
- * 2) enqueue request if cache is alive.
- * Otherwise the request may be enqueued after xarray has been
- * flushed, leaving the orphan request never being completed.
- *
- * CPU 1 CPU 2
- * ===== =====
- * test CACHEFILES_DEAD bit
- * set CACHEFILES_DEAD bit
- * flush requests in the xarray
- * enqueue the request
- */
- xas_lock(&xas);
-
- if (test_bit(CACHEFILES_DEAD, &cache->flags) ||
- cachefiles_ondemand_object_is_dropping(object)) {
- xas_unlock(&xas);
- ret = -EIO;
- goto out;
- }
-
- /* coupled with the barrier in cachefiles_flush_reqs() */
- smp_mb();
-
- if (opcode == CACHEFILES_OP_CLOSE &&
- !cachefiles_ondemand_object_is_open(object)) {
- WARN_ON_ONCE(object->ondemand->ondemand_id == 0);
- xas_unlock(&xas);
- ret = -EIO;
- goto out;
- }
-
- /*
- * Cyclically find a free xas to avoid msg_id reuse that would
- * cause the daemon to successfully copen a stale msg_id.
- */
- xas.xa_index = cache->msg_id_next;
- xas_find_marked(&xas, UINT_MAX, XA_FREE_MARK);
- if (xas.xa_node == XAS_RESTART) {
- xas.xa_index = 0;
- xas_find_marked(&xas, cache->msg_id_next - 1, XA_FREE_MARK);
- }
- if (xas.xa_node == XAS_RESTART)
- xas_set_err(&xas, -EBUSY);
-
- xas_store(&xas, req);
- if (xas_valid(&xas)) {
- cache->msg_id_next = xas.xa_index + 1;
- xas_clear_mark(&xas, XA_FREE_MARK);
- xas_set_mark(&xas, CACHEFILES_REQ_NEW);
- }
- xas_unlock(&xas);
- } while (xas_nomem(&xas, GFP_KERNEL));
-
- ret = xas_error(&xas);
- if (ret)
- goto out;
-
- wake_up_all(&cache->daemon_pollwq);
-wait:
- ret = wait_for_completion_killable(&req->done);
- if (!ret) {
- ret = req->error;
- } else {
- ret = -EINTR;
- if (!cachefiles_ondemand_finish_req(req, &xas, ret)) {
- /* Someone will complete it soon. */
- cpu_relax();
- goto wait;
- }
- }
- cachefiles_req_put(req);
- return ret;
-out:
- /* Reset the object to close state in error handling path.
- * If error occurs after creating the anonymous fd,
- * cachefiles_ondemand_fd_release() will set object to close.
- */
- if (opcode == CACHEFILES_OP_OPEN &&
- !cachefiles_ondemand_object_is_dropping(object))
- cachefiles_ondemand_set_object_close(object);
- kfree(req);
- return ret;
-}
-
-static int cachefiles_ondemand_init_open_req(struct cachefiles_req *req,
- void *private)
-{
- struct cachefiles_object *object = req->object;
- struct fscache_cookie *cookie = object->cookie;
- struct fscache_volume *volume = object->volume->vcookie;
- struct cachefiles_open *load = (void *)req->msg.data;
- size_t volume_key_size, cookie_key_size;
- void *volume_key, *cookie_key;
-
- /*
- * Volume key is a NUL-terminated string. key[0] stores strlen() of the
- * string, followed by the content of the string (excluding '\0').
- */
- volume_key_size = volume->key[0] + 1;
- volume_key = volume->key + 1;
-
- /* Cookie key is binary data, which is netfs specific. */
- cookie_key_size = cookie->key_len;
- cookie_key = fscache_get_key(cookie);
-
- if (!(object->cookie->advice & FSCACHE_ADV_WANT_CACHE_SIZE)) {
- pr_err("WANT_CACHE_SIZE is needed for on-demand mode\n");
- return -EINVAL;
- }
-
- load->volume_key_size = volume_key_size;
- load->cookie_key_size = cookie_key_size;
- memcpy(load->data, volume_key, volume_key_size);
- memcpy(load->data + volume_key_size, cookie_key, cookie_key_size);
-
- return 0;
-}
-
-static int cachefiles_ondemand_init_close_req(struct cachefiles_req *req,
- void *private)
-{
- struct cachefiles_object *object = req->object;
-
- if (!cachefiles_ondemand_object_is_open(object))
- return -ENOENT;
-
- trace_cachefiles_ondemand_close(object, &req->msg);
- return 0;
-}
-
-struct cachefiles_read_ctx {
- loff_t off;
- size_t len;
-};
-
-static int cachefiles_ondemand_init_read_req(struct cachefiles_req *req,
- void *private)
-{
- struct cachefiles_object *object = req->object;
- struct cachefiles_read *load = (void *)req->msg.data;
- struct cachefiles_read_ctx *read_ctx = private;
-
- load->off = read_ctx->off;
- load->len = read_ctx->len;
- trace_cachefiles_ondemand_read(object, &req->msg, load);
- return 0;
-}
-
-int cachefiles_ondemand_init_object(struct cachefiles_object *object)
-{
- struct fscache_cookie *cookie = object->cookie;
- struct fscache_volume *volume = object->volume->vcookie;
- size_t volume_key_size, cookie_key_size, data_len;
-
- if (!object->ondemand)
- return 0;
-
- /*
- * CacheFiles will firstly check the cache file under the root cache
- * directory. If the coherency check failed, it will fallback to
- * creating a new tmpfile as the cache file. Reuse the previously
- * allocated object ID if any.
- */
- if (cachefiles_ondemand_object_is_open(object))
- return 0;
-
- volume_key_size = volume->key[0] + 1;
- cookie_key_size = cookie->key_len;
- data_len = sizeof(struct cachefiles_open) +
- volume_key_size + cookie_key_size;
-
- return cachefiles_ondemand_send_req(object, CACHEFILES_OP_OPEN,
- data_len, cachefiles_ondemand_init_open_req, NULL);
-}
-
-void cachefiles_ondemand_clean_object(struct cachefiles_object *object)
-{
- unsigned long index;
- struct cachefiles_req *req;
- struct cachefiles_cache *cache;
-
- if (!object->ondemand)
- return;
-
- cachefiles_ondemand_send_req(object, CACHEFILES_OP_CLOSE, 0,
- cachefiles_ondemand_init_close_req, NULL);
-
- if (!object->ondemand->ondemand_id)
- return;
-
- /* Cancel all requests for the object that is being dropped. */
- cache = object->volume->cache;
- xa_lock(&cache->reqs);
- cachefiles_ondemand_set_object_dropping(object);
- xa_for_each(&cache->reqs, index, req) {
- if (req->object == object) {
- req->error = -EIO;
- complete(&req->done);
- __xa_erase(&cache->reqs, index);
- }
- }
- xa_unlock(&cache->reqs);
-
- /* Wait for ondemand_object_worker() to finish to avoid UAF. */
- cancel_work_sync(&object->ondemand->ondemand_work);
-}
-
-int cachefiles_ondemand_init_obj_info(struct cachefiles_object *object,
- struct cachefiles_volume *volume)
-{
- if (!cachefiles_in_ondemand_mode(volume->cache))
- return 0;
-
- object->ondemand = kzalloc_obj(struct cachefiles_ondemand_info);
- if (!object->ondemand)
- return -ENOMEM;
-
- object->ondemand->object = object;
- spin_lock_init(&object->ondemand->lock);
- INIT_WORK(&object->ondemand->ondemand_work, ondemand_object_worker);
- return 0;
-}
-
-void cachefiles_ondemand_deinit_obj_info(struct cachefiles_object *object)
-{
- kfree(object->ondemand);
- object->ondemand = NULL;
-}
-
-int cachefiles_ondemand_read(struct cachefiles_object *object,
- loff_t pos, size_t len)
-{
- struct cachefiles_read_ctx read_ctx = {pos, len};
-
- return cachefiles_ondemand_send_req(object, CACHEFILES_OP_READ,
- sizeof(struct cachefiles_read),
- cachefiles_ondemand_init_read_req, &read_ctx);
-}
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index cc8d6500d059..a8702bd52d3a 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -191,7 +191,6 @@ struct netfs_io_subrequest {
#define NETFS_SREQ_COPY_TO_CACHE 0 /* Set if should copy the data to the cache */
#define NETFS_SREQ_CLEAR_TAIL 1 /* Set if the rest of the read should be cleared */
#define NETFS_SREQ_MADE_PROGRESS 4 /* Set if we transferred at least some data */
-#define NETFS_SREQ_ONDEMAND 5 /* Set if it's from on-demand read mode */
#define NETFS_SREQ_BOUNDARY 6 /* Set if ends on hard boundary (eg. ceph object) */
#define NETFS_SREQ_HIT_EOF 7 /* Set if short due to EOF */
#define NETFS_SREQ_IN_PROGRESS 8 /* Unlocked when the subrequest completes */
@@ -374,14 +373,6 @@ struct netfs_cache_ops {
loff_t *_start, size_t *_len, size_t upper_len,
loff_t i_size, bool no_space_allocated_yet);
- /* Prepare an on-demand read operation, shortening it to a cached/uncached
- * boundary as appropriate.
- */
- enum netfs_io_source (*prepare_ondemand_read)(struct netfs_cache_resources *cres,
- loff_t start, size_t *_len,
- loff_t i_size,
- unsigned long *_flags, ino_t ino);
-
/* Query the occupancy of the cache in a region, returning where the
* next chunk of data starts and how long it is.
*/
diff --git a/include/trace/events/cachefiles.h b/include/trace/events/cachefiles.h
index 6e3b1424eea4..9259bc71049e 100644
--- a/include/trace/events/cachefiles.h
+++ b/include/trace/events/cachefiles.h
@@ -31,10 +31,6 @@ enum cachefiles_obj_ref_trace {
cachefiles_obj_see_lookup_failed,
cachefiles_obj_see_withdraw_cookie,
cachefiles_obj_see_withdrawal,
- cachefiles_obj_get_ondemand_fd,
- cachefiles_obj_put_ondemand_fd,
- cachefiles_obj_get_read_req,
- cachefiles_obj_put_read_req,
};
enum fscache_why_object_killed {
@@ -129,11 +125,7 @@ enum cachefiles_error_trace {
EM(cachefiles_obj_see_lookup_cookie, "SEE lookup_cookie") \
EM(cachefiles_obj_see_lookup_failed, "SEE lookup_failed") \
EM(cachefiles_obj_see_withdraw_cookie, "SEE withdraw_cookie") \
- EM(cachefiles_obj_see_withdrawal, "SEE withdrawal") \
- EM(cachefiles_obj_get_ondemand_fd, "GET ondemand_fd") \
- EM(cachefiles_obj_put_ondemand_fd, "PUT ondemand_fd") \
- EM(cachefiles_obj_get_read_req, "GET read_req") \
- E_(cachefiles_obj_put_read_req, "PUT read_req")
+ E_(cachefiles_obj_see_withdrawal, "SEE withdrawal")
#define cachefiles_coherency_traces \
EM(cachefiles_coherency_check_aux, "BAD aux ") \
@@ -687,180 +679,6 @@ TRACE_EVENT(cachefiles_io_error,
__entry->error)
);
-TRACE_EVENT(cachefiles_ondemand_open,
- TP_PROTO(struct cachefiles_object *obj, struct cachefiles_msg *msg,
- struct cachefiles_open *load),
-
- TP_ARGS(obj, msg, load),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, msg_id)
- __field(unsigned int, object_id)
- __field(unsigned int, fd)
- __field(unsigned int, flags)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->msg_id = msg->msg_id;
- __entry->object_id = msg->object_id;
- __entry->fd = load->fd;
- __entry->flags = load->flags;
- ),
-
- TP_printk("o=%08x mid=%x oid=%x fd=%d f=%x",
- __entry->obj,
- __entry->msg_id,
- __entry->object_id,
- __entry->fd,
- __entry->flags)
- );
-
-TRACE_EVENT(cachefiles_ondemand_copen,
- TP_PROTO(struct cachefiles_object *obj, unsigned int msg_id,
- long len),
-
- TP_ARGS(obj, msg_id, len),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, msg_id)
- __field(long, len)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->msg_id = msg_id;
- __entry->len = len;
- ),
-
- TP_printk("o=%08x mid=%x l=%lx",
- __entry->obj,
- __entry->msg_id,
- __entry->len)
- );
-
-TRACE_EVENT(cachefiles_ondemand_close,
- TP_PROTO(struct cachefiles_object *obj, struct cachefiles_msg *msg),
-
- TP_ARGS(obj, msg),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, msg_id)
- __field(unsigned int, object_id)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->msg_id = msg->msg_id;
- __entry->object_id = msg->object_id;
- ),
-
- TP_printk("o=%08x mid=%x oid=%x",
- __entry->obj,
- __entry->msg_id,
- __entry->object_id)
- );
-
-TRACE_EVENT(cachefiles_ondemand_read,
- TP_PROTO(struct cachefiles_object *obj, struct cachefiles_msg *msg,
- struct cachefiles_read *load),
-
- TP_ARGS(obj, msg, load),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, msg_id)
- __field(unsigned int, object_id)
- __field(loff_t, start)
- __field(size_t, len)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->msg_id = msg->msg_id;
- __entry->object_id = msg->object_id;
- __entry->start = load->off;
- __entry->len = load->len;
- ),
-
- TP_printk("o=%08x mid=%x oid=%x s=%llx l=%zx",
- __entry->obj,
- __entry->msg_id,
- __entry->object_id,
- __entry->start,
- __entry->len)
- );
-
-TRACE_EVENT(cachefiles_ondemand_cread,
- TP_PROTO(struct cachefiles_object *obj, unsigned int msg_id),
-
- TP_ARGS(obj, msg_id),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, msg_id)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->msg_id = msg_id;
- ),
-
- TP_printk("o=%08x mid=%x",
- __entry->obj,
- __entry->msg_id)
- );
-
-TRACE_EVENT(cachefiles_ondemand_fd_write,
- TP_PROTO(struct cachefiles_object *obj, struct inode *backer,
- loff_t start, size_t len),
-
- TP_ARGS(obj, backer, start, len),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, backer)
- __field(loff_t, start)
- __field(size_t, len)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->backer = backer->i_ino;
- __entry->start = start;
- __entry->len = len;
- ),
-
- TP_printk("o=%08x iB=%x s=%llx l=%zx",
- __entry->obj,
- __entry->backer,
- __entry->start,
- __entry->len)
- );
-
-TRACE_EVENT(cachefiles_ondemand_fd_release,
- TP_PROTO(struct cachefiles_object *obj, int object_id),
-
- TP_ARGS(obj, object_id),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, object_id)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->object_id = object_id;
- ),
-
- TP_printk("o=%08x oid=%x",
- __entry->obj,
- __entry->object_id)
- );
-
#endif /* _TRACE_CACHEFILES_H */
/* This part must be outside protection */
diff --git a/include/uapi/linux/cachefiles.h b/include/uapi/linux/cachefiles.h
deleted file mode 100644
index 78caa73e5343..000000000000
--- a/include/uapi/linux/cachefiles.h
+++ /dev/null
@@ -1,68 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-#ifndef _LINUX_CACHEFILES_H
-#define _LINUX_CACHEFILES_H
-
-#include <linux/types.h>
-#include <linux/ioctl.h>
-
-/*
- * Fscache ensures that the maximum length of cookie key is 255. The volume key
- * is controlled by netfs, and generally no bigger than 255.
- */
-#define CACHEFILES_MSG_MAX_SIZE 1024
-
-enum cachefiles_opcode {
- CACHEFILES_OP_OPEN,
- CACHEFILES_OP_CLOSE,
- CACHEFILES_OP_READ,
-};
-
-/*
- * Message Header
- *
- * @msg_id a unique ID identifying this message
- * @opcode message type, CACHEFILE_OP_*
- * @len message length, including message header and following data
- * @object_id a unique ID identifying a cache file
- * @data message type specific payload
- */
-struct cachefiles_msg {
- __u32 msg_id;
- __u32 opcode;
- __u32 len;
- __u32 object_id;
- __u8 data[];
-};
-
-/*
- * @data contains the volume_key followed directly by the cookie_key. volume_key
- * is a NUL-terminated string; @volume_key_size indicates the size of the volume
- * key in bytes. cookie_key is binary data, which is netfs specific;
- * @cookie_key_size indicates the size of the cookie key in bytes.
- *
- * @fd identifies an anon_fd referring to the cache file.
- */
-struct cachefiles_open {
- __u32 volume_key_size;
- __u32 cookie_key_size;
- __u32 fd;
- __u32 flags;
- __u8 data[];
-};
-
-/*
- * @off indicates the starting offset of the requested file range
- * @len indicates the length of the requested file range
- */
-struct cachefiles_read {
- __u64 off;
- __u64 len;
-};
-
-/*
- * Reply for READ request
- * @arg for this ioctl is the @id field of READ request.
- */
-#define CACHEFILES_IOC_READ_COMPLETE _IOW(0x98, 1, int)
-
-#endif
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index c4673a54c4ba..7e25051dc1ea 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -6676,13 +6676,6 @@ static const struct bpf_raw_tp_null_args raw_tp_null_args[] = {
{ "cachefiles_mark_inactive", 0x1 },
{ "cachefiles_vfs_error", 0x1 },
{ "cachefiles_io_error", 0x1 },
- { "cachefiles_ondemand_open", 0x1 },
- { "cachefiles_ondemand_copen", 0x1 },
- { "cachefiles_ondemand_close", 0x1 },
- { "cachefiles_ondemand_read", 0x1 },
- { "cachefiles_ondemand_cread", 0x1 },
- { "cachefiles_ondemand_fd_write", 0x1 },
- { "cachefiles_ondemand_fd_release", 0x1 },
/* ext4, from ext4__mballoc event class */
{ "ext4_mballoc_discard", 0x10 },
{ "ext4_mballoc_free", 0x10 },
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 04/35] cachefiles: Fix potential UAF/KASAN warning
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (2 preceding siblings ...)
2026-08-24 14:40 ` [PATCH v10 03/35] cachefiles,netfs: sunset ondemand mode David Howells
@ 2026-08-24 14:40 ` David Howells
2026-08-24 14:40 ` [PATCH v10 05/35] netfs: Use uoff_t instead of unsigned long long and loff_t David Howells
` (31 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:40 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Currently, trace_cachefiles_coherency() is being passed a pointer to a
__be64 lain over the coherency data in struct cachefiles_xattr so that it
can display the first 8 bytes. However, the data is of variable length and
could even be 0 bytes. This could lead to a UAF or KASAN warning.
Fix this by making sure the buffer has room for at least 8 bytes and that
those 8 bytes are pre-cleared.
Further, those bytes are not 8-byte aligned, so fix the tracepoint to
extract the data as four 2-byte words (they are 2-byte aligned) and
reassemble the __be64. The compiler will convert this into a single 8-byte
load where the CPU supports it.
Fixes: 229105e5cfd9 ("cachefiles: Add auxiliary data trace")
Link: https://sashiko.dev/#/patchset/20260810144746.574036-1-dhowells%40redhat.com
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/cachefiles/xattr.c | 16 ++++++++--------
include/trace/events/cachefiles.h | 19 +++++++++++++++++--
2 files changed, 25 insertions(+), 10 deletions(-)
diff --git a/fs/cachefiles/xattr.c b/fs/cachefiles/xattr.c
index f8ae78b3f7b6..92990cfdcca7 100644
--- a/fs/cachefiles/xattr.c
+++ b/fs/cachefiles/xattr.c
@@ -13,6 +13,7 @@
#include <linux/quotaops.h>
#include <linux/xattr.h>
#include <linux/slab.h>
+#include <linux/unaligned.h>
#include "internal.h"
#define CACHEFILES_COOKIE_TYPE_DATA 1
@@ -50,7 +51,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
_enter("%x,#%d", object->debug_id, len);
- buf = kmalloc(sizeof(struct cachefiles_xattr) + len, GFP_KERNEL);
+ buf = kmalloc(sizeof(struct cachefiles_xattr) + min(len, sizeof(__be64)), GFP_KERNEL);
if (!buf)
return -ENOMEM;
@@ -60,6 +61,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
buf->content = object->content_info;
if (test_bit(FSCACHE_COOKIE_LOCAL_WRITE, &object->cookie->flags))
buf->content = CACHEFILES_CONTENT_DIRTY;
+ put_unaligned_be64(0, (__be64 *)buf->data);
if (len > 0)
memcpy(buf->data, fscache_get_aux(object->cookie), len);
@@ -77,8 +79,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
trace_cachefiles_vfs_error(object, file_inode(file), ret,
cachefiles_trace_setxattr_error);
trace_cachefiles_coherency(object, file_inode(file)->i_ino,
- be64_to_cpup((__be64 *)buf->data),
- buf->content,
+ buf->data, buf->content,
cachefiles_coherency_set_fail);
if (ret != -ENOMEM)
cachefiles_io_error_obj(
@@ -86,8 +87,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
"Failed to set xattr with error %d", ret);
} else {
trace_cachefiles_coherency(object, file_inode(file)->i_ino,
- be64_to_cpup((__be64 *)buf->data),
- buf->content,
+ buf->data, buf->content,
cachefiles_coherency_set_ok);
}
@@ -110,9 +110,10 @@ int cachefiles_check_auxdata(struct cachefiles_object *object, struct file *file
int ret = -ESTALE;
tlen = sizeof(struct cachefiles_xattr) + len;
- buf = kmalloc(tlen, GFP_KERNEL);
+ buf = kmalloc(sizeof(struct cachefiles_xattr) + min(len, sizeof(__be64)), GFP_KERNEL);
if (!buf)
return -ENOMEM;
+ put_unaligned_be64(0, (__be64 *)buf->data);
xlen = cachefiles_inject_read_error();
if (xlen == 0)
@@ -148,8 +149,7 @@ int cachefiles_check_auxdata(struct cachefiles_object *object, struct file *file
out:
trace_cachefiles_coherency(object, file_inode(file)->i_ino,
- be64_to_cpup((__be64 *)buf->data),
- buf->content, why);
+ buf->data, buf->content, why);
kfree(buf);
return ret;
}
diff --git a/include/trace/events/cachefiles.h b/include/trace/events/cachefiles.h
index 9259bc71049e..927338f8fe85 100644
--- a/include/trace/events/cachefiles.h
+++ b/include/trace/events/cachefiles.h
@@ -372,7 +372,7 @@ TRACE_EVENT(cachefiles_rename,
TRACE_EVENT(cachefiles_coherency,
TP_PROTO(struct cachefiles_object *obj,
ino_t ino,
- u64 disk_aux,
+ const void *disk_aux,
enum cachefiles_content content,
enum cachefiles_coherency_trace why),
@@ -389,12 +389,27 @@ TRACE_EVENT(cachefiles_coherency,
),
TP_fast_assign(
+ union {
+ __be16 s[4];
+ __be64 ll;
+ } x;
+
__entry->obj = obj->debug_id;
__entry->why = why;
__entry->content = content;
__entry->ino = ino;
__entry->aux = be64_to_cpup((__be64 *)obj->cookie->inline_aux);
- __entry->disk_aux = disk_aux;
+
+ /* cachefiles_xattr::data is not 64-byte aligned. */
+ if (disk_aux) {
+ x.s[0] = ((__be16 *)disk_aux)[0];
+ x.s[1] = ((__be16 *)disk_aux)[1];
+ x.s[2] = ((__be16 *)disk_aux)[2];
+ x.s[3] = ((__be16 *)disk_aux)[3];
+ __entry->disk_aux = be64_to_cpu(x.ll);
+ } else {
+ __entry->disk_aux = 0;
+ }
),
TP_printk("o=%08x %s B=%llx c=%u aux=%llx dsk=%llx",
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 05/35] netfs: Use uoff_t instead of unsigned long long and loff_t
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (3 preceding siblings ...)
2026-08-24 14:40 ` [PATCH v10 04/35] cachefiles: Fix potential UAF/KASAN warning David Howells
@ 2026-08-24 14:40 ` David Howells
2026-08-24 14:40 ` [PATCH v10 06/35] mm: Make readahead store folio count in readahead_control David Howells
` (30 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:40 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Use uoff_t instead of unsigned long long and loff_t for file positions that
can't be negative.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/afs/file.c | 4 +-
fs/cachefiles/interface.c | 10 ++--
fs/cachefiles/internal.h | 4 +-
fs/cachefiles/io.c | 24 +++++-----
fs/ceph/addr.c | 4 +-
fs/netfs/buffered_read.c | 24 +++++-----
fs/netfs/buffered_write.c | 10 ++--
fs/netfs/direct_read.c | 2 +-
fs/netfs/direct_write.c | 8 ++--
fs/netfs/fscache_cookie.c | 8 ++--
fs/netfs/fscache_io.c | 8 ++--
fs/netfs/internal.h | 10 ++--
fs/netfs/iterator.c | 2 +-
fs/netfs/misc.c | 8 ++--
fs/netfs/objects.c | 2 +-
fs/netfs/read_collect.c | 4 +-
fs/netfs/read_pgpriv2.c | 8 ++--
fs/netfs/read_retry.c | 2 +-
fs/netfs/write_collect.c | 10 ++--
fs/netfs/write_issue.c | 10 ++--
fs/netfs/write_retry.c | 2 +-
include/linux/fscache-cache.h | 2 +-
include/linux/fscache.h | 36 +++++++--------
include/linux/netfs.h | 74 ++++++++++++++---------------
include/trace/events/cachefiles.h | 30 ++++++------
include/trace/events/fscache.h | 10 ++--
include/trace/events/netfs.h | 77 +++++++++++++++----------------
27 files changed, 196 insertions(+), 197 deletions(-)
diff --git a/fs/afs/file.c b/fs/afs/file.c
index 0467742bfeee..3380a0d20c0f 100644
--- a/fs/afs/file.c
+++ b/fs/afs/file.c
@@ -413,7 +413,7 @@ static int afs_init_request(struct netfs_io_request *rreq, struct file *file)
return 0;
}
-static int afs_check_write_begin(struct file *file, loff_t pos, unsigned len,
+static int afs_check_write_begin(struct file *file, uoff_t pos, unsigned len,
struct folio **foliop, void **_fsdata)
{
struct afs_vnode *vnode = AFS_FS_I(file_inode(file));
@@ -451,7 +451,7 @@ void afs_set_i_size(struct afs_vnode *vnode, loff_t new_i_size)
fscache_update_cookie(afs_vnode_cache(vnode), NULL, &new_i_size);
}
-static void afs_update_i_size(struct inode *inode, loff_t new_i_size)
+static void afs_update_i_size(struct inode *inode, uoff_t new_i_size)
{
afs_set_i_size(AFS_FS_I(inode), new_i_size);
}
diff --git a/fs/cachefiles/interface.c b/fs/cachefiles/interface.c
index 50a000310a8c..a160d5c3e74c 100644
--- a/fs/cachefiles/interface.c
+++ b/fs/cachefiles/interface.c
@@ -111,7 +111,7 @@ static int cachefiles_adjust_size(struct cachefiles_object *object)
struct iattr newattrs;
struct file *file = object->file;
uint64_t ni_size;
- loff_t oi_size;
+ uoff_t oi_size;
int ret;
ni_size = object->cookie->object_size;
@@ -225,11 +225,11 @@ static bool cachefiles_lookup_cookie(struct fscache_cookie *cookie)
* any unused granules.
*/
static bool cachefiles_shorten_object(struct cachefiles_object *object,
- struct file *file, loff_t new_size)
+ struct file *file, uoff_t new_size)
{
struct cachefiles_cache *cache = object->volume->cache;
struct inode *inode = file_inode(file);
- loff_t i_size, dio_size;
+ uoff_t i_size, dio_size;
int ret;
dio_size = round_up(new_size, CACHEFILES_DIO_BLOCK_SIZE);
@@ -271,14 +271,14 @@ static bool cachefiles_shorten_object(struct cachefiles_object *object,
* Resize the backing object.
*/
static void cachefiles_resize_cookie(struct netfs_cache_resources *cres,
- loff_t new_size)
+ uoff_t new_size)
{
struct cachefiles_object *object = cachefiles_cres_object(cres);
struct cachefiles_cache *cache = object->volume->cache;
struct fscache_cookie *cookie = object->cookie;
const struct cred *saved_cred;
struct file *file = cachefiles_cres_file(cres);
- loff_t old_size = cookie->object_size;
+ uoff_t old_size = cookie->object_size;
_enter("%llu->%llu", old_size, new_size);
diff --git a/fs/cachefiles/internal.h b/fs/cachefiles/internal.h
index c93324e0f98c..60bd801ada04 100644
--- a/fs/cachefiles/internal.h
+++ b/fs/cachefiles/internal.h
@@ -203,11 +203,11 @@ extern bool cachefiles_begin_operation(struct netfs_cache_resources *cres,
enum fscache_want_state want_state);
extern int __cachefiles_prepare_write(struct cachefiles_object *object,
struct file *file,
- loff_t *_start, size_t *_len, size_t upper_len,
+ uoff_t *_start, size_t *_len, size_t upper_len,
bool no_space_allocated_yet);
extern int __cachefiles_write(struct cachefiles_object *object,
struct file *file,
- loff_t start_pos,
+ uoff_t start_pos,
struct iov_iter *iter,
netfs_io_terminated_t term_func,
void *term_func_priv);
diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c
index 9540ec25b3cb..7de8069d15b6 100644
--- a/fs/cachefiles/io.c
+++ b/fs/cachefiles/io.c
@@ -19,7 +19,7 @@
struct cachefiles_kiocb {
struct kiocb iocb;
refcount_t ki_refcnt;
- loff_t start;
+ uoff_t start;
union {
size_t skipped;
size_t len;
@@ -73,7 +73,7 @@ static void cachefiles_read_complete(struct kiocb *iocb, long ret)
* Initiate a read from the cache.
*/
static int cachefiles_read(struct netfs_cache_resources *cres,
- loff_t start_pos,
+ uoff_t start_pos,
struct iov_iter *iter,
enum netfs_read_from_hole read_hole,
netfs_io_terminated_t term_func,
@@ -197,8 +197,8 @@ static int cachefiles_read(struct netfs_cache_resources *cres,
* of data starts and how long it is.
*/
static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
- loff_t start, size_t len, size_t granularity,
- loff_t *_data_start, size_t *_data_len)
+ uoff_t start, size_t len, size_t granularity,
+ uoff_t *_data_start, size_t *_data_len)
{
struct cachefiles_object *object;
struct file *file;
@@ -280,7 +280,7 @@ static void cachefiles_write_complete(struct kiocb *iocb, long ret)
*/
int __cachefiles_write(struct cachefiles_object *object,
struct file *file,
- loff_t start_pos,
+ uoff_t start_pos,
struct iov_iter *iter,
netfs_io_terminated_t term_func,
void *term_func_priv)
@@ -357,7 +357,7 @@ int __cachefiles_write(struct cachefiles_object *object,
}
static int cachefiles_write(struct netfs_cache_resources *cres,
- loff_t start_pos,
+ uoff_t start_pos,
struct iov_iter *iter,
netfs_io_terminated_t term_func,
void *term_func_priv)
@@ -377,7 +377,7 @@ static int cachefiles_write(struct netfs_cache_resources *cres,
static inline enum netfs_io_source
cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
- loff_t start, size_t *_len, loff_t i_size,
+ uoff_t start, size_t *_len, loff_t i_size,
unsigned long *_flags, ino_t netfs_ino)
{
enum cachefiles_prepare_read_trace why;
@@ -483,7 +483,7 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
* boundary as appropriate.
*/
static enum netfs_io_source cachefiles_prepare_read(struct netfs_io_subrequest *subreq,
- unsigned long long i_size)
+ uoff_t i_size)
{
return cachefiles_do_prepare_read(&subreq->rreq->cache_resources,
subreq->start, &subreq->len, i_size,
@@ -495,7 +495,7 @@ static enum netfs_io_source cachefiles_prepare_read(struct netfs_io_subrequest *
*/
int __cachefiles_prepare_write(struct cachefiles_object *object,
struct file *file,
- loff_t *_start, size_t *_len, size_t upper_len,
+ uoff_t *_start, size_t *_len, size_t upper_len,
bool no_space_allocated_yet)
{
struct cachefiles_cache *cache = object->volume->cache;
@@ -577,8 +577,8 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
}
static int cachefiles_prepare_write(struct netfs_cache_resources *cres,
- loff_t *_start, size_t *_len, size_t upper_len,
- loff_t i_size, bool no_space_allocated_yet)
+ uoff_t *_start, size_t *_len, size_t upper_len,
+ uoff_t i_size, bool no_space_allocated_yet)
{
struct cachefiles_object *object = cachefiles_cres_object(cres);
struct cachefiles_cache *cache = object->volume->cache;
@@ -628,7 +628,7 @@ static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)
struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
const struct cred *saved_cred;
size_t off, pre, post, len = subreq->len;
- loff_t start = subreq->start;
+ uoff_t start = subreq->start;
int ret;
_enter("W=%x[%x] %llx-%llx",
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index ecf33b66610c..6586f6c1dc73 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -65,7 +65,7 @@
(CONGESTION_ON_THRESH(congestion_kb) - \
(CONGESTION_ON_THRESH(congestion_kb) >> 2))
-static int ceph_netfs_check_write_begin(struct file *file, loff_t pos, unsigned int len,
+static int ceph_netfs_check_write_begin(struct file *file, uoff_t pos, unsigned int len,
struct folio **foliop, void **_fsdata);
static inline struct ceph_snap_context *page_snap_context(struct page *page)
@@ -1854,7 +1854,7 @@ ceph_find_incompatible(struct folio *folio)
return NULL;
}
-static int ceph_netfs_check_write_begin(struct file *file, loff_t pos, unsigned int len,
+static int ceph_netfs_check_write_begin(struct file *file, uoff_t pos, unsigned int len,
struct folio **foliop, void **_fsdata)
{
struct inode *inode = file_inode(file);
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 3c32ef41a27f..7e58aec6cdff 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -10,9 +10,9 @@
#include "internal.h"
static void netfs_cache_expand_readahead(struct netfs_io_request *rreq,
- unsigned long long *_start,
- unsigned long long *_len,
- unsigned long long i_size)
+ uoff_t *_start,
+ uoff_t *_len,
+ uoff_t i_size)
{
struct netfs_cache_resources *cres = &rreq->cache_resources;
@@ -134,7 +134,7 @@ static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq,
static enum netfs_io_source netfs_cache_prepare_read(struct netfs_io_request *rreq,
struct netfs_io_subrequest *subreq,
- loff_t i_size)
+ uoff_t i_size)
{
struct netfs_cache_resources *cres = &rreq->cache_resources;
enum netfs_io_source source;
@@ -214,8 +214,8 @@ static void netfs_issue_read(struct netfs_io_request *rreq,
static void netfs_read_to_pagecache(struct netfs_io_request *rreq,
struct readahead_control *ractl)
{
- unsigned long long start = rreq->start;
ssize_t size = rreq->len;
+ uoff_t start = rreq->start;
int ret = 0;
do {
@@ -237,8 +237,8 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq,
source = netfs_cache_prepare_read(rreq, subreq, rreq->i_size);
subreq->source = source;
if (source == NETFS_DOWNLOAD_FROM_SERVER) {
- unsigned long long zero_point = netfs_read_zero_point(rreq->inode);
- unsigned long long zp = umin(zero_point, rreq->i_size);
+ uoff_t zero_point = netfs_read_zero_point(rreq->inode);
+ uoff_t zp = umin(zero_point, rreq->i_size);
size_t len = subreq->len;
if (unlikely(rreq->origin == NETFS_READ_SINGLE))
@@ -342,7 +342,7 @@ void netfs_readahead(struct readahead_control *ractl)
{
struct netfs_io_request *rreq;
struct netfs_inode *ictx = netfs_inode(ractl->mapping->host);
- unsigned long long start = readahead_pos(ractl);
+ uoff_t start = readahead_pos(ractl);
size_t size = readahead_length(ractl);
int ret;
@@ -564,11 +564,11 @@ EXPORT_SYMBOL(netfs_read_folio);
* If any of these criteria are met, then zero out the unwritten parts
* of the folio and return true. Otherwise, return false.
*/
-static bool netfs_skip_folio_read(struct folio *folio, loff_t pos, size_t len,
+static bool netfs_skip_folio_read(struct folio *folio, uoff_t pos, size_t len,
bool always_fill)
{
struct inode *inode = folio_inode(folio);
- loff_t i_size = i_size_read(inode);
+ uoff_t i_size = i_size_read(inode);
size_t offset = offset_in_folio(folio, pos);
size_t plen = folio_size(folio);
@@ -633,7 +633,7 @@ static bool netfs_skip_folio_read(struct folio *folio, loff_t pos, size_t len,
*/
int netfs_write_begin(struct netfs_inode *ctx,
struct file *file, struct address_space *mapping,
- loff_t pos, unsigned int len, struct folio **_folio,
+ uoff_t pos, unsigned int len, struct folio **_folio,
void **_fsdata)
{
struct netfs_io_request *rreq;
@@ -729,7 +729,7 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,
struct netfs_io_request *rreq;
struct address_space *mapping = folio->mapping;
struct netfs_inode *ctx = netfs_inode(mapping->host);
- unsigned long long start = folio_pos(folio);
+ uoff_t start = folio_pos(folio);
size_t flen = folio_size(folio);
int ret;
diff --git a/fs/netfs/buffered_write.c b/fs/netfs/buffered_write.c
index 2cdb68e6b16f..df496873e4f4 100644
--- a/fs/netfs/buffered_write.c
+++ b/fs/netfs/buffered_write.c
@@ -17,7 +17,7 @@
* as possible to hold as much of the remaining length as possible in one go.
*/
static struct folio *netfs_grab_folio_for_write(struct address_space *mapping,
- loff_t pos, size_t part)
+ uoff_t pos, size_t part)
{
pgoff_t index = pos / PAGE_SIZE;
fgf_t fgp_flags = FGP_WRITEBEGIN;
@@ -35,9 +35,9 @@ static struct folio *netfs_grab_folio_for_write(struct address_space *mapping,
* the values actually are.
*/
void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode,
- loff_t pos, size_t copied)
+ uoff_t pos, size_t copied)
{
- loff_t i_size, end = pos + copied;
+ uoff_t i_size, end = pos + copied;
blkcnt_t add;
size_t gap;
@@ -102,7 +102,7 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,
struct folio *folio = NULL, *writethrough = NULL;
unsigned int bdp_flags = (iocb->ki_flags & IOCB_NOWAIT) ? BDP_ASYNC : 0;
ssize_t written = 0, ret, ret2;
- loff_t pos = iocb->ki_pos;
+ uoff_t pos = iocb->ki_pos;
size_t max_chunk = mapping_max_folio_size(mapping);
bool maybe_trouble = false;
@@ -134,7 +134,7 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,
enum netfs_folio_trace trace;
struct netfs_folio *finfo;
struct netfs_group *group;
- unsigned long long fpos;
+ uoff_t fpos;
size_t flen;
size_t offset; /* Offset into pagecache folio */
size_t part; /* Bytes to write to folio */
diff --git a/fs/netfs/direct_read.c b/fs/netfs/direct_read.c
index 6a8fb0d55e04..aa10af5171a8 100644
--- a/fs/netfs/direct_read.c
+++ b/fs/netfs/direct_read.c
@@ -47,8 +47,8 @@ static void netfs_prepare_dio_read_iterator(struct netfs_io_subrequest *subreq)
*/
static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
{
- unsigned long long start = rreq->start;
ssize_t size = rreq->len;
+ uoff_t start = rreq->start;
int ret;
do {
diff --git a/fs/netfs/direct_write.c b/fs/netfs/direct_write.c
index b04019097ab8..47ec47e3a95b 100644
--- a/fs/netfs/direct_write.c
+++ b/fs/netfs/direct_write.c
@@ -216,9 +216,9 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
struct netfs_group *netfs_group)
{
struct netfs_io_request *wreq;
- unsigned long long start = iocb->ki_pos;
- unsigned long long end = start + iov_iter_count(iter);
ssize_t ret, n;
+ uoff_t start = iocb->ki_pos;
+ uoff_t end = start + iov_iter_count(iter);
size_t len = iov_iter_count(iter);
bool async = !is_sync_kiocb(iocb);
@@ -327,8 +327,8 @@ ssize_t netfs_unbuffered_write_iter(struct kiocb *iocb, struct iov_iter *from)
struct inode *inode = mapping->host;
struct netfs_inode *ictx = netfs_inode(inode);
ssize_t ret;
- loff_t pos = iocb->ki_pos;
- unsigned long long end = pos + iov_iter_count(from) - 1;
+ uoff_t pos = iocb->ki_pos;
+ uoff_t end = pos + iov_iter_count(from) - 1;
_enter("%llx,%zx,%llx", pos, iov_iter_count(from), i_size_read(inode));
diff --git a/fs/netfs/fscache_cookie.c b/fs/netfs/fscache_cookie.c
index 3d56fc73435f..5a226f9cbdea 100644
--- a/fs/netfs/fscache_cookie.c
+++ b/fs/netfs/fscache_cookie.c
@@ -327,7 +327,7 @@ static struct fscache_cookie *fscache_alloc_cookie(
u8 advice,
const void *index_key, size_t index_key_len,
const void *aux_data, size_t aux_data_len,
- loff_t object_size)
+ uoff_t object_size)
{
struct fscache_cookie *cookie;
@@ -452,7 +452,7 @@ struct fscache_cookie *__fscache_acquire_cookie(
u8 advice,
const void *index_key, size_t index_key_len,
const void *aux_data, size_t aux_data_len,
- loff_t object_size)
+ uoff_t object_size)
{
struct fscache_cookie *cookie;
@@ -663,7 +663,7 @@ static void fscache_unuse_cookie_locked(struct fscache_cookie *cookie)
* Stop using the cookie for I/O.
*/
void __fscache_unuse_cookie(struct fscache_cookie *cookie,
- const void *aux_data, const loff_t *object_size)
+ const void *aux_data, const uoff_t *object_size)
{
unsigned int debug_id = cookie->debug_id;
unsigned int r = refcount_read(&cookie->ref);
@@ -1049,7 +1049,7 @@ static void fscache_perform_invalidation(struct fscache_cookie *cookie)
* Invalidate an object.
*/
void __fscache_invalidate(struct fscache_cookie *cookie,
- const void *aux_data, loff_t new_size,
+ const void *aux_data, uoff_t new_size,
unsigned int flags)
{
bool is_caching;
diff --git a/fs/netfs/fscache_io.c b/fs/netfs/fscache_io.c
index 37f05b4d3469..8bca63721eeb 100644
--- a/fs/netfs/fscache_io.c
+++ b/fs/netfs/fscache_io.c
@@ -162,7 +162,7 @@ EXPORT_SYMBOL(__fscache_begin_write_operation);
struct fscache_write_request {
struct netfs_cache_resources cache_resources;
struct address_space *mapping;
- loff_t start;
+ uoff_t start;
size_t len;
bool set_bits;
bool using_pgpriv2;
@@ -171,7 +171,7 @@ struct fscache_write_request {
};
void __fscache_clear_page_bits(struct address_space *mapping,
- loff_t start, size_t len)
+ uoff_t start, size_t len)
{
pgoff_t first = start / PAGE_SIZE;
pgoff_t last = (start + len - 1) / PAGE_SIZE;
@@ -208,7 +208,7 @@ static void fscache_wreq_done(void *priv, ssize_t transferred_or_error)
void __fscache_write_to_cache(struct fscache_cookie *cookie,
struct address_space *mapping,
- loff_t start, size_t len, loff_t i_size,
+ uoff_t start, size_t len, uoff_t i_size,
netfs_io_terminated_t term_func,
void *term_func_priv,
bool using_pgpriv2, bool cond)
@@ -267,7 +267,7 @@ EXPORT_SYMBOL(__fscache_write_to_cache);
/*
* Change the size of a backing object.
*/
-void __fscache_resize_cookie(struct fscache_cookie *cookie, loff_t new_size)
+void __fscache_resize_cookie(struct fscache_cookie *cookie, uoff_t new_size)
{
struct netfs_cache_resources cres;
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index f92281a611de..4ea3ef8b3bb0 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -33,7 +33,7 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,
* buffered_write.c
*/
void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode,
- loff_t pos, size_t copied);
+ uoff_t pos, size_t copied);
/*
* main.c
@@ -85,7 +85,7 @@ void netfs_wait_for_paused_write(struct netfs_io_request *rreq);
*/
struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
struct file *file,
- loff_t start, size_t len,
+ uoff_t start, size_t len,
enum netfs_io_origin origin);
void netfs_get_request(struct netfs_io_request *rreq, enum netfs_rreq_ref_trace what);
void netfs_clear_subrequests(struct netfs_io_request *rreq);
@@ -201,11 +201,11 @@ void netfs_write_collection_worker(struct work_struct *work);
*/
struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
struct file *file,
- loff_t start,
+ uoff_t start,
enum netfs_io_origin origin);
void netfs_prepare_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
- loff_t start);
+ uoff_t start);
void netfs_reissue_write(struct netfs_io_stream *stream,
struct netfs_io_subrequest *subreq,
struct iov_iter *source);
@@ -213,7 +213,7 @@ void netfs_issue_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream);
size_t netfs_advance_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
- loff_t start, size_t len, bool to_eof);
+ uoff_t start, size_t len, bool to_eof);
struct netfs_io_request *netfs_begin_writethrough(struct kiocb *iocb, size_t len);
int netfs_advance_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,
struct folio *folio, size_t copied, bool to_page_end,
diff --git a/fs/netfs/iterator.c b/fs/netfs/iterator.c
index b375567e0520..eb1efb17f53a 100644
--- a/fs/netfs/iterator.c
+++ b/fs/netfs/iterator.c
@@ -209,7 +209,7 @@ static size_t netfs_limit_xarray(const struct iov_iter *iter, size_t start_offse
{
struct folio *folio;
unsigned int nsegs = 0;
- loff_t pos = iter->xarray_start + iter->iov_offset;
+ uoff_t pos = iter->xarray_start + iter->iov_offset;
pgoff_t index = pos / PAGE_SIZE;
size_t span = 0, n = iter->count;
diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c
index 5d554512ed23..f58a0d45e614 100644
--- a/fs/netfs/misc.c
+++ b/fs/netfs/misc.c
@@ -193,7 +193,7 @@ void netfs_clear_inode_writeback(struct inode *inode, const void *aux)
struct fscache_cookie *cookie = netfs_i_cookie(netfs_inode(inode));
if (inode_state_read_once(inode) & I_PINNING_NETFS_WB) {
- loff_t i_size = i_size_read(inode);
+ uoff_t i_size = i_size_read(inode);
fscache_unuse_cookie(cookie, aux, &i_size);
}
}
@@ -218,8 +218,8 @@ void netfs_invalidate_folio(struct folio *folio, size_t offset, size_t length)
_enter("{%lx},%zx,%zx", folio->index, offset, length);
if (offset == 0 && length == flen) {
- unsigned long long i_size, remote_i_size, zero_point;
- unsigned long long fpos = folio_pos(folio), end;
+ uoff_t i_size, remote_i_size, zero_point;
+ uoff_t fpos = folio_pos(folio), end;
netfs_read_sizes(inode, &i_size, &remote_i_size, &zero_point);
end = umin(fpos + flen, i_size);
@@ -305,7 +305,7 @@ bool netfs_release_folio(struct folio *folio, gfp_t gfp)
{
struct inode *inode = folio_inode(folio);
struct netfs_inode *ctx = netfs_inode(inode);
- unsigned long long i_size, remote_i_size, zero_point, end;
+ uoff_t i_size, remote_i_size, zero_point, end;
if (folio_test_dirty(folio))
return false;
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index 7f6a3e912602..3460aa1c4af1 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -16,7 +16,7 @@ static void netfs_free_request(struct work_struct *work);
*/
struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
struct file *file,
- loff_t start, size_t len,
+ uoff_t start, size_t len,
enum netfs_io_origin origin)
{
static atomic_t debug_ids;
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index 723b479ef606..a29b694ce1e8 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -130,8 +130,8 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
unsigned int *notes)
{
struct folio_queue *folioq = rreq->buffer.tail;
- unsigned long long collected_to = rreq->collected_to;
unsigned int slot = rreq->buffer.first_tail_slot;
+ uoff_t collected_to = rreq->collected_to;
if (rreq->cleaned_to >= rreq->collected_to)
return;
@@ -149,7 +149,7 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
for (;;) {
struct folio *folio;
- unsigned long long fpos = rreq->cleaned_to, fend;
+ uoff_t fpos = rreq->cleaned_to, fend;
size_t fsize;
if (*notes & COPY_TO_CACHE)
diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c
index c31190993b76..ac16bb8210d1 100644
--- a/fs/netfs/read_pgpriv2.c
+++ b/fs/netfs/read_pgpriv2.c
@@ -20,7 +20,7 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
{
struct netfs_io_stream *cache = &creq->io_streams[1];
size_t fsize = folio_size(folio), flen = fsize;
- loff_t fpos = folio_pos(folio), i_size;
+ uoff_t fpos = folio_pos(folio), i_size;
bool to_eof = false;
_enter("");
@@ -172,8 +172,8 @@ void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)
bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *creq)
{
struct folio_queue *folioq = creq->buffer.tail;
- unsigned long long collected_to = creq->collected_to;
unsigned int slot = creq->buffer.first_tail_slot;
+ uoff_t collected_to = creq->collected_to;
bool made_progress = false;
if (slot >= folioq_nr_slots(folioq)) {
@@ -183,7 +183,7 @@ bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *creq)
for (;;) {
struct folio *folio;
- unsigned long long fpos, fend;
+ uoff_t fpos, fend;
size_t fsize, flen;
folio = folioq_folio(folioq, slot);
@@ -196,7 +196,7 @@ bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *creq)
fsize = folio_size(folio);
flen = fsize;
- fend = min_t(unsigned long long, fpos + flen, creq->i_size);
+ fend = min_t(uoff_t, fpos + flen, creq->i_size);
trace_netfs_collect_folio(creq, folio, fend, collected_to);
diff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c
index 2b42758e01ec..f5c5ed12dd02 100644
--- a/fs/netfs/read_retry.c
+++ b/fs/netfs/read_retry.c
@@ -75,7 +75,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
do {
struct netfs_io_subrequest *from, *to, *tmp;
struct iov_iter source;
- unsigned long long start, len;
+ uoff_t start, len;
size_t part;
bool boundary = false, subreq_superfluous = false;
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index 210eb8f3958d..100a5038c61e 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -67,7 +67,7 @@ int netfs_folio_written_back(struct folio *folio)
/* Streaming writes cannot be redirtied whilst under writeback,
* so discard the streaming record.
*/
- unsigned long long fend;
+ uoff_t fend;
fend = folio_pos(folio) + finfo->dirty_offset + finfo->dirty_len;
spin_lock(&ictx->inode.i_lock);
@@ -115,8 +115,8 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
unsigned int *notes)
{
struct folio_queue *folioq = wreq->buffer.tail;
- unsigned long long collected_to = wreq->collected_to;
unsigned int slot = wreq->buffer.first_tail_slot;
+ uoff_t collected_to = wreq->collected_to;
if (WARN_ON_ONCE(!folioq)) {
pr_err("[!] Writeback unlock found empty rolling buffer!\n");
@@ -140,7 +140,7 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
for (;;) {
struct folio *folio;
struct netfs_folio *finfo;
- unsigned long long fpos, fend;
+ uoff_t fpos, fend;
size_t fsize, flen;
folio = folioq_folio(folioq, slot);
@@ -154,7 +154,7 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
finfo = netfs_folio_info(folio);
flen = finfo ? finfo->dirty_offset + finfo->dirty_len : fsize;
- fend = min_t(unsigned long long, fpos + flen, wreq->i_size);
+ fend = min_t(uoff_t, fpos + flen, wreq->i_size);
trace_netfs_collect_folio(wreq, folio, fend, collected_to);
@@ -201,8 +201,8 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
{
struct netfs_io_subrequest *front, *remove;
struct netfs_io_stream *stream;
- unsigned long long collected_to, issued_to;
unsigned int notes;
+ uoff_t collected_to, issued_to;
int s;
_enter("%llx-%llx", wreq->start, wreq->start + wreq->len);
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index 2d9cfcd43658..1a1f4efcaefd 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -89,7 +89,7 @@ static void netfs_kill_dirty_pages(struct address_space *mapping,
*/
struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
struct file *file,
- loff_t start,
+ uoff_t start,
enum netfs_io_origin origin)
{
struct netfs_io_request *wreq;
@@ -156,7 +156,7 @@ EXPORT_SYMBOL(netfs_prepare_write_failed);
*/
void netfs_prepare_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
- loff_t start)
+ uoff_t start)
{
struct netfs_io_subrequest *subreq;
struct iov_iter *wreq_iter = &wreq->buffer.iter;
@@ -277,7 +277,7 @@ void netfs_issue_write(struct netfs_io_request *wreq,
*/
size_t netfs_advance_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
- loff_t start, size_t len, bool to_eof)
+ uoff_t start, size_t len, bool to_eof)
{
struct netfs_io_subrequest *subreq = stream->construct;
size_t part;
@@ -328,7 +328,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
struct netfs_folio *finfo;
size_t iter_off = 0;
size_t fsize = folio_size(folio), flen = fsize, foff = 0;
- loff_t fpos = folio_pos(folio), i_size;
+ uoff_t fpos = folio_pos(folio), i_size;
bool to_eof = false, streamw = false;
bool debug = false;
@@ -719,7 +719,7 @@ static int netfs_write_folio_single(struct netfs_io_request *wreq,
struct netfs_io_stream *stream;
size_t iter_off = 0;
size_t fsize = folio_size(folio), flen;
- loff_t fpos = folio_pos(folio);
+ uoff_t fpos = folio_pos(folio);
ssize_t ret;
bool to_eof = false;
bool no_debug = false;
diff --git a/fs/netfs/write_retry.c b/fs/netfs/write_retry.c
index 058bc7a166a5..6cd584242af2 100644
--- a/fs/netfs/write_retry.c
+++ b/fs/netfs/write_retry.c
@@ -55,7 +55,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
do {
struct netfs_io_subrequest *subreq = NULL, *from, *to, *tmp;
struct iov_iter source;
- unsigned long long start, len;
+ uoff_t start, len;
size_t part;
bool boundary = false;
diff --git a/include/linux/fscache-cache.h b/include/linux/fscache-cache.h
index 4c91a019972b..ee524c863fa9 100644
--- a/include/linux/fscache-cache.h
+++ b/include/linux/fscache-cache.h
@@ -67,7 +67,7 @@ struct fscache_cache_ops {
/* Change the size of a data object */
void (*resize_cookie)(struct netfs_cache_resources *cres,
- loff_t new_size);
+ uoff_t new_size);
/* Invalidate an object */
bool (*invalidate_cookie)(struct fscache_cookie *cookie);
diff --git a/include/linux/fscache.h b/include/linux/fscache.h
index 58fdb9605425..e19fca38382b 100644
--- a/include/linux/fscache.h
+++ b/include/linux/fscache.h
@@ -112,7 +112,7 @@ struct fscache_cookie {
struct list_head proc_link; /* Link in proc list */
struct list_head commit_link; /* Link in commit queue */
struct work_struct work; /* Commit/relinq/withdraw work */
- loff_t object_size; /* Size of the netfs object */
+ uoff_t object_size; /* Size of the netfs object */
unsigned long unused_at; /* Time at which unused (jiffies) */
unsigned long flags;
#define FSCACHE_COOKIE_RELINQUISHED 0 /* T if cookie has been relinquished */
@@ -163,22 +163,22 @@ extern struct fscache_cookie *__fscache_acquire_cookie(
u8,
const void *, size_t,
const void *, size_t,
- loff_t);
+ uoff_t);
extern void __fscache_use_cookie(struct fscache_cookie *, bool);
-extern void __fscache_unuse_cookie(struct fscache_cookie *, const void *, const loff_t *);
+extern void __fscache_unuse_cookie(struct fscache_cookie *, const void *, const uoff_t *);
extern void __fscache_relinquish_cookie(struct fscache_cookie *, bool);
-extern void __fscache_resize_cookie(struct fscache_cookie *, loff_t);
-extern void __fscache_invalidate(struct fscache_cookie *, const void *, loff_t, unsigned int);
+extern void __fscache_resize_cookie(struct fscache_cookie *, uoff_t);
+extern void __fscache_invalidate(struct fscache_cookie *, const void *, uoff_t, unsigned int);
extern int __fscache_begin_read_operation(struct netfs_cache_resources *, struct fscache_cookie *);
extern int __fscache_begin_write_operation(struct netfs_cache_resources *, struct fscache_cookie *);
void __fscache_write_to_cache(struct fscache_cookie *cookie,
struct address_space *mapping,
- loff_t start, size_t len, loff_t i_size,
+ uoff_t start, size_t len, uoff_t i_size,
netfs_io_terminated_t term_func,
void *term_func_priv,
bool using_pgpriv2, bool cond);
-extern void __fscache_clear_page_bits(struct address_space *, loff_t, size_t);
+extern void __fscache_clear_page_bits(struct address_space *, uoff_t, size_t);
/**
* fscache_acquire_volume - Register a volume as desiring caching services
@@ -249,7 +249,7 @@ struct fscache_cookie *fscache_acquire_cookie(struct fscache_volume *volume,
size_t index_key_len,
const void *aux_data,
size_t aux_data_len,
- loff_t object_size)
+ uoff_t object_size)
{
if (!fscache_volume_valid(volume))
return NULL;
@@ -286,7 +286,7 @@ static inline void fscache_use_cookie(struct fscache_cookie *cookie,
*/
static inline void fscache_unuse_cookie(struct fscache_cookie *cookie,
const void *aux_data,
- const loff_t *object_size)
+ const uoff_t *object_size)
{
if (fscache_cookie_valid(cookie))
__fscache_unuse_cookie(cookie, aux_data, object_size);
@@ -327,7 +327,7 @@ static inline void *fscache_get_aux(struct fscache_cookie *cookie)
*/
static inline
void fscache_update_aux(struct fscache_cookie *cookie,
- const void *aux_data, const loff_t *object_size)
+ const void *aux_data, const uoff_t *object_size)
{
void *p = fscache_get_aux(cookie);
@@ -343,7 +343,7 @@ extern atomic_t fscache_n_updates;
static inline
void __fscache_update_cookie(struct fscache_cookie *cookie, const void *aux_data,
- const loff_t *object_size)
+ const uoff_t *object_size)
{
#ifdef CONFIG_FSCACHE_STATS
atomic_inc(&fscache_n_updates);
@@ -369,7 +369,7 @@ void __fscache_update_cookie(struct fscache_cookie *cookie, const void *aux_data
*/
static inline
void fscache_update_cookie(struct fscache_cookie *cookie, const void *aux_data,
- const loff_t *object_size)
+ const uoff_t *object_size)
{
if (fscache_cookie_enabled(cookie))
__fscache_update_cookie(cookie, aux_data, object_size);
@@ -386,7 +386,7 @@ void fscache_update_cookie(struct fscache_cookie *cookie, const void *aux_data,
* description.
*/
static inline
-void fscache_resize_cookie(struct fscache_cookie *cookie, loff_t new_size)
+void fscache_resize_cookie(struct fscache_cookie *cookie, uoff_t new_size)
{
if (fscache_cookie_enabled(cookie))
__fscache_resize_cookie(cookie, new_size);
@@ -413,7 +413,7 @@ void fscache_resize_cookie(struct fscache_cookie *cookie, loff_t new_size)
*/
static inline
void fscache_invalidate(struct fscache_cookie *cookie,
- const void *aux_data, loff_t size, unsigned int flags)
+ const void *aux_data, uoff_t size, unsigned int flags)
{
if (fscache_cookie_enabled(cookie))
__fscache_invalidate(cookie, aux_data, size, flags);
@@ -502,7 +502,7 @@ static inline void fscache_end_operation(struct netfs_cache_resources *cres)
*/
static inline
int fscache_read(struct netfs_cache_resources *cres,
- loff_t start_pos,
+ uoff_t start_pos,
struct iov_iter *iter,
enum netfs_read_from_hole read_hole,
netfs_io_terminated_t term_func,
@@ -561,7 +561,7 @@ int fscache_begin_write_operation(struct netfs_cache_resources *cres,
*/
static inline
int fscache_write(struct netfs_cache_resources *cres,
- loff_t start_pos,
+ uoff_t start_pos,
struct iov_iter *iter,
netfs_io_terminated_t term_func,
void *term_func_priv)
@@ -581,7 +581,7 @@ int fscache_write(struct netfs_cache_resources *cres,
* waiting.
*/
static inline void fscache_clear_page_bits(struct address_space *mapping,
- loff_t start, size_t len,
+ uoff_t start, size_t len,
bool caching)
{
if (caching)
@@ -615,7 +615,7 @@ static inline void fscache_clear_page_bits(struct address_space *mapping,
*/
static inline void fscache_write_to_cache(struct fscache_cookie *cookie,
struct address_space *mapping,
- loff_t start, size_t len, loff_t i_size,
+ uoff_t start, size_t len, uoff_t i_size,
netfs_io_terminated_t term_func,
void *term_func_priv,
bool using_pgpriv2, bool caching)
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index a8702bd52d3a..8c78c91fefd9 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -62,8 +62,8 @@ struct netfs_inode {
struct fscache_cookie *cache;
#endif
struct list_head wb_queue; /* Queue of processes wanting to do writeback */
- loff_t _remote_i_size; /* Size of the remote file */
- loff_t _zero_point; /* Size after which we assume there's no data
+ uoff_t _remote_i_size; /* Size of the remote file */
+ uoff_t _zero_point; /* Size after which we assume there's no data
* on the server */
spinlock_t lock; /* Lock covering wb_queue */
atomic_t io_count; /* Number of outstanding reqs */
@@ -142,7 +142,7 @@ struct netfs_io_stream {
void (*issue_write)(struct netfs_io_subrequest *subreq);
/* Collection tracking */
struct list_head subrequests; /* Contributory I/O operations */
- unsigned long long collected_to; /* Position we've collected results to */
+ uoff_t collected_to; /* Position we've collected results to */
size_t transferred; /* The amount transferred from this stream */
unsigned short error; /* Aggregate error for the stream */
enum netfs_io_source source; /* Where to read from/write to */
@@ -177,7 +177,7 @@ struct netfs_io_subrequest {
struct work_struct work;
struct list_head rreq_link; /* Link in rreq->subrequests */
struct iov_iter io_iter; /* Iterator for this subrequest */
- unsigned long long start; /* Where to start the I/O */
+ uoff_t start; /* Where to start the I/O */
size_t len; /* Size of the I/O */
size_t transferred; /* Amount of data transferred */
refcount_t ref;
@@ -243,17 +243,17 @@ struct netfs_io_request {
void *netfs_priv; /* Private data for the netfs */
void *netfs_priv2; /* Private data for the netfs */
struct bio_vec *direct_bv; /* DIO buffer list (when handling iovec-iter) */
- unsigned long long submitted; /* Amount submitted for I/O so far */
- unsigned long long len; /* Length of the request */
+ uoff_t submitted; /* Amount submitted for I/O so far */
+ uoff_t len; /* Length of the request */
size_t transferred; /* Amount to be indicated as transferred */
size_t progress_at; /* Report read progress when hit this much read */
long error; /* 0 or error that occurred */
- unsigned long long i_size; /* Size of the file */
- unsigned long long start; /* Start position */
+ uoff_t i_size; /* Size of the file */
+ uoff_t start; /* Start position */
atomic64_t issued_to; /* Write issuer folio cursor */
- unsigned long long collected_to; /* Point we've collected to */
- unsigned long long cleaned_to; /* Position we've cleaned folios to */
- unsigned long long abandon_to; /* Position to abandon folios to */
+ uoff_t collected_to; /* Point we've collected to */
+ uoff_t cleaned_to; /* Position we've cleaned folios to */
+ uoff_t abandon_to; /* Position to abandon folios to */
const struct folio *no_unlock_folio; /* Don't unlock this folio after read */
gfp_t gfp; /* GFP flags to use */
unsigned int direct_bv_count; /* Number of elements in direct_bv[] */
@@ -298,12 +298,12 @@ struct netfs_request_ops {
int (*prepare_read)(struct netfs_io_subrequest *subreq);
void (*issue_read)(struct netfs_io_subrequest *subreq);
bool (*is_still_valid)(struct netfs_io_request *rreq);
- int (*check_write_begin)(struct file *file, loff_t pos, unsigned len,
+ int (*check_write_begin)(struct file *file, uoff_t pos, unsigned len,
struct folio **foliop, void **_fsdata);
void (*done)(struct netfs_io_request *rreq);
/* Modification handling */
- void (*update_i_size)(struct inode *inode, loff_t i_size);
+ void (*update_i_size)(struct inode *inode, uoff_t i_size);
void (*post_modify)(struct inode *inode);
/* Write request handling */
@@ -331,7 +331,7 @@ struct netfs_cache_ops {
/* Read data from the cache */
int (*read)(struct netfs_cache_resources *cres,
- loff_t start_pos,
+ uoff_t start_pos,
struct iov_iter *iter,
enum netfs_read_from_hole read_hole,
netfs_io_terminated_t term_func,
@@ -339,7 +339,7 @@ struct netfs_cache_ops {
/* Write data to the cache */
int (*write)(struct netfs_cache_resources *cres,
- loff_t start_pos,
+ uoff_t start_pos,
struct iov_iter *iter,
netfs_io_terminated_t term_func,
void *term_func_priv);
@@ -349,15 +349,15 @@ struct netfs_cache_ops {
/* Expand readahead request */
void (*expand_readahead)(struct netfs_cache_resources *cres,
- unsigned long long *_start,
- unsigned long long *_len,
- unsigned long long i_size);
+ uoff_t *_start,
+ uoff_t *_len,
+ uoff_t i_size);
/* Prepare a read operation, shortening it to a cached/uncached
* boundary as appropriate.
*/
enum netfs_io_source (*prepare_read)(struct netfs_io_subrequest *subreq,
- unsigned long long i_size);
+ uoff_t i_size);
/* Prepare a write subrequest, working out if we're allowed to do it
* and finding out the maximum amount of data to gather before
@@ -370,15 +370,15 @@ struct netfs_cache_ops {
* actually do.
*/
int (*prepare_write)(struct netfs_cache_resources *cres,
- loff_t *_start, size_t *_len, size_t upper_len,
- loff_t i_size, bool no_space_allocated_yet);
+ uoff_t *_start, size_t *_len, size_t upper_len,
+ uoff_t i_size, bool no_space_allocated_yet);
/* Query the occupancy of the cache in a region, returning where the
* next chunk of data starts and how long it is.
*/
int (*query_occupancy)(struct netfs_cache_resources *cres,
- loff_t start, size_t len, size_t granularity,
- loff_t *_data_start, size_t *_data_len);
+ uoff_t start, size_t len, size_t granularity,
+ uoff_t *_data_start, size_t *_data_len);
};
/* High-level read API. */
@@ -409,7 +409,7 @@ struct readahead_control;
void netfs_readahead(struct readahead_control *);
int netfs_read_folio(struct file *, struct folio *);
int netfs_write_begin(struct netfs_inode *, struct file *,
- struct address_space *, loff_t pos, unsigned int len,
+ struct address_space *, uoff_t pos, unsigned int len,
struct folio **, void **fsdata);
int netfs_writepages(struct address_space *mapping,
struct writeback_control *wbc);
@@ -487,10 +487,10 @@ static inline struct netfs_inode *netfs_inode(struct inode *inode)
* cmpxchg8b without the need of the lock prefix). For SMP compiles and 64bit
* archs it makes no difference if preempt is enabled or not.
*/
-static inline unsigned long long netfs_read_remote_i_size(const struct inode *inode)
+static inline uoff_t netfs_read_remote_i_size(const struct inode *inode)
{
const struct netfs_inode *ictx = container_of(inode, struct netfs_inode, inode);
- unsigned long long remote_i_size;
+ uoff_t remote_i_size;
#if BITS_PER_LONG==32 && defined(CONFIG_SMP)
unsigned int seq;
@@ -525,7 +525,7 @@ static inline unsigned long long netfs_read_remote_i_size(const struct inode *in
* spinning forever.
*/
static inline void netfs_write_remote_i_size(struct inode *inode,
- unsigned long long remote_i_size)
+ uoff_t remote_i_size)
{
struct netfs_inode *ictx = netfs_inode(inode);
@@ -562,10 +562,10 @@ static inline void netfs_write_remote_i_size(struct inode *inode,
* cmpxchg8b without the need of the lock prefix). For SMP compiles and 64bit
* archs it makes no difference if preempt is enabled or not.
*/
-static inline unsigned long long netfs_read_zero_point(const struct inode *inode)
+static inline uoff_t netfs_read_zero_point(const struct inode *inode)
{
struct netfs_inode *ictx = container_of(inode, struct netfs_inode, inode);
- unsigned long long zero_point;
+ uoff_t zero_point;
#if BITS_PER_LONG==32 && defined(CONFIG_SMP)
unsigned int seq;
@@ -600,7 +600,7 @@ static inline unsigned long long netfs_read_zero_point(const struct inode *inode
* forever.
*/
static inline void netfs_write_zero_point(struct inode *inode,
- unsigned long long zero_point)
+ uoff_t zero_point)
{
struct netfs_inode *ictx = netfs_inode(inode);
@@ -641,9 +641,9 @@ static inline void netfs_write_zero_point(struct inode *inode,
* archs it makes no difference if preempt is enabled or not.
*/
static inline void netfs_read_sizes(const struct inode *inode,
- unsigned long long *i_size,
- unsigned long long *remote_i_size,
- unsigned long long *zero_point)
+ uoff_t *i_size,
+ uoff_t *remote_i_size,
+ uoff_t *zero_point)
{
const struct netfs_inode *ictx = container_of(inode, struct netfs_inode, inode);
#if BITS_PER_LONG==32 && defined(CONFIG_SMP)
@@ -689,9 +689,9 @@ static inline void netfs_read_sizes(const struct inode *inode,
* forever.
*/
static inline void netfs_write_sizes(struct inode *inode,
- unsigned long long i_size,
- unsigned long long remote_i_size,
- unsigned long long zero_point)
+ uoff_t i_size,
+ uoff_t remote_i_size,
+ uoff_t zero_point)
{
struct netfs_inode *ictx = netfs_inode(inode);
@@ -759,7 +759,7 @@ static inline void netfs_inode_init(struct netfs_inode *ctx,
* Inform the netfs lib that a file got resized so that it can adjust its state.
*/
static inline void netfs_resize_file(struct netfs_inode *ictx,
- unsigned long long new_i_size,
+ uoff_t new_i_size,
bool changed_on_server)
{
#if BITS_PER_LONG==32 && defined(CONFIG_SMP)
diff --git a/include/trace/events/cachefiles.h b/include/trace/events/cachefiles.h
index 927338f8fe85..4412d8344182 100644
--- a/include/trace/events/cachefiles.h
+++ b/include/trace/events/cachefiles.h
@@ -449,7 +449,7 @@ TRACE_EVENT(cachefiles_vol_coherency,
TRACE_EVENT(cachefiles_prep_read,
TP_PROTO(struct cachefiles_object *obj,
- loff_t start,
+ uoff_t start,
size_t len,
unsigned short flags,
enum netfs_io_source source,
@@ -464,7 +464,7 @@ TRACE_EVENT(cachefiles_prep_read,
__field(enum netfs_io_source, source)
__field(enum cachefiles_prepare_read_trace, why)
__field(size_t, len)
- __field(loff_t, start)
+ __field(uoff_t, start)
__field(unsigned int, netfs_inode)
__field(unsigned int, cache_inode)
),
@@ -492,16 +492,16 @@ TRACE_EVENT(cachefiles_prep_read,
TRACE_EVENT(cachefiles_read,
TP_PROTO(struct cachefiles_object *obj,
struct inode *backer,
- loff_t start,
+ uoff_t start,
size_t len),
TP_ARGS(obj, backer, start, len),
TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, backer)
- __field(size_t, len)
- __field(loff_t, start)
+ __field(unsigned int, obj)
+ __field(unsigned int, backer)
+ __field(size_t, len)
+ __field(uoff_t, start)
),
TP_fast_assign(
@@ -521,16 +521,16 @@ TRACE_EVENT(cachefiles_read,
TRACE_EVENT(cachefiles_write,
TP_PROTO(struct cachefiles_object *obj,
struct inode *backer,
- loff_t start,
+ uoff_t start,
size_t len),
TP_ARGS(obj, backer, start, len),
TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, backer)
- __field(size_t, len)
- __field(loff_t, start)
+ __field(unsigned int, obj)
+ __field(unsigned int, backer)
+ __field(size_t, len)
+ __field(uoff_t, start)
),
TP_fast_assign(
@@ -549,7 +549,7 @@ TRACE_EVENT(cachefiles_write,
TRACE_EVENT(cachefiles_trunc,
TP_PROTO(struct cachefiles_object *obj, struct inode *backer,
- loff_t from, loff_t to, enum cachefiles_trunc_trace why),
+ uoff_t from, uoff_t to, enum cachefiles_trunc_trace why),
TP_ARGS(obj, backer, from, to, why),
@@ -557,8 +557,8 @@ TRACE_EVENT(cachefiles_trunc,
__field(unsigned int, obj)
__field(unsigned int, backer)
__field(enum cachefiles_trunc_trace, why)
- __field(loff_t, from)
- __field(loff_t, to)
+ __field(uoff_t, from)
+ __field(uoff_t, to)
),
TP_fast_assign(
diff --git a/include/trace/events/fscache.h b/include/trace/events/fscache.h
index f1a73aa83fbb..8735d428ebd9 100644
--- a/include/trace/events/fscache.h
+++ b/include/trace/events/fscache.h
@@ -460,13 +460,13 @@ TRACE_EVENT(fscache_relinquish,
);
TRACE_EVENT(fscache_invalidate,
- TP_PROTO(struct fscache_cookie *cookie, loff_t new_size),
+ TP_PROTO(struct fscache_cookie *cookie, uoff_t new_size),
TP_ARGS(cookie, new_size),
TP_STRUCT__entry(
__field(unsigned int, cookie )
- __field(loff_t, new_size )
+ __field(uoff_t, new_size )
),
TP_fast_assign(
@@ -479,14 +479,14 @@ TRACE_EVENT(fscache_invalidate,
);
TRACE_EVENT(fscache_resize,
- TP_PROTO(struct fscache_cookie *cookie, loff_t new_size),
+ TP_PROTO(struct fscache_cookie *cookie, uoff_t new_size),
TP_ARGS(cookie, new_size),
TP_STRUCT__entry(
__field(unsigned int, cookie )
- __field(loff_t, old_size )
- __field(loff_t, new_size )
+ __field(uoff_t, old_size )
+ __field(uoff_t, new_size )
),
TP_fast_assign(
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 8ec10c076875..4f5a447d92b1 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -296,7 +296,7 @@ netfs_folioq_traces;
TRACE_EVENT(netfs_read,
TP_PROTO(struct netfs_io_request *rreq,
- loff_t start, size_t len,
+ uoff_t start, size_t len,
enum netfs_read_trace what),
TP_ARGS(rreq, start, len, what),
@@ -304,8 +304,8 @@ TRACE_EVENT(netfs_read,
TP_STRUCT__entry(
__field(unsigned int, rreq)
__field(unsigned int, cookie)
- __field(loff_t, i_size)
- __field(loff_t, start)
+ __field(uoff_t, i_size)
+ __field(uoff_t, start)
__field(size_t, len)
__field(enum netfs_read_trace, what)
__field(u64, netfs_inode)
@@ -372,7 +372,7 @@ TRACE_EVENT(netfs_sreq,
__field(u8, slot)
__field(size_t, len)
__field(size_t, transferred)
- __field(loff_t, start)
+ __field(uoff_t, start)
),
TP_fast_assign(
@@ -413,7 +413,7 @@ TRACE_EVENT(netfs_failure,
__field(enum netfs_failure, what)
__field(size_t, len)
__field(size_t, transferred)
- __field(loff_t, start)
+ __field(uoff_t, start)
),
TP_fast_assign(
@@ -519,10 +519,10 @@ TRACE_EVENT(netfs_write_iter,
TP_ARGS(iocb, from),
TP_STRUCT__entry(
- __field(unsigned long long, start)
- __field(size_t, len)
- __field(unsigned int, flags)
- __field(unsigned int, ino)
+ __field(uoff_t, start)
+ __field(size_t, len)
+ __field(unsigned int, flags)
+ __field(unsigned int, ino)
),
TP_fast_assign(
@@ -547,8 +547,8 @@ TRACE_EVENT(netfs_write,
__field(unsigned int, cookie)
__field(unsigned int, ino)
__field(enum netfs_write_trace, what)
- __field(unsigned long long, start)
- __field(unsigned long long, len)
+ __field(uoff_t, start)
+ __field(uoff_t, len)
),
TP_fast_assign(
@@ -577,10 +577,10 @@ TRACE_EVENT(netfs_copy2cache,
TP_ARGS(rreq, creq),
TP_STRUCT__entry(
- __field(unsigned int, rreq)
- __field(unsigned int, creq)
- __field(unsigned int, cookie)
- __field(unsigned int, ino)
+ __field(unsigned int, rreq)
+ __field(unsigned int, creq)
+ __field(unsigned int, cookie)
+ __field(unsigned int, ino)
),
TP_fast_assign(
@@ -605,10 +605,10 @@ TRACE_EVENT(netfs_collect,
TP_ARGS(wreq),
TP_STRUCT__entry(
- __field(unsigned int, wreq)
- __field(unsigned int, len)
- __field(unsigned long long, transferred)
- __field(unsigned long long, start)
+ __field(unsigned int, wreq)
+ __field(unsigned int, len)
+ __field(uoff_t, transferred)
+ __field(uoff_t, start)
),
TP_fast_assign(
@@ -631,12 +631,12 @@ TRACE_EVENT(netfs_collect_sreq,
TP_ARGS(wreq, subreq),
TP_STRUCT__entry(
- __field(unsigned int, wreq)
- __field(unsigned int, subreq)
- __field(unsigned int, stream)
- __field(unsigned int, len)
- __field(unsigned int, transferred)
- __field(unsigned long long, start)
+ __field(unsigned int, wreq)
+ __field(unsigned int, subreq)
+ __field(unsigned int, stream)
+ __field(unsigned int, len)
+ __field(unsigned int, transferred)
+ __field(uoff_t, start)
),
TP_fast_assign(
@@ -656,17 +656,16 @@ TRACE_EVENT(netfs_collect_sreq,
TRACE_EVENT(netfs_collect_folio,
TP_PROTO(const struct netfs_io_request *wreq,
const struct folio *folio,
- unsigned long long fend,
- unsigned long long collected_to),
+ uoff_t fend, uoff_t collected_to),
TP_ARGS(wreq, folio, fend, collected_to),
TP_STRUCT__entry(
__field(unsigned int, wreq)
__field(unsigned long, index)
- __field(unsigned long long, fend)
- __field(unsigned long long, cleaned_to)
- __field(unsigned long long, collected_to)
+ __field(uoff_t, fend)
+ __field(uoff_t, cleaned_to)
+ __field(uoff_t, collected_to)
),
TP_fast_assign(
@@ -679,13 +678,13 @@ TRACE_EVENT(netfs_collect_folio,
TP_printk("R=%08x ix=%05lx r=%llx-%llx t=%llx/%llx",
__entry->wreq, __entry->index,
- (unsigned long long)__entry->index * PAGE_SIZE, __entry->fend,
+ (uoff_t)__entry->index * PAGE_SIZE, __entry->fend,
__entry->cleaned_to, __entry->collected_to)
);
TRACE_EVENT(netfs_collect_state,
TP_PROTO(const struct netfs_io_request *wreq,
- unsigned long long collected_to,
+ uoff_t collected_to,
unsigned int notes),
TP_ARGS(wreq, collected_to, notes),
@@ -693,8 +692,8 @@ TRACE_EVENT(netfs_collect_state,
TP_STRUCT__entry(
__field(unsigned int, wreq)
__field(unsigned int, notes)
- __field(unsigned long long, collected_to)
- __field(unsigned long long, cleaned_to)
+ __field(uoff_t, collected_to)
+ __field(uoff_t, cleaned_to)
),
TP_fast_assign(
@@ -713,7 +712,7 @@ TRACE_EVENT(netfs_collect_state,
TRACE_EVENT(netfs_collect_gap,
TP_PROTO(const struct netfs_io_request *wreq,
const struct netfs_io_stream *stream,
- unsigned long long jump_to, char type),
+ uoff_t jump_to, char type),
TP_ARGS(wreq, stream, jump_to, type),
@@ -721,8 +720,8 @@ TRACE_EVENT(netfs_collect_gap,
__field(unsigned int, wreq)
__field(unsigned char, stream)
__field(unsigned char, type)
- __field(unsigned long long, from)
- __field(unsigned long long, to)
+ __field(uoff_t, from)
+ __field(uoff_t, to)
),
TP_fast_assign(
@@ -747,8 +746,8 @@ TRACE_EVENT(netfs_collect_stream,
TP_STRUCT__entry(
__field(unsigned int, wreq)
__field(unsigned char, stream)
- __field(unsigned long long, collected_to)
- __field(unsigned long long, issued_to)
+ __field(uoff_t, collected_to)
+ __field(uoff_t, issued_to)
),
TP_fast_assign(
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 06/35] mm: Make readahead store folio count in readahead_control
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (4 preceding siblings ...)
2026-08-24 14:40 ` [PATCH v10 05/35] netfs: Use uoff_t instead of unsigned long long and loff_t David Howells
@ 2026-08-24 14:40 ` David Howells
2026-08-24 14:41 ` [PATCH v10 07/35] netfs: Bulk load the readahead-provided folios up front David Howells
` (29 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:40 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, linux-mm
Make readahead store folio count in readahead_control so that the
filesystem can know in advance how many folios it needs to keep track of.
This is cleared by read_pages() in case it is called from a loop.
The count is accessed by the filesystem with readahead_folio_count().
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara (Red Hat) <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-mm@kvack.org
cc: linux-fsdevel@vger.kernel.org
---
include/linux/pagemap.h | 10 ++++++++++
mm/readahead.c | 5 +++++
2 files changed, 15 insertions(+)
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 2c3718d592d6..e1e51bace388 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1348,6 +1348,7 @@ struct readahead_control {
struct file_ra_state *ra;
/* private: use the readahead_* accessors instead */
pgoff_t _index;
+ unsigned int _nr_folios;
unsigned int _nr_pages;
unsigned int _batch_count;
bool dropbehind;
@@ -1527,6 +1528,15 @@ static inline size_t readahead_batch_length(const struct readahead_control *rac)
return rac->_batch_count * PAGE_SIZE;
}
+/**
+ * readahead_folio_count - Get the number of folios in this readahead request.
+ * @rac: The readahead request.
+ */
+static inline unsigned int readahead_folio_count(const struct readahead_control *rac)
+{
+ return rac->_nr_folios;
+}
+
static inline unsigned long dir_pages(const struct inode *inode)
{
return (unsigned long)(inode->i_size + PAGE_SIZE - 1) >>
diff --git a/mm/readahead.c b/mm/readahead.c
index 558c92957518..069ded56fd80 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -188,6 +188,7 @@ static void read_pages(struct readahead_control *rac)
if (unlikely(rac->_workingset))
psi_memstall_leave(&rac->_pflags);
rac->_workingset = false;
+ rac->_nr_folios = 0;
BUG_ON(readahead_count(rac));
}
@@ -303,6 +304,7 @@ void page_cache_ra_unbounded(struct readahead_control *ractl,
if (i == mark)
folio_set_readahead(folio);
ractl->_workingset |= folio_test_workingset(folio);
+ ractl->_nr_folios++;
ractl->_nr_pages += min_nrpages;
i += min_nrpages;
}
@@ -473,6 +475,7 @@ static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index,
return err;
}
+ ractl->_nr_folios++;
ractl->_nr_pages += 1UL << order;
ractl->_workingset |= folio_test_workingset(folio);
return 0;
@@ -822,6 +825,7 @@ void readahead_expand(struct readahead_control *ractl,
ractl->_workingset = true;
psi_memstall_enter(&ractl->_pflags);
}
+ ractl->_nr_folios++;
ractl->_nr_pages += min_nrpages;
ractl->_index = folio->index;
}
@@ -851,6 +855,7 @@ void readahead_expand(struct readahead_control *ractl,
ractl->_workingset = true;
psi_memstall_enter(&ractl->_pflags);
}
+ ractl->_nr_folios++;
ractl->_nr_pages += min_nrpages;
if (ra) {
ra->size += min_nrpages;
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 07/35] netfs: Bulk load the readahead-provided folios up front
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (5 preceding siblings ...)
2026-08-24 14:40 ` [PATCH v10 06/35] mm: Make readahead store folio count in readahead_control David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 08/35] Add a function to kmap one page of a multipage bio_vec David Howells
` (28 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, linux-mm
Load all the folios by the VM for readahead up front into the folio queue.
With the number of folios provided by the VM, the folio queue can be fully
allocated first and then the loading happen in one go inside the RCU read
lock. The folio refs acquired from readahead are dropped in bulk once the
first subrequest is dispatched as it's quite a slow operation. The
collector waits for NETFS_RREQ_NEED_PUT_RA_REFS to be cleared so that it
doesn't unlock folios before the xarray has been scanned for them.
This simplifies the buffer handling later and isn't noticeably slower as
the xarray doesn't need to be modified and the folios are all already
pre-locked.
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Paulo Alcantara (Red Hat) <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-mm@kvack.org
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/buffered_read.c | 103 +++++++++++++++++++--------------
fs/netfs/internal.h | 1 +
fs/netfs/misc.c | 19 ++++++
fs/netfs/read_collect.c | 7 +++
fs/netfs/read_retry.c | 7 +++
fs/netfs/rolling_buffer.c | 75 ++++++++++++++++++++++++
include/linux/netfs.h | 1 +
include/linux/rolling_buffer.h | 3 +
include/trace/events/netfs.h | 3 +
9 files changed, 177 insertions(+), 42 deletions(-)
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 7e58aec6cdff..77f7dfc816a8 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -54,6 +54,42 @@ static void netfs_rreq_expand(struct netfs_io_request *rreq,
}
}
+/*
+ * Drop the folio refs acquired from the readahead API.
+ */
+static void netfs_bulk_drop_ra_refs(struct netfs_io_request *rreq)
+{
+ struct folio_batch fbatch;
+ struct folio *folio;
+ pgoff_t nr_pages = DIV_ROUND_UP(rreq->len, PAGE_SIZE);
+ pgoff_t first = rreq->start / PAGE_SIZE;
+ XA_STATE(xas, &rreq->mapping->i_pages, first);
+
+ folio_batch_init(&fbatch);
+
+ rcu_read_lock();
+
+ xas_for_each(&xas, folio, first + nr_pages - 1) {
+ if (xas_retry(&xas, folio))
+ continue;
+
+ if (!folio_batch_add(&fbatch, folio))
+ folio_batch_release(&fbatch);
+ }
+
+ rcu_read_unlock();
+ folio_batch_release(&fbatch);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_ra_put_ref);
+ clear_bit_unlock(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags);
+ wake_up(&rreq->waitq);
+}
+
+static void netfs_maybe_bulk_drop_ra_refs(struct netfs_io_request *rreq)
+{
+ if (test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags))
+ netfs_bulk_drop_ra_refs(rreq);
+}
+
/*
* Begin an operation, and fetch the stored zero point value from the cookie if
* available.
@@ -74,12 +110,8 @@ static int netfs_begin_cache_read(struct netfs_io_request *rreq, struct netfs_in
*
* Returns the limited size if successful and -ENOMEM if insufficient memory
* available.
- *
- * [!] NOTE: This must be run in the same thread as ->issue_read() was called
- * in as we access the readahead_control struct.
*/
-static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq,
- struct readahead_control *ractl)
+static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
size_t rsize = subreq->len;
@@ -87,33 +119,6 @@ static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq,
if (subreq->source == NETFS_DOWNLOAD_FROM_SERVER)
rsize = umin(rsize, rreq->io_streams[0].sreq_max_len);
- if (ractl) {
- /* If we don't have sufficient folios in the rolling buffer,
- * extract a folioq's worth from the readahead region at a time
- * into the buffer. Note that this acquires a ref on each page
- * that we will need to release later - but we don't want to do
- * that until after we've started the I/O.
- */
- struct folio_batch put_batch;
-
- folio_batch_init(&put_batch);
- while (rreq->submitted < subreq->start + rsize) {
- ssize_t added;
-
- added = rolling_buffer_load_from_ra(&rreq->buffer, ractl,
- &put_batch);
- if (added < 0) {
- folio_batch_release(&put_batch);
- return added;
- }
-
- if (!rreq->progress_at)
- netfs_read_set_unlock_at(rreq);
- rreq->submitted += added;
- }
- folio_batch_release(&put_batch);
- }
-
subreq->len = rsize;
if (unlikely(rreq->io_streams[0].sreq_max_segs)) {
size_t limit = netfs_limit_iter(&rreq->buffer.iter, 0, rsize,
@@ -211,8 +216,7 @@ static void netfs_issue_read(struct netfs_io_request *rreq,
* slicing up the region to be read according to available cache blocks and
* network rsize.
*/
-static void netfs_read_to_pagecache(struct netfs_io_request *rreq,
- struct readahead_control *ractl)
+static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
{
ssize_t size = rreq->len;
uoff_t start = rreq->start;
@@ -291,7 +295,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq,
break;
issue:
- slice = netfs_prepare_read_iterator(subreq, ractl);
+ slice = netfs_prepare_read_iterator(subreq);
if (slice < 0) {
ret = slice;
netfs_cancel_read(subreq, ret);
@@ -305,6 +309,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq,
}
netfs_issue_read(rreq, subreq);
+ netfs_maybe_bulk_drop_ra_refs(rreq);
if (test_bit(NETFS_RREQ_PAUSE, &rreq->flags))
netfs_wait_for_paused_read(rreq);
@@ -342,6 +347,7 @@ void netfs_readahead(struct readahead_control *ractl)
{
struct netfs_io_request *rreq;
struct netfs_inode *ictx = netfs_inode(ractl->mapping->host);
+ ssize_t added;
uoff_t start = readahead_pos(ractl);
size_t size = readahead_length(ractl);
int ret;
@@ -363,11 +369,24 @@ void netfs_readahead(struct readahead_control *ractl)
netfs_rreq_expand(rreq, ractl);
- rreq->submitted = rreq->start;
- if (rolling_buffer_init(&rreq->buffer, rreq->debug_id, ITER_DEST, rreq->gfp) < 0)
+ /* Load the folios to be read into a bvecq chain. Note that this
+ * acquires a ref on each folio that we will need to release later -
+ * but we don't want to do that until after we've started the I/O.
+ */
+ added = rolling_buffer_bulk_load_from_ra(&rreq->buffer, ractl,
+ rreq->debug_id, rreq->gfp);
+ if (added < 0) {
+ ret = added;
goto cleanup_free;
- netfs_read_to_pagecache(rreq, ractl);
+ }
+ __set_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags);
+
+ rreq->submitted = rreq->start + added;
+ rreq->cleaned_to = rreq->start;
+ netfs_read_set_unlock_at(rreq);
+ netfs_read_to_pagecache(rreq);
+ netfs_maybe_bulk_drop_ra_refs(rreq);
return netfs_put_request(rreq, netfs_rreq_trace_put_return);
cleanup_free:
@@ -461,7 +480,7 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
iov_iter_bvec(&rreq->buffer.iter, ITER_DEST, bvec, i, rreq->len);
rreq->submitted = rreq->start + flen;
- netfs_read_to_pagecache(rreq, NULL);
+ netfs_read_to_pagecache(rreq);
ret = netfs_wait_for_read(rreq);
if (ret >= 0) {
@@ -536,7 +555,7 @@ int netfs_read_folio(struct file *file, struct folio *folio)
if (ret < 0)
goto discard;
- netfs_read_to_pagecache(rreq, NULL);
+ netfs_read_to_pagecache(rreq);
ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret < 0 ? ret : 0;
@@ -693,7 +712,7 @@ int netfs_write_begin(struct netfs_inode *ctx,
if (ret < 0)
goto error_put;
- netfs_read_to_pagecache(rreq, NULL);
+ netfs_read_to_pagecache(rreq);
ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
if (ret < 0)
@@ -758,7 +777,7 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,
if (ret < 0)
goto error_put;
- netfs_read_to_pagecache(rreq, NULL);
+ netfs_read_to_pagecache(rreq);
ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret < 0 ? ret : 0;
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 4ea3ef8b3bb0..5680e37fc70d 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -79,6 +79,7 @@ ssize_t netfs_wait_for_read(struct netfs_io_request *rreq);
ssize_t netfs_wait_for_write(struct netfs_io_request *rreq);
void netfs_wait_for_paused_read(struct netfs_io_request *rreq);
void netfs_wait_for_paused_write(struct netfs_io_request *rreq);
+void netfs_wait_for_put_ra_refs(struct netfs_io_request *rreq);
/*
* objects.c
diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c
index f58a0d45e614..eafc4edae6a0 100644
--- a/fs/netfs/misc.c
+++ b/fs/netfs/misc.c
@@ -563,3 +563,22 @@ void netfs_wait_for_paused_write(struct netfs_io_request *rreq)
{
return netfs_wait_for_pause(rreq, netfs_write_collection);
}
+
+/*
+ * Wait for the readahead-acquired refs to be put.
+ */
+void netfs_wait_for_put_ra_refs(struct netfs_io_request *rreq)
+{
+ DEFINE_WAIT(myself);
+
+ for (;;) {
+ trace_netfs_rreq(rreq, netfs_rreq_trace_wait_put_ra_refs);
+ prepare_to_wait(&rreq->waitq, &myself, TASK_UNINTERRUPTIBLE);
+ if (!test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags))
+ break;
+ schedule();
+ }
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_waited_put_ra_refs);
+ finish_wait(&rreq->waitq, &myself);
+}
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index a29b694ce1e8..72afc17ecbb3 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -147,6 +147,13 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
slot = 0;
}
+ /* We have to wait for readahead refs to have been released before we
+ * can unlock any folios as the ref-dropper walks i_pages and the only
+ * thing preventing these folios from being removed is the folio lock.
+ */
+ if (test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags))
+ netfs_wait_for_put_ra_refs(rreq);
+
for (;;) {
struct folio *folio;
uoff_t fpos = rreq->cleaned_to, fend;
diff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c
index f5c5ed12dd02..61dbd644d6bb 100644
--- a/fs/netfs/read_retry.c
+++ b/fs/netfs/read_retry.c
@@ -292,6 +292,13 @@ void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq)
{
struct folio_queue *p;
+ /* We have to wait for readahead refs to have been released before we
+ * can unlock any folios as the ref-dropper walks i_pages and the only
+ * thing preventing these folios from being removed is the folio lock.
+ */
+ if (test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags))
+ netfs_wait_for_put_ra_refs(rreq);
+
for (p = rreq->buffer.tail; p; p = p->next) {
for (int slot = 0; slot < folioq_count(p); slot++) {
struct folio *folio = folioq_folio(p, slot);
diff --git a/fs/netfs/rolling_buffer.c b/fs/netfs/rolling_buffer.c
index 8c0026836f9c..ff6c7ee23f4d 100644
--- a/fs/netfs/rolling_buffer.c
+++ b/fs/netfs/rolling_buffer.c
@@ -153,6 +153,81 @@ ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
return size;
}
+/*
+ * Decant the entire list of folios to read into a rolling buffer.
+ */
+ssize_t rolling_buffer_bulk_load_from_ra(struct rolling_buffer *roll,
+ struct readahead_control *ractl,
+ unsigned int rreq_id, gfp_t gfp)
+{
+ XA_STATE(xas, &ractl->mapping->i_pages, ractl->_index);
+ struct folio_queue *fq;
+ struct folio *folio;
+ ssize_t loaded = 0;
+ int nr, slot = 0, npages = 0;
+
+ /* First allocate all the folioqs we're going to need to avoid having
+ * to deal with ENOMEM later.
+ */
+ nr = ractl->_nr_folios;
+ do {
+ fq = netfs_folioq_alloc(rreq_id, gfp,
+ netfs_trace_folioq_make_space);
+ if (!fq) {
+ rolling_buffer_clear(roll);
+ return -ENOMEM;
+ }
+ fq->prev = roll->head;
+ if (!roll->tail)
+ roll->tail = fq;
+ else
+ roll->head->next = fq;
+ roll->head = fq;
+
+ nr -= folioq_nr_slots(fq);
+ } while (nr > 0);
+
+ rcu_read_lock();
+
+ fq = roll->tail;
+ xas_for_each(&xas, folio, ractl->_index + ractl->_nr_pages - 1) {
+ unsigned int order;
+
+ if (xas_retry(&xas, folio))
+ continue;
+ VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+
+ order = folio_order(folio);
+ fq->orders[slot] = order;
+ fq->vec.folios[slot] = folio;
+ loaded += PAGE_SIZE << order;
+ npages += 1 << order;
+ trace_netfs_folio(folio, netfs_folio_trace_read);
+
+ slot++;
+ if (slot >= folioq_nr_slots(fq)) {
+ fq->vec.nr = slot;
+ fq = fq->next;
+ if (!fq) {
+ WARN_ON_ONCE(npages < readahead_count(ractl));
+ break;
+ }
+ slot = 0;
+ }
+ }
+
+ rcu_read_unlock();
+
+ if (fq)
+ fq->vec.nr = slot;
+
+ WRITE_ONCE(roll->iter.count, loaded);
+ iov_iter_folio_queue(&roll->iter, ITER_DEST, roll->tail, 0, 0, loaded);
+ ractl->_index += npages;
+ ractl->_nr_pages -= npages;
+ return loaded;
+}
+
/*
* Append a folio to the rolling buffer.
*/
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 8c78c91fefd9..16cc0858c613 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -278,6 +278,7 @@ struct netfs_io_request {
#define NETFS_RREQ_FOLIO_COPY_TO_CACHE 10 /* Copy current folio to cache from read */
#define NETFS_RREQ_UPLOAD_TO_SERVER 11 /* Need to write to the server */
#define NETFS_RREQ_USE_IO_ITER 12 /* Use ->io_iter rather than ->i_pages */
+#define NETFS_RREQ_NEED_PUT_RA_REFS 17 /* Need to put the folio refs RA gave us */
#define NETFS_RREQ_USE_PGPRIV2 31 /* [DEPRECATED] Use PG_private_2 to mark
* write to cache on read */
const struct netfs_request_ops *netfs_ops;
diff --git a/include/linux/rolling_buffer.h b/include/linux/rolling_buffer.h
index 9e5dad29669c..761735bdf3ab 100644
--- a/include/linux/rolling_buffer.h
+++ b/include/linux/rolling_buffer.h
@@ -48,6 +48,9 @@ int rolling_buffer_make_space(struct rolling_buffer *roll, gfp_t gfp);
ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
struct readahead_control *ractl,
struct folio_batch *put_batch);
+ssize_t rolling_buffer_bulk_load_from_ra(struct rolling_buffer *roll,
+ struct readahead_control *ractl,
+ unsigned int rreq_id, gfp_t gfp);
ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,
unsigned int flags, gfp_t gfp);
struct folio_queue *rolling_buffer_delete_spent(struct rolling_buffer *roll);
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 4f5a447d92b1..0202ba4ccbd0 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -59,6 +59,7 @@
EM(netfs_rreq_trace_free, "FREE ") \
EM(netfs_rreq_trace_intr, "INTR ") \
EM(netfs_rreq_trace_ki_complete, "KI-CMPL") \
+ EM(netfs_rreq_trace_ra_put_ref, "RA-PUT ") \
EM(netfs_rreq_trace_recollect, "RECLLCT") \
EM(netfs_rreq_trace_redirty, "REDIRTY") \
EM(netfs_rreq_trace_resubmit, "RESUBMT") \
@@ -70,9 +71,11 @@
EM(netfs_rreq_trace_unpause, "UNPAUSE") \
EM(netfs_rreq_trace_wait_ip, "WAIT-IP") \
EM(netfs_rreq_trace_wait_pause, "--PAUSED--") \
+ EM(netfs_rreq_trace_wait_put_ra_refs, "WAIT-P-RA") \
EM(netfs_rreq_trace_wait_quiesce, "WAIT-QUIESCE") \
EM(netfs_rreq_trace_waited_ip, "DONE-IP") \
EM(netfs_rreq_trace_waited_pause, "--UNPAUSED--") \
+ EM(netfs_rreq_trace_waited_put_ra_refs, "DONE-P-RA") \
EM(netfs_rreq_trace_waited_quiesce, "DONE-QUIESCE") \
EM(netfs_rreq_trace_wake_ip, "WAKE-IP") \
EM(netfs_rreq_trace_wake_queue, "WAKE-Q ") \
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 08/35] Add a function to kmap one page of a multipage bio_vec
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (6 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 07/35] netfs: Bulk load the readahead-provided folios up front David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 09/35] iov_iter: Make iov_iter_get_pages*() wrap iov_iter_extract_pages() David Howells
` (27 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, linux-block
Add a function to kmap one page of a multipage bio_vec by offset (which is
added to the offset in the bio_vec internally). The caller is responsible
for calculating how much of the page is then available.
Signed-off-by: David Howells <dhowells@redhat.com>
Acked-by: Paulo Alcantara (Red Hat) <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: linux-block@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
include/linux/bvec.h | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/include/linux/bvec.h b/include/linux/bvec.h
index 92837e2743f1..53cf36e73967 100644
--- a/include/linux/bvec.h
+++ b/include/linux/bvec.h
@@ -343,4 +343,22 @@ static inline phys_addr_t bvec_phys(const struct bio_vec *bvec)
return page_to_phys(bvec->bv_page) + bvec->bv_offset;
}
+/**
+ * bvec_kmap_partial - Map part of a bvec into the kernel virtual address space
+ * @bvec: bvec to map
+ * @offset: Offset into bvec
+ *
+ * Map the page containing the byte at @offset into the kernel virtual address
+ * space. The caller is responsible for making sure this doesn't overrun.
+ *
+ * Call kunmap_local on the returned address to unmap.
+ */
+static inline void *bvec_kmap_partial(struct bio_vec *bvec, size_t offset)
+{
+ offset += bvec->bv_offset;
+
+ return kmap_local_page(bvec->bv_page + (offset >> PAGE_SHIFT)) +
+ (offset & ~PAGE_MASK);
+}
+
#endif /* __LINUX_BVEC_H */
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 09/35] iov_iter: Make iov_iter_get_pages*() wrap iov_iter_extract_pages()
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (7 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 08/35] Add a function to kmap one page of a multipage bio_vec David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 10/35] iov_iter: Add a segmented queue of bio_vec[] David Howells
` (26 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, linux-block
Make iov_iter_get_pages*() wrap iov_iter_extract_pages() for kernel
iterator types (e.g. ITER_BVEC, ITER_FOLIOQ, ITER_XARRAY). The pages
obtained have their refcounts incremented afterwards if they're not slab
pages. ITER_KVEC is left returning -EFAULT.
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Paulo Alcantara (Red Hat) <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: linux-block@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
lib/iov_iter.c | 164 ++++++-------------------------------------------
1 file changed, 19 insertions(+), 145 deletions(-)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index c2484551a4e8..15a50d538c65 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -910,118 +910,34 @@ static int want_pages_array(struct page ***res, size_t size,
return count;
}
-static ssize_t iter_folioq_get_pages(struct iov_iter *iter,
+/*
+ * Wrap iov_iter_extract_pages() and then pin the non-slab pages we got back.
+ * This only works for non-user iterator types as get_pages uses get_user_pages
+ * not pin_user_pages.
+ */
+static ssize_t iter_get_kernel_pages(struct iov_iter *iter,
struct page ***ppages, size_t maxsize,
unsigned maxpages, size_t *_start_offset)
{
- const struct folio_queue *folioq = iter->folioq;
struct page **pages;
- unsigned int slot = iter->folioq_slot;
- size_t extracted = 0, count = iter->count, iov_offset = iter->iov_offset;
+ ssize_t ret, done;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
- if (WARN_ON(iov_offset != 0))
- return -EIO;
- }
+ ret = iov_iter_extract_pages(iter, ppages, maxsize, maxpages,
+ 0, _start_offset);
+ if (ret <= 0)
+ return ret;
- maxpages = want_pages_array(ppages, maxsize, iov_offset & ~PAGE_MASK, maxpages);
- if (!maxpages)
- return -ENOMEM;
- *_start_offset = iov_offset & ~PAGE_MASK;
pages = *ppages;
+ for (done = ret + *_start_offset; done > 0; done -= PAGE_SIZE) {
+ struct folio *folio = page_folio(*pages);
- for (;;) {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t offset = iov_offset, fsize = folioq_folio_size(folioq, slot);
- size_t part = PAGE_SIZE - offset % PAGE_SIZE;
-
- if (offset < fsize) {
- part = umin(part, umin(maxsize - extracted, fsize - offset));
- count -= part;
- iov_offset += part;
- extracted += part;
-
- *pages = folio_page(folio, offset / PAGE_SIZE);
- get_page(*pages);
- pages++;
- maxpages--;
- }
-
- if (maxpages == 0 || extracted >= maxsize)
- break;
-
- if (iov_offset >= fsize) {
- iov_offset = 0;
- slot++;
- if (slot == folioq_nr_slots(folioq) && folioq->next) {
- folioq = folioq->next;
- slot = 0;
- }
- }
- }
-
- iter->count = count;
- iter->iov_offset = iov_offset;
- iter->folioq = folioq;
- iter->folioq_slot = slot;
- return extracted;
-}
-
-static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
- pgoff_t index, unsigned int nr_pages)
-{
- XA_STATE(xas, xa, index);
- struct folio *folio;
- unsigned int ret = 0;
-
- rcu_read_lock();
- for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) {
- if (xas_retry(&xas, folio))
- continue;
-
- /* Has the folio moved or been split? */
- if (unlikely(folio != xas_reload(&xas))) {
- xas_reset(&xas);
- continue;
- }
-
- pages[ret] = folio_file_page(folio, xas.xa_index);
- folio_get(folio);
- if (++ret == nr_pages)
- break;
+ if (!folio_test_slab(folio) && !folio_test_large_kmalloc(folio))
+ folio_get(folio);
+ pages++;
}
- rcu_read_unlock();
return ret;
}
-static ssize_t iter_xarray_get_pages(struct iov_iter *i,
- struct page ***pages, size_t maxsize,
- unsigned maxpages, size_t *_start_offset)
-{
- unsigned nr, offset, count;
- pgoff_t index;
- loff_t pos;
-
- pos = i->xarray_start + i->iov_offset;
- index = pos >> PAGE_SHIFT;
- offset = pos & ~PAGE_MASK;
- *_start_offset = offset;
-
- count = want_pages_array(pages, maxsize, offset, maxpages);
- if (!count)
- return -ENOMEM;
- nr = iter_xarray_populate_pages(*pages, i->xarray, index, count);
- if (nr == 0)
- return 0;
-
- maxsize = min_t(size_t, nr * PAGE_SIZE - offset, maxsize);
- i->iov_offset += maxsize;
- i->count -= maxsize;
- return maxsize;
-}
-
/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */
static unsigned long first_iovec_segment(const struct iov_iter *i, size_t *size)
{
@@ -1044,22 +960,6 @@ static unsigned long first_iovec_segment(const struct iov_iter *i, size_t *size)
BUG(); // if it had been empty, we wouldn't get called
}
-/* must be done on non-empty ITER_BVEC one */
-static struct page *first_bvec_segment(const struct iov_iter *i,
- size_t *size, size_t *start)
-{
- struct page *page;
- size_t skip = i->iov_offset, len;
-
- len = i->bvec->bv_len - skip;
- if (*size > len)
- *size = len;
- skip += i->bvec->bv_offset;
- page = i->bvec->bv_page + skip / PAGE_SIZE;
- *start = skip % PAGE_SIZE;
- return page;
-}
-
static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,
struct page ***pages, size_t maxsize,
unsigned int maxpages, size_t *start)
@@ -1095,36 +995,10 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,
iov_iter_advance(i, maxsize);
return maxsize;
}
- if (iov_iter_is_bvec(i)) {
- struct page **p;
- struct page *page;
- page = first_bvec_segment(i, &maxsize, start);
- n = want_pages_array(pages, maxsize, *start, maxpages);
- if (!n)
- return -ENOMEM;
- p = *pages;
- for (int k = 0; k < n; k++) {
- struct folio *folio = page_folio(page + k);
- p[k] = page + k;
- if (!folio_test_slab(folio))
- folio_get(folio);
- }
- maxsize = min_t(size_t, maxsize, n * PAGE_SIZE - *start);
- i->count -= maxsize;
- i->iov_offset += maxsize;
- if (i->iov_offset == i->bvec->bv_len) {
- i->iov_offset = 0;
- i->bvec++;
- i->nr_segs--;
- }
- return maxsize;
- }
- if (iov_iter_is_folioq(i))
- return iter_folioq_get_pages(i, pages, maxsize, maxpages, start);
- if (iov_iter_is_xarray(i))
- return iter_xarray_get_pages(i, pages, maxsize, maxpages, start);
- return -EFAULT;
+ if (iov_iter_is_kvec(i))
+ return -EFAULT;
+ return iter_get_kernel_pages(i, pages, maxsize, maxpages, start);
}
ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages,
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 10/35] iov_iter: Add a segmented queue of bio_vec[]
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (8 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 09/35] iov_iter: Make iov_iter_get_pages*() wrap iov_iter_extract_pages() David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 11/35] netfs: Add some tools for managing bvecq chains David Howells
` (25 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, linux-block
Add the concept of a segmented queue of bio_vec[] arrays. This allows an
indefinite quantity of elements to be handled and allows things like
network filesystems and crypto drivers to glue bits on the ends without
having to reallocate the array.
The bvecq struct that defines each segment also carries capacity/usage
information along with flags indicating whether the constituent memory
regions need freeing or unpinning. The bvecq structs are refcounted to
allow a queue to be extracted in batches and split between a number of
subrequests.
The bvecq can have the bio_vec[] it manages allocated in with it, but this
is not required. A flag is provided for if this is the case as comparing
->bv to ->__bv is not sufficient to detect this case.
Add an iterator type ITER_BVECQ for it. This is intended to replace
ITER_FOLIOQ (and ITER_XARRAY).
Note that the prev pointer is only really needed for iov_iter_revert() and
could be dispensed with if struct iov_iter contained the head information
as well as the current point.
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Paulo Alcantara (Red Hat) <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: linux-block@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
include/linux/bvecq.h | 50 ++++++
include/linux/iov_iter.h | 74 ++++++++-
include/linux/uio.h | 11 ++
lib/iov_iter.c | 333 ++++++++++++++++++++++++++++++++++++-
lib/scatterlist.c | 67 +++++++-
lib/tests/kunit_iov_iter.c | 260 +++++++++++++++++++++++++++++
6 files changed, 789 insertions(+), 6 deletions(-)
create mode 100644 include/linux/bvecq.h
diff --git a/include/linux/bvecq.h b/include/linux/bvecq.h
new file mode 100644
index 000000000000..77fd07852c33
--- /dev/null
+++ b/include/linux/bvecq.h
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Implementation of a segmented queue of bio_vec[].
+ *
+ * Copyright (C) 2026 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+
+#ifndef _LINUX_BVECQ_H
+#define _LINUX_BVECQ_H
+
+#include <linux/bvec.h>
+
+/*
+ * The type of memory retention used by the elements in bvecq->bv[] and how to
+ * clean it up.
+ */
+enum bvecq_mem {
+ BVECQ_MEM_EXTERNAL, /* Externally retained memory - no freeing */
+ BVECQ_MEM_PAGECACHE, /* Ref'd pagecache pages - must put */
+ BVECQ_MEM_GUP, /* Pinned memory from get_user_pages() - unpin */
+ BVECQ_MEM_ALLOCED, /* Memory alloc'd by bvecq - can be freed/pooled */
+} __mode(byte);
+
+/*
+ * Segmented bio_vec queue.
+ *
+ * These can be linked together to form messages of indefinite length and
+ * iterated over with an ITER_BVECQ iterator. The list is non-circular; next
+ * and prev are NULL at the ends.
+ *
+ * The bv pointer points to the bio_vec array; this may be __bv if allocated
+ * together. The caller is responsible for determining whether or not this is
+ * the case as the array pointed to by bv may be follow on directly from the
+ * bvecq by accident of allocation (ie. ->bv == ->__bv is *not* sufficient to
+ * determine this).
+ */
+struct bvecq {
+ struct bvecq *next; /* Next bvec in the list or NULL */
+ struct bvecq *prev; /* Prev bvec in the list or NULL */
+ refcount_t ref;
+ u32 priv; /* Private data */
+ u16 nr_slots; /* Number of elements in bv[] used */
+ u16 max_slots; /* Number of elements allocated in bv[] */
+ enum bvecq_mem mem_type:3; /* What sort of memory and how to free it */
+ bool inline_bv:1; /* T if __bv[] is being used */
+ struct bio_vec *bv; /* Pointer to array of page fragments */
+ struct bio_vec __bv[]; /* Default array (if ->inline_bv) */
+};
+
+#endif /* _LINUX_BVECQ_H */
diff --git a/include/linux/iov_iter.h b/include/linux/iov_iter.h
index f9a17fbbd398..ec0aa2893689 100644
--- a/include/linux/iov_iter.h
+++ b/include/linux/iov_iter.h
@@ -10,6 +10,7 @@
#include <linux/uio.h>
#include <linux/bvec.h>
+#include <linux/bvecq.h>
#include <linux/folio_queue.h>
typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len,
@@ -141,6 +142,71 @@ size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2,
return progress;
}
+/*
+ * Handle ITER_BVECQ.
+ */
+static __always_inline
+size_t iterate_bvecq(struct iov_iter *iter, size_t len, void *priv, void *priv2,
+ iov_step_f step)
+{
+ const struct bvecq *bq = iter->bvecq;
+ unsigned int slot = iter->bvecq_slot;
+ size_t progress = 0, skip = iter->iov_offset;
+
+ do {
+ const struct bio_vec *bvec;
+ struct page *page;
+ size_t poff, plen;
+ void *base;
+
+ if (slot >= bq->nr_slots) {
+ if (!bq->next)
+ break;
+ bq = bq->next;
+ slot = 0;
+ continue;
+ }
+
+ bvec = &bq->bv[slot];
+ /*
+ * The caller must ensure that a slot with bv_len>0 has a valid
+ * bv_page.
+ */
+ page = bvec->bv_page + (bvec->bv_offset + skip) / PAGE_SIZE;
+ poff = (bvec->bv_offset + skip) % PAGE_SIZE;
+ plen = min(bvec->bv_len - skip, len);
+
+ while (plen > 0) {
+ size_t part, remain, consumed;
+
+ part = min(plen, PAGE_SIZE - poff);
+ base = kmap_local_page(page) + poff;
+ remain = step(base, progress, part, priv, priv2);
+ kunmap_local(base);
+
+ consumed = part - remain;
+ progress += consumed;
+ skip += consumed;
+ len -= consumed;
+ if (!len || remain)
+ goto stop;
+ page++;
+ poff = 0;
+ plen -= consumed;
+ }
+
+ skip = 0;
+ slot++;
+ } while (len);
+
+stop:
+ iter->bvecq_slot = slot;
+ iter->bvecq = bq;
+ iter->iov_offset = skip;
+ iter->count -= progress;
+ return progress;
+}
+
/*
* Handle ITER_FOLIOQ.
*/
@@ -306,6 +372,8 @@ size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv,
return iterate_bvec(iter, len, priv, priv2, step);
if (iov_iter_is_kvec(iter))
return iterate_kvec(iter, len, priv, priv2, step);
+ if (iov_iter_is_bvecq(iter))
+ return iterate_bvecq(iter, len, priv, priv2, step);
if (iov_iter_is_folioq(iter))
return iterate_folioq(iter, len, priv, priv2, step);
if (iov_iter_is_xarray(iter))
@@ -342,8 +410,8 @@ size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv,
* buffer is presented in segments, which for kernel iteration are broken up by
* physical pages and mapped, with the mapped address being presented.
*
- * [!] Note This will only handle BVEC, KVEC, FOLIOQ, XARRAY and DISCARD-type
- * iterators; it will not handle UBUF or IOVEC-type iterators.
+ * [!] Note This will only handle BVEC, KVEC, BVECQ, FOLIOQ, XARRAY and
+ * DISCARD-type iterators; it will not handle UBUF or IOVEC-type iterators.
*
* A step functions, @step, must be provided, one for handling mapped kernel
* addresses and the other is given user addresses which have the potential to
@@ -370,6 +438,8 @@ size_t iterate_and_advance_kernel(struct iov_iter *iter, size_t len, void *priv,
return iterate_bvec(iter, len, priv, priv2, step);
if (iov_iter_is_kvec(iter))
return iterate_kvec(iter, len, priv, priv2, step);
+ if (iov_iter_is_bvecq(iter))
+ return iterate_bvecq(iter, len, priv, priv2, step);
if (iov_iter_is_folioq(iter))
return iterate_folioq(iter, len, priv, priv2, step);
if (iov_iter_is_xarray(iter))
diff --git a/include/linux/uio.h b/include/linux/uio.h
index a9bc5b3067e3..f7cfa6ea8213 100644
--- a/include/linux/uio.h
+++ b/include/linux/uio.h
@@ -26,6 +26,7 @@ enum iter_type {
ITER_IOVEC,
ITER_BVEC,
ITER_KVEC,
+ ITER_BVECQ,
ITER_FOLIOQ,
ITER_XARRAY,
ITER_DISCARD,
@@ -68,6 +69,7 @@ struct iov_iter {
const struct iovec *__iov;
const struct kvec *kvec;
const struct bio_vec *bvec;
+ const struct bvecq *bvecq;
const struct folio_queue *folioq;
struct xarray *xarray;
void __user *ubuf;
@@ -77,6 +79,7 @@ struct iov_iter {
};
union {
unsigned long nr_segs;
+ u16 bvecq_slot;
u8 folioq_slot;
loff_t xarray_start;
};
@@ -145,6 +148,11 @@ static inline bool iov_iter_is_discard(const struct iov_iter *i)
return iov_iter_type(i) == ITER_DISCARD;
}
+static inline bool iov_iter_is_bvecq(const struct iov_iter *i)
+{
+ return iov_iter_type(i) == ITER_BVECQ;
+}
+
static inline bool iov_iter_is_folioq(const struct iov_iter *i)
{
return iov_iter_type(i) == ITER_FOLIOQ;
@@ -295,6 +303,9 @@ void iov_iter_kvec(struct iov_iter *i, unsigned int direction, const struct kvec
void iov_iter_bvec(struct iov_iter *i, unsigned int direction, const struct bio_vec *bvec,
unsigned long nr_segs, size_t count);
void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count);
+void iov_iter_bvec_queue(struct iov_iter *i, unsigned int direction,
+ const struct bvecq *bvecq,
+ unsigned int first_slot, unsigned int offset, size_t count);
void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,
const struct folio_queue *folioq,
unsigned int first_slot, unsigned int offset, size_t count);
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 15a50d538c65..c7f7ff9a9b37 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -538,6 +538,40 @@ static void iov_iter_iovec_advance(struct iov_iter *i, size_t size)
i->__iov = iov;
}
+static void iov_iter_bvecq_advance(struct iov_iter *i, size_t by)
+{
+ const struct bvecq *bq = i->bvecq;
+ unsigned int slot = i->bvecq_slot;
+
+ if (!i->count)
+ return;
+ i->count -= by;
+
+ by += i->iov_offset; /* From beginning of current segment. */
+ do {
+ size_t len;
+
+ if (slot >= bq->nr_slots) {
+ if (!bq->next)
+ break;
+ bq = bq->next;
+ slot = 0;
+ continue;
+ }
+
+ len = bq->bv[slot].bv_len;
+
+ if (likely(by < len))
+ break;
+ by -= len;
+ slot++;
+ } while (by);
+
+ i->iov_offset = by;
+ i->bvecq_slot = slot;
+ i->bvecq = bq;
+}
+
static void iov_iter_folioq_advance(struct iov_iter *i, size_t size)
{
const struct folio_queue *folioq = i->folioq;
@@ -583,6 +617,8 @@ void iov_iter_advance(struct iov_iter *i, size_t size)
iov_iter_iovec_advance(i, size);
} else if (iov_iter_is_bvec(i)) {
iov_iter_bvec_advance(i, size);
+ } else if (iov_iter_is_bvecq(i)) {
+ iov_iter_bvecq_advance(i, size);
} else if (iov_iter_is_folioq(i)) {
iov_iter_folioq_advance(i, size);
} else if (iov_iter_is_discard(i)) {
@@ -591,6 +627,33 @@ void iov_iter_advance(struct iov_iter *i, size_t size)
}
EXPORT_SYMBOL(iov_iter_advance);
+static void iov_iter_bvecq_revert(struct iov_iter *i, size_t unroll)
+{
+ const struct bvecq *bq = i->bvecq;
+ unsigned int slot = i->bvecq_slot;
+
+ for (;;) {
+ size_t len;
+
+ if (slot == 0) {
+ bq = bq->prev;
+ slot = bq->nr_slots;
+ continue;
+ }
+ slot--;
+
+ len = bq->bv[slot].bv_len;
+ if (unroll <= len) {
+ i->iov_offset = len - unroll;
+ break;
+ }
+ unroll -= len;
+ }
+
+ i->bvecq_slot = slot;
+ i->bvecq = bq;
+}
+
static void iov_iter_folioq_revert(struct iov_iter *i, size_t unroll)
{
const struct folio_queue *folioq = i->folioq;
@@ -648,6 +711,9 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll)
}
unroll -= n;
}
+ } else if (iov_iter_is_bvecq(i)) {
+ i->iov_offset = 0;
+ iov_iter_bvecq_revert(i, unroll);
} else if (iov_iter_is_folioq(i)) {
i->iov_offset = 0;
iov_iter_folioq_revert(i, unroll);
@@ -678,9 +744,30 @@ size_t iov_iter_single_seg_count(const struct iov_iter *i)
if (iov_iter_is_bvec(i))
return min(i->count, i->bvec->bv_len - i->iov_offset);
}
+ if (!i->count)
+ return 0;
+ if (unlikely(iov_iter_is_bvecq(i))) {
+ const struct bvecq *bq = i->bvecq;
+ unsigned int slot = i->bvecq_slot;
+ size_t offset = i->iov_offset;
+
+ for (;;) {
+ while (slot >= bq->nr_slots) {
+ bq = bq->next;
+ if (!bq)
+ return 0;
+ slot = 0;
+ offset = 0;
+ }
+ if (bq->bv[slot].bv_len > offset)
+ break;
+ slot++;
+ offset = 0;
+ }
+ return min(i->count, bq->bv[slot].bv_len - offset);
+ }
if (unlikely(iov_iter_is_folioq(i)))
- return !i->count ? 0 :
- umin(folioq_folio_size(i->folioq, i->folioq_slot), i->count);
+ return umin(folioq_folio_size(i->folioq, i->folioq_slot), i->count);
return i->count;
}
EXPORT_SYMBOL(iov_iter_single_seg_count);
@@ -717,6 +804,35 @@ void iov_iter_bvec(struct iov_iter *i, unsigned int direction,
}
EXPORT_SYMBOL(iov_iter_bvec);
+/**
+ * iov_iter_bvec_queue - Initialise an I/O iterator to use a segmented bvec queue
+ * @i: The iterator to initialise.
+ * @direction: The direction of the transfer.
+ * @bvecq: The starting point in the bvec queue.
+ * @first_slot: The first slot in the bvec queue to use
+ * @offset: The offset into the bvec in the first slot to start at
+ * @count: The size of the I/O buffer in bytes.
+ *
+ * Set up an I/O iterator to either draw data out of the buffers attached to an
+ * inode or to inject data into those buffers. The pages *must* be prevented
+ * from evaporation, either by the caller.
+ */
+void iov_iter_bvec_queue(struct iov_iter *i, unsigned int direction,
+ const struct bvecq *bvecq, unsigned int first_slot,
+ unsigned int offset, size_t count)
+{
+ WARN_ON(direction & ~(READ | WRITE));
+ *i = (struct iov_iter) {
+ .iter_type = ITER_BVECQ,
+ .data_source = direction,
+ .bvecq = bvecq,
+ .bvecq_slot = first_slot,
+ .count = count,
+ .iov_offset = offset,
+ };
+}
+EXPORT_SYMBOL(iov_iter_bvec_queue);
+
/**
* iov_iter_folio_queue - Initialise an I/O iterator to use the folios in a folio queue
* @i: The iterator to initialise.
@@ -839,6 +955,39 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)
return res;
}
+static unsigned long iov_iter_alignment_bvecq(const struct iov_iter *iter)
+{
+ const struct bvecq *bq;
+ unsigned long res = 0;
+ unsigned int slot = iter->bvecq_slot;
+ size_t skip = iter->iov_offset;
+ size_t size = iter->count;
+
+ if (!size)
+ return res;
+
+ for (bq = iter->bvecq; bq; bq = bq->next) {
+ for (; slot < bq->nr_slots; slot++) {
+ const struct bio_vec *bvec = &bq->bv[slot];
+ size_t part = min(bvec->bv_len - skip, size);
+
+ if (part) {
+ res |= bvec->bv_offset + skip;
+ res |= part;
+ }
+
+ size -= part;
+ if (size == 0)
+ return res;
+ skip = 0;
+ }
+
+ slot = 0;
+ }
+
+ return res;
+}
+
unsigned long iov_iter_alignment(const struct iov_iter *i)
{
if (likely(iter_is_ubuf(i))) {
@@ -854,6 +1003,8 @@ unsigned long iov_iter_alignment(const struct iov_iter *i)
if (iov_iter_is_bvec(i))
return iov_iter_alignment_bvec(i);
+ if (iov_iter_is_bvecq(i))
+ return iov_iter_alignment_bvecq(i);
/* With both xarray and folioq types, we're dealing with whole folios. */
if (iov_iter_is_folioq(i))
@@ -1066,6 +1217,38 @@ static int bvec_npages(const struct iov_iter *i, int maxpages)
return npages;
}
+static size_t iov_npages_bvecq(const struct iov_iter *iter, size_t maxpages)
+{
+ const struct bvecq *bq;
+ unsigned int slot = iter->bvecq_slot;
+ size_t npages = 0;
+ size_t skip = iter->iov_offset;
+ size_t size = iter->count;
+
+ for (bq = iter->bvecq; bq; bq = bq->next) {
+ for (; slot < bq->nr_slots; slot++) {
+ const struct bio_vec *bvec = &bq->bv[slot];
+ size_t offs = (bvec->bv_offset + skip) % PAGE_SIZE;
+ size_t part = min(bvec->bv_len - skip, size);
+
+ if (part) {
+ npages += DIV_ROUND_UP(offs + part, PAGE_SIZE);
+ if (npages >= maxpages)
+ goto out;
+ }
+
+ size -= part;
+ if (!size)
+ goto out;
+ skip = 0;
+ }
+
+ slot = 0;
+ }
+out:
+ return umin(npages, maxpages);
+}
+
int iov_iter_npages(const struct iov_iter *i, int maxpages)
{
if (unlikely(!i->count))
@@ -1080,6 +1263,8 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages)
return iov_npages(i, maxpages);
if (iov_iter_is_bvec(i))
return bvec_npages(i, maxpages);
+ if (iov_iter_is_bvecq(i))
+ return iov_npages_bvecq(i, maxpages);
if (iov_iter_is_folioq(i)) {
unsigned offset = i->iov_offset % PAGE_SIZE;
int npages = DIV_ROUND_UP(offset + i->count, PAGE_SIZE);
@@ -1366,6 +1551,146 @@ void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state)
i->nr_segs = state->nr_segs;
}
+/*
+ * Count the number of virtually contiguous pages coming up next in an
+ * ITER_BVECQ iterator, up to the specified maxima.
+ */
+static unsigned int iter_count_bvecq_pages(const struct iov_iter *iter,
+ size_t maxsize,
+ unsigned int maxpages)
+{
+ const struct bvecq *bvecq = iter->bvecq;
+ unsigned int slot = iter->bvecq_slot;
+ ssize_t remain = umin(maxsize, iter->count);
+ size_t count = 0, offset = iter->iov_offset;
+
+ do {
+ const struct bio_vec *bv;
+ size_t boff, blen;
+
+ if (slot >= bvecq->nr_slots) {
+ if (!bvecq->next) {
+ WARN_ON_ONCE(remain > 0);
+ break;
+ }
+ bvecq = bvecq->next;
+ slot = 0;
+ offset = 0;
+ continue;
+ }
+
+ bv = &bvecq->bv[slot++];
+ boff = bv->bv_offset;
+ blen = bv->bv_len;
+
+ /* bv_page is not allowed to be NULL unless bv_len == 0. */
+ if (WARN_ON_ONCE(!bv->bv_page && blen > 0))
+ break;
+ if (!PAGE_ALIGNED(boff) && count > 0)
+ break;
+
+ boff += offset;
+ blen -= offset;
+ offset = 0;
+ if (!blen)
+ continue;
+
+ blen = umin(blen, remain);
+ remain -= blen;
+ blen += offset_in_page(boff);
+ count += DIV_ROUND_UP(blen, PAGE_SIZE);
+
+ if (!PAGE_ALIGNED(blen))
+ break;
+ } while (remain > 0 && count < maxpages);
+
+ return umin(count, maxpages);
+}
+
+/*
+ * Extract a list of virtually contiguous pages from an ITER_BVECQ iterator.
+ * This does not get references on the pages, nor does it get a pin on them.
+ */
+static ssize_t iov_iter_extract_bvecq_pages(struct iov_iter *iter,
+ struct page ***pages, size_t maxsize,
+ unsigned int maxpages,
+ iov_iter_extraction_t extraction_flags,
+ size_t *offset0)
+{
+ const struct bvecq *bvecq;
+ struct page **p;
+ unsigned int slot, nr = 0;
+ size_t extracted = 0, offset;
+
+ /* Count the next run of virtually contiguous pages. */
+ maxpages = iter_count_bvecq_pages(iter, maxsize, maxpages);
+ if (!maxpages)
+ return 0;
+
+ if (!*pages) {
+ *pages = kvmalloc_array(maxpages, sizeof(struct page *), GFP_KERNEL);
+ if (!*pages)
+ return -ENOMEM;
+ }
+
+ p = *pages;
+
+ /* Now transcribe the page pointers. */
+ extracted = 0;
+ bvecq = iter->bvecq;
+ offset = iter->iov_offset;
+ slot = iter->bvecq_slot;
+
+ do {
+ const struct bio_vec *bv;
+ size_t boff, blen;
+
+ if (slot >= bvecq->nr_slots) {
+ if (!bvecq->next) {
+ WARN_ON_ONCE(extracted < iter->count);
+ break;
+ }
+ bvecq = bvecq->next;
+ slot = 0;
+ offset = 0;
+ continue;
+ }
+
+ bv = &bvecq->bv[slot];
+ boff = bv->bv_offset;
+ blen = bv->bv_len;
+
+ /* bv_page is not allowed to be NULL unless bv_len == 0. */
+
+ if (offset < blen) {
+ size_t part = umin(maxsize - extracted, blen - offset);
+ size_t poff = (boff + offset) % PAGE_SIZE;
+ size_t pix = (boff + offset) / PAGE_SIZE;
+
+ if (poff + part > PAGE_SIZE)
+ part = PAGE_SIZE - poff;
+
+ if (!extracted)
+ *offset0 = poff;
+
+ p[nr++] = bv->bv_page + pix;
+ offset += part;
+ extracted += part;
+ }
+
+ if (offset >= blen) {
+ offset = 0;
+ slot++;
+ }
+ } while (nr < maxpages && extracted < maxsize);
+
+ iter->bvecq = bvecq;
+ iter->bvecq_slot = slot;
+ iter->iov_offset = offset;
+ iter->count -= extracted;
+ return extracted;
+}
+
/*
* Extract a list of contiguous pages from an ITER_FOLIOQ iterator. This does
* not get references on the pages, nor does it get a pin on them.
@@ -1726,6 +2051,10 @@ ssize_t iov_iter_extract_pages(struct iov_iter *i,
return iov_iter_extract_bvec_pages(i, pages, maxsize,
maxpages, extraction_flags,
offset0);
+ if (iov_iter_is_bvecq(i))
+ return iov_iter_extract_bvecq_pages(i, pages, maxsize,
+ maxpages, extraction_flags,
+ offset0);
if (iov_iter_is_folioq(i))
return iov_iter_extract_folioq_pages(i, pages, maxsize,
maxpages, extraction_flags,
diff --git a/lib/scatterlist.c b/lib/scatterlist.c
index 6ea40d2e6247..23e5a180103b 100644
--- a/lib/scatterlist.c
+++ b/lib/scatterlist.c
@@ -10,6 +10,7 @@
#include <linux/highmem.h>
#include <linux/kmemleak.h>
#include <linux/bvec.h>
+#include <linux/bvecq.h>
#include <linux/uio.h>
#include <linux/folio_queue.h>
@@ -1267,6 +1268,65 @@ static ssize_t extract_kvec_to_sg(struct iov_iter *iter,
return ret;
}
+/*
+ * Extract up to sg_max folios from an BVECQ-type iterator and add them to
+ * the scatterlist. The pages are not pinned.
+ */
+static ssize_t extract_bvecq_to_sg(struct iov_iter *iter,
+ ssize_t maxsize,
+ struct sg_table *sgtable,
+ unsigned int sg_max,
+ iov_iter_extraction_t extraction_flags)
+{
+ const struct bvecq *bvecq = iter->bvecq;
+ struct scatterlist *sg = sgtable->sgl + sgtable->nents;
+ unsigned int slot = iter->bvecq_slot;
+ ssize_t ret = 0;
+ size_t offset = iter->iov_offset;
+
+ maxsize = umin(maxsize, iov_iter_count(iter));
+
+ while (sg_max > 0 && ret < maxsize) {
+ const struct bio_vec *bv;
+ size_t blen, part;
+
+ if (slot >= bvecq->nr_slots) {
+ if (!bvecq->next) {
+ WARN_ON_ONCE(ret < iter->count);
+ break;
+ }
+ bvecq = bvecq->next;
+ slot = 0;
+ offset = 0;
+ continue;
+ }
+
+ bv = &bvecq->bv[slot];
+ blen = bv->bv_len;
+
+ if (offset >= blen) {
+ offset = 0;
+ slot++;
+ continue;
+ }
+
+ part = umin(maxsize - ret, blen - offset);
+
+ sg_set_page(sg, bv->bv_page, part, bv->bv_offset + offset);
+ sgtable->nents++;
+ sg++;
+ sg_max--;
+ offset += part;
+ ret += part;
+ }
+
+ iter->bvecq = bvecq;
+ iter->bvecq_slot = slot;
+ iter->iov_offset = offset;
+ iter->count -= ret;
+ return ret;
+}
+
/*
* Extract up to sg_max folios from an FOLIOQ-type iterator and add them to
* the scatterlist. The pages are not pinned.
@@ -1391,8 +1451,8 @@ static ssize_t extract_xarray_to_sg(struct iov_iter *iter,
* addition of @sg_max elements.
*
* The pages referred to by UBUF- and IOVEC-type iterators are extracted and
- * pinned; BVEC-, KVEC-, FOLIOQ- and XARRAY-type are extracted but aren't
- * pinned; DISCARD-type is not supported.
+ * pinned; BVEC-, BVECQ-, KVEC-, FOLIOQ- and XARRAY-type are extracted but
+ * aren't pinned; DISCARD-type is not supported.
*
* No end mark is placed on the scatterlist; that's left to the caller.
*
@@ -1424,6 +1484,9 @@ ssize_t extract_iter_to_sg(struct iov_iter *iter, size_t maxsize,
case ITER_KVEC:
return extract_kvec_to_sg(iter, maxsize, sgtable, sg_max,
extraction_flags);
+ case ITER_BVECQ:
+ return extract_bvecq_to_sg(iter, maxsize, sgtable, sg_max,
+ extraction_flags);
case ITER_FOLIOQ:
return extract_folioq_to_sg(iter, maxsize, sgtable, sg_max,
extraction_flags);
diff --git a/lib/tests/kunit_iov_iter.c b/lib/tests/kunit_iov_iter.c
index d9690ba1db88..f6413e6b925a 100644
--- a/lib/tests/kunit_iov_iter.c
+++ b/lib/tests/kunit_iov_iter.c
@@ -12,6 +12,7 @@
#include <linux/mm.h>
#include <linux/uio.h>
#include <linux/bvec.h>
+#include <linux/bvecq.h>
#include <linux/folio_queue.h>
#include <linux/scatterlist.h>
#include <linux/minmax.h>
@@ -552,6 +553,183 @@ static void __init iov_kunit_copy_from_folioq(struct kunit *test)
KUNIT_SUCCEED(test);
}
+static void iov_kunit_destroy_bvecq(void *data)
+{
+ struct bvecq *bq, *next;
+
+ for (bq = data; bq; bq = next) {
+ next = bq->next;
+ /* The pages are freed by vmap with VM_MAP_PUT_PAGES. */
+ kfree(bq);
+ }
+}
+
+static struct bvecq *iov_kunit_alloc_bvecq(struct kunit *test, unsigned int max_slots)
+{
+ struct bvecq *bq;
+
+ bq = kzalloc(struct_size(bq, __bv, max_slots), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bq);
+ bq->max_slots = max_slots;
+ bq->bv = bq->__bv;
+ bq->inline_bv = true;
+ return bq;
+}
+
+static struct bvecq *iov_kunit_create_bvecq(struct kunit *test, unsigned int max_slots)
+{
+ struct bvecq *bq;
+
+ bq = iov_kunit_alloc_bvecq(test, max_slots);
+ kunit_add_action_or_reset(test, iov_kunit_destroy_bvecq, bq);
+ return bq;
+}
+
+static void __init iov_kunit_load_bvecq(struct kunit *test,
+ struct iov_iter *iter, int dir,
+ struct bvecq *bq_head,
+ struct page **pages, size_t npages)
+{
+ struct bvecq *bq = bq_head;
+ size_t size = 0;
+
+ for (int i = 0; i < npages; i++) {
+ if (bq->nr_slots >= bq->max_slots) {
+ bq->next = iov_kunit_alloc_bvecq(test, 13);
+ bq->next->prev = bq;
+ bq = bq->next;
+ }
+ bvec_set_page(&bq->bv[bq->nr_slots], pages[i], PAGE_SIZE, 0);
+ bq->nr_slots++;
+ size += PAGE_SIZE;
+ }
+ iov_iter_bvec_queue(iter, dir, bq_head, 0, 0, size);
+}
+
+/*
+ * Test copying to a ITER_BVECQ-type iterator.
+ */
+static void __init iov_kunit_copy_to_bvecq(struct kunit *test)
+{
+ const struct kvec_test_range *pr;
+ struct iov_iter iter;
+ struct bvecq *bq;
+ struct page **spages, **bpages;
+ u8 *scratch, *buffer;
+ size_t bufsize, npages, size, copied;
+ int i, patt;
+
+ bufsize = 0x100000;
+ npages = bufsize / PAGE_SIZE;
+
+ bq = iov_kunit_create_bvecq(test, 13);
+
+ scratch = iov_kunit_create_buffer(test, &spages, npages);
+ for (i = 0; i < bufsize; i++)
+ scratch[i] = pattern(i);
+
+ buffer = iov_kunit_create_buffer(test, &bpages, npages);
+ memset(buffer, 0, bufsize);
+
+ iov_kunit_load_bvecq(test, &iter, READ, bq, bpages, npages);
+
+ i = 0;
+ for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
+ size = pr->to - pr->from;
+ KUNIT_ASSERT_LE(test, pr->to, bufsize);
+
+ iov_iter_bvec_queue(&iter, READ, bq, 0, 0, pr->to);
+ iov_iter_advance(&iter, pr->from);
+ copied = copy_to_iter(scratch + i, size, &iter);
+
+ KUNIT_EXPECT_EQ(test, copied, size);
+ KUNIT_EXPECT_EQ(test, iter.count, 0);
+ i += size;
+ if (test->status == KUNIT_FAILURE)
+ goto stop;
+ }
+
+ /* Build the expected image in the scratch buffer. */
+ patt = 0;
+ memset(scratch, 0, bufsize);
+ for (pr = kvec_test_ranges; pr->from >= 0; pr++)
+ for (i = pr->from; i < pr->to; i++)
+ scratch[i] = pattern(patt++);
+
+ /* Compare the images */
+ for (i = 0; i < bufsize; i++) {
+ KUNIT_EXPECT_EQ_MSG(test, buffer[i], scratch[i], "at i=%x", i);
+ if (buffer[i] != scratch[i])
+ return;
+ }
+
+stop:
+ KUNIT_SUCCEED(test);
+}
+
+/*
+ * Test copying from a ITER_BVECQ-type iterator.
+ */
+static void __init iov_kunit_copy_from_bvecq(struct kunit *test)
+{
+ const struct kvec_test_range *pr;
+ struct iov_iter iter;
+ struct bvecq *bq;
+ struct page **spages, **bpages;
+ u8 *scratch, *buffer;
+ size_t bufsize, npages, size, copied;
+ int i, j;
+
+ bufsize = 0x100000;
+ npages = bufsize / PAGE_SIZE;
+
+ bq = iov_kunit_create_bvecq(test, 13);
+
+ buffer = iov_kunit_create_buffer(test, &bpages, npages);
+ for (i = 0; i < bufsize; i++)
+ buffer[i] = pattern(i);
+
+ scratch = iov_kunit_create_buffer(test, &spages, npages);
+ memset(scratch, 0, bufsize);
+
+ iov_kunit_load_bvecq(test, &iter, READ, bq, bpages, npages);
+
+ i = 0;
+ for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
+ size = pr->to - pr->from;
+ KUNIT_ASSERT_LE(test, pr->to, bufsize);
+
+ iov_iter_bvec_queue(&iter, WRITE, bq, 0, 0, pr->to);
+ iov_iter_advance(&iter, pr->from);
+ copied = copy_from_iter(scratch + i, size, &iter);
+
+ KUNIT_EXPECT_EQ(test, copied, size);
+ KUNIT_EXPECT_EQ(test, iter.count, 0);
+ i += size;
+ }
+
+ /* Build the expected image in the main buffer. */
+ i = 0;
+ memset(buffer, 0, bufsize);
+ for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
+ for (j = pr->from; j < pr->to; j++) {
+ buffer[i++] = pattern(j);
+ if (i >= bufsize)
+ goto stop;
+ }
+ }
+stop:
+
+ /* Compare the images */
+ for (i = 0; i < bufsize; i++) {
+ KUNIT_EXPECT_EQ_MSG(test, scratch[i], buffer[i], "at i=%x", i);
+ if (scratch[i] != buffer[i])
+ return;
+ }
+
+ KUNIT_SUCCEED(test);
+}
+
static void iov_kunit_destroy_xarray(void *data)
{
struct xarray *xarray = data;
@@ -867,6 +1045,85 @@ static void __init iov_kunit_extract_pages_bvec(struct kunit *test)
KUNIT_SUCCEED(test);
}
+/*
+ * Test the extraction of ITER_BVECQ-type iterators.
+ */
+static void __init iov_kunit_extract_pages_bvecq(struct kunit *test)
+{
+ const struct kvec_test_range *pr;
+ struct iov_iter iter;
+ struct bvecq *bq;
+ struct page **bpages, *pagelist[8], **pages = pagelist;
+ ssize_t len;
+ size_t bufsize, size = 0, npages;
+ int i, from;
+
+ bufsize = 0x100000;
+ npages = bufsize / PAGE_SIZE;
+
+ bq = iov_kunit_create_bvecq(test, 13);
+
+ iov_kunit_create_buffer(test, &bpages, npages);
+ iov_kunit_load_bvecq(test, &iter, READ, bq, bpages, npages);
+
+ for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
+ from = pr->from;
+ size = pr->to - from;
+ KUNIT_ASSERT_LE(test, pr->to, bufsize);
+
+ iov_iter_bvec_queue(&iter, WRITE, bq, 0, 0, pr->to);
+ iov_iter_advance(&iter, from);
+
+ do {
+ size_t offset0 = LONG_MAX;
+
+ for (i = 0; i < ARRAY_SIZE(pagelist); i++)
+ pagelist[i] = (void *)(unsigned long)0xaa55aa55aa55aa55ULL;
+
+ len = iov_iter_extract_pages(&iter, &pages, 100 * 1024,
+ ARRAY_SIZE(pagelist), 0, &offset0);
+ KUNIT_EXPECT_GE(test, len, 0);
+ if (len < 0)
+ break;
+ KUNIT_EXPECT_LE(test, len, size);
+ KUNIT_EXPECT_EQ(test, iter.count, size - len);
+ if (len == 0)
+ break;
+ size -= len;
+ KUNIT_EXPECT_GE(test, (ssize_t)offset0, 0);
+ KUNIT_EXPECT_LT(test, offset0, PAGE_SIZE);
+
+ for (i = 0; i < ARRAY_SIZE(pagelist); i++) {
+ struct page *p;
+ ssize_t part = min_t(ssize_t, len, PAGE_SIZE - offset0);
+ int ix;
+
+ KUNIT_ASSERT_GE(test, part, 0);
+ ix = from / PAGE_SIZE;
+ KUNIT_ASSERT_LT(test, ix, npages);
+ p = bpages[ix];
+ KUNIT_EXPECT_PTR_EQ(test, pagelist[i], p);
+ KUNIT_EXPECT_EQ(test, offset0, from % PAGE_SIZE);
+ from += part;
+ len -= part;
+ KUNIT_ASSERT_GE(test, len, 0);
+ if (len == 0)
+ break;
+ offset0 = 0;
+ }
+
+ if (test->status == KUNIT_FAILURE)
+ goto stop;
+ } while (iov_iter_count(&iter) > 0);
+
+ KUNIT_EXPECT_EQ(test, size, 0);
+ KUNIT_EXPECT_EQ(test, iter.count, 0);
+ }
+
+stop:
+ KUNIT_SUCCEED(test);
+}
+
/*
* Test the extraction of ITER_FOLIOQ-type iterators.
*/
@@ -1226,12 +1483,15 @@ static struct kunit_case __refdata iov_kunit_cases[] = {
KUNIT_CASE(iov_kunit_copy_from_kvec),
KUNIT_CASE(iov_kunit_copy_to_bvec),
KUNIT_CASE(iov_kunit_copy_from_bvec),
+ KUNIT_CASE(iov_kunit_copy_to_bvecq),
+ KUNIT_CASE(iov_kunit_copy_from_bvecq),
KUNIT_CASE(iov_kunit_copy_to_folioq),
KUNIT_CASE(iov_kunit_copy_from_folioq),
KUNIT_CASE(iov_kunit_copy_to_xarray),
KUNIT_CASE(iov_kunit_copy_from_xarray),
KUNIT_CASE(iov_kunit_extract_pages_kvec),
KUNIT_CASE(iov_kunit_extract_pages_bvec),
+ KUNIT_CASE(iov_kunit_extract_pages_bvecq),
KUNIT_CASE(iov_kunit_extract_pages_folioq),
KUNIT_CASE(iov_kunit_extract_pages_xarray),
KUNIT_CASE(iov_kunit_iter_to_sg_kvec),
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 11/35] netfs: Add some tools for managing bvecq chains
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (9 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 10/35] iov_iter: Add a segmented queue of bio_vec[] David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 12/35] netfs: Make mempool available for bvecq David Howells
` (24 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Provide a selection of tools for managing bvec queue chains. This
includes:
(1) Allocation, prepopulation, expansion, shortening and refcounting of
bvecqs and bvecq chains.
This can be used to do things like creating an encryption buffer in
cifs or a directory content buffer in afs. The memory segments will
be appropriate disposed off according to the flags on the bvecq.
(2) Management of a bvecq chain as a rolling buffer and the management of
positions within it.
(3) Loading folios, slicing chains and clearing content.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/Makefile | 1 +
fs/netfs/bvecq.c | 830 +++++++++++++++++++++++++++++++++++
fs/netfs/internal.h | 1 +
fs/netfs/stats.c | 4 +-
include/linux/bvecq.h | 308 +++++++++++++
include/linux/netfs.h | 1 +
include/trace/events/netfs.h | 24 +
7 files changed, 1168 insertions(+), 1 deletion(-)
create mode 100644 fs/netfs/bvecq.c
diff --git a/fs/netfs/Makefile b/fs/netfs/Makefile
index b43188d64bd8..e1f12ecb5abf 100644
--- a/fs/netfs/Makefile
+++ b/fs/netfs/Makefile
@@ -3,6 +3,7 @@
netfs-y := \
buffered_read.o \
buffered_write.o \
+ bvecq.o \
direct_read.o \
direct_write.o \
iterator.o \
diff --git a/fs/netfs/bvecq.c b/fs/netfs/bvecq.c
new file mode 100644
index 000000000000..6905c84ea351
--- /dev/null
+++ b/fs/netfs/bvecq.c
@@ -0,0 +1,830 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Buffering helpers for bvec queues
+ *
+ * Copyright (C) 2026 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+
+#include <linux/bvecq.h>
+#include "internal.h"
+
+void bvecq_dump(const struct bvecq *bq)
+{
+ int b = 0;
+
+ for (; bq; bq = bvecq_next(bq), b++) {
+ int skipz = 0;
+
+ pr_notice("BQ[%u] %u/%u\n", b, bq->nr_slots, bq->max_slots);
+ for (int s = 0; s < bq->nr_slots; s++) {
+ const struct bio_vec *bv = &bq->bv[s];
+
+ if (!bv->bv_page && !bv->bv_len && skipz < 2) {
+ skipz = 1;
+ continue;
+ }
+ if (skipz == 1)
+ pr_notice("BQ[%u:00-%02u] ...\n", b, s - 1);
+ skipz = 2;
+ pr_notice("BQ[%u:%02u] %10lx %04x %04x %u\n",
+ b, s,
+ bv->bv_page ? page_to_pfn(bv->bv_page) : 0,
+ bv->bv_offset, bv->bv_len,
+ bv->bv_page ? page_count(bv->bv_page) : 0);
+ }
+ }
+}
+EXPORT_SYMBOL(bvecq_dump);
+
+/**
+ * bvecq_alloc_one - Allocate a single bvecq node with unpopulated slots
+ * @nr_slots: Number of slots to allocate
+ * @gfp: The allocation constraints.
+ * @for_writeback: True if allocating for writeback
+ *
+ * Allocate a single bvecq node and initialise the header. A number of inline
+ * slots are also allocated, rounded up to fit after the header in a power-of-2
+ * slab object of up to 512 bytes (up to 29 slots on a 64-bit cpu). The slot
+ * array is not initialised.
+ *
+ * Return: The node pointer or NULL on allocation failure.
+ */
+struct bvecq *bvecq_alloc_one(size_t nr_slots, gfp_t gfp, bool for_writeback)
+{
+ struct bvecq *bq;
+ const size_t max_size = 512;
+ const size_t max_slots = (max_size - sizeof(*bq)) / sizeof(bq->__bv[0]);
+ size_t part = min(nr_slots, max_slots);
+ size_t size = roundup_pow_of_two(struct_size(bq, __bv, part));
+
+ bq = kmalloc(size, gfp & ~GFP_ZONEMASK);
+ if (!bq)
+ return bq;
+
+ *bq = (struct bvecq) {
+ .ref = REFCOUNT_INIT(1),
+ .bv = bq->__bv,
+ .inline_bv = true,
+ .max_slots = (size - sizeof(*bq)) / sizeof(bq->__bv[0]),
+ };
+ netfs_stat(&netfs_n_bvecq);
+ return bq;
+}
+EXPORT_SYMBOL(bvecq_alloc_one);
+
+/**
+ * bvecq_alloc_chain - Allocate an unpopulated bvecq chain
+ * @nr_slots: Number of slots to allocate
+ * @gfp: The allocation constraints.
+ * @for_writeback: True if allocating for writeback
+ *
+ * Allocate a chain of bvecq nodes providing at least the requested cumulative
+ * number of slots.
+ *
+ * Return: The first node pointer or NULL on allocation failure.
+ */
+struct bvecq *bvecq_alloc_chain(size_t nr_slots, gfp_t gfp, bool for_writeback)
+{
+ struct bvecq *head = NULL, *tail = NULL;
+
+ _enter("%zu", nr_slots);
+
+ for (;;) {
+ struct bvecq *bq;
+
+ bq = bvecq_alloc_one(nr_slots, gfp, for_writeback);
+ if (!bq)
+ goto oom;
+
+ if (tail)
+ bvecq_append(tail, bq);
+ else
+ head = bq;
+ tail = bq;
+ if (tail->max_slots >= nr_slots)
+ break;
+ nr_slots -= tail->max_slots;
+ }
+
+ return head;
+oom:
+ bvecq_put(head);
+ return NULL;
+}
+EXPORT_SYMBOL(bvecq_alloc_chain);
+
+/**
+ * bvecq_alloc_buffer2 - Allocate a bvecq chain and populate with buffers
+ * @size: Target size of the buffer (can be 0 for an empty buffer)
+ * @pre_slots: Number of preamble slots to set aside
+ * @gfp: The allocation constraints.
+ * @for_writeback: True if allocating for writeback
+ *
+ * Allocate a chain of bvecq nodes and populate the slots with sufficient pages
+ * to provide at least the requested amount of space, leaving the first
+ * @pre_slots slots unset. The pre-slots must all fit into the the first
+ * bvecq.
+ *
+ * The pages allocated may be compound pages larger than PAGE_SIZE and thus
+ * occupy fewer slots. The pages have their refcounts set to 1 and can be
+ * passed to MSG_SPLICE_PAGES.
+ *
+ * Return: The first node pointer or NULL on allocation failure.
+ */
+struct bvecq *bvecq_alloc_buffer2(size_t size, unsigned int pre_slots, gfp_t gfp,
+ bool for_writeback)
+{
+ struct bvecq *head = NULL, *p = NULL;
+ size_t nr_per_bq = BVECQ_STD_SLOTS;
+ size_t count = pre_slots + DIV_ROUND_UP(size, PAGE_SIZE);
+
+ _enter("%zx,%zx,%u", size, count, pre_slots);
+
+ if (WARN_ON_ONCE(pre_slots > nr_per_bq))
+ return NULL;
+
+ head = bvecq_alloc_chain(count, gfp, for_writeback);
+ if (!head)
+ return NULL;
+
+ p = head;
+ do {
+ struct page **pages;
+ size_t unused, want, got, slot;
+
+ if (!count)
+ break;
+ if (WARN_ON_ONCE(!p))
+ goto oom;
+
+ if (p->nr_slots == 0) {
+ /* Need to clear pre slots and pages[], so just clear all. */
+ memset(p->bv, 0, p->max_slots * sizeof(p->bv[0]));
+ p->mem_type = BVECQ_MEM_ALLOCED;
+ p->nr_slots = pre_slots;
+ count -= pre_slots;
+ pre_slots = 0;
+ if (!count)
+ break;
+ }
+
+ if (p->nr_slots >= p->max_slots) {
+ p = p->next;
+ continue;
+ }
+ unused = p->max_slots - p->nr_slots;
+
+ pages = (struct page **)&p->bv[p->max_slots];
+ pages -= unused;
+
+ want = min(count, unused);
+ got = alloc_pages_bulk(gfp, want, pages);
+ if (!got)
+ goto oom;
+
+ slot = p->nr_slots;
+ for (int i = 0; i < got; i++) {
+ set_page_count(pages[i], 1);
+ bvec_set_page(&p->bv[slot++], pages[i], PAGE_SIZE, 0);
+ }
+
+ bvecq_filled_to(p, slot);
+ count -= got;
+ } while (count > 0);
+
+ return head;
+oom:
+ bvecq_put(head);
+ return NULL;
+}
+EXPORT_SYMBOL(bvecq_alloc_buffer2);
+
+/*
+ * Free the page pointed to by a slot as necessary.
+ */
+static void bvecq_free_slot(struct bvecq *bq, unsigned int slot)
+{
+ struct page *page = bq->bv[slot].bv_page;
+
+ if (!page)
+ return;
+
+ switch (bq->mem_type) {
+ case BVECQ_MEM_EXTERNAL:
+ break;
+ case BVECQ_MEM_PAGECACHE:
+ put_page(page);
+ break;
+ case BVECQ_MEM_GUP:
+ unpin_user_page(page);
+ break;
+ case BVECQ_MEM_ALLOCED:
+ __free_pages(page, compound_order(page));
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
+}
+
+/**
+ * bvecq_put - Put a ref on a bvec queue
+ * @bq: The start of the folio queue to free
+ *
+ * Put the ref(s) on the nodes in a bvec queue, freeing up the node and the
+ * page fragments it points to as the refcounts become zero.
+ */
+void bvecq_put(struct bvecq *bq)
+{
+ struct bvecq *next;
+
+ for (; bq; bq = next) {
+ if (!refcount_dec_and_test(&bq->ref))
+ break;
+ for (int slot = 0; slot < bq->nr_slots; slot++)
+ bvecq_free_slot(bq, slot);
+ next = bq->next;
+ netfs_stat_d(&netfs_n_bvecq);
+ kfree(bq);
+ }
+}
+EXPORT_SYMBOL(bvecq_put);
+
+/**
+ * bvecq_expand_buffer - Allocate buffer space into a bvec queue
+ * @_buffer: Pointer to the bvecq chain to expand (may point to a NULL; updated).
+ * @_cur_size: Current size of the buffer (updated).
+ * @size: Target size of the buffer.
+ * @gfp: The allocation constraints.
+ *
+ * Append extra pages to a buffer to increase its capacity to the @size
+ * specified. If the current tail has space, but is not of the
+ * BVECQ_MEM_ALLOCED memory type, a separate bvecq will be allocated to hold
+ * the new memory.
+ */
+int bvecq_expand_buffer(struct bvecq **_buffer, size_t *_cur_size, size_t size, gfp_t gfp)
+{
+ struct bvecq *tail = *_buffer;
+
+ size = round_up(size, PAGE_SIZE);
+ if (tail)
+ while (tail->next)
+ tail = tail->next;
+
+ while (*_cur_size < size) {
+ struct page *page;
+ size_t need = size - *_cur_size;
+ int order = 0;
+
+ if (!tail || bvecq_is_full(tail) || tail->mem_type != BVECQ_MEM_ALLOCED) {
+ struct bvecq *p;
+
+ p = bvecq_alloc_one(BVECQ_STD_SLOTS, gfp, false);
+ if (!p)
+ return -ENOMEM;
+ if (tail)
+ bvecq_append(tail, p);
+ else
+ *_buffer = p;
+ tail = p;
+ p->mem_type = BVECQ_MEM_ALLOCED;
+ }
+
+ if (need > PAGE_SIZE)
+ order = umin(ilog2(need) - PAGE_SHIFT, MAX_PAGECACHE_ORDER);
+
+ page = alloc_pages(gfp | __GFP_COMP, order);
+ if (!page && order > 0) {
+ page = alloc_pages(gfp | __GFP_COMP, 0);
+ order = 0;
+ }
+ if (!page)
+ return -ENOMEM;
+
+ bvec_set_page(&tail->bv[tail->nr_slots], page, PAGE_SIZE << order, 0);
+ *_cur_size += PAGE_SIZE << order;
+ bvecq_filled_to(tail, tail->nr_slots + 1);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(bvecq_expand_buffer);
+
+/**
+ * bvecq_shorten_buffer - Shorten a bvec queue buffer
+ * @bq: The start of the buffer to shorten
+ * @slot: The slot to start from
+ * @size: The size to retain
+ *
+ * Shorten the content of a bvec queue down to the minimum number of slots,
+ * starting at the specified slot, to retain the specified size.
+ *
+ * Return: 0 if successful; -EMSGSIZE if there is insufficient content.
+ */
+int bvecq_shorten_buffer(struct bvecq *bq, unsigned int slot, size_t size)
+{
+ struct bvecq *next;
+
+ /* Skip through the segments we want to keep. */
+ for (; bq; bq = bq->next) {
+ for (; slot < bq->nr_slots; slot++) {
+ if (size < bq->bv[slot].bv_len)
+ goto found;
+ size -= bq->bv[slot].bv_len;
+ }
+ slot = 0;
+ }
+ if (WARN_ON_ONCE(size > 0))
+ return -EMSGSIZE;
+ return 0;
+
+found:
+ /* Shorten any partial entry and clean the rest of this bvecq. */
+ if (size > 0) {
+ bq->bv[slot].bv_len = size;
+ slot++;
+ }
+ for (int i = slot; i < bq->nr_slots; i++)
+ bvecq_free_slot(bq, i);
+ bq->nr_slots = slot;
+
+ /* Free the queue tail. */
+ next = bq->next;
+ bq->next = NULL;
+ bvecq_put(next);
+ return 0;
+}
+EXPORT_SYMBOL(bvecq_shorten_buffer);
+
+/**
+ * bvecq_buffer_init - Initialise a buffer and set position
+ * @pos: The position to point at the new buffer.
+ * @gfp: The allocation constraints.
+ * @for_writeback: True if allocating for writeback
+ *
+ * Initialise a rolling buffer. We allocate an unpopulated bvecq node to so
+ * that the pointers can be independently driven by the producer and the
+ * consumer.
+ *
+ * Return 0 if successful; -ENOMEM on allocation failure.
+ */
+int bvecq_buffer_init(struct bvecq_pos *pos, gfp_t gfp, bool for_writeback)
+{
+ struct bvecq *bq;
+
+ bq = bvecq_alloc_one(BVECQ_STD_SLOTS, gfp, for_writeback);
+ if (!bq)
+ return -ENOMEM;
+
+ pos->bvecq = bq; /* Comes with a ref. */
+ pos->slot = 0;
+ pos->offset = 0;
+ return 0;
+}
+
+/**
+ * bvecq_buffer_append - Append a new bvecq node to a buffer
+ * @pos: The position of the last node.
+ * @bq: The buffer to add.
+ *
+ * Add a new node on to the buffer chain at the specified position, either
+ * because the previous one is full or because we have a discontiguity to
+ * contend with, and update @pos to point to it.
+ */
+void bvecq_buffer_append(struct bvecq_pos *pos, struct bvecq *bq)
+{
+ struct bvecq *head = pos->bvecq;
+
+ pos->bvecq = bvecq_get(bq);
+ pos->slot = 0;
+ pos->offset = 0;
+
+ /* [!] NOTE: After we set head->next, the consumer is at liberty to
+ * immediately delete the old head.
+ */
+ bvecq_append(head, bq);
+ bvecq_put(head);
+}
+
+/**
+ * bvecq_pos_advance - Advance a bvecq position
+ * @pos: The position to advance.
+ * @amount: The amount of bytes to advance by.
+ *
+ * Advance the specified bvecq position by @amount bytes. @pos is updated and
+ * bvecq ref counts may have been manipulated. If the position hits the end of
+ * the queue, then it is left pointing beyond the last slot of the last bvecq
+ * so that it doesn't break the chain.
+ */
+void bvecq_pos_advance(struct bvecq_pos *pos, size_t amount)
+{
+ struct bvecq *bq = pos->bvecq, *next;
+ unsigned int slot = pos->slot;
+ size_t offset = pos->offset;
+
+ while (amount) {
+ size_t part;
+
+ if (!bvecq_acquire_slot(bq, slot)) {
+ next = bvecq_next(bq);
+ if (!next) {
+ WARN_ON_ONCE(amount > 0);
+ break;
+ }
+ if (bvecq_acquire_slot(bq, slot))
+ continue; /* More slots got added. */
+ bq = next;
+ slot = 0;
+ offset = 0;
+ continue;
+ }
+
+ part = bq->bv[slot].bv_len - offset;
+
+ if (part > amount) {
+ offset += amount;
+ break;
+ }
+ amount -= part;
+ offset = 0;
+ slot++;
+ }
+
+ pos->slot = slot;
+ pos->offset = offset;
+ bvecq_pos_move(pos, bq);
+}
+
+/*
+ * Clear part of the memory pointed to by a bio_vec.
+ */
+static void bvec_zero(const struct bio_vec *bv, size_t offset, size_t len)
+{
+ struct page *page = bv->bv_page;
+
+ offset += bv->bv_offset;
+
+ page += offset / PAGE_SIZE;
+ offset = offset % PAGE_SIZE;
+
+ while (len) {
+ size_t part = min(len, PAGE_SIZE - offset);
+ char *p = kmap_local_page(page);
+
+ memset(p + offset, 0, part);
+ kunmap_local(p);
+
+ len -= part;
+ offset = 0;
+ page++;
+ }
+}
+
+/**
+ * bvecq_zero - Clear memory starting at the bvecq position.
+ * @pos: The position in the bvecq chain to start clearing.
+ * @amount: The number of bytes to clear.
+ *
+ * Clear memory fragments pointed to by a bvec queue. @pos is updated and
+ * bvecq ref counts may have been manipulated. If the position hits the end of
+ * the queue, then it is left pointing beyond the last slot of the last bvecq
+ * so that it doesn't break the chain.
+ *
+ * Return: The number of bytes cleared.
+ */
+ssize_t bvecq_zero(struct bvecq_pos *pos, size_t amount)
+{
+ struct bvecq *bq = pos->bvecq, *next;
+ unsigned int slot = pos->slot;
+ ssize_t cleared = 0;
+ size_t offset = pos->offset;
+
+ while (amount) {
+ const struct bio_vec *bv;
+ size_t part;
+
+ if (!bvecq_acquire_slot(bq, slot)) {
+ next = bvecq_next(bq);
+ if (!next) {
+ WARN_ON_ONCE(amount > 0);
+ break;
+ }
+ if (bvecq_acquire_slot(bq, slot))
+ continue; /* More slots got added. */
+ bq = next;
+ slot = 0;
+ offset = 0;
+ continue;
+ }
+
+ bv = &bq->bv[slot];
+ if (offset >= bv->bv_len) {
+ slot++;
+ offset = 0;
+ continue;
+ }
+
+ part = min(bv->bv_len - offset, amount);
+ bvec_zero(bv, offset, part);
+ cleared += part;
+ offset += part;
+ amount -= part;
+ }
+
+ pos->slot = slot;
+ pos->offset = offset;
+ bvecq_pos_move(pos, bq);
+ return cleared;
+}
+
+/**
+ * bvecq_slice - Find a slice of a bvecq queue
+ * @pos: The position to start at.
+ * @max_size: The maximum size of the slice (or ULONG_MAX).
+ * @max_slots: The maximum number of slots in the slice (or INT_MAX).
+ * @_nr_slots: Where to put the number of slots (updated).
+ *
+ * Determine the size and number of slots that can be obtained the next slice
+ * of bvec queue up to the maximum size and slot count specified.
+ *
+ * @pos is updated to the end of the slice. If the position hits the end of
+ * the queue, then it is left pointing beyond the last slot of the last bvecq
+ * so that it doesn't break the chain.
+ *
+ * Return: The number of bytes in the slice.
+ */
+size_t bvecq_slice(struct bvecq_pos *pos, size_t max_size,
+ unsigned int max_slots, unsigned int *_nr_slots)
+{
+ struct bvecq *bq, *next;
+ unsigned int slot = pos->slot, nslots = 0;
+ size_t size = 0, offset = pos->offset;
+
+ bq = pos->bvecq;
+ for (;;) {
+ for (; slot < bvecq_nr_slots_acquire(bq); slot++) {
+ const struct bio_vec *bvec = &bq->bv[slot];
+
+ if (offset < bvec->bv_len && bvec->bv_page) {
+ size_t part = min(bvec->bv_len - offset, max_size);
+
+ size += part;
+ offset += part;
+ max_size -= part;
+ nslots++;
+ if (!max_size || nslots >= max_slots)
+ goto out;
+ }
+ offset = 0;
+ }
+
+ /* pos->bvecq isn't allowed to go NULL as the queue may get
+ * extended and we would lose our place.
+ */
+ next = bvecq_next(bq);
+ if (!next)
+ break;
+ if (bvecq_acquire_slot(bq, slot))
+ continue; /* More slots got added. */
+ slot = 0;
+ bq = next;
+ }
+
+out:
+ *_nr_slots = nslots;
+ if (slot == bvecq_nr_slots_acquire(bq)) {
+ next = bvecq_next(bq);
+ if (next) {
+ bq = next;
+ slot = 0;
+ offset = 0;
+ }
+ }
+ bvecq_pos_move(pos, bq);
+ pos->slot = slot;
+ pos->offset = offset;
+ return size;
+}
+
+/**
+ * bvecq_extract - Extract a slice of a bvecq queue into a new bvecq queue
+ * @pos: The position to start at.
+ * @max_size: The maximum size of the slice (or ULONG_MAX).
+ * @max_slots: The maximum number of slots in the slice (or INT_MAX).
+ * @to: Where to put the extraction bvecq chain head (updated).
+ * @for_writeback: True if allocating for writeback
+ *
+ * Allocate a new bvecq and extract into it memory fragments from a slice of
+ * bvec queue, starting at @pos. No refs are taken on the page.
+ *
+ * @pos is updated to the end of the slice. If the position hits the end of
+ * the queue, then it is left pointing beyond the last slot of the last bvecq
+ * so that it doesn't break the chain.
+ *
+ * If successful, *@to is set to point to the head of the newly allocated chain
+ * and the caller inherits a ref to it.
+ *
+ * Return: The number of bytes extracted; -ENOMEM on allocation failure or -EIO
+ * if no slots were available to extract.
+ */
+ssize_t bvecq_extract(struct bvecq_pos *pos, size_t max_size, unsigned int max_slots,
+ struct bvecq **to, bool for_writeback)
+{
+ struct bvecq_pos tmp_pos;
+ struct bvecq *src, *dst = NULL, *next;
+ unsigned int slot = pos->slot, dslot = 0, nslots;
+ ssize_t extracted = 0;
+ size_t offset = pos->offset, amount;
+
+ *to = NULL;
+ if (WARN_ON_ONCE(!max_slots))
+ max_slots = INT_MAX;
+
+ bvecq_pos_set(&tmp_pos, pos);
+ amount = bvecq_slice(&tmp_pos, max_size, max_slots, &nslots);
+ bvecq_pos_unset(&tmp_pos);
+ if (nslots == 0)
+ return -EIO;
+
+ dst = bvecq_alloc_chain(nslots, GFP_NOFS, for_writeback);
+ if (!dst)
+ return -ENOMEM;
+ *to = dst;
+ max_slots = nslots;
+ nslots = 0;
+
+ /* Transcribe the slots */
+ src = pos->bvecq;
+ for (;;) {
+ for (; slot < bvecq_nr_slots_acquire(src); slot++) {
+ const struct bio_vec *sv = &src->bv[slot];
+ struct bio_vec *dv = &dst->bv[dslot];
+
+ _debug("EXTR BQ=%x[%x] off=%zx am=%zx p=%lx",
+ src->priv, slot, offset, amount, page_to_pfn(sv->bv_page));
+
+ if (offset < sv->bv_len && sv->bv_page) {
+ size_t part = min(sv->bv_len - offset, amount);
+
+ bvec_set_page(dv, sv->bv_page, part,
+ sv->bv_offset + offset);
+ extracted += part;
+ amount -= part;
+ offset += part;
+ trace_netfs_bv_slot(dst, dslot);
+ dslot++;
+ nslots++;
+ if (dslot >= dst->max_slots) {
+ bvecq_filled_to(dst, dslot);
+ dst = dst->next;
+ dslot = 0;
+ }
+ if (nslots >= max_slots)
+ goto out;
+ if (amount == 0)
+ goto out;
+ }
+ offset = 0;
+ }
+
+ /* pos->bvecq isn't allowed to go NULL as the queue may get
+ * extended and we would lose our place.
+ */
+ next = bvecq_next(src);
+ if (!next)
+ break;
+ if (bvecq_acquire_slot(src, slot))
+ continue; /* More slots got added. */
+ slot = 0;
+ src = next;
+ if (extracted > 0)
+ break;
+ }
+
+out:
+ if (dst)
+ bvecq_filled_to(dst, dslot);
+ if (slot == bvecq_nr_slots_acquire(src)) {
+ next = bvecq_next(src);
+ if (next) {
+ src = next;
+ slot = 0;
+ offset = 0;
+ }
+ }
+ bvecq_pos_move(pos, src);
+ pos->slot = slot;
+ pos->offset = offset;
+ return extracted;
+}
+
+/**
+ * bvecq_load_from_ra - Allocate a bvecq chain and load from readahead
+ * @pos: Blank position object to attach the new chain to.
+ * @ractl: The readahead control context.
+ *
+ * Decant the set of folios to be read from the readahead context into a bvecq
+ * chain. Each folio occupies one bio_vec element.
+ *
+ * Return: Amount of data loaded or -ENOMEM on allocation failure.
+ */
+ssize_t bvecq_load_from_ra(struct bvecq_pos *pos, struct readahead_control *ractl)
+{
+ XA_STATE(xas, &ractl->mapping->i_pages, ractl->_index);
+ struct folio *folio;
+ struct bvecq *bq;
+ unsigned int slot = 0;
+ size_t loaded = 0;
+
+ bq = bvecq_alloc_chain(ractl->_nr_folios, GFP_KERNEL, false);
+ if (!bq)
+ return -ENOMEM;
+
+ pos->bvecq = bq;
+ pos->slot = 0;
+ pos->offset = 0;
+
+ rcu_read_lock();
+
+ xas_for_each(&xas, folio, ractl->_index + ractl->_nr_pages - 1) {
+ size_t len;
+
+ if (xas_retry(&xas, folio))
+ continue;
+ VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+
+ len = folio_size(folio);
+ bvec_set_folio(&bq->bv[slot], folio, len, 0);
+ loaded += len;
+ slot++;
+ trace_netfs_folio(folio, netfs_folio_trace_read);
+
+ if (slot >= bq->max_slots) {
+ bvecq_filled_to(bq, slot);
+ bq = bq->next;
+ if (!bq)
+ break;
+ slot = 0;
+ }
+ }
+
+ rcu_read_unlock();
+
+ if (bq)
+ bvecq_filled_to(bq, slot);
+
+ ractl->_index += ractl->_nr_pages;
+ ractl->_nr_pages = 0;
+ return loaded;
+}
+
+/*
+ * Add space to a buffer.
+ */
+static int bvecq_buffer_make_space(struct bvecq_pos *pos, gfp_t gfp, bool for_writeback)
+{
+ struct bvecq *bq;
+
+ bq = bvecq_alloc_one(BVECQ_STD_SLOTS, gfp, for_writeback);
+ if (!bq)
+ return -ENOMEM;
+
+ bvecq_buffer_append(pos, bq);
+ return 0;
+}
+
+/**
+ * bvecq_append_page - Add part of a page to a buffer and advance to it
+ * @pos: The current position in the buffer
+ * @page: The page to add
+ * @offset: The offset of the page part to include
+ * @len: The length of the page part to include
+ * @gfp: The allocation flags
+ * @for_writeback: True if allocating for writeback
+ *
+ * Add part of a page to a buffer, extending the buffer if necessary. The
+ * position in the buffer is updated on return.
+ *
+ * Return: 0 or -ENOMEM on allocation failure.
+ */
+int bvecq_append_page(struct bvecq_pos *pos, struct page *page,
+ size_t offset, size_t len, gfp_t gfp, bool for_writeback)
+{
+ struct bvecq *bq = pos->bvecq;
+ int slot = pos->slot;
+
+ WARN_ON_ONCE(slot != bq->nr_slots);
+
+ if (slot >= bq->max_slots) {
+ if (bvecq_buffer_make_space(pos, gfp, for_writeback) < 0)
+ return -ENOMEM;
+ bq = pos->bvecq;
+ slot = pos->slot;
+ }
+
+ bvec_set_page(&bq->bv[slot++], page, len, offset);
+ bvecq_filled_to(bq, slot);
+ pos->slot = slot;
+ return 0;
+}
+EXPORT_SYMBOL(bvecq_append_page);
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 5680e37fc70d..8db2c82004a4 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -172,6 +172,7 @@ extern atomic_t netfs_n_wh_retry_write_subreq;
extern atomic_t netfs_n_wb_lock_skip;
extern atomic_t netfs_n_wb_lock_wait;
extern atomic_t netfs_n_folioq;
+extern atomic_t netfs_n_bvecq;
int netfs_stats_show(struct seq_file *m, void *v);
diff --git a/fs/netfs/stats.c b/fs/netfs/stats.c
index ab6b916addc4..84c2a4bcc762 100644
--- a/fs/netfs/stats.c
+++ b/fs/netfs/stats.c
@@ -48,6 +48,7 @@ atomic_t netfs_n_wh_retry_write_subreq;
atomic_t netfs_n_wb_lock_skip;
atomic_t netfs_n_wb_lock_wait;
atomic_t netfs_n_folioq;
+atomic_t netfs_n_bvecq;
int netfs_stats_show(struct seq_file *m, void *v)
{
@@ -90,9 +91,10 @@ int netfs_stats_show(struct seq_file *m, void *v)
atomic_read(&netfs_n_rh_retry_read_subreq),
atomic_read(&netfs_n_wh_retry_write_req),
atomic_read(&netfs_n_wh_retry_write_subreq));
- seq_printf(m, "Objs : rr=%u sr=%u foq=%u wsc=%u\n",
+ seq_printf(m, "Objs : rr=%u sr=%u bq=%u foq=%u wsc=%u\n",
atomic_read(&netfs_n_rh_rreq),
atomic_read(&netfs_n_rh_sreq),
+ atomic_read(&netfs_n_bvecq),
atomic_read(&netfs_n_folioq),
atomic_read(&netfs_n_wh_wstream_conflict));
seq_printf(m, "WbLock : skip=%u wait=%u\n",
diff --git a/include/linux/bvecq.h b/include/linux/bvecq.h
index 77fd07852c33..22fc7995f3ee 100644
--- a/include/linux/bvecq.h
+++ b/include/linux/bvecq.h
@@ -47,4 +47,312 @@ struct bvecq {
struct bio_vec __bv[]; /* Default array (if ->inline_bv) */
};
+#if BITS_PER_LONG == 64
+/* Number of slots in __bv[] for a bvecq in a 512-byte kmalloc block. */
+#define BVECQ_STD_SLOTS 29 /* 2 words/slot; 32 slots; bvecq is 6 words (3 slots) */
+#elif BITS_PER_LONG == 32
+/* Number of slots in __bv[] for a bvecq in a 256-byte kmalloc block. */
+#define BVECQ_STD_SLOTS 18 /* 3 words/slot; 21 slots; bvecq is 9 words (3 slots) */
+#else
+#error BVECQ_STD_SLOTS undetermined
+#endif
+
+/*
+ * Position in a bio_vec queue. The bvecq holds a ref on the queue segment it
+ * points to.
+ */
+struct bvecq_pos {
+ struct bvecq *bvecq; /* The first bvecq */
+ unsigned int offset; /* The offset within the starting slot */
+ u16 slot; /* The starting slot */
+};
+
+void bvecq_dump(const struct bvecq *bq);
+struct bvecq *bvecq_alloc_one(size_t nr_slots, gfp_t gfp, bool for_writeback);
+struct bvecq *bvecq_alloc_chain(size_t nr_slots, gfp_t gfp, bool for_writeback);
+struct bvecq *bvecq_alloc_buffer2(size_t size, unsigned int pre_slots, gfp_t gfp,
+ bool for_writeback);
+void bvecq_put(struct bvecq *bq);
+int bvecq_expand_buffer(struct bvecq **_buffer, size_t *_cur_size, size_t size, gfp_t gfp);
+int bvecq_shorten_buffer(struct bvecq *bq, unsigned int slot, size_t size);
+int bvecq_buffer_init(struct bvecq_pos *pos, gfp_t gfp, bool for_writeback);
+void bvecq_buffer_append(struct bvecq_pos *pos, struct bvecq *bq);
+void bvecq_pos_advance(struct bvecq_pos *pos, size_t amount);
+ssize_t bvecq_zero(struct bvecq_pos *pos, size_t amount);
+size_t bvecq_slice(struct bvecq_pos *pos, size_t max_size,
+ unsigned int max_slots, unsigned int *_nr_slots);
+ssize_t bvecq_extract(struct bvecq_pos *pos, size_t max_size, unsigned int max_slots,
+ struct bvecq **to, bool for_writeback);
+ssize_t bvecq_load_from_ra(struct bvecq_pos *pos, struct readahead_control *ractl);
+int bvecq_append_page(struct bvecq_pos *pos, struct page *page,
+ size_t offset, size_t len, gfp_t gfp, bool for_writeback);
+
+/**
+ * bvecq_alloc_buffer - Allocate a bvecq chain and populate with buffers
+ * @size: Target size of the buffer (can be 0 for an empty buffer)
+ * @gfp: The allocation constraints.
+ * @for_writeback: True if allocating for writeback
+ *
+ * Wrapper around %bvecq_alloc_buffer2().
+ */
+static inline struct bvecq *bvecq_alloc_buffer(size_t size, gfp_t gfp, bool for_writeback)
+{
+ return bvecq_alloc_buffer2(size, 0, gfp, for_writeback);
+}
+
+/**
+ * bvecq_get - Get a ref on a bvecq
+ * @bq: The bvecq to get a ref on
+ */
+static inline struct bvecq *bvecq_get(struct bvecq *bq)
+{
+ refcount_inc(&bq->ref);
+ return bq;
+}
+
+/**
+ * bvecq_is_full - Determine if a bvecq is full
+ * @bvecq: The object to query
+ *
+ * Return: true if full; false if not.
+ */
+static inline bool bvecq_is_full(const struct bvecq *bvecq)
+{
+ return bvecq->nr_slots >= bvecq->max_slots;
+}
+
+/**
+ * bvecq_filled_to - Release filled slots with release barrier
+ * @bvecq: The object modified
+ * @to: The latest slot filled + 1
+ */
+static inline void bvecq_filled_to(struct bvecq *bvecq, unsigned int to)
+{
+ /* Set the slot counter after filling the slot */
+ smp_store_release(&bvecq->nr_slots, to);
+}
+
+/**
+ * bvecq_nr_slots_acquire - Get the number of filled slots with acquire barrier
+ * @bvecq: The object to query
+ *
+ * Return: The number of filled slots
+ */
+static inline unsigned int bvecq_nr_slots_acquire(const struct bvecq *bvecq)
+{
+ /* Read the slot counter before looking at the slot */
+ return smp_load_acquire(&bvecq->nr_slots);
+}
+
+/**
+ * bvecq_acquire_slot - Determine if a slot is valid with acquire barrier
+ * @bvecq: The object to query
+ * @slot: The next slot
+ *
+ * Return: true if valid; false if might not be valid
+ */
+static inline bool bvecq_acquire_slot(const struct bvecq *bvecq, unsigned int slot)
+{
+ /* Read the slot counter before looking at the slot */
+ return slot < bvecq_nr_slots_acquire(bvecq);
+}
+
+/**
+ * bvecq_append - Get the next bvecq with appropriate barrier
+ * @to: The bvecq to append to
+ * @add: The bvecq to append
+ *
+ * Attach a new bvecq to a chain using an appropriate barrier to protect the
+ * write.
+ *
+ * [!] Note that this function transfers the caller's ref to the chain.
+ */
+static inline void bvecq_append(struct bvecq *to, struct bvecq *add)
+{
+ add->prev = to;
+
+ /* Make sure the initialisation is stored before the next pointer. */
+ smp_store_release(&to->next, add);
+}
+
+/**
+ * bvecq_next - Get the next bvecq with appropriate barrier
+ * @bq: The bvecq to start from
+ *
+ * Return the next bvecq in a chain, using an appropriate barrier to protect
+ * the access.
+ */
+static inline struct bvecq *bvecq_next(const struct bvecq *bq)
+{
+ /* Read the contents of the next node after the pointer to it. */
+ return smp_load_acquire(&bq->next);
+}
+
+/**
+ * bvecq_pos_set - Set one position to be the same as another
+ * @pos: The position object to set
+ * @at: The source position.
+ *
+ * Set @pos to have the same position as @at. This may take a ref on the
+ * bvecq pointed to.
+ */
+static inline void bvecq_pos_set(struct bvecq_pos *pos, const struct bvecq_pos *at)
+{
+ *pos = *at;
+ bvecq_get(pos->bvecq);
+}
+
+/**
+ * bvecq_pos_unset - Unset a position
+ * @pos: The position object to unset
+ *
+ * Unset @pos. This does any needed ref cleanup.
+ */
+static inline void bvecq_pos_unset(struct bvecq_pos *pos)
+{
+ bvecq_put(pos->bvecq);
+ pos->bvecq = NULL;
+ pos->slot = 0;
+ pos->offset = 0;
+}
+
+/**
+ * bvecq_pos_transfer - Transfer one position to another, clearing the first
+ * @pos: The position object to set
+ * @from: The source position to clear.
+ *
+ * Set @pos to have the same position as @from and then clear @from. This may
+ * transfer a ref on the bvecq pointed to.
+ */
+static inline void bvecq_pos_transfer(struct bvecq_pos *pos, struct bvecq_pos *from)
+{
+ *pos = *from;
+ from->bvecq = NULL;
+ from->slot = 0;
+ from->offset = 0;
+}
+
+/**
+ * bvecq_pos_move - Update a position to a new bvecq
+ * @pos: The position object to update.
+ * @to: The new bvecq to point at.
+ *
+ * Update @pos to point to @to if it doesn't already do so. This may
+ * manipulate refs on the bvecqs pointed to.
+ */
+static inline void bvecq_pos_move(struct bvecq_pos *pos, struct bvecq *to)
+{
+ struct bvecq *old = pos->bvecq;
+
+ if (old != to) {
+ pos->bvecq = bvecq_get(to);
+ bvecq_put(old);
+ }
+}
+
+/**
+ * bvecq_pos_nudge - Nudge a position onto the next segment if current used up
+ * @pos: The position object to nudge.
+ *
+ * Update @pos to point to the next segment in the chain if we've used up the
+ * current segment. This may manipulate refs on the bvecqs pointed to.
+ *
+ * Return: true if found a new segment, false if hit the end.
+ */
+static inline bool bvecq_pos_nudge(struct bvecq_pos *pos)
+{
+ struct bvecq *bq = pos->bvecq;
+
+ for (;;) {
+ if (!bvecq_acquire_slot(bq, pos->slot)) {
+ bq = bvecq_next(bq);
+ if (!bq)
+ return false;
+ if (bvecq_acquire_slot(bq, pos->slot))
+ continue; /* More slots got added. */
+ bvecq_pos_move(pos, bq);
+ pos->slot = 0;
+ pos->offset = 0;
+ continue;
+ }
+ if (pos->offset >= bq->bv[pos->slot].bv_len) {
+ pos->slot++;
+ pos->offset = 0;
+ continue;
+ }
+ return true;
+ }
+}
+
+/**
+ * bvecq_pos_step - Step a position to the next slot if possible
+ * @pos: The position object to step.
+ *
+ * Update @pos to point to the next slot in the queue if not at the end. This
+ * may manipulate refs on the bvecqs pointed to.
+ *
+ * Return: true if successful, false if was at the end.
+ */
+static inline bool bvecq_pos_step(struct bvecq_pos *pos)
+{
+ struct bvecq *bq = pos->bvecq, *next;
+
+ pos->slot++;
+ pos->offset = 0;
+ if (bvecq_acquire_slot(bq, pos->slot))
+ return true;
+ next = bvecq_next(bq);
+ if (!next)
+ return false;
+ if (bvecq_acquire_slot(bq, pos->slot))
+ return true;
+ bvecq_pos_move(pos, next);
+ pos->slot = 0;
+ return true;
+}
+
+/**
+ * bvecq_delete_spent - Delete the bvecq at the front if possible
+ * @pos: The position object to update.
+ *
+ * Delete the used up bvecq at the front of the queue that @pos points to if it
+ * is not the last node in the queue; if it is the last node in the queue, it
+ * is kept so that the queue doesn't become detached from the other end. This
+ * may manipulate refs on the bvecqs pointed to. It is also possible that the
+ * producer will fill more slots in the current bvecq.
+ *
+ * Also, we have to be very careful: the consumer can catch the producer, which
+ * could lead to us having nothing left in the queue, causing the front and
+ * back pointers to end up on different tracks. To avoid this, we must always
+ * keep at least one segment in the queue.
+ *
+ * The caller must reload from @pos after calling this.
+ *
+ * Return: true if there's more available; false if not.
+ */
+static inline bool bvecq_delete_spent(struct bvecq_pos *pos)
+{
+ struct bvecq *spent = pos->bvecq;
+ struct bvecq *next;
+ unsigned int slot = pos->slot;
+
+again:
+ /* Read the contents of the queue node after the pointer to it. */
+ next = bvecq_next(spent);
+ if (!next)
+ return false; /* Nothing more to consume at the moment. */
+ if (slot < bvecq_nr_slots_acquire(spent))
+ return true; /* The producer added more. */
+ next->prev = NULL;
+ bvecq_pos_move(pos, next);
+ pos->slot = 0;
+ pos->offset = 0;
+ if (!bvecq_acquire_slot(next, 0)) {
+ spent = next;
+ slot = 0;
+ goto again;
+ }
+ return true;
+}
+
#endif /* _LINUX_BVECQ_H */
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 16cc0858c613..c0eb80e49f5c 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -17,6 +17,7 @@
#include <linux/workqueue.h>
#include <linux/fs.h>
#include <linux/pagemap.h>
+#include <linux/bvecq.h>
#include <linux/uio.h>
#include <linux/rolling_buffer.h>
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 0202ba4ccbd0..111c02817bea 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -809,6 +809,30 @@ TRACE_EVENT(netfs_read_progress_at,
__entry->rreq, __entry->cleaned_to, __entry->progress_at)
);
+TRACE_EVENT(netfs_bv_slot,
+ TP_PROTO(const struct bvecq *bq, int slot),
+
+ TP_ARGS(bq, slot),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, pfn)
+ __field(unsigned int, offset)
+ __field(unsigned int, len)
+ __field(unsigned int, slot)
+ ),
+
+ TP_fast_assign(
+ __entry->slot = slot;
+ __entry->pfn = page_to_pfn(bq->bv[slot].bv_page);
+ __entry->offset = bq->bv[slot].bv_offset;
+ __entry->len = bq->bv[slot].bv_len;
+ ),
+
+ TP_printk("bq[%x] p=%lx %x-%x",
+ __entry->slot,
+ __entry->pfn, __entry->offset, __entry->offset + __entry->len)
+ );
+
#undef EM
#undef E_
#endif /* _TRACE_NETFS_H */
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 12/35] netfs: Make mempool available for bvecq
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (10 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 11/35] netfs: Add some tools for managing bvecq chains David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 13/35] netfs: Add a function to extract from an iter into a bvecq David Howells
` (23 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Make a mempool available for allocating bvecq structs. Use it
automatically if anything other than GFP_KERNEL (with GFP_ZONEMASK flags
masked off) is specified. Reading from a file can use GFP_KERNEL as the
failure mode is straightforward and the same for DIO reads and writes.
When it comes to writeback, however, the writeback_iter() API does not
permit temporary failure, including ENOMEM, if WB_SYNC_ALL is set and the
caller must process all folios to completion. (I'm not sure if EINTR
constitutes an acceptable failure).
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/bvecq.c | 39 +++++++++++++++++++++++++++++++++------
fs/netfs/internal.h | 1 +
fs/netfs/main.c | 7 +++++++
include/linux/bvecq.h | 3 +++
4 files changed, 44 insertions(+), 6 deletions(-)
diff --git a/fs/netfs/bvecq.c b/fs/netfs/bvecq.c
index 6905c84ea351..7b0aedafeda9 100644
--- a/fs/netfs/bvecq.c
+++ b/fs/netfs/bvecq.c
@@ -44,8 +44,9 @@ EXPORT_SYMBOL(bvecq_dump);
*
* Allocate a single bvecq node and initialise the header. A number of inline
* slots are also allocated, rounded up to fit after the header in a power-of-2
- * slab object of up to 512 bytes (up to 29 slots on a 64-bit cpu). The slot
- * array is not initialised.
+ * slab object of up to 512 bytes (up to 29 slots on a 64-bit cpu). The caller
+ * should be aware that the number of slots allocated may be more or less than
+ * the number requested. The slot array is not initialised.
*
* Return: The node pointer or NULL on allocation failure.
*/
@@ -56,16 +57,39 @@ struct bvecq *bvecq_alloc_one(size_t nr_slots, gfp_t gfp, bool for_writeback)
const size_t max_slots = (max_size - sizeof(*bq)) / sizeof(bq->__bv[0]);
size_t part = min(nr_slots, max_slots);
size_t size = roundup_pow_of_two(struct_size(bq, __bv, part));
+ bool from_pool = false;
- bq = kmalloc(size, gfp & ~GFP_ZONEMASK);
- if (!bq)
- return bq;
+ gfp &= ~(GFP_ZONEMASK | __GFP_THISNODE);
+
+ if (for_writeback) {
+ if (size != BVECQ_STD_SIZE) {
+ gfp_t gfp_temp = gfp;
+ gfp_temp |= __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN;
+ gfp_temp &= ~(__GFP_DIRECT_RECLAIM | __GFP_IO);
+ bq = kmalloc(size, gfp_temp);
+ if (bq)
+ goto success;
+ }
+
+ bq = mempool_alloc(&netfs_bvecq_pool, gfp);
+ if (!bq)
+ return bq;
+ from_pool = true;
+ size = BVECQ_STD_SIZE;
+ } else {
+ bq = kmalloc(size, gfp);
+ if (!bq)
+ return bq;
+ }
+
+success:
*bq = (struct bvecq) {
.ref = REFCOUNT_INIT(1),
.bv = bq->__bv,
.inline_bv = true,
.max_slots = (size - sizeof(*bq)) / sizeof(bq->__bv[0]),
+ .from_pool = from_pool,
};
netfs_stat(&netfs_n_bvecq);
return bq;
@@ -245,7 +269,10 @@ void bvecq_put(struct bvecq *bq)
bvecq_free_slot(bq, slot);
next = bq->next;
netfs_stat_d(&netfs_n_bvecq);
- kfree(bq);
+ if (bq->from_pool)
+ mempool_free(bq, &netfs_bvecq_pool);
+ else
+ kfree(bq);
}
}
EXPORT_SYMBOL(bvecq_put);
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 8db2c82004a4..431ec77b1305 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -43,6 +43,7 @@ extern struct list_head netfs_io_requests;
extern spinlock_t netfs_proc_lock;
extern mempool_t netfs_request_pool;
extern mempool_t netfs_subrequest_pool;
+extern mempool_t netfs_bvecq_pool;
extern mempool_t netfs_folioq_pool;
#ifdef CONFIG_PROC_FS
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index 927badf3989d..9f72e5054aff 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -28,6 +28,7 @@ static struct kmem_cache *netfs_request_slab;
static struct kmem_cache *netfs_subrequest_slab;
mempool_t netfs_request_pool;
mempool_t netfs_subrequest_pool;
+mempool_t netfs_bvecq_pool;
mempool_t netfs_folioq_pool;
#ifdef CONFIG_PROC_FS
@@ -112,6 +113,9 @@ static int __init netfs_init(void)
if (mempool_init_kmalloc_pool(&netfs_folioq_pool, 100, sizeof(struct folio_queue)) < 0)
goto error_folioq_pool;
+ if (mempool_init_kmalloc_pool(&netfs_bvecq_pool, 100, BVECQ_STD_SIZE) < 0)
+ goto error_bvecq_pool;
+
netfs_request_slab = kmem_cache_create("netfs_request",
sizeof(struct netfs_io_request), 0,
SLAB_HWCACHE_ALIGN | SLAB_ACCOUNT,
@@ -164,6 +168,8 @@ static int __init netfs_init(void)
error_reqpool:
kmem_cache_destroy(netfs_request_slab);
error_req:
+ mempool_exit(&netfs_bvecq_pool);
+error_bvecq_pool:
mempool_exit(&netfs_folioq_pool);
error_folioq_pool:
return ret;
@@ -178,6 +184,7 @@ static void __exit netfs_exit(void)
kmem_cache_destroy(netfs_subrequest_slab);
mempool_exit(&netfs_request_pool);
kmem_cache_destroy(netfs_request_slab);
+ mempool_exit(&netfs_bvecq_pool);
mempool_exit(&netfs_folioq_pool);
}
module_exit(netfs_exit);
diff --git a/include/linux/bvecq.h b/include/linux/bvecq.h
index 22fc7995f3ee..8adfdd43b865 100644
--- a/include/linux/bvecq.h
+++ b/include/linux/bvecq.h
@@ -43,15 +43,18 @@ struct bvecq {
u16 max_slots; /* Number of elements allocated in bv[] */
enum bvecq_mem mem_type:3; /* What sort of memory and how to free it */
bool inline_bv:1; /* T if __bv[] is being used */
+ bool from_pool:1; /* T if bvecq from mempool */
struct bio_vec *bv; /* Pointer to array of page fragments */
struct bio_vec __bv[]; /* Default array (if ->inline_bv) */
};
#if BITS_PER_LONG == 64
/* Number of slots in __bv[] for a bvecq in a 512-byte kmalloc block. */
+#define BVECQ_STD_SIZE 512
#define BVECQ_STD_SLOTS 29 /* 2 words/slot; 32 slots; bvecq is 6 words (3 slots) */
#elif BITS_PER_LONG == 32
/* Number of slots in __bv[] for a bvecq in a 256-byte kmalloc block. */
+#define BVECQ_STD_SIZE 256
#define BVECQ_STD_SLOTS 18 /* 3 words/slot; 21 slots; bvecq is 9 words (3 slots) */
#else
#error BVECQ_STD_SLOTS undetermined
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 13/35] netfs: Add a function to extract from an iter into a bvecq
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (11 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 12/35] netfs: Make mempool available for bvecq David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 14/35] afs: Use a bvecq to hold dir content rather than folioq David Howells
` (22 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Add a function to extract a slice of data from an iterator of any type into
a bvec queue chain.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/iterator.c | 143 ++++++++++++++++++++++++++++++++++++++++++
include/linux/netfs.h | 3 +
2 files changed, 146 insertions(+)
diff --git a/fs/netfs/iterator.c b/fs/netfs/iterator.c
index eb1efb17f53a..e66c589ca3b0 100644
--- a/fs/netfs/iterator.c
+++ b/fs/netfs/iterator.c
@@ -13,6 +13,149 @@
#include <linux/netfs.h>
#include "internal.h"
+/**
+ * netfs_extract_iter - Extract virtually contiguous pages from an iterator into a bvecq
+ * @orig: The original iterator
+ * @max_len: Maximum number of bytes to extract
+ * @max_pages: Maximum number of pages to extract
+ * @_bvecq_head: Where to cache the bvec queue
+ * @extraction_flags: Flags to qualify the request
+ * @gfp: Allocation mode for bvecq structs.
+ *
+ * Extract virtually contiguous page fragments from the source iterator up to
+ * the given maxima and build bvec queue that refers to all of those bits.
+ * This allows the original iterator to disposed of.
+ *
+ * @extraction_flags can have ITER_ALLOW_P2PDMA set to request peer-to-peer DMA be
+ * allowed on the pages extracted.
+ *
+ * On success or partial success, the amount of data in the bvec is returned,
+ * the original iterator will have been advanced by the amount extracted.
+ *
+ * If an error occurs and no pages are extracted, an error will be returned and
+ * any allocated bvecq will be freed. If there is no data to be extracted (or
+ * @max_len or @max_pages are zero), a single empty bvecq will be returned.
+ *
+ * The bvecq segments are marked with indications on how to get clean up the
+ * extracted fragments.
+ */
+ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages,
+ struct bvecq **_bvecq_head,
+ iov_iter_extraction_t extraction_flags, gfp_t gfp)
+{
+ struct bvecq *bq_tail = NULL, *bq;
+ ssize_t ret = 0;
+ size_t extracted = 0;
+
+ _enter("{%u,%zx},%zx", orig->iter_type, orig->count, max_len);
+
+ *_bvecq_head = NULL;
+ if (max_len > orig->count)
+ max_len = orig->count;
+ if (!max_len || !max_pages)
+ goto alloc_empty;
+ if (WARN_ON_ONCE(max_pages > INT_MAX))
+ max_pages = INT_MAX; /* Protect iov_iter_npages(). */
+
+ max_pages = iov_iter_npages(orig, max_pages);
+ if (!max_pages)
+ goto alloc_empty;
+
+ do {
+ bq = bvecq_alloc_one(max_pages, gfp, false);
+ if (!bq) {
+ ret = -ENOMEM;
+ break;
+ }
+ if (user_backed_iter(orig))
+ bq->mem_type = iov_iter_extract_will_pin(orig) ?
+ BVECQ_MEM_GUP : BVECQ_MEM_PAGECACHE;
+
+ if (bq_tail)
+ bvecq_append(bq_tail, bq);
+ else
+ *_bvecq_head = bq;
+ bq_tail = bq;
+
+ if (max_len == 0)
+ break;
+
+ struct bio_vec *bv = bq->bv;
+ unsigned int slot = 0;
+ do {
+ struct page **pages;
+ ssize_t got;
+ size_t offset;
+ size_t space = bq->max_slots - slot;
+ size_t bv_size = array_size(bq->max_slots, sizeof(*bv));
+ size_t pg_size = array_size(space, sizeof(*pages));
+
+ /* Put the page list at the end of the bvec list
+ * storage. bvec elements are larger than page
+ * pointers, so as long as we work 0->last, we should
+ * be fine.
+ */
+ pages = (void *)bv + bv_size - pg_size;
+
+ got = iov_iter_extract_pages(orig, &pages, max_len,
+ min(space, max_pages),
+ extraction_flags, &offset);
+ if (got < 0) {
+ ret = got;
+ goto out;
+ }
+
+ if (got == 0) {
+ pr_err("extract_pages gave nothing from %zx, %zx\n",
+ extracted, max_len);
+ ret = -EIO;
+ goto out;
+ }
+
+ if (WARN(got > max_len,
+ "%s: extract_pages overrun %zx > %zx bytes\n",
+ __func__, got, max_len)) {
+ ret = -EIO;
+ goto out;
+ }
+
+ extracted += got;
+ max_len -= got;
+
+ do {
+ size_t len = umin(got, PAGE_SIZE - offset);
+
+ BUG_ON(slot >= bq->max_slots);
+
+ bvec_set_page(&bq->bv[slot], *pages++, len, offset);
+ slot++;
+ max_pages--;
+ got -= len;
+ offset = 0;
+ } while (got > 0);
+
+ bvecq_filled_to(bq, slot);
+ } while (max_len > 0 && max_pages > 0 && !bvecq_is_full(bq));
+
+ } while (max_len > 0 && max_pages > 0);
+
+out:
+ if (extracted || ret == 0)
+ return extracted;
+ bvecq_put(*_bvecq_head);
+ *_bvecq_head = NULL;
+ return ret;
+
+alloc_empty:
+ bq = bvecq_alloc_one(1, gfp, false);
+ if (!bq)
+ return -ENOMEM;
+ *_bvecq_head = bq;
+ return 0;
+
+}
+EXPORT_SYMBOL_GPL(netfs_extract_iter);
+
/**
* netfs_extract_user_iter - Extract the pages from a user iterator into a bvec
* @orig: The original iterator
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index c0eb80e49f5c..14a24bb80af5 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -431,6 +431,9 @@ void netfs_get_subrequest(struct netfs_io_subrequest *subreq,
enum netfs_sreq_ref_trace what);
void netfs_put_subrequest(struct netfs_io_subrequest *subreq,
enum netfs_sreq_ref_trace what);
+ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages,
+ struct bvecq **_bvecq_head,
+ iov_iter_extraction_t extraction_flags, gfp_t gfp);
ssize_t netfs_extract_user_iter(struct iov_iter *orig, size_t orig_len,
struct iov_iter *new,
iov_iter_extraction_t extraction_flags);
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 14/35] afs: Use a bvecq to hold dir content rather than folioq
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (12 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 13/35] netfs: Add a function to extract from an iter into a bvecq David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 15/35] cifs: Use a bvecq for buffering instead of a folioq David Howells
` (21 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Use a bvecq to hold the contents of a directory rather than the folioq so
that the latter can be phased out.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Marc Dionne <marc.dionne@auristor.com>
cc: linux-afs@lists.infradead.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/afs/dir.c | 35 +++++----
fs/afs/dir_edit.c | 42 +++++------
fs/afs/dir_search.c | 33 ++++-----
fs/afs/inode.c | 2 +-
fs/afs/internal.h | 6 +-
fs/afs/symlink.c | 34 ++++-----
fs/netfs/write_issue.c | 163 ++++++-----------------------------------
7 files changed, 93 insertions(+), 222 deletions(-)
diff --git a/fs/afs/dir.c b/fs/afs/dir.c
index 6df56fe9163f..1dff50323b56 100644
--- a/fs/afs/dir.c
+++ b/fs/afs/dir.c
@@ -140,9 +140,9 @@ static void afs_dir_dump(struct afs_vnode *dvnode)
pr_warn("DIR %llx:%llx is=%llx\n",
dvnode->fid.vid, dvnode->fid.vnode, i_size);
- iov_iter_folio_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
- iterate_folioq(&iter, iov_iter_count(&iter), NULL, NULL,
- afs_dir_dump_step);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
+ iterate_bvecq(&iter, iov_iter_count(&iter), NULL, NULL,
+ afs_dir_dump_step);
}
/*
@@ -203,9 +203,9 @@ static int afs_dir_check(struct afs_vnode *dvnode)
if (unlikely(!i_size))
return 0;
- iov_iter_folio_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
- checked = iterate_folioq(&iter, iov_iter_count(&iter), dvnode, NULL,
- afs_dir_check_step);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
+ checked = iterate_bvecq(&iter, iov_iter_count(&iter), dvnode, NULL,
+ afs_dir_check_step);
if (checked != i_size) {
afs_dir_dump(dvnode);
return -EIO;
@@ -250,15 +250,14 @@ static ssize_t afs_do_read_single(struct afs_vnode *dvnode, struct file *file)
if (dvnode->directory_size < i_size) {
size_t cur_size = dvnode->directory_size;
- ret = netfs_alloc_folioq_buffer(NULL,
- &dvnode->directory, &cur_size, i_size,
- mapping_gfp_mask(dvnode->netfs.inode.i_mapping));
+ ret = bvecq_expand_buffer(&dvnode->directory, &cur_size,
+ round_up(i_size, PAGE_SIZE), GFP_KERNEL);
dvnode->directory_size = cur_size;
if (ret < 0)
return ret;
}
- iov_iter_folio_queue(&iter, ITER_DEST, dvnode->directory, 0, 0, dvnode->directory_size);
+ iov_iter_bvec_queue(&iter, ITER_DEST, dvnode->directory, 0, 0, dvnode->directory_size);
/* AFS requires us to perform the read of a directory synchronously as
* a single unit to avoid issues with the directory contents being
@@ -294,8 +293,8 @@ static ssize_t afs_read_single(struct afs_vnode *dvnode, struct file *file)
}
/*
- * Read the directory into a folio_queue buffer in one go, scrubbing the
- * previous contents. We return -ESTALE if the caller needs to call us again.
+ * Read the directory into the buffer in one go, scrubbing the previous
+ * contents. We return -ESTALE if the caller needs to call us again.
*/
ssize_t afs_read_dir(struct afs_vnode *dvnode, struct file *file)
__acquires(&dvnode->validate_lock)
@@ -483,7 +482,7 @@ static size_t afs_dir_iterate_step(void *iter_base, size_t progress, size_t len,
}
/*
- * Iterate through the directory folios.
+ * Iterate through the directory content.
*/
static int afs_dir_iterate_contents(struct inode *dir, struct dir_context *dir_ctx)
{
@@ -498,11 +497,11 @@ static int afs_dir_iterate_contents(struct inode *dir, struct dir_context *dir_c
if (i_size <= 0 || dir_ctx->pos >= i_size)
return 0;
- iov_iter_folio_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
iov_iter_advance(&iter, round_down(dir_ctx->pos, AFS_DIR_BLOCK_SIZE));
- iterate_folioq(&iter, iov_iter_count(&iter), dvnode, &ctx,
- afs_dir_iterate_step);
+ iterate_bvecq(&iter, iov_iter_count(&iter), dvnode, &ctx,
+ afs_dir_iterate_step);
if (ctx.error == -ESTALE)
afs_invalidate_dir(dvnode, afs_dir_invalid_iter_stale);
@@ -2228,8 +2227,8 @@ static int afs_dir_writepages(struct address_space *mapping,
}
if (test_bit(AFS_VNODE_DIR_VALID, &dvnode->flags)) {
- iov_iter_folio_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0,
- i_size_read(&dvnode->netfs.inode));
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0,
+ i_size_read(&dvnode->netfs.inode));
ret = netfs_writeback_single(mapping, wbc, &iter);
if (ret == 1)
ret = 0; /* Skipped write due to lock conflict. */
diff --git a/fs/afs/dir_edit.c b/fs/afs/dir_edit.c
index 3ead36a07048..01c6265e8865 100644
--- a/fs/afs/dir_edit.c
+++ b/fs/afs/dir_edit.c
@@ -110,9 +110,8 @@ static void afs_clear_contig_bits(union afs_xdr_dir_block *block,
*/
static union afs_xdr_dir_block *afs_dir_get_block(struct afs_dir_iter *iter, size_t block)
{
- struct folio_queue *fq;
struct afs_vnode *dvnode = iter->dvnode;
- struct folio *folio;
+ struct bvecq *bq;
size_t blpos = block * AFS_DIR_BLOCK_SIZE;
size_t blend = (block + 1) * AFS_DIR_BLOCK_SIZE, fpos = iter->fpos;
int ret;
@@ -120,41 +119,38 @@ static union afs_xdr_dir_block *afs_dir_get_block(struct afs_dir_iter *iter, siz
if (dvnode->directory_size < blend) {
size_t cur_size = dvnode->directory_size;
- ret = netfs_alloc_folioq_buffer(
- NULL, &dvnode->directory, &cur_size, blend,
- mapping_gfp_mask(dvnode->netfs.inode.i_mapping));
+ ret = bvecq_expand_buffer(&dvnode->directory, &cur_size, blend,
+ GFP_KERNEL);
dvnode->directory_size = cur_size;
if (ret < 0)
goto fail;
}
- fq = iter->fq;
- if (!fq)
- fq = dvnode->directory;
+ bq = iter->bq;
+ if (!bq)
+ bq = dvnode->directory;
- /* Search the folio queue for the folio containing the block... */
- for (; fq; fq = fq->next) {
- for (int s = iter->fq_slot; s < folioq_count(fq); s++) {
- size_t fsize = folioq_folio_size(fq, s);
+ /* Search the contents for the region containing the block... */
+ for (; bq; bq = bq->next) {
+ for (int s = iter->bq_slot; s < bq->nr_slots; s++) {
+ struct bio_vec *bv = &bq->bv[s];
+ size_t bsize = bv->bv_len;
- if (blend <= fpos + fsize) {
+ if (blend <= fpos + bsize) {
/* ... and then return the mapped block. */
- folio = folioq_folio(fq, s);
- if (WARN_ON_ONCE(folio_pos(folio) != fpos))
- goto fail;
- iter->fq = fq;
- iter->fq_slot = s;
+ iter->bq = bq;
+ iter->bq_slot = s;
iter->fpos = fpos;
- return kmap_local_folio(folio, blpos - fpos);
+ return bvec_kmap_partial(bv, blpos - fpos);
}
- fpos += fsize;
+ fpos += bsize;
}
- iter->fq_slot = 0;
+ iter->bq_slot = 0;
}
fail:
- iter->fq = NULL;
- iter->fq_slot = 0;
+ iter->bq = NULL;
+ iter->bq_slot = 0;
afs_invalidate_dir(dvnode, afs_dir_invalid_edit_get_block);
return NULL;
}
diff --git a/fs/afs/dir_search.c b/fs/afs/dir_search.c
index 104411c0692f..01751c680115 100644
--- a/fs/afs/dir_search.c
+++ b/fs/afs/dir_search.c
@@ -66,12 +66,11 @@ bool afs_dir_init_iter(struct afs_dir_iter *iter, const struct qstr *name)
*/
union afs_xdr_dir_block *afs_dir_find_block(struct afs_dir_iter *iter, size_t block)
{
- struct folio_queue *fq = iter->fq;
struct afs_vnode *dvnode = iter->dvnode;
- struct folio *folio;
+ struct bvecq *bq = iter->bq;
size_t blpos = block * AFS_DIR_BLOCK_SIZE;
size_t blend = (block + 1) * AFS_DIR_BLOCK_SIZE, fpos = iter->fpos;
- int slot = iter->fq_slot;
+ int slot = iter->bq_slot;
_enter("%zx,%d", block, slot);
@@ -83,36 +82,34 @@ union afs_xdr_dir_block *afs_dir_find_block(struct afs_dir_iter *iter, size_t bl
if (dvnode->directory_size < blend)
goto fail;
- if (!fq || blpos < fpos) {
- fq = dvnode->directory;
+ if (!bq || blpos < fpos) {
+ bq = dvnode->directory;
slot = 0;
fpos = 0;
}
/* Search the folio queue for the folio containing the block... */
- for (; fq; fq = fq->next) {
- for (; slot < folioq_count(fq); slot++) {
- size_t fsize = folioq_folio_size(fq, slot);
+ for (; bq; bq = bq->next) {
+ for (; slot < bq->nr_slots; slot++) {
+ struct bio_vec *bv = &bq->bv[slot];
+ size_t bsize = bv->bv_len;
- if (blend <= fpos + fsize) {
+ if (blend <= fpos + bsize) {
/* ... and then return the mapped block. */
- folio = folioq_folio(fq, slot);
- if (WARN_ON_ONCE(folio_pos(folio) != fpos))
- goto fail;
- iter->fq = fq;
- iter->fq_slot = slot;
+ iter->bq = bq;
+ iter->bq_slot = slot;
iter->fpos = fpos;
- iter->block = kmap_local_folio(folio, blpos - fpos);
+ iter->block = bvec_kmap_partial(bv, blpos - fpos);
return iter->block;
}
- fpos += fsize;
+ fpos += bsize;
}
slot = 0;
}
fail:
- iter->fq = NULL;
- iter->fq_slot = 0;
+ iter->bq = NULL;
+ iter->bq_slot = 0;
afs_invalidate_dir(dvnode, afs_dir_invalid_edit_get_block);
return NULL;
}
diff --git a/fs/afs/inode.c b/fs/afs/inode.c
index 14f39a9bea6c..634fbf8eb212 100644
--- a/fs/afs/inode.c
+++ b/fs/afs/inode.c
@@ -683,7 +683,7 @@ void afs_evict_inode(struct inode *inode)
flush_delayed_work(&vnode->lock_work);
netfs_wait_for_outstanding_io(inode);
truncate_inode_pages_final(&inode->i_data);
- netfs_free_folioq_buffer(vnode->directory);
+ bvecq_put(vnode->directory);
if (vnode->symlink)
afs_evict_symlink(vnode);
diff --git a/fs/afs/internal.h b/fs/afs/internal.h
index 290873bac89b..556e7e7777f2 100644
--- a/fs/afs/internal.h
+++ b/fs/afs/internal.h
@@ -710,7 +710,7 @@ struct afs_vnode {
#define AFS_VNODE_MODIFYING 10 /* Set if we're performing a modification op */
#define AFS_VNODE_DIR_READ 11 /* Set if we've read a dir's contents */
- struct folio_queue *directory; /* Directory contents */
+ struct bvecq *directory; /* Directory contents */
struct afs_symlink __rcu *symlink; /* Symlink content */
struct list_head wb_keys; /* List of keys available for writeback */
struct list_head pending_locks; /* locks waiting to be granted */
@@ -991,9 +991,9 @@ static inline void afs_invalidate_cache(struct afs_vnode *vnode, unsigned int fl
struct afs_dir_iter {
struct afs_vnode *dvnode;
union afs_xdr_dir_block *block;
- struct folio_queue *fq;
+ struct bvecq *bq;
unsigned int fpos;
- int fq_slot;
+ int bq_slot;
unsigned int loop_check;
u8 nr_slots;
u8 bucket;
diff --git a/fs/afs/symlink.c b/fs/afs/symlink.c
index 16b4823cb7b7..7fd4f985c441 100644
--- a/fs/afs/symlink.c
+++ b/fs/afs/symlink.c
@@ -56,7 +56,6 @@ void afs_evict_symlink(struct afs_vnode *vnode)
void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op)
{
struct afs_symlink *symlink = op->create.symlink;
- size_t dsize = 0;
size_t size = strlen(symlink->content) + 1;
char *p;
@@ -66,13 +65,15 @@ void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op)
if (!fscache_cookie_enabled(netfs_i_cookie(&vnode->netfs)))
return;
- if (netfs_alloc_folioq_buffer(NULL, &vnode->directory, &dsize, size,
- mapping_gfp_mask(vnode->netfs.inode.i_mapping)) < 0)
+ vnode->directory = bvecq_alloc_buffer(PAGE_SIZE, GFP_KERNEL, false);
+ if (!vnode->directory)
return;
- vnode->directory_size = dsize;
- p = kmap_local_folio(folioq_folio(vnode->directory, 0), 0);
+ vnode->directory_size = size;
+ p = bvec_kmap_partial(&vnode->directory->bv[0], 0);
memcpy(p, symlink->content, size);
+ if (size < PAGE_SIZE)
+ memset(p + size, 0, PAGE_SIZE - size);
kunmap_local(p);
netfs_single_mark_inode_dirty(&vnode->netfs.inode);
}
@@ -94,17 +95,12 @@ static ssize_t afs_do_read_symlink(struct afs_vnode *vnode)
}
if (!vnode->directory) {
- size_t cur_size = 0;
-
- ret = netfs_alloc_folioq_buffer(NULL,
- &vnode->directory, &cur_size, PAGE_SIZE,
- mapping_gfp_mask(vnode->netfs.inode.i_mapping));
- vnode->directory_size = PAGE_SIZE - 1;
- if (ret < 0)
- return ret;
+ vnode->directory = bvecq_alloc_buffer(PAGE_SIZE, GFP_KERNEL, false);
+ if (!vnode->directory)
+ return -ENOMEM;
}
- iov_iter_folio_queue(&iter, ITER_DEST, vnode->directory, 0, 0, PAGE_SIZE);
+ iov_iter_bvec_queue(&iter, ITER_DEST, vnode->directory, 0, 0, PAGE_SIZE);
/* AFS requires us to perform the read of a symlink as a single unit to
* avoid issues with the content being changed between reads.
@@ -127,7 +123,7 @@ static ssize_t afs_do_read_symlink(struct afs_vnode *vnode)
refcount_set(&symlink->ref, 1);
symlink->content[i_size] = 0;
- const char *s = kmap_local_folio(folioq_folio(vnode->directory, 0), 0);
+ const char *s = bvec_kmap_partial(&vnode->directory->bv[0], 0);
memcpy(symlink->content, s, i_size);
kunmap_local(s);
@@ -136,7 +132,7 @@ static ssize_t afs_do_read_symlink(struct afs_vnode *vnode)
}
if (!fscache_cookie_enabled(netfs_i_cookie(&vnode->netfs))) {
- netfs_free_folioq_buffer(vnode->directory);
+ bvecq_put(vnode->directory);
vnode->directory = NULL;
vnode->directory_size = 0;
}
@@ -249,14 +245,14 @@ int afs_symlink_writepages(struct address_space *mapping,
if (vnode->directory &&
atomic64_read(&vnode->cb_expires_at) != AFS_NO_CB_PROMISE) {
- iov_iter_folio_queue(&iter, ITER_SOURCE, vnode->directory, 0, 0,
- i_size_read(&vnode->netfs.inode));
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, vnode->directory, 0, 0,
+ i_size_read(&vnode->netfs.inode));
ret = netfs_writeback_single(mapping, wbc, &iter);
}
if (ret == 0) {
netfs_wb_begin(&vnode->netfs, false);
- netfs_free_folioq_buffer(vnode->directory);
+ bvecq_put(vnode->directory);
vnode->directory = NULL;
vnode->directory_size = 0;
netfs_wb_end(&vnode->netfs);
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index 1a1f4efcaefd..d68d130d6da0 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -708,129 +708,11 @@ ssize_t netfs_end_writethrough(struct netfs_io_request *wreq, struct writeback_c
return ret;
}
-/*
- * Write some of a pending folio data back to the server and/or the cache.
- */
-static int netfs_write_folio_single(struct netfs_io_request *wreq,
- struct folio *folio)
-{
- struct netfs_io_stream *upload = &wreq->io_streams[0];
- struct netfs_io_stream *cache = &wreq->io_streams[1];
- struct netfs_io_stream *stream;
- size_t iter_off = 0;
- size_t fsize = folio_size(folio), flen;
- uoff_t fpos = folio_pos(folio);
- ssize_t ret;
- bool to_eof = false;
- bool no_debug = false;
-
- _enter("");
-
- flen = folio_size(folio);
- if (flen > wreq->i_size - fpos) {
- flen = wreq->i_size - fpos;
- folio_zero_segment(folio, flen, fsize);
- to_eof = true;
- } else if (flen == wreq->i_size - fpos) {
- to_eof = true;
- }
-
- _debug("folio %zx/%zx", flen, fsize);
-
- if (!upload->avail && !cache->avail) {
- trace_netfs_folio(folio, netfs_folio_trace_cancel_store);
- return 0;
- }
-
- if (!upload->construct)
- trace_netfs_folio(folio, netfs_folio_trace_store);
- else
- trace_netfs_folio(folio, netfs_folio_trace_store_plus);
-
- /* Attach the folio to the rolling buffer. */
- folio_get(folio);
- ret = rolling_buffer_append(&wreq->buffer, folio, NETFS_ROLLBUF_PUT_MARK, wreq->gfp);
- if (ret < 0) {
- folio_put(folio);
- return ret;
- }
-
- /* Move the submission point forward to allow for write-streaming data
- * not starting at the front of the page. We don't do write-streaming
- * with the cache as the cache requires DIO alignment.
- *
- * Also skip uploading for data that's been read and just needs copying
- * to the cache.
- */
- for (int s = 0; s < NR_IO_STREAMS; s++) {
- stream = &wreq->io_streams[s];
- stream->submit_off = 0;
- stream->submit_len = flen;
- if (!stream->avail) {
- stream->submit_off = UINT_MAX;
- stream->submit_len = 0;
- }
- }
-
- /* Attach the folio to one or more subrequests. For a big folio, we
- * could end up with thousands of subrequests if the wsize is small -
- * but we might need to wait during the creation of subrequests for
- * network resources (eg. SMB credits).
- */
- for (;;) {
- ssize_t part;
- size_t lowest_off = ULONG_MAX;
- int choose_s = -1;
-
- /* Always add to the lowest-submitted stream first. */
- for (int s = 0; s < NR_IO_STREAMS; s++) {
- stream = &wreq->io_streams[s];
- if (stream->submit_len > 0 &&
- stream->submit_off < lowest_off) {
- lowest_off = stream->submit_off;
- choose_s = s;
- }
- }
-
- if (choose_s < 0)
- break;
- stream = &wreq->io_streams[choose_s];
-
- /* Advance the iterator(s). */
- if (stream->submit_off > iter_off) {
- rolling_buffer_advance(&wreq->buffer, stream->submit_off - iter_off);
- iter_off = stream->submit_off;
- }
-
- atomic64_set(&wreq->issued_to, fpos + stream->submit_off);
- stream->submit_extendable_to = fsize - stream->submit_off;
- part = netfs_advance_write(wreq, stream, fpos + stream->submit_off,
- stream->submit_len, to_eof);
- stream->submit_off += part;
- if (part > stream->submit_len)
- stream->submit_len = 0;
- else
- stream->submit_len -= part;
- if (part > 0)
- no_debug = true;
- }
-
- wreq->buffer.iter.iov_offset = 0;
- if (fsize > iter_off)
- rolling_buffer_advance(&wreq->buffer, fsize - iter_off);
- atomic64_set(&wreq->issued_to, fpos + fsize);
-
- if (!no_debug)
- kdebug("R=%x: No submit", wreq->debug_id);
- _leave(" = 0");
- return 0;
-}
-
/**
* netfs_writeback_single - Write back a monolithic payload
* @mapping: The mapping to write from
* @wbc: Hints from the VM
- * @iter: Data to write, must be ITER_FOLIOQ.
+ * @iter: Data to write.
*
* Write a monolithic, non-pagecache object back to the server and/or
* the cache.
@@ -844,13 +726,8 @@ int netfs_writeback_single(struct address_space *mapping,
{
struct netfs_io_request *wreq;
struct netfs_inode *ictx = netfs_inode(mapping->host);
- struct folio_queue *fq;
- size_t size = iov_iter_count(iter);
int ret;
- if (WARN_ON_ONCE(!iov_iter_is_folioq(iter)))
- return -EIO;
-
if (!netfs_wb_begin(ictx, wbc->sync_mode == WB_SYNC_NONE)) {
/* The VFS will have undirtied the inode. */
netfs_single_mark_inode_dirty(&ictx->inode);
@@ -863,37 +740,43 @@ int netfs_writeback_single(struct address_space *mapping,
goto couldnt_start;
}
+ wreq->buffer.iter = *iter;
+ wreq->len = iov_iter_count(iter);
+
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags);
trace_netfs_write(wreq, netfs_write_trace_writeback_single);
netfs_stat(&netfs_n_wh_writepages);
- if (__test_and_set_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags))
+ if (test_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags))
wreq->netfs_ops->begin_writeback(wreq);
- for (fq = (struct folio_queue *)iter->folioq; fq; fq = fq->next) {
- for (int slot = 0; slot < folioq_count(fq); slot++) {
- struct folio *folio = folioq_folio(fq, slot);
- size_t part = umin(folioq_folio_size(fq, slot), size);
+ for (int s = 0; s < NR_IO_STREAMS; s++) {
+ struct netfs_io_subrequest *subreq;
+ struct netfs_io_stream *stream = &wreq->io_streams[s];
- _debug("wbiter %lx %llx", folio->index, atomic64_read(&wreq->issued_to));
+ if (!stream->avail)
+ continue;
- ret = netfs_write_folio_single(wreq, folio);
- if (ret < 0)
- goto stop;
- size -= part;
- if (size <= 0)
- goto stop;
- }
+ netfs_prepare_write(wreq, stream, 0);
+
+ subreq = stream->construct;
+ subreq->len = wreq->len;
+ stream->submit_len = subreq->len;
+ stream->submit_extendable_to = round_up(wreq->len, PAGE_SIZE);
+
+ netfs_issue_write(wreq, stream);
}
-stop:
- for (int s = 0; s < NR_IO_STREAMS; s++)
- netfs_issue_write(wreq, &wreq->io_streams[s]);
smp_wmb(); /* Write lists before ALL_QUEUED. */
set_bit(NETFS_RREQ_ALL_QUEUED, &wreq->flags);
netfs_wake_collector(wreq);
+ /* TODO: Might want to be async here if WB_SYNC_NONE, but then need to
+ * wait before modifying.
+ */
+ ret = netfs_wait_for_write(wreq);
+
netfs_put_request(wreq, netfs_rreq_trace_put_return);
_leave(" = %d", ret);
return ret;
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 15/35] cifs: Use a bvecq for buffering instead of a folioq
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (13 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 14/35] afs: Use a bvecq to hold dir content rather than folioq David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 16/35] smbdirect: Support ITER_BVECQ in smbdirect_map_sges_from_iter() David Howells
` (20 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Use a bvecq for internal buffering for crypto purposes instead of a folioq
so that the latter can be phased out.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Namjae Jeon <linkinjeon@kernel.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: linux-cifs@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/smb/client/cifsglob.h | 2 +-
fs/smb/client/smb2ops.c | 76 +++++++++++++++++++---------------------
2 files changed, 37 insertions(+), 41 deletions(-)
diff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h
index 79e4e84f8985..c8b1c23ef494 100644
--- a/fs/smb/client/cifsglob.h
+++ b/fs/smb/client/cifsglob.h
@@ -289,7 +289,7 @@ struct smb_rqst {
struct kvec *rq_iov; /* array of kvecs */
unsigned int rq_nvec; /* number of kvecs in array */
struct iov_iter rq_iter; /* Data iterator */
- struct folio_queue *rq_buffer; /* Buffer for encryption */
+ struct bvecq *rq_buffer; /* Buffer for encryption */
};
struct mid_q_entry;
diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c
index 192649fec25d..885f0cc861ed 100644
--- a/fs/smb/client/smb2ops.c
+++ b/fs/smb/client/smb2ops.c
@@ -4683,19 +4683,18 @@ crypt_message(struct TCP_Server_Info *server, int num_rqst,
}
/*
- * Copy data from an iterator to the folios in a folio queue buffer.
+ * Copy data from an iterator to the pages in a bvec queue buffer.
*/
-static bool cifs_copy_iter_to_folioq(struct iov_iter *iter, size_t size,
- struct folio_queue *buffer)
+static bool cifs_copy_iter_to_bvecq(struct iov_iter *iter, size_t size,
+ struct bvecq *buffer)
{
for (; buffer; buffer = buffer->next) {
- for (int s = 0; s < folioq_count(buffer); s++) {
- struct folio *folio = folioq_folio(buffer, s);
- size_t part = folioq_folio_size(buffer, s);
+ for (int s = 0; s < buffer->nr_slots; s++) {
+ struct bio_vec *bv = &buffer->bv[s];
+ size_t part = umin(bv->bv_len, size);
- part = umin(part, size);
-
- if (copy_folio_from_iter(folio, 0, part, iter) != part)
+ if (copy_page_from_iter(bv->bv_page, bv->bv_offset,
+ part, iter) != part)
return false;
size -= part;
}
@@ -4707,7 +4706,7 @@ void
smb3_free_compound_rqst(int num_rqst, struct smb_rqst *rqst)
{
for (int i = 0; i < num_rqst; i++)
- netfs_free_folioq_buffer(rqst[i].rq_buffer);
+ bvecq_put(rqst[i].rq_buffer);
}
/*
@@ -4734,7 +4733,7 @@ smb3_init_transform_rq(struct TCP_Server_Info *server, int num_rqst,
for (int i = 1; i < num_rqst; i++) {
struct smb_rqst *old = &old_rq[i - 1];
struct smb_rqst *new = &new_rq[i];
- struct folio_queue *buffer = NULL;
+ struct bvecq *buffer = NULL;
size_t size = iov_iter_count(&old->rq_iter);
orig_len += smb_rqst_len(server, old);
@@ -4742,17 +4741,16 @@ smb3_init_transform_rq(struct TCP_Server_Info *server, int num_rqst,
new->rq_nvec = old->rq_nvec;
if (size > 0) {
- size_t cur_size = 0;
- rc = netfs_alloc_folioq_buffer(NULL, &buffer, &cur_size,
- size, GFP_NOFS);
- new->rq_buffer = buffer;
- if (rc < 0)
+ rc = -ENOMEM;
+ buffer = bvecq_alloc_buffer(size, GFP_NOFS, true);
+ if (!buffer)
goto err_free;
- iov_iter_folio_queue(&new->rq_iter, ITER_SOURCE,
- buffer, 0, 0, size);
+ new->rq_buffer = buffer;
+ iov_iter_bvec_queue(&new->rq_iter, ITER_SOURCE,
+ buffer, 0, 0, size);
- if (!cifs_copy_iter_to_folioq(&old->rq_iter, size, buffer)) {
+ if (!cifs_copy_iter_to_bvecq(&old->rq_iter, size, buffer)) {
rc = smb_EIO1(smb_eio_trace_tx_copy_iter_to_buf, size);
goto err_free;
}
@@ -4842,22 +4840,20 @@ decrypt_raw_data(struct TCP_Server_Info *server, char *buf,
}
static int
-cifs_copy_folioq_to_iter(struct folio_queue *folioq, size_t data_size,
- size_t skip, struct iov_iter *iter)
+cifs_copy_bvecq_to_iter(struct bvecq *bq, size_t data_size,
+ size_t skip, struct iov_iter *iter)
{
- for (; folioq; folioq = folioq->next) {
- for (int s = 0; s < folioq_count(folioq); s++) {
- struct folio *folio;
- size_t fsize, n, len;
+ for (; bq; bq = bq->next) {
+ for (int s = 0; s < bq->nr_slots; s++) {
+ struct bio_vec *bv = &bq->bv[s];
+ size_t n, len;
if (data_size == 0)
return 0;
- folio = folioq_folio(folioq, s);
- fsize = folio_size(folio);
- len = umin(fsize - skip, data_size);
+ len = umin(bv->bv_len - skip, data_size);
- n = copy_folio_to_iter(folio, skip, len, iter);
+ n = copy_page_to_iter(bv->bv_page, bv->bv_offset + skip, len, iter);
if (n != len) {
cifs_dbg(VFS, "%s: something went wrong\n", __func__);
return smb_EIO2(smb_eio_trace_rx_copy_to_iter,
@@ -4879,7 +4875,7 @@ cifs_copy_folioq_to_iter(struct folio_queue *folioq, size_t data_size,
static int
handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
- char *buf, unsigned int buf_len, struct folio_queue *buffer,
+ char *buf, unsigned int buf_len, struct bvecq *buffer,
unsigned int buffer_len, bool is_offloaded)
{
unsigned int data_offset;
@@ -4989,8 +4985,8 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
}
/* Copy the data to the output I/O iterator. */
- rdata->result = cifs_copy_folioq_to_iter(buffer, data_len,
- cur_off, &rdata->subreq.io_iter);
+ rdata->result = cifs_copy_bvecq_to_iter(buffer, data_len,
+ cur_off, &rdata->subreq.io_iter);
if (rdata->result != 0) {
if (is_offloaded)
mid->mid_state = MID_RESPONSE_MALFORMED;
@@ -5029,7 +5025,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
struct smb2_decrypt_work {
struct work_struct decrypt;
struct TCP_Server_Info *server;
- struct folio_queue *buffer;
+ struct bvecq *buffer;
char *buf;
unsigned int len;
};
@@ -5043,7 +5039,7 @@ static void smb2_decrypt_offload(struct work_struct *work)
struct mid_q_entry *mid;
struct iov_iter iter;
- iov_iter_folio_queue(&iter, ITER_DEST, dw->buffer, 0, 0, dw->len);
+ iov_iter_bvec_queue(&iter, ITER_DEST, dw->buffer, 0, 0, dw->len);
rc = decrypt_raw_data(dw->server, dw->buf, dw->server->vals->read_rsp_size,
&iter, true);
if (rc) {
@@ -5092,7 +5088,7 @@ static void smb2_decrypt_offload(struct work_struct *work)
}
free_pages:
- netfs_free_folioq_buffer(dw->buffer);
+ bvecq_put(dw->buffer);
cifs_small_buf_release(dw->buf);
kfree(dw);
}
@@ -5138,12 +5134,12 @@ receive_encrypted_read(struct TCP_Server_Info *server, struct mid_q_entry **mid,
dw->len = len;
len = round_up(dw->len, PAGE_SIZE);
- size_t cur_size = 0;
- rc = netfs_alloc_folioq_buffer(NULL, &dw->buffer, &cur_size, len, GFP_NOFS);
- if (rc < 0)
+ rc = -ENOMEM;
+ dw->buffer = bvecq_alloc_buffer(len, GFP_NOFS, false);
+ if (!dw->buffer)
goto discard_data;
- iov_iter_folio_queue(&iter, ITER_DEST, dw->buffer, 0, 0, len);
+ iov_iter_bvec_queue(&iter, ITER_DEST, dw->buffer, 0, 0, len);
/* Read the data into the buffer and clear excess bufferage. */
rc = cifs_read_iter_from_socket(server, &iter, dw->len);
@@ -5201,7 +5197,7 @@ receive_encrypted_read(struct TCP_Server_Info *server, struct mid_q_entry **mid,
}
free_pages:
- netfs_free_folioq_buffer(dw->buffer);
+ bvecq_put(dw->buffer);
free_dw:
kfree(dw);
return rc;
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 16/35] smbdirect: Support ITER_BVECQ in smbdirect_map_sges_from_iter()
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (14 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 15/35] cifs: Use a bvecq for buffering instead of a folioq David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 17/35] netfs: Remove the writethrough code David Howells
` (19 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, Shyam Prasad N, Tom Talpey
Add support for ITER_BVECQ to smbdirect_map_sges_from_iter().
Signed-off-by: David Howells <dhowells@redhat.com>
Acked-by: Stefan Metzmacher <metze@samba.org>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Namjae Jeon <linkinjeon@kernel.org>
cc: Shyam Prasad N <sprasad@microsoft.com>
cc: Tom Talpey <tom@talpey.com>
cc: linux-cifs@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/smb/smbdirect/connection.c | 67 +++++++++++++++++++++++++++++++++++
1 file changed, 67 insertions(+)
diff --git a/fs/smb/smbdirect/connection.c b/fs/smb/smbdirect/connection.c
index 8adf58097534..f5e7e75dd809 100644
--- a/fs/smb/smbdirect/connection.c
+++ b/fs/smb/smbdirect/connection.c
@@ -5,6 +5,7 @@
*/
#include "internal.h"
+#include <linux/bvecq.h>
#include <linux/folio_queue.h>
struct smbdirect_map_sges {
@@ -2006,6 +2007,69 @@ static ssize_t smbdirect_map_sges_from_bvec(struct iov_iter *iter,
return ret;
}
+/*
+ * Extract memory fragments from a BVECQ-class iterator and add them to an RDMA
+ * list. The fragments are not pinned.
+ */
+static ssize_t smbdirect_map_sges_from_bvecq(struct iov_iter *iter,
+ struct smbdirect_map_sges *state,
+ ssize_t maxsize)
+{
+ const struct bvecq *bq = iter->bvecq, *next;
+ unsigned int slot = iter->bvecq_slot;
+ ssize_t extracted = 0;
+ size_t offset = iter->iov_offset;
+
+ maxsize = umin(maxsize, iov_iter_count(iter));
+
+ do {
+ struct bio_vec *bv;
+ size_t bsize;
+
+ while (slot >= bq->nr_slots) {
+ next = bvecq_next(bq);
+ if (!next) {
+ if (WARN_ON_ONCE(maxsize > 0))
+ return -EIO;
+ goto out;
+ }
+ bq = next;
+ slot = 0;
+ }
+
+ bv = &bq->bv[slot];
+ bsize = bv->bv_len;
+
+ if (offset < bsize) {
+ size_t part = umin(maxsize, bsize - offset);
+ bool ok;
+
+ ok = smbdirect_map_sges_single_page(state,
+ bv->bv_page,
+ bv->bv_offset + offset,
+ part);
+ if (!ok)
+ return -EIO;
+
+ offset += part;
+ extracted += part;
+ maxsize -= part;
+ }
+
+ if (offset >= bsize) {
+ offset = 0;
+ slot++;
+ }
+ } while (state->num_sge < state->max_sge && maxsize > 0);
+
+out:
+ iter->bvecq = bq;
+ iter->bvecq_slot = slot;
+ iter->iov_offset = offset;
+ iter->count -= extracted;
+ return extracted;
+}
+
/*
* Extract fragments from a KVEC-class iterator and add them to an ib_sge list.
* This can deal with vmalloc'd buffers as well as kmalloc'd or static buffers.
@@ -2155,6 +2219,9 @@ static ssize_t smbdirect_map_sges_from_iter(struct iov_iter *iter, size_t len,
case ITER_BVEC:
ret = smbdirect_map_sges_from_bvec(iter, state, len);
break;
+ case ITER_BVECQ:
+ ret = smbdirect_map_sges_from_bvecq(iter, state, len);
+ break;
case ITER_KVEC:
ret = smbdirect_map_sges_from_kvec(iter, state, len);
break;
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 17/35] netfs: Remove the writethrough code
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (15 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 16/35] smbdirect: Support ITER_BVECQ in smbdirect_map_sges_from_iter() David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 18/35] netfs: trace: Change the "clear" folio traces to "endwb" David Howells
` (18 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Remove the netfs writethrough code as it's very tricky to get the locking
right and it will probably deadlock if used in conjunction with Ceph
snapshots because it excludes writeback for the duration, but to flush out
old snapshots, it does a synchronous flush that invokes writeback.
Instead, O_SYNC writes do a flush after performing the write - which is
already there as the callers of netfs_perform_write() all call
generic_write_sync().
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/9p/vfs_addr.c | 1 -
fs/afs/file.c | 1 -
fs/netfs/buffered_write.c | 56 ++-----------------
fs/netfs/internal.h | 7 ---
fs/netfs/main.c | 1 -
fs/netfs/stats.c | 4 +-
fs/netfs/write_collect.c | 2 -
fs/netfs/write_issue.c | 104 +----------------------------------
include/linux/netfs.h | 1 -
include/trace/events/netfs.h | 8 +--
10 files changed, 9 insertions(+), 176 deletions(-)
diff --git a/fs/9p/vfs_addr.c b/fs/9p/vfs_addr.c
index 1ac0b3dcc077..2129fcb0f65c 100644
--- a/fs/9p/vfs_addr.c
+++ b/fs/9p/vfs_addr.c
@@ -124,7 +124,6 @@ static int v9fs_init_request(struct netfs_io_request *rreq, struct file *file)
struct p9_fid *fid;
struct dentry *dentry;
bool writing = (rreq->origin == NETFS_READ_FOR_WRITE ||
- rreq->origin == NETFS_WRITETHROUGH ||
rreq->origin == NETFS_UNBUFFERED_WRITE ||
rreq->origin == NETFS_DIO_WRITE);
diff --git a/fs/afs/file.c b/fs/afs/file.c
index 3380a0d20c0f..99987f6bbc36 100644
--- a/fs/afs/file.c
+++ b/fs/afs/file.c
@@ -400,7 +400,6 @@ static int afs_init_request(struct netfs_io_request *rreq, struct file *file)
}
break;
case NETFS_WRITEBACK:
- case NETFS_WRITETHROUGH:
case NETFS_UNBUFFERED_WRITE:
case NETFS_DIO_WRITE:
if (S_ISREG(rreq->inode->i_mode))
diff --git a/fs/netfs/buffered_write.c b/fs/netfs/buffered_write.c
index df496873e4f4..ead22980075f 100644
--- a/fs/netfs/buffered_write.c
+++ b/fs/netfs/buffered_write.c
@@ -91,44 +91,14 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,
struct inode *inode = file_inode(file);
struct address_space *mapping = inode->i_mapping;
struct netfs_inode *ctx = netfs_inode(inode);
- struct writeback_control wbc = {
- .sync_mode = WB_SYNC_NONE,
- .for_sync = true,
- .nr_to_write = LONG_MAX,
- .range_start = iocb->ki_pos,
- .range_end = iocb->ki_pos + iter->count,
- };
- struct netfs_io_request *wreq = NULL;
- struct folio *folio = NULL, *writethrough = NULL;
+ struct folio *folio = NULL;
unsigned int bdp_flags = (iocb->ki_flags & IOCB_NOWAIT) ? BDP_ASYNC : 0;
- ssize_t written = 0, ret, ret2;
+ ssize_t written = 0, ret;
uoff_t pos = iocb->ki_pos;
size_t max_chunk = mapping_max_folio_size(mapping);
bool maybe_trouble = false;
- if (unlikely(iocb->ki_flags & (IOCB_DSYNC | IOCB_SYNC))
- ) {
- wbc_attach_fdatawrite_inode(&wbc, mapping->host);
-
- ret = filemap_write_and_wait_range(mapping, pos, pos + iter->count);
- if (ret < 0) {
- wbc_detach_inode(&wbc);
- goto out;
- }
-
- wreq = netfs_begin_writethrough(iocb, iter->count);
- if (IS_ERR(wreq)) {
- wbc_detach_inode(&wbc);
- ret = PTR_ERR(wreq);
- wreq = NULL;
- goto out;
- }
- if (!is_sync_kiocb(iocb))
- wreq->iocb = iocb;
- netfs_stat(&netfs_n_wh_writethrough);
- } else {
- netfs_stat(&netfs_n_wh_buffered_write);
- }
+ netfs_stat(&netfs_n_wh_buffered_write);
do {
enum netfs_folio_trace trace;
@@ -390,15 +360,8 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,
pos += copied;
written += copied;
- if (likely(!wreq)) {
- folio_mark_dirty(folio);
- folio_unlock(folio);
- } else {
- netfs_advance_writethrough(wreq, &wbc, folio, copied,
- offset + copied == flen,
- &writethrough);
- /* Folio unlocked */
- }
+ folio_mark_dirty(folio);
+ folio_unlock(folio);
retry:
folio_put(folio);
folio = NULL;
@@ -420,15 +383,6 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,
ctx->ops->post_modify(inode);
}
- if (unlikely(wreq)) {
- ret2 = netfs_end_writethrough(wreq, &wbc, writethrough);
- wbc_detach_inode(&wbc);
- if (ret2 == -EIOCBQUEUED)
- return ret2;
- if (ret == 0 && ret2 < 0)
- ret = ret2;
- }
-
iocb->ki_pos += written;
_leave(" = %zd [%zd]", written, ret);
return written ? written : ret;
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 431ec77b1305..09e86183b26b 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -157,7 +157,6 @@ extern atomic_t netfs_n_rh_write_zskip;
extern atomic_t netfs_n_rh_retry_read_req;
extern atomic_t netfs_n_rh_retry_read_subreq;
extern atomic_t netfs_n_wh_buffered_write;
-extern atomic_t netfs_n_wh_writethrough;
extern atomic_t netfs_n_wh_dio_write;
extern atomic_t netfs_n_wh_writepages;
extern atomic_t netfs_n_wh_copy_to_cache;
@@ -217,12 +216,6 @@ void netfs_issue_write(struct netfs_io_request *wreq,
size_t netfs_advance_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
uoff_t start, size_t len, bool to_eof);
-struct netfs_io_request *netfs_begin_writethrough(struct kiocb *iocb, size_t len);
-int netfs_advance_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,
- struct folio *folio, size_t copied, bool to_page_end,
- struct folio **writethrough_cache);
-ssize_t netfs_end_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,
- struct folio *writethrough_cache);
/*
* write_retry.c
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index 9f72e5054aff..3908ea2422c8 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -45,7 +45,6 @@ static const char *netfs_origins[nr__netfs_io_origin] = {
[NETFS_DIO_READ] = "DR",
[NETFS_WRITEBACK] = "WB",
[NETFS_WRITEBACK_SINGLE] = "W1",
- [NETFS_WRITETHROUGH] = "WT",
[NETFS_UNBUFFERED_WRITE] = "UW",
[NETFS_DIO_WRITE] = "DW",
[NETFS_PGPRIV2_COPY_TO_CACHE] = "2C",
diff --git a/fs/netfs/stats.c b/fs/netfs/stats.c
index 84c2a4bcc762..a10d34f88597 100644
--- a/fs/netfs/stats.c
+++ b/fs/netfs/stats.c
@@ -32,7 +32,6 @@ atomic_t netfs_n_rh_write_zskip;
atomic_t netfs_n_rh_retry_read_req;
atomic_t netfs_n_rh_retry_read_subreq;
atomic_t netfs_n_wh_buffered_write;
-atomic_t netfs_n_wh_writethrough;
atomic_t netfs_n_wh_dio_write;
atomic_t netfs_n_wh_writepages;
atomic_t netfs_n_wh_copy_to_cache;
@@ -59,9 +58,8 @@ int netfs_stats_show(struct seq_file *m, void *v)
atomic_read(&netfs_n_rh_read_single),
atomic_read(&netfs_n_rh_write_begin),
atomic_read(&netfs_n_rh_write_zskip));
- seq_printf(m, "Writes : BW=%u WT=%u DW=%u WP=%u 2C=%u\n",
+ seq_printf(m, "Writes : BW=%u DW=%u WP=%u 2C=%u\n",
atomic_read(&netfs_n_wh_buffered_write),
- atomic_read(&netfs_n_wh_writethrough),
atomic_read(&netfs_n_wh_dio_write),
atomic_read(&netfs_n_wh_writepages),
atomic_read(&netfs_n_wh_copy_to_cache));
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index 100a5038c61e..244a68e04624 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -214,7 +214,6 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
smp_rmb();
collected_to = ULLONG_MAX;
if (wreq->origin == NETFS_WRITEBACK ||
- wreq->origin == NETFS_WRITETHROUGH ||
wreq->origin == NETFS_PGPRIV2_COPY_TO_CACHE)
notes = NEED_UNLOCK;
else
@@ -411,7 +410,6 @@ bool netfs_write_collection(struct netfs_io_request *wreq)
switch (wreq->origin) {
case NETFS_WRITEBACK:
case NETFS_WRITEBACK_SINGLE:
- case NETFS_WRITETHROUGH:
netfs_wb_end(ictx);
break;
default:
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index d68d130d6da0..6d0f72797351 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -96,7 +96,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
struct netfs_inode *ictx;
bool is_cacheable = (origin == NETFS_WRITEBACK ||
origin == NETFS_WRITEBACK_SINGLE ||
- origin == NETFS_WRITETHROUGH ||
origin == NETFS_PGPRIV2_COPY_TO_CACHE);
wreq = netfs_alloc_request(mapping, file, start, 0, origin);
@@ -365,11 +364,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
streamw = true;
}
- if (wreq->origin == NETFS_WRITETHROUGH) {
- to_eof = false;
- if (flen > i_size - fpos)
- flen = i_size - fpos;
- } else if (flen > i_size - fpos) {
+ if (flen > i_size - fpos) {
flen = i_size - fpos;
if (!streamw)
folio_zero_segment(folio, flen, fsize);
@@ -611,103 +606,6 @@ int netfs_writepages(struct address_space *mapping,
}
EXPORT_SYMBOL(netfs_writepages);
-/*
- * Begin a write operation for writing through the pagecache.
- */
-struct netfs_io_request *netfs_begin_writethrough(struct kiocb *iocb, size_t len)
-{
- struct netfs_io_request *wreq = NULL;
- struct netfs_inode *ictx = netfs_inode(file_inode(iocb->ki_filp));
-
- netfs_wb_begin(ictx, false);
-
- wreq = netfs_create_write_req(iocb->ki_filp->f_mapping, iocb->ki_filp,
- iocb->ki_pos, NETFS_WRITETHROUGH);
- if (IS_ERR(wreq)) {
- netfs_wb_end(ictx);
- return wreq;
- }
-
- wreq->io_streams[0].avail = true;
- __set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags);
- trace_netfs_write(wreq, netfs_write_trace_writethrough);
- return wreq;
-}
-
-/*
- * Advance the state of the write operation used when writing through the
- * pagecache. Data has been copied into the pagecache that we need to append
- * to the request. If we've added more than wsize then we need to create a new
- * subrequest.
- */
-int netfs_advance_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,
- struct folio *folio, size_t copied, bool to_page_end,
- struct folio **writethrough_cache)
-{
- int ret;
-
- _enter("R=%x ic=%zu ws=%u cp=%zu tp=%u",
- wreq->debug_id, wreq->buffer.iter.count, wreq->wsize, copied, to_page_end);
-
- /* The folio is locked. */
-
- if (*writethrough_cache != folio) {
- if (*writethrough_cache) {
- /* Did the folio get moved? */
- folio_put(*writethrough_cache);
- *writethrough_cache = NULL;
- }
- /* We can make multiple writes to the folio... */
- if (wreq->len == 0)
- trace_netfs_folio(folio, netfs_folio_trace_wthru);
- else
- trace_netfs_folio(folio, netfs_folio_trace_wthru_plus);
- *writethrough_cache = folio;
- folio_get(folio);
- }
-
- wreq->len += copied;
-
- if (!to_page_end) {
- folio_mark_dirty(folio);
- folio_unlock(folio);
- return 0;
- }
-
- ret = netfs_write_folio(wreq, wbc, folio);
- folio_put(*writethrough_cache);
- *writethrough_cache = NULL;
- wreq->submitted = wreq->len;
- return ret;
-}
-
-/*
- * End a write operation used when writing through the pagecache.
- */
-ssize_t netfs_end_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,
- struct folio *writethrough_cache)
-{
- ssize_t ret;
-
- _enter("R=%x", wreq->debug_id);
-
- if (writethrough_cache) {
- folio_lock(writethrough_cache);
- netfs_write_folio(wreq, wbc, writethrough_cache);
- folio_put(writethrough_cache);
- wreq->submitted = wreq->len;
- }
-
- netfs_end_issue_write(wreq);
-
- if (wreq->iocb)
- ret = -EIOCBQUEUED;
- else
- ret = netfs_wait_for_write(wreq);
- netfs_put_request(wreq, netfs_rreq_trace_put_return);
- return ret;
-}
-
/**
* netfs_writeback_single - Write back a monolithic payload
* @mapping: The mapping to write from
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 14a24bb80af5..b6e5f6dc54e2 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -209,7 +209,6 @@ enum netfs_io_origin {
NETFS_DIO_READ, /* This is a direct I/O read */
NETFS_WRITEBACK, /* This write was triggered by writepages */
NETFS_WRITEBACK_SINGLE, /* This monolithic write was triggered by writepages */
- NETFS_WRITETHROUGH, /* This write was made by netfs_perform_write() */
NETFS_UNBUFFERED_WRITE, /* This is an unbuffered write */
NETFS_DIO_WRITE, /* This is a direct I/O write */
NETFS_PGPRIV2_COPY_TO_CACHE, /* [DEPRECATED] This is writing read data to the cache */
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 111c02817bea..ccb9f3c30432 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -30,8 +30,7 @@
EM(netfs_write_trace_dio_write, "DIO-WRITE") \
EM(netfs_write_trace_unbuffered_write, "UNB-WRITE") \
EM(netfs_write_trace_writeback, "WRITEBACK") \
- EM(netfs_write_trace_writeback_single, "WB-SINGLE") \
- E_(netfs_write_trace_writethrough, "WRITETHRU")
+ E_(netfs_write_trace_writeback_single, "WB-SINGLE")
#define netfs_rreq_origins \
EM(NETFS_READAHEAD, "RA") \
@@ -43,7 +42,6 @@
EM(NETFS_DIO_READ, "DR") \
EM(NETFS_WRITEBACK, "WB") \
EM(NETFS_WRITEBACK_SINGLE, "W1") \
- EM(NETFS_WRITETHROUGH, "WT") \
EM(NETFS_UNBUFFERED_WRITE, "UW") \
EM(NETFS_DIO_WRITE, "DW") \
E_(NETFS_PGPRIV2_COPY_TO_CACHE, "2C")
@@ -221,9 +219,7 @@
EM(netfs_folio_trace_redirtied, "redirtied") \
EM(netfs_folio_trace_store, "store") \
EM(netfs_folio_trace_store_copy, "store-copy") \
- EM(netfs_folio_trace_store_plus, "store+") \
- EM(netfs_folio_trace_wthru, "wthru") \
- E_(netfs_folio_trace_wthru_plus, "wthru+")
+ E_(netfs_folio_trace_store_plus, "store+")
#define netfs_collect_contig_traces \
EM(netfs_contig_trace_collect, "Collect") \
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 18/35] netfs: trace: Change the "clear" folio traces to "endwb"
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (16 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 17/35] netfs: Remove the writethrough code David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 19/35] netfs: trace: Rejig a couple of the tracepoints David Howells
` (17 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Change the "clear" folio traces to "endwb" as it's more obvious what it
means.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/write_collect.c | 8 ++++----
include/trace/events/netfs.h | 8 ++++----
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index 244a68e04624..6114bdf27ce0 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -56,7 +56,7 @@ static void netfs_dump_request(const struct netfs_io_request *rreq)
*/
int netfs_folio_written_back(struct folio *folio)
{
- enum netfs_folio_trace why = netfs_folio_trace_clear;
+ enum netfs_folio_trace why = netfs_folio_trace_endwb;
struct inode *inode = folio_inode(folio);
struct netfs_inode *ictx = netfs_inode(inode);
struct netfs_folio *finfo;
@@ -79,13 +79,13 @@ int netfs_folio_written_back(struct folio *folio)
group = finfo->netfs_group;
gcount++;
kfree(finfo);
- why = netfs_folio_trace_clear_s;
+ why = netfs_folio_trace_endwb_s;
goto end_wb;
}
if ((group = netfs_folio_group(folio))) {
if (group == NETFS_FOLIO_COPY_TO_CACHE) {
- why = netfs_folio_trace_clear_cc;
+ why = netfs_folio_trace_endwb_cc;
folio_detach_private(folio);
goto end_wb;
}
@@ -98,7 +98,7 @@ int netfs_folio_written_back(struct folio *folio)
if (!folio_test_dirty(folio)) {
folio_detach_private(folio);
gcount++;
- why = netfs_folio_trace_clear_g;
+ why = netfs_folio_trace_endwb_g;
}
}
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index ccb9f3c30432..e23300c56d00 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -192,12 +192,12 @@
EM(netfs_folio_trace_alloc_buffer, "alloc-buf") \
EM(netfs_folio_trace_cancel_copy, "cancel-copy") \
EM(netfs_folio_trace_cancel_store, "cancel-store") \
- EM(netfs_folio_trace_clear, "clear") \
- EM(netfs_folio_trace_clear_cc, "clear-cc") \
- EM(netfs_folio_trace_clear_g, "clear-g") \
- EM(netfs_folio_trace_clear_s, "clear-s") \
EM(netfs_folio_trace_copy_to_cache, "mark-copy") \
EM(netfs_folio_trace_end_copy, "end-copy") \
+ EM(netfs_folio_trace_endwb, "endwb") \
+ EM(netfs_folio_trace_endwb_cc, "endwb-cc") \
+ EM(netfs_folio_trace_endwb_g, "endwb-g") \
+ EM(netfs_folio_trace_endwb_s, "endwb-s") \
EM(netfs_folio_trace_filled_gaps, "filled-gaps") \
EM(netfs_folio_trace_invalidate_all, "inval-all") \
EM(netfs_folio_trace_invalidate_front, "inval-front") \
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 19/35] netfs: trace: Rejig a couple of the tracepoints
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (17 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 18/35] netfs: trace: Change the "clear" folio traces to "endwb" David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 20/35] netfs: Add some functions to wrap the all-queued handling David Howells
` (16 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Rejig the following tracepoints:
(1) Change netfs_folio to show the pfn.
(2) Change netfs_collect_folio to show a folio index range rather than
file position range and don't show the cleaned_to or collected_to
points.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/read_collect.c | 2 +-
fs/netfs/read_pgpriv2.c | 2 +-
fs/netfs/write_collect.c | 2 +-
include/trace/events/netfs.h | 23 ++++++++++-------------
4 files changed, 13 insertions(+), 16 deletions(-)
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index 72afc17ecbb3..94e180ec6e41 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -172,7 +172,7 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
fpos = folio_pos(folio);
fend = fpos + fsize;
- trace_netfs_collect_folio(rreq, folio, fend, collected_to);
+ trace_netfs_collect_folio(rreq, folio);
/* Unlock any folio we've transferred all of. */
if (collected_to < fend)
diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c
index ac16bb8210d1..883843270699 100644
--- a/fs/netfs/read_pgpriv2.c
+++ b/fs/netfs/read_pgpriv2.c
@@ -198,7 +198,7 @@ bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *creq)
fend = min_t(uoff_t, fpos + flen, creq->i_size);
- trace_netfs_collect_folio(creq, folio, fend, collected_to);
+ trace_netfs_collect_folio(creq, folio);
/* Unlock any folio we've transferred all of. */
if (collected_to < fend)
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index 6114bdf27ce0..7194182b975c 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -156,7 +156,7 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
fend = min_t(uoff_t, fpos + flen, wreq->i_size);
- trace_netfs_collect_folio(wreq, folio, fend, collected_to);
+ trace_netfs_collect_folio(wreq, folio);
/* Unlock any folio we've transferred all of. */
if (collected_to < fend)
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index e23300c56d00..6439210d0705 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -495,6 +495,7 @@ TRACE_EVENT(netfs_folio,
TP_STRUCT__entry(
__field(u64, ino)
__field(pgoff_t, index)
+ __field(unsigned long, pfn)
__field(unsigned int, nr)
__field(enum netfs_folio_trace, why)
),
@@ -505,9 +506,11 @@ TRACE_EVENT(netfs_folio,
__entry->why = why;
__entry->index = folio->index;
__entry->nr = folio_nr_pages(folio);
+ __entry->pfn = folio_pfn(folio);
),
- TP_printk("i=%05llx ix=%05lx-%05lx %s",
+ TP_printk("p=%lx i=%05llx ix=%05lx-%05lx %s",
+ __entry->pfn,
__entry->ino, __entry->index, __entry->index + __entry->nr - 1,
__print_symbolic(__entry->why, netfs_folio_traces))
);
@@ -654,31 +657,25 @@ TRACE_EVENT(netfs_collect_sreq,
TRACE_EVENT(netfs_collect_folio,
TP_PROTO(const struct netfs_io_request *wreq,
- const struct folio *folio,
- uoff_t fend, uoff_t collected_to),
+ const struct folio *folio),
- TP_ARGS(wreq, folio, fend, collected_to),
+ TP_ARGS(wreq, folio),
TP_STRUCT__entry(
__field(unsigned int, wreq)
__field(unsigned long, index)
- __field(uoff_t, fend)
- __field(uoff_t, cleaned_to)
- __field(uoff_t, collected_to)
+ __field(unsigned int, nr)
),
TP_fast_assign(
__entry->wreq = wreq->debug_id;
__entry->index = folio->index;
- __entry->fend = fend;
- __entry->cleaned_to = wreq->cleaned_to;
- __entry->collected_to = collected_to;
+ __entry->nr = folio_nr_pages(folio);
),
- TP_printk("R=%08x ix=%05lx r=%llx-%llx t=%llx/%llx",
+ TP_printk("R=%08x ix=%05lx-%05lx",
__entry->wreq, __entry->index,
- (uoff_t)__entry->index * PAGE_SIZE, __entry->fend,
- __entry->cleaned_to, __entry->collected_to)
+ __entry->index + __entry->nr - 1)
);
TRACE_EVENT(netfs_collect_state,
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 20/35] netfs: Add some functions to wrap the all-queued handling
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (18 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 19/35] netfs: trace: Rejig a couple of the tracepoints David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 21/35] netfs: Make deprecated PG_private_2 support optional David Howells
` (15 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Add some helper functions to wrap the handling of the NETFS_RREQ_ALL_QUEUED
flag and to insert the appropriate barriers.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/buffered_read.c | 9 +++------
fs/netfs/direct_read.c | 9 +++------
fs/netfs/internal.h | 19 +++++++++++++++++++
fs/netfs/misc.c | 2 +-
fs/netfs/read_collect.c | 4 +---
fs/netfs/read_pgpriv2.c | 3 +--
fs/netfs/read_single.c | 9 +++------
fs/netfs/write_collect.c | 3 +--
fs/netfs/write_issue.c | 7 ++-----
include/trace/events/netfs.h | 1 +
10 files changed, 35 insertions(+), 31 deletions(-)
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 77f7dfc816a8..03e095060162 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -303,10 +303,8 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
}
start += slice;
size -= slice;
- if (size <= 0) {
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
- }
+ if (size <= 0)
+ netfs_all_subreqs_queued(rreq);
netfs_issue_read(rreq, subreq);
netfs_maybe_bulk_drop_ra_refs(rreq);
@@ -319,8 +317,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
} while (size > 0);
if (unlikely(size > 0)) {
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+ netfs_all_subreqs_queued(rreq);
netfs_wake_collector(rreq);
}
diff --git a/fs/netfs/direct_read.c b/fs/netfs/direct_read.c
index aa10af5171a8..5405e108b7a3 100644
--- a/fs/netfs/direct_read.c
+++ b/fs/netfs/direct_read.c
@@ -84,10 +84,8 @@ static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
size -= slice;
start += slice;
rreq->submitted += slice;
- if (size <= 0) {
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
- }
+ if (size <= 0)
+ netfs_all_subreqs_queued(rreq);
rreq->netfs_ops->issue_read(subreq);
@@ -99,8 +97,7 @@ static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
} while (size > 0);
if (unlikely(size > 0)) {
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+ netfs_all_subreqs_queued(rreq);
netfs_wake_collector(rreq);
}
}
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 09e86183b26b..df04ee82e34b 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -314,6 +314,25 @@ static inline bool netfs_check_subreq_in_progress(const struct netfs_io_subreque
return test_bit_acquire(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
}
+/*
+ * Indicate that we've generated and queued all the subrequests we're going to.
+ */
+static inline void netfs_all_subreqs_queued(struct netfs_io_request *rreq)
+{
+ smp_wmb(); /* Write lists before ALL_QUEUED. */
+ set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_all_queued);
+}
+
+/*
+ * Query if all subrequests are queued.
+ */
+static inline bool netfs_are_all_subreqs_queued(const struct netfs_io_request *rreq)
+{
+ /* Read lists after ALL_QUEUED. */
+ return test_bit_acquire(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+}
+
/*
* fscache-cache.c
*/
diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c
index eafc4edae6a0..a3cd76d584b8 100644
--- a/fs/netfs/misc.c
+++ b/fs/netfs/misc.c
@@ -424,7 +424,7 @@ static int netfs_collect_in_app(struct netfs_io_request *rreq,
need_collect = true;
break;
}
- if (subreq || !test_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags))
+ if (subreq || !netfs_are_all_subreqs_queued(rreq))
done = false;
}
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index 94e180ec6e41..f758ab0aaf5d 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -447,10 +447,8 @@ bool netfs_read_collection(struct netfs_io_request *rreq)
/* We're done when the app thread has finished posting subreqs and the
* queue is empty.
*/
- if (!test_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags))
+ if (!netfs_are_all_subreqs_queued(rreq))
return false;
- smp_rmb(); /* Read ALL_QUEUED before subreq lists. */
-
if (!list_empty(&stream->subrequests))
return false;
diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c
index 883843270699..aab156d00031 100644
--- a/fs/netfs/read_pgpriv2.c
+++ b/fs/netfs/read_pgpriv2.c
@@ -155,8 +155,7 @@ void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)
return;
netfs_issue_write(creq, &creq->io_streams[1]);
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &creq->flags);
+ netfs_all_subreqs_queued(creq);
trace_netfs_rreq(rreq, netfs_rreq_trace_end_copy_to_cache);
if (list_empty_careful(&creq->io_streams[1].subrequests))
netfs_wake_collector(creq);
diff --git a/fs/netfs/read_single.c b/fs/netfs/read_single.c
index de67ac41548d..ccb5fc809d99 100644
--- a/fs/netfs/read_single.c
+++ b/fs/netfs/read_single.c
@@ -113,14 +113,12 @@ static int netfs_single_dispatch_read(struct netfs_io_request *rreq)
goto cancel;
}
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+ netfs_all_subreqs_queued(rreq);
rreq->netfs_ops->issue_read(subreq);
rreq->submitted += subreq->len;
break;
case NETFS_READ_FROM_CACHE:
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+ netfs_all_subreqs_queued(rreq);
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
netfs_single_read_cache(rreq, subreq);
rreq->submitted += subreq->len;
@@ -136,8 +134,7 @@ static int netfs_single_dispatch_read(struct netfs_io_request *rreq)
return ret;
cancel:
netfs_cancel_read(subreq, ret);
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+ netfs_all_subreqs_queued(rreq);
netfs_wake_collector(rreq);
return ret;
}
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index 7194182b975c..6d99d4a6f780 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -371,9 +371,8 @@ bool netfs_write_collection(struct netfs_io_request *wreq)
/* We're done when the app thread has finished posting subreqs and all
* the queues in all the streams are empty.
*/
- if (!test_bit(NETFS_RREQ_ALL_QUEUED, &wreq->flags))
+ if (!netfs_are_all_subreqs_queued(wreq))
return false;
- smp_rmb(); /* Read ALL_QUEUED before lists. */
transferred = LONG_MAX;
for (s = 0; s < NR_IO_STREAMS; s++) {
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index 6d0f72797351..4ed50ac7c1f8 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -518,8 +518,7 @@ static void netfs_end_issue_write(struct netfs_io_request *wreq)
{
bool needs_poke = true;
- smp_wmb(); /* Write subreq lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &wreq->flags);
+ netfs_all_subreqs_queued(wreq);
for (int s = 0; s < NR_IO_STREAMS; s++) {
struct netfs_io_stream *stream = &wreq->io_streams[s];
@@ -665,9 +664,7 @@ int netfs_writeback_single(struct address_space *mapping,
netfs_issue_write(wreq, stream);
}
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &wreq->flags);
-
+ netfs_all_subreqs_queued(wreq);
netfs_wake_collector(wreq);
/* TODO: Might want to be async here if WB_SYNC_NONE, but then need to
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 6439210d0705..1687b974900c 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -47,6 +47,7 @@
E_(NETFS_PGPRIV2_COPY_TO_CACHE, "2C")
#define netfs_rreq_traces \
+ EM(netfs_rreq_trace_all_queued, "ALL-Q ") \
EM(netfs_rreq_trace_assess, "ASSESS ") \
EM(netfs_rreq_trace_collect, "COLLECT") \
EM(netfs_rreq_trace_complete, "COMPLET") \
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 21/35] netfs: Make deprecated PG_private_2 support optional
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (19 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 20/35] netfs: Add some functions to wrap the all-queued handling David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 22/35] cachefiles: Don't rely on backing fs storage map for most use cases David Howells
` (14 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, Trond Myklebust, Anna Schumaker,
Alex Markuze, Viacheslav Dubeyko
Make the deprecated PG_private_2 support optional, requiring it to be
selected by the filesystems that might want to use it.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Trond Myklebust <trondmy@kernel.org>
cc: Anna Schumaker <anna@kernel.org>
cc: Ilya Dryomov <idryomov@gmail.com>
cc: Alex Markuze <amarkuze@redhat.com>
cc: Viacheslav Dubeyko <slava@dubeyko.com>
cc: netfs@lists.linux.dev
cc: linux-nfs@vger.kernel.org
cc: ceph-devel@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
---
fs/ceph/Kconfig | 1 +
fs/netfs/Kconfig | 3 +++
fs/netfs/Makefile | 2 +-
fs/netfs/internal.h | 28 ++++++++++++++++++++++++++++
fs/netfs/read_collect.c | 2 +-
fs/nfs/Kconfig | 1 +
include/linux/netfs.h | 2 ++
7 files changed, 37 insertions(+), 2 deletions(-)
diff --git a/fs/ceph/Kconfig b/fs/ceph/Kconfig
index 3d64a316ca31..aa6ccd7794d2 100644
--- a/fs/ceph/Kconfig
+++ b/fs/ceph/Kconfig
@@ -4,6 +4,7 @@ config CEPH_FS
depends on INET
select CEPH_LIB
select NETFS_SUPPORT
+ select NETFS_PGPRIV2
select FS_ENCRYPTION_ALGS if FS_ENCRYPTION
default n
help
diff --git a/fs/netfs/Kconfig b/fs/netfs/Kconfig
index 7701c037c328..d0e7b0971fa3 100644
--- a/fs/netfs/Kconfig
+++ b/fs/netfs/Kconfig
@@ -22,6 +22,9 @@ config NETFS_STATS
between CPUs. On the other hand, the stats are very useful for
debugging purposes. Saying 'Y' here is recommended.
+config NETFS_PGPRIV2
+ bool
+
config NETFS_DEBUG
bool "Enable dynamic debugging netfslib and FS-Cache"
depends on NETFS_SUPPORT
diff --git a/fs/netfs/Makefile b/fs/netfs/Makefile
index e1f12ecb5abf..b1ea4439c1bb 100644
--- a/fs/netfs/Makefile
+++ b/fs/netfs/Makefile
@@ -12,7 +12,6 @@ netfs-y := \
misc.o \
objects.o \
read_collect.o \
- read_pgpriv2.o \
read_retry.o \
read_single.o \
rolling_buffer.o \
@@ -20,6 +19,7 @@ netfs-y := \
write_issue.o \
write_retry.o
+netfs-$(CONFIG_NETFS_PGPRIV2) += read_pgpriv2.o
netfs-$(CONFIG_NETFS_STATS) += stats.o
netfs-$(CONFIG_FSCACHE) += \
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index df04ee82e34b..c50750b72262 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -120,9 +120,37 @@ void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error);
/*
* read_pgpriv2.c
*/
+#ifdef CONFIG_NETFS_PGPRIV2
+int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *folio);
void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq);
bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq);
+static inline bool netfs_using_pgpriv2(const struct netfs_io_request *rreq)
+{
+ return test_bit(NETFS_RREQ_USE_PGPRIV2, &rreq->flags);
+}
+#else
+static inline int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ return -EIO;
+}
+static inline void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *folio)
+{
+}
+static inline void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)
+{
+}
+static inline bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq)
+{
+ return true;
+}
+static inline bool netfs_using_pgpriv2(const struct netfs_io_request *rreq)
+{
+ return false;
+}
+#endif
/*
* read_retry.c
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index f758ab0aaf5d..f773579692ab 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -54,7 +54,7 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
flush_dcache_folio(folio);
folio_mark_uptodate(folio);
- if (!test_bit(NETFS_RREQ_USE_PGPRIV2, &rreq->flags)) {
+ if (!netfs_using_pgpriv2(rreq)) {
finfo = netfs_folio_info(folio);
if (finfo) {
trace_netfs_folio(folio, netfs_folio_trace_filled_gaps);
diff --git a/fs/nfs/Kconfig b/fs/nfs/Kconfig
index 6bb30543eff0..e7862f35b72c 100644
--- a/fs/nfs/Kconfig
+++ b/fs/nfs/Kconfig
@@ -174,6 +174,7 @@ config NFS_FSCACHE
bool "Provide NFS client caching support"
depends on NFS_FS
select NETFS_SUPPORT
+ select NETFS_PGPRIV2
select FSCACHE
help
Say Y here if you want NFS data to be cached locally on disc through
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index b6e5f6dc54e2..7cdb14a130d9 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -279,8 +279,10 @@ struct netfs_io_request {
#define NETFS_RREQ_UPLOAD_TO_SERVER 11 /* Need to write to the server */
#define NETFS_RREQ_USE_IO_ITER 12 /* Use ->io_iter rather than ->i_pages */
#define NETFS_RREQ_NEED_PUT_RA_REFS 17 /* Need to put the folio refs RA gave us */
+#ifdef CONFIG_NETFS_PGPRIV2
#define NETFS_RREQ_USE_PGPRIV2 31 /* [DEPRECATED] Use PG_private_2 to mark
* write to cache on read */
+#endif
const struct netfs_request_ops *netfs_ops;
};
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 22/35] cachefiles: Don't rely on backing fs storage map for most use cases
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (20 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 21/35] netfs: Make deprecated PG_private_2 support optional David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 23/35] netfs: Add the cache object ID to netfs_read/write tracepoints David Howells
` (13 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Cachefiles currently uses the backing filesystem's idea of what data is
held in a backing file and queries this by means of SEEK_DATA and
SEEK_HOLE. However, this means it does two seek operations on the backing
file for each individual read call it wants to prepare (unless the first
returns -ENXIO). Worse, the backing filesystem is at liberty to insert or
remove blocks of zeros in order to optimise its layout which may cause
false positives and false negatives.
The problem is that keeping track of what is dirty is tricky (if storing
info in xattrs, which may have limited capacity and must be read and
written as one piece) and expensive (in terms of diskspace at least) and is
basically duplicating what a filesystem does.
However, the most common write case, in which the application does {
open(O_TRUNC); write(); write(); ... write(); close(); } where each write
follows directly on from the previous and leaves no gaps in the file is
reasonably easy to detect and can be noted in the primary xattr as
CACHEFILES_CONTENT_ALL, indicating we have everything up to the object size
stored.
In this specific case, given that it is known that there are no holes in
the file, there's no need to call SEEK_DATA/HOLE or use any other mechanism
to track the contents. That speeds things up enormously.
Even when it is necessary to use SEEK_DATA/HOLE, it may not be necessary to
call it for each cache read subrequest generated.
Implement this by adding support for the CACHEFILES_CONTENT_ALL content
type (which is defined, but currently unused), which requires a slight
adjustment in how backing files are managed. Specifically, the driver
needs to know how much of the tail block is data and whether storing more
data will create a hole.
To this end, the way that the size of a backing file is managed is changed.
Currently, the backing file is expanded to strictly match the size of the
network file, but this can be changed to carry more useful information.
This makes two pieces of metadata available: xattr.object_size and the
backing file's i_size. Apply the following schema:
(a) i_size is always a multiple of the DIO block size.
(b) i_size is only updated to the end of the highest write stored. This
is used to work out if we are following on without leaving a hole.
(c) xattr.object_size is the size of the network filesystem file cached
in this backing file.
(d) xattr.object_size must point after the start of the last block
(unless both are 0).
(e) If xattr.object_size is at or after the block at the current end of
the backing file (ie. i_size), then we have all the contents of the
block (if xattr.content == CACHEFILES_CONTENT_ALL).
(f) If xattr.object_size is somewhere in the middle of the last block,
then the data following it is invalid and must be ignored.
(g) If data is added to the last block, then that block must be fetched,
modified and rewritten (it must be a buffered write through the
pagecache and not DIO).
(h) Writes to cache are rounded out to blocks on both sides and the
folios used as sources must contain data for any lower gap and must
have been cleared for any upper gap, and so will rewrite any
non-data area in the tail block.
To implement this, the following changes are made:
(1) cookie->object_size is no longer updated when writes are copied into
the pagecache, but rather only updated when a write request completes.
This prevents object size miscomparison when checking the xattr
causing the backing file to be invalidated (opening and marking the
backing file and modifying the pagecache run in parallel).
(2) The cache's current idea of the amount of data that should be stored
in the backing file is kept track of in object->object_size.
Possibly this is redundant with cookie->object_size, but the latter
gets updated in some addition circumstances.
(3) The size of the backing file at the start of a request is now tracked
in struct netfs_cache_resources so that the partial EOF block can be
located and cleaned.
(4) The cache block size is now used consistently rather than using
CACHEFILES_DIO_BLOCK_SIZE (4096).
(5) The backing file size is no longer adjusted when looking up an object.
(6) When shortening a file, if the new size is not block aligned, the part
beyond the new size is cleared. If the file is truncated to zero, the
content_info gets reset to CACHEFILES_CONTENT_NO_DATA.
(7) A new struct, fscache_occupancy, is instituted to track the region
being read. Netfslib allocates it and fills in the start and end of
the region to be read then calls the ->query_occupancy() method to
find and fill in the extents. It also indicates whether a recorded
extent contains data or just contains a region that's all zeros
(FSCACHE_EXTENT_DATA or FSCACHE_EXTENT_ZERO).
(8) The ->prepare_read() cache method is changed such that, if given, it
just limits the amount that can be read from the cache in one go. It
no longer indicates what source of read should be done; that
information is now obtained from ->query_occupancy().
(9) A new cache method, ->collect_write(), is added that is called when a
contiguous series of writes have completed and a discontiguity or the
end of the request has been hit. It it supplied with the start and
length of the write made to the backing file and can use this
information to update the cache metadata.
(10) cachefiles_query_occupancy() is altered to find the next two "extents"
of data stored in the backing file by doing SEEK_DATA/HOLE between the
bounds set - unless it is known that there are no holes, in which case
a whole-file first extent can be set.
(11) cachefiles_collect_write() is implemented to take the collated write
completion information and use this to update the cache metadata, in
particular working out whether there's now a hole in the backing file
requiring future use of SEEK_DATA/HOLE instead of just assuming the
data is all present.
It also uses fallocate(FALLOC_FL_ZERO_RANGE) to clean the part of a
partial block that extended beyond the old object size. It might be
better to perform a synchronous DIO write for this purpose, but that
would mandate an RMW cycle. Ideally, it should be all zeros anyway,
but, unfortunately, shared-writable mmap can interfere.
(12) cachefiles_begin_operation() is updated to note the current backing
file size and the cache DIO size.
(13) cachefiles_create_tmpfile() no longer expands the backing file when it
creates it.
(14) cachefiles_set_object_xattr() is changed to use object->object_size
rather than cookie->object_size.
(15) cachefiles_check_auxdata() is altered to actually store the content
type and to also set object->object_size. The cachefiles_coherency
tracepoint is also modified to display xattr.object_size.
(16) netfs_read_to_pagecache() is reworked. The cache ->prepare_read()
method is replaced with ->query_occupancy() as the arbiter of what
region of the file is read from where, and that retrieves up to two
occupied extents of the backing file at once.
The cache ->prepare_read() method is now repurposed to be the same as
the equivalent network filesystem method and allows the cache to limit
the size of the read before the iterator is prepared.
netfs_single_dispatch_read() is similarly modified.
(17) netfs_update_i_size() and afs_update_i_size() no longer call
fscache_update_cookie() to update cookie->object_size.
(18) Write collection now collates contiguous sequences of writes to the
cache and calls the cache ->collect_write() method.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: linux-cifs@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/afs/file.c | 1 -
fs/cachefiles/interface.c | 90 ++------
fs/cachefiles/internal.h | 13 +-
fs/cachefiles/io.c | 341 ++++++++++++++++++++++++------
fs/cachefiles/namei.c | 19 +-
fs/cachefiles/xattr.c | 23 +-
fs/netfs/buffered_read.c | 176 +++++++++------
fs/netfs/buffered_write.c | 3 -
fs/netfs/internal.h | 2 +
fs/netfs/objects.c | 1 +
fs/netfs/read_retry.c | 2 +
fs/netfs/read_single.c | 39 ++--
fs/netfs/write_collect.c | 132 ++++++++++--
fs/netfs/write_issue.c | 18 ++
fs/netfs/write_retry.c | 3 +
include/linux/fscache.h | 17 ++
include/linux/netfs.h | 40 +++-
include/trace/events/cachefiles.h | 21 +-
include/trace/events/netfs.h | 9 +-
19 files changed, 662 insertions(+), 288 deletions(-)
diff --git a/fs/afs/file.c b/fs/afs/file.c
index 99987f6bbc36..11695be8c4a5 100644
--- a/fs/afs/file.c
+++ b/fs/afs/file.c
@@ -447,7 +447,6 @@ void afs_set_i_size(struct afs_vnode *vnode, loff_t new_i_size)
}
spin_unlock(&inode->i_lock);
write_sequnlock(&vnode->cb_lock);
- fscache_update_cookie(afs_vnode_cache(vnode), NULL, &new_i_size);
}
static void afs_update_i_size(struct inode *inode, uoff_t new_i_size)
diff --git a/fs/cachefiles/interface.c b/fs/cachefiles/interface.c
index a160d5c3e74c..789ff6abe926 100644
--- a/fs/cachefiles/interface.c
+++ b/fs/cachefiles/interface.c
@@ -99,73 +99,6 @@ void cachefiles_put_object(struct cachefiles_object *object,
_leave("");
}
-/*
- * Adjust the size of a cache file if necessary to match the DIO size. We keep
- * the EOF marker a multiple of DIO blocks so that we don't fall back to doing
- * non-DIO for a partial block straddling the EOF, but we also have to be
- * careful of someone expanding the file and accidentally accreting the
- * padding.
- */
-static int cachefiles_adjust_size(struct cachefiles_object *object)
-{
- struct iattr newattrs;
- struct file *file = object->file;
- uint64_t ni_size;
- uoff_t oi_size;
- int ret;
-
- ni_size = object->cookie->object_size;
- ni_size = round_up(ni_size, CACHEFILES_DIO_BLOCK_SIZE);
-
- _enter("{OBJ%x},[%llu]",
- object->debug_id, (unsigned long long) ni_size);
-
- if (!file)
- return -ENOBUFS;
-
- oi_size = i_size_read(file_inode(file));
- if (oi_size == ni_size)
- return 0;
-
- inode_lock(file_inode(file));
-
- /* if there's an extension to a partial page at the end of the backing
- * file, we need to discard the partial page so that we pick up new
- * data after it */
- if (oi_size & ~PAGE_MASK && ni_size > oi_size) {
- _debug("discard tail %llx", oi_size);
- newattrs.ia_valid = ATTR_SIZE;
- newattrs.ia_size = oi_size & PAGE_MASK;
- ret = cachefiles_inject_remove_error();
- if (ret == 0)
- ret = notify_change(&nop_mnt_idmap, file->f_path.dentry,
- &newattrs, NULL);
- if (ret < 0)
- goto truncate_failed;
- }
-
- newattrs.ia_valid = ATTR_SIZE;
- newattrs.ia_size = ni_size;
- ret = cachefiles_inject_write_error();
- if (ret == 0)
- ret = notify_change(&nop_mnt_idmap, file->f_path.dentry,
- &newattrs, NULL);
-
-truncate_failed:
- inode_unlock(file_inode(file));
-
- if (ret < 0)
- trace_cachefiles_io_error(NULL, file_inode(file), ret,
- cachefiles_trace_notify_change_error);
- if (ret == -EIO) {
- cachefiles_io_error_obj(object, "Size set failed");
- ret = -ENOBUFS;
- }
-
- _leave(" = %d", ret);
- return ret;
-}
-
/*
* Attempt to look up the nominated node in this cache
*/
@@ -198,7 +131,6 @@ static bool cachefiles_lookup_cookie(struct fscache_cookie *cookie)
spin_lock(&cache->object_list_lock);
list_add(&object->cache_link, &cache->object_list);
spin_unlock(&cache->object_list_lock);
- cachefiles_adjust_size(object);
cachefiles_end_secure(cache, saved_cred);
_leave(" = t");
@@ -232,7 +164,7 @@ static bool cachefiles_shorten_object(struct cachefiles_object *object,
uoff_t i_size, dio_size;
int ret;
- dio_size = round_up(new_size, CACHEFILES_DIO_BLOCK_SIZE);
+ dio_size = round_up(new_size, cache->bsize);
i_size = i_size_read(inode);
trace_cachefiles_trunc(object, inode, i_size, dio_size,
@@ -264,6 +196,7 @@ static bool cachefiles_shorten_object(struct cachefiles_object *object,
}
}
+ object->object_size = new_size;
return true;
}
@@ -278,22 +211,31 @@ static void cachefiles_resize_cookie(struct netfs_cache_resources *cres,
struct fscache_cookie *cookie = object->cookie;
const struct cred *saved_cred;
struct file *file = cachefiles_cres_file(cres);
- uoff_t old_size = cookie->object_size;
+ uoff_t i_size = i_size_read(file_inode(file));
- _enter("%llu->%llu", old_size, new_size);
+ _enter("%llu->%llu", object->object_size, new_size);
- if (new_size < old_size) {
+ /* If the file is being shrunk, we need to downsize the backing file
+ * and clear the end of the final block.
+ */
+ if (new_size < object->object_size) {
+ if (new_size >= i_size)
+ goto out;
cachefiles_begin_secure(cache, &saved_cred);
cachefiles_shorten_object(object, file, new_size);
cachefiles_end_secure(cache, saved_cred);
object->cookie->object_size = new_size;
+ if (new_size == 0)
+ object->content_info = CACHEFILES_CONTENT_NO_DATA;
return;
}
/* The file is being expanded. We don't need to do anything
- * particularly. cookie->initial_size doesn't change and so the point
- * at which we have to download before doesn't change.
+ * particularly. The tail of the last block should have been cleared
+ * both when it is written and when it is shrunk.
*/
+out:
+ object->object_size = new_size;
cookie->object_size = new_size;
}
diff --git a/fs/cachefiles/internal.h b/fs/cachefiles/internal.h
index 60bd801ada04..b2605111fd56 100644
--- a/fs/cachefiles/internal.h
+++ b/fs/cachefiles/internal.h
@@ -16,8 +16,6 @@
#include <linux/cred.h>
#include <linux/security.h>
-#define CACHEFILES_DIO_BLOCK_SIZE 4096
-
struct cachefiles_cache;
struct cachefiles_object;
@@ -51,12 +49,17 @@ struct cachefiles_object {
struct list_head cache_link; /* Link in cache->*_list */
struct file *file; /* The file representing this object */
char *d_name; /* Backing file name */
+ unsigned long flags;
+#define CACHEFILES_OBJECT_USING_TMPFILE 0 /* Have an unlinked tmpfile */
+ uoff_t object_size; /* Size of the object stored
+ * (independent of cookie->object_size for
+ * coherency reasons)
+ */
+ atomic64_t read_limit; /* Point beyond which uncommitted writes */
int debug_id;
spinlock_t lock;
refcount_t ref;
- enum cachefiles_content content_info:8; /* Info about content presence */
- unsigned long flags;
-#define CACHEFILES_OBJECT_USING_TMPFILE 0 /* Have an unlinked tmpfile */
+ enum cachefiles_content content_info; /* Info about content presence */
};
/*
diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c
index 7de8069d15b6..b33ad1d21d44 100644
--- a/fs/cachefiles/io.c
+++ b/fs/cachefiles/io.c
@@ -32,6 +32,8 @@ struct cachefiles_kiocb {
u64 b_writing;
};
+#define IS_ERR_VALUE_LL(x) unlikely((x) >= (unsigned long long)-MAX_ERRNO)
+
static inline void cachefiles_put_kiocb(struct cachefiles_kiocb *ki)
{
if (refcount_dec_and_test(&ki->ki_refcnt)) {
@@ -193,60 +195,81 @@ static int cachefiles_read(struct netfs_cache_resources *cres,
}
/*
- * Query the occupancy of the cache in a region, returning where the next chunk
- * of data starts and how long it is.
+ * Query the occupancy of the cache in a region, returning the extent of the
+ * next two chunks of cached data and the next hole.
*/
static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
- uoff_t start, size_t len, size_t granularity,
- uoff_t *_data_start, size_t *_data_len)
+ struct fscache_occupancy *occ)
{
struct cachefiles_object *object;
+ struct inode *inode;
struct file *file;
- loff_t off, off2;
-
- *_data_start = -1;
- *_data_len = 0;
+ uoff_t read_limit;
+ loff_t ret;
+ int i;
if (!fscache_wait_for_operation(cres, FSCACHE_WANT_READ))
return -ENOBUFS;
object = cachefiles_cres_object(cres);
file = cachefiles_cres_file(cres);
- granularity = max_t(size_t, object->volume->cache->bsize, granularity);
+ inode = file_inode(file);
+ occ->granularity = object->volume->cache->bsize;
+ /* Read read_limit before content_info. */
+ read_limit = atomic64_read_acquire(&object->read_limit);
+
+ _enter("%pD,%llu,%llx-%llx/%llx",
+ file, inode->i_ino, occ->query_from, occ->query_to, read_limit);
+
+ if (read_limit == 0)
+ goto done;
+
+ switch (READ_ONCE(object->content_info)) {
+ case CACHEFILES_CONTENT_ALL:
+ case CACHEFILES_CONTENT_SINGLE:
+ if (read_limit > occ->query_from) {
+ occ->cached_from[0] = 0;
+ occ->cached_to[0] = read_limit;
+ occ->cached_type[0] = FSCACHE_EXTENT_DATA;
+ occ->query_from = ULLONG_MAX;
+ }
+ goto done;
+ default:
+ break;
+ }
- _enter("%pD,%llu,%llx,%zx/%llx",
- file, file_inode(file)->i_ino, start, len,
- i_size_read(file_inode(file)));
+ for (i = 0; i < ARRAY_SIZE(occ->cached_from); i++) {
+ ret = cachefiles_inject_read_error();
+ if (ret == 0)
+ ret = vfs_llseek(file, occ->query_from, SEEK_DATA);
+ if (IS_ERR_VALUE_LL(ret)) {
+ if (ret != -ENXIO)
+ return ret;
+ occ->query_from = ULLONG_MAX;
+ goto done;
+ }
+ occ->cached_type[i] = FSCACHE_EXTENT_DATA;
+ occ->cached_from[i] = ret;
+ occ->query_from = ret;
+
+ ret = cachefiles_inject_read_error();
+ if (ret == 0)
+ ret = vfs_llseek(file, occ->query_from, SEEK_HOLE);
+ if (IS_ERR_VALUE_LL(ret)) {
+ if (ret != -ENXIO)
+ return ret;
+ occ->query_from = ULLONG_MAX;
+ goto done;
+ }
+ occ->cached_to[i] = ret;
+ occ->query_from = ret;
+ if (occ->query_from >= occ->query_to)
+ break;
+ }
- off = cachefiles_inject_read_error();
- if (off == 0)
- off = vfs_llseek(file, start, SEEK_DATA);
- if (off == -ENXIO)
- return -ENODATA; /* Beyond EOF */
- if (off < 0 && off >= (loff_t)-MAX_ERRNO)
- return -ENOBUFS; /* Error. */
- if (round_up(off, granularity) >= start + len)
- return -ENODATA; /* No data in range */
-
- off2 = cachefiles_inject_read_error();
- if (off2 == 0)
- off2 = vfs_llseek(file, off, SEEK_HOLE);
- if (off2 == -ENXIO)
- return -ENODATA; /* Beyond EOF */
- if (off2 < 0 && off2 >= (loff_t)-MAX_ERRNO)
- return -ENOBUFS; /* Error. */
-
- /* Round away partial blocks */
- off = round_up(off, granularity);
- off2 = round_down(off2, granularity);
- if (off2 <= off)
- return -ENODATA;
-
- *_data_start = off;
- if (off2 > start + len)
- *_data_len = len;
- else
- *_data_len = off2 - off;
+done:
+ _debug("query[0] %llx-%llx", occ->cached_from[0], occ->cached_to[0]);
+ _debug("query[1] %llx-%llx", occ->cached_from[1], occ->cached_to[1]);
return 0;
}
@@ -377,7 +400,7 @@ static int cachefiles_write(struct netfs_cache_resources *cres,
static inline enum netfs_io_source
cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
- uoff_t start, size_t *_len, loff_t i_size,
+ uoff_t start, size_t *_len, uoff_t i_size,
unsigned long *_flags, ino_t netfs_ino)
{
enum cachefiles_prepare_read_trace why;
@@ -478,18 +501,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
return ret;
}
-/*
- * Prepare a read operation, shortening it to a cached/uncached
- * boundary as appropriate.
- */
-static enum netfs_io_source cachefiles_prepare_read(struct netfs_io_subrequest *subreq,
- uoff_t i_size)
-{
- return cachefiles_do_prepare_read(&subreq->rreq->cache_resources,
- subreq->start, &subreq->len, i_size,
- &subreq->flags, subreq->rreq->inode->i_ino);
-}
-
/*
* Prepare for a write to occur.
*/
@@ -504,7 +515,7 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
int ret;
/* Round to DIO size */
- start = round_down(*_start, PAGE_SIZE);
+ start = round_down(*_start, cache->bsize);
if (start != *_start || *_len > upper_len) {
/* Probably asked to cache a streaming write written into the
* pagecache when the cookie was temporarily out of service to
@@ -514,7 +525,7 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
return -ENOBUFS;
}
- *_len = round_up(len, PAGE_SIZE);
+ *_len = round_up(len, cache->bsize);
/* We need to work out whether there's sufficient disk space to perform
* the write - but we can skip that check if we have space already
@@ -540,7 +551,7 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
* space, we need to see if it's fully allocated. If it's not, we may
* want to cull it.
*/
- if (cachefiles_has_space(cache, 0, *_len / PAGE_SIZE,
+ if (cachefiles_has_space(cache, 0, *_len / cache->bsize,
cachefiles_has_space_check) == 0)
return 0; /* Enough space to simply overwrite the whole block */
@@ -572,7 +583,7 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
return ret;
check_space:
- return cachefiles_has_space(cache, 0, *_len / PAGE_SIZE,
+ return cachefiles_has_space(cache, 0, *_len / cache->bsize,
cachefiles_has_space_for_write);
}
@@ -635,9 +646,9 @@ static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)
wreq->debug_id, subreq->debug_index, start, start + len - 1);
/* We need to start on the cache granularity boundary */
- off = start & (CACHEFILES_DIO_BLOCK_SIZE - 1);
+ off = start & (cache->bsize - 1);
if (off) {
- pre = CACHEFILES_DIO_BLOCK_SIZE - off;
+ pre = cache->bsize - off;
if (pre >= len) {
fscache_count_dio_misfit();
netfs_write_subrequest_terminated(subreq, len);
@@ -651,8 +662,8 @@ static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)
/* We also need to end on the cache granularity boundary */
if (start + len == wreq->i_size) {
- size_t part = len % CACHEFILES_DIO_BLOCK_SIZE;
- size_t need = CACHEFILES_DIO_BLOCK_SIZE - part;
+ size_t part = len & (cache->bsize - 1);
+ size_t need = cache->bsize - part;
if (part && stream->submit_extendable_to >= need) {
len += need;
@@ -661,7 +672,7 @@ static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)
}
}
- post = len & (CACHEFILES_DIO_BLOCK_SIZE - 1);
+ post = len & (cache->bsize - 1);
if (post) {
len -= post;
if (len == 0) {
@@ -688,6 +699,198 @@ static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)
netfs_write_subrequest_terminated, subreq);
}
+/*
+ * Collect the result of buffered writeback to the cache. This includes
+ * copying a read to the cache. Netfslib collates the results, which might
+ * occur out of order, and delivers them to the cache so that it can update its
+ * content record.
+ *
+ * block_type is one of:
+ * - NETFS_CACHE_COLLECT_WRITE_DATA for a contiguous block of data
+ * - NETFS_CACHE_COLLECT_WRITE_GAP if a discontiguity was skipped
+ * - NETFS_CACHE_COLLECT_WRITE_CANCEL for a hole due to a failed/cancelled write
+ *
+ * The writes we made are all rounded out at both sides to the nearest DIO
+ * block boundary, so if the final block contains the EOF in the middle of it
+ * (rather than at the end), padding will have been written to the file. The
+ * backing file's filesize will have been updated if the write extended the
+ * file; the filesize may still change due to outstanding subreqs.
+ *
+ * The metadata in the cache file xattr records the size of the object we have
+ * stored, but the cache file EOF only goes up to where we've cached data to
+ * and, furthermore, is rounded up to the nearest DIO block boundary.
+ *
+ * Concurrent updates should be protected against by the caller. Netfslib
+ * holds NETFS_ICTX_WB_LOCK as a lock on writeback requests. DIO writes
+ * invalidate the cookie and caching is kept disabled until all users have
+ * unused the cookie.
+ */
+static void cachefiles_collect_write(struct netfs_io_request *wreq,
+ uoff_t start, size_t len,
+ enum netfs_cache_collect block_type)
+{
+ struct netfs_cache_resources *cres = &wreq->cache_resources;
+ struct cachefiles_object *object = cachefiles_cres_object(cres);
+ struct cachefiles_cache *cache = object->volume->cache;
+ struct inode *inode;
+ struct file *file = cachefiles_cres_file(cres);
+ uoff_t read_limit;
+ uoff_t old_size = cres->cache_i_size;
+ uoff_t new_size;
+ uoff_t data_to = object->object_size;
+ uoff_t end = start + len;
+ int ret;
+
+ if (!file)
+ return;
+
+ inode = file_inode(file);
+ new_size = i_size_read(inode);
+
+ _enter("%llx,%zx,%x", start, len, cache->bsize);
+
+ if (WARN_ON(old_size & (cache->bsize - 1)) ||
+ WARN_ON(new_size & (cache->bsize - 1)) ||
+ WARN_ON(start & (cache->bsize - 1)) ||
+ WARN_ON(len & (cache->bsize - 1))) {
+ trace_cachefiles_io_error(object, inode, -EIO,
+ cachefiles_trace_alignment_error);
+ cachefiles_remove_object_xattr(cache, object, file->f_path.dentry);
+ return;
+ }
+
+ /* If this is recording a gap, due to discontiguous writes or lack of
+ * cache space, then a hole may have been introduced into the backing
+ * file. Treat it as a zero-length data block.
+ */
+ if (block_type == NETFS_CACHE_COLLECT_WRITE_GAP ||
+ block_type == NETFS_CACHE_COLLECT_WRITE_CANCEL) {
+ start = end;
+ len = 0;
+ }
+
+ /* Zeroth case: Single monolithic files are handled specially.
+ */
+ if (wreq->origin == NETFS_WRITEBACK_SINGLE) {
+ if (block_type == NETFS_CACHE_COLLECT_WRITE_GAP ||
+ block_type == NETFS_CACHE_COLLECT_WRITE_CANCEL) {
+ trace_cachefiles_trunc(object, inode, data_to, 0,
+ cachefiles_trunc_zap);
+ ret = cachefiles_inject_remove_error();
+ if (ret == 0)
+ ret = vfs_truncate(&file->f_path, 0);
+ if (ret < 0) {
+ trace_cachefiles_io_error(object, inode, ret,
+ cachefiles_trace_trunc_error);
+ cachefiles_io_error_obj(object, "truncate failed %d", ret);
+ cachefiles_remove_object_xattr(cache, object, file->f_path.dentry);
+ return;
+ }
+
+ object->content_info = CACHEFILES_CONTENT_NO_DATA;
+ read_limit = 0;
+ } else {
+ object->content_info = CACHEFILES_CONTENT_SINGLE;
+ read_limit = len;
+ }
+ goto update_sizes_2;
+ }
+
+ /* First case: The backing file was empty. */
+ if (old_size == 0) {
+ if (start == 0)
+ object->content_info = CACHEFILES_CONTENT_ALL;
+ else
+ object->content_info = CACHEFILES_CONTENT_BACKFS_MAP;
+ goto update_sizes;
+ }
+
+ /* Second case: The backing file is entirely within the old object size
+ * and thus there can be no partial tail block to deal with in the
+ * cache file.
+ */
+ if (old_size <= data_to) {
+ if (start > old_size)
+ goto discontiguous;
+ goto update_sizes;
+ }
+
+ /* Third case: The write happened entirely within the bounds of the
+ * current cache file's size.
+ */
+ if (end <= old_size)
+ goto update_sizes;
+
+ /* Fourth case: The write overwrote the partial tail block and extended
+ * the file. We only need to update the object size because netfslib
+ * rounds out/pads cache writes to whole disk blocks.
+ */
+ if (start < old_size)
+ goto update_sizes;
+
+ /* Fifth case: The write started from the end of the whole tail block
+ * and extended the file. Just extend our notion of the filesize.
+ */
+ if (start == old_size && old_size == data_to)
+ goto update_sizes;
+
+ /* Sixth case: The write continued on from the partial tail block and
+ * extended the file. Need to clear the gap.
+ */
+ if (start == old_size && old_size > data_to)
+ goto clear_gap;
+
+discontiguous:
+ /* Seventh case: The write was beyond the EOF on the cache file, so now
+ * there's a hole in the file and we can no longer say in the metadata
+ * that we can assume we have it all. We may also need to clear the
+ * end of the partial tail block.
+ */
+ /* TODO: For the moment, we will have to use SEEK_HOLE/SEEK_DATA. */
+ if (object->content_info != CACHEFILES_CONTENT_BACKFS_MAP) {
+ object->content_info = CACHEFILES_CONTENT_BACKFS_MAP;
+ trace_cachefiles_coherency(object, inode->i_ino, data_to, NULL,
+ CACHEFILES_CONTENT_BACKFS_MAP,
+ cachefiles_coherency_discontiguous);
+ }
+
+clear_gap:
+ /* We need to clear any partial padding that got jumped over. It
+ * *should* be all zeros, but shared-writable mmap exists...
+ */
+ if (old_size > data_to) {
+ trace_cachefiles_trunc(object, inode, data_to, old_size,
+ cachefiles_trunc_clear_padding);
+ ret = cachefiles_inject_write_error();
+ if (ret == 0)
+ ret = vfs_fallocate(file, FALLOC_FL_ZERO_RANGE,
+ data_to, old_size - data_to);
+ if (ret < 0) {
+ trace_cachefiles_io_error(object, inode, ret,
+ cachefiles_trace_fallocate_error);
+ cachefiles_io_error_obj(object, "fallocate zero pad failed %d", ret);
+ cachefiles_remove_object_xattr(cache, object, file->f_path.dentry);
+ return;
+ }
+ }
+
+update_sizes:
+ read_limit = umax(old_size, end);
+update_sizes_2:
+ cres->cache_i_size = read_limit;
+
+ /* We need to be careful setting the object_size: we may have written
+ * more to the cache than to the server (due to cache DIO rounding) and
+ * the i_size set on the netfs inode may include unwritten data that
+ * the server doesn't know about yet.
+ */
+ object->object_size = umin(read_limit, wreq->i_size);
+
+ /* Raise the limit at which reads can access the file. */
+ /* Update read_limit after content_info */
+ atomic64_set_release(&object->read_limit, read_limit);
+}
+
/*
* Clean up an operation.
*/
@@ -705,10 +908,10 @@ static const struct netfs_cache_ops cachefiles_netfs_cache_ops = {
.read = cachefiles_read,
.write = cachefiles_write,
.issue_write = cachefiles_issue_write,
- .prepare_read = cachefiles_prepare_read,
.prepare_write = cachefiles_prepare_write,
.prepare_write_subreq = cachefiles_prepare_write_subreq,
.query_occupancy = cachefiles_query_occupancy,
+ .collect_write = cachefiles_collect_write,
};
/*
@@ -718,13 +921,19 @@ bool cachefiles_begin_operation(struct netfs_cache_resources *cres,
enum fscache_want_state want_state)
{
struct cachefiles_object *object = cachefiles_cres_object(cres);
+ struct file *file;
+
+ cres->dio_size = object->volume->cache->bsize;
if (!cachefiles_cres_file(cres)) {
cres->ops = &cachefiles_netfs_cache_ops;
if (object->file) {
spin_lock(&object->lock);
- if (!cres->cache_priv2 && object->file)
- cres->cache_priv2 = get_file(object->file);
+ file = object->file;
+ if (!cres->cache_priv2 && file) {
+ cres->cache_priv2 = get_file(file);
+ cres->cache_i_size = i_size_read(file_inode(file));
+ }
spin_unlock(&object->lock);
}
}
diff --git a/fs/cachefiles/namei.c b/fs/cachefiles/namei.c
index 88955249a1a6..a30df0f91e5e 100644
--- a/fs/cachefiles/namei.c
+++ b/fs/cachefiles/namei.c
@@ -414,7 +414,6 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)
struct dentry *fan = volume->fanout[(u8)object->cookie->key_hash];
struct file *file;
const struct path parentpath = { .mnt = cache->mnt, .dentry = fan };
- uint64_t ni_size;
long ret;
@@ -442,23 +441,6 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)
if (!cachefiles_mark_inode_in_use(object, file_inode(file)))
WARN_ON(1);
- ni_size = object->cookie->object_size;
- ni_size = round_up(ni_size, CACHEFILES_DIO_BLOCK_SIZE);
-
- if (ni_size > 0) {
- trace_cachefiles_trunc(object, file_inode(file), 0, ni_size,
- cachefiles_trunc_expand_tmpfile);
- ret = cachefiles_inject_write_error();
- if (ret == 0)
- ret = vfs_truncate(&file->f_path, ni_size);
- if (ret < 0) {
- trace_cachefiles_vfs_error(
- object, file_inode(file), ret,
- cachefiles_trace_trunc_error);
- goto err_unuse;
- }
- }
-
ret = -EINVAL;
if (unlikely(!file->f_op->read_iter) ||
unlikely(!file->f_op->write_iter)) {
@@ -467,6 +449,7 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)
}
out:
cachefiles_end_secure(cache, saved_cred);
+ object->content_info = CACHEFILES_CONTENT_ALL;
return file;
err_unuse:
diff --git a/fs/cachefiles/xattr.c b/fs/cachefiles/xattr.c
index 92990cfdcca7..5c1794beca93 100644
--- a/fs/cachefiles/xattr.c
+++ b/fs/cachefiles/xattr.c
@@ -43,6 +43,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
struct dentry *dentry;
struct file *file = object->file;
unsigned int len = object->cookie->aux_len;
+ uoff_t object_size = object->cookie->object_size;
int ret;
if (!file)
@@ -55,7 +56,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
if (!buf)
return -ENOMEM;
- buf->object_size = cpu_to_be64(object->cookie->object_size);
+ buf->object_size = cpu_to_be64(object_size);
buf->zero_point = 0;
buf->type = CACHEFILES_COOKIE_TYPE_DATA;
buf->content = object->content_info;
@@ -79,7 +80,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
trace_cachefiles_vfs_error(object, file_inode(file), ret,
cachefiles_trace_setxattr_error);
trace_cachefiles_coherency(object, file_inode(file)->i_ino,
- buf->data, buf->content,
+ object_size, buf->data, buf->content,
cachefiles_coherency_set_fail);
if (ret != -ENOMEM)
cachefiles_io_error_obj(
@@ -87,7 +88,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
"Failed to set xattr with error %d", ret);
} else {
trace_cachefiles_coherency(object, file_inode(file)->i_ino,
- buf->data, buf->content,
+ object_size, buf->data, buf->content,
cachefiles_coherency_set_ok);
}
@@ -103,10 +104,12 @@ int cachefiles_check_auxdata(struct cachefiles_object *object, struct file *file
{
struct cachefiles_xattr *buf;
struct dentry *dentry = file->f_path.dentry;
+ struct inode *inode = file_inode(file);
unsigned int len = object->cookie->aux_len, tlen;
const void *p = fscache_get_aux(object->cookie);
enum cachefiles_coherency_trace why;
ssize_t xlen;
+ uoff_t obj_size;
int ret = -ESTALE;
tlen = sizeof(struct cachefiles_xattr) + len;
@@ -121,34 +124,39 @@ int cachefiles_check_auxdata(struct cachefiles_object *object, struct file *file
if (xlen != tlen) {
if (xlen < 0) {
ret = xlen;
- trace_cachefiles_vfs_error(object, file_inode(file), xlen,
+ trace_cachefiles_vfs_error(object, inode, xlen,
cachefiles_trace_getxattr_error);
}
if (xlen == -EIO)
cachefiles_io_error_obj(
object,
"Failed to read aux with error %zd", xlen);
+ obj_size = 0;
why = cachefiles_coherency_check_xattr;
goto out;
}
+ obj_size = be64_to_cpu(buf->object_size);
if (buf->type != CACHEFILES_COOKIE_TYPE_DATA) {
why = cachefiles_coherency_check_type;
} else if (memcmp(buf->data, p, len) != 0) {
why = cachefiles_coherency_check_aux;
- } else if (be64_to_cpu(buf->object_size) != object->cookie->object_size) {
+ } else if (obj_size != object->cookie->object_size) {
why = cachefiles_coherency_check_objsize;
} else if (buf->content == CACHEFILES_CONTENT_DIRTY) {
// TODO: Begin conflict resolution
pr_warn("Dirty object in cache\n");
why = cachefiles_coherency_check_dirty;
} else {
+ object->content_info = buf->content;
+ object->object_size = obj_size;
+ atomic64_set(&object->read_limit, i_size_read(inode));
why = cachefiles_coherency_check_ok;
ret = 0;
}
out:
- trace_cachefiles_coherency(object, file_inode(file)->i_ino,
+ trace_cachefiles_coherency(object, inode->i_ino, obj_size,
buf->data, buf->content, why);
kfree(buf);
return ret;
@@ -163,6 +171,9 @@ int cachefiles_remove_object_xattr(struct cachefiles_cache *cache,
{
int ret;
+ trace_cachefiles_coherency(object, d_inode(dentry)->i_ino, 0, NULL, 0,
+ cachefiles_coherency_remove);
+
ret = cachefiles_inject_remove_error();
if (ret == 0) {
ret = mnt_want_write(cache->mnt);
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 03e095060162..5af849b2f283 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -137,21 +137,6 @@ static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq)
return subreq->len;
}
-static enum netfs_io_source netfs_cache_prepare_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq,
- uoff_t i_size)
-{
- struct netfs_cache_resources *cres = &rreq->cache_resources;
- enum netfs_io_source source;
-
- if (!cres->ops)
- return NETFS_DOWNLOAD_FROM_SERVER;
- source = cres->ops->prepare_read(subreq, i_size);
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
- return source;
-
-}
-
/*
* Issue a read against the cache.
* - Eats the caller's ref on subreq.
@@ -166,6 +151,19 @@ static void netfs_read_cache_to_pagecache(struct netfs_io_request *rreq,
netfs_cache_read_terminated, subreq);
}
+int netfs_read_query_cache(struct netfs_io_request *rreq, struct fscache_occupancy *occ)
+{
+ struct netfs_cache_resources *cres = &rreq->cache_resources;
+
+ occ->granularity = PAGE_SIZE;
+ if (occ->query_from >= occ->query_to)
+ return 0;
+ if (!cres->ops)
+ return 0;
+ occ->query_from = round_up(occ->query_from, occ->granularity);
+ return cres->ops->query_occupancy(cres, occ);
+}
+
void netfs_queue_read(struct netfs_io_request *rreq,
struct netfs_io_subrequest *subreq)
{
@@ -218,14 +216,53 @@ static void netfs_issue_read(struct netfs_io_request *rreq,
*/
static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
{
+ struct fscache_occupancy _occ = {
+ .query_from = rreq->start,
+ .query_to = rreq->start + rreq->len,
+ .cached_from[0] = 0,
+ .cached_to[0] = 0,
+ .cached_from[1] = ULLONG_MAX,
+ .cached_to[1] = ULLONG_MAX,
+ };
+ struct fscache_occupancy *occ = &_occ;
ssize_t size = rreq->len;
uoff_t start = rreq->start;
int ret = 0;
do {
+ int (*prepare_read)(struct netfs_io_subrequest *subreq) = NULL;
struct netfs_io_subrequest *subreq;
- enum netfs_io_source source = NETFS_SOURCE_UNKNOWN;
ssize_t slice;
+ uoff_t hole_to, cache_to;
+
+ /* If we don't have any, find out the next couple of data
+ * extents from the cache, containing of following the
+ * specified start offset. Holes have to be fetched from the
+ * server; data regions from the cache.
+ */
+ hole_to = occ->cached_from[0];
+ cache_to = occ->cached_to[0];
+ if (start >= cache_to) {
+ /* Extent exhausted; shuffle down. */
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(occ->cached_from) - 1; i++) {
+ occ->cached_from[i] = occ->cached_from[i + 1];
+ occ->cached_to[i] = occ->cached_to[i + 1];
+ occ->cached_type[i] = occ->cached_type[i + 1];
+ }
+ occ->cached_from[i] = ULLONG_MAX;
+ occ->cached_to[i] = ULLONG_MAX;
+
+ if (occ->cached_from[0] != ULLONG_MAX)
+ continue;
+
+ /* Get new extents */
+ ret = netfs_read_query_cache(rreq, occ);
+ if (ret < 0)
+ break;
+ continue;
+ }
subreq = netfs_alloc_subrequest(rreq);
if (!subreq) {
@@ -238,63 +275,75 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
netfs_queue_read(rreq, subreq);
- source = netfs_cache_prepare_read(rreq, subreq, rreq->i_size);
- subreq->source = source;
- if (source == NETFS_DOWNLOAD_FROM_SERVER) {
- uoff_t zero_point = netfs_read_zero_point(rreq->inode);
- uoff_t zp = umin(zero_point, rreq->i_size);
- size_t len = subreq->len;
-
- if (unlikely(rreq->origin == NETFS_READ_SINGLE))
- zp = rreq->i_size;
- if (subreq->start >= zp) {
- subreq->source = source = NETFS_FILL_WITH_ZEROES;
- goto fill_with_zeroes;
+ uoff_t zero_point = netfs_read_zero_point(rreq->inode);
+ uoff_t zlimit = umin(zero_point, rreq->i_size);
+
+ _debug("rsub %llx %llx-%llx", subreq->start, hole_to, cache_to);
+
+ if (start >= hole_to && start < cache_to) {
+ /* Overlap with a cached region, where the cache may
+ * record a block of zeroes.
+ */
+ _debug("cached s=%llx c=%llx l=%zx", start, cache_to, size);
+ subreq->len = umin(cache_to - start, size);
+ subreq->len = round_up(subreq->len, occ->granularity);
+ if (occ->cached_type[0] == FSCACHE_EXTENT_ZERO) {
+ subreq->source = NETFS_FILL_WITH_ZEROES;
+ netfs_stat(&netfs_n_rh_zero);
+ } else {
+ subreq->source = NETFS_READ_FROM_CACHE;
+ prepare_read = rreq->cache_resources.ops->prepare_read;
}
- if (len > zp - subreq->start)
- len = zp - subreq->start;
- if (len == 0) {
- pr_err("ZERO-LEN READ: R=%08x[%x] l=%zx/%zx s=%llx z=%llx i=%llx",
- rreq->debug_id, subreq->debug_index,
- subreq->len, size,
- subreq->start, zero_point, rreq->i_size);
- netfs_cancel_read(subreq, ret);
- break;
- }
- subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
- netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read) {
- ret = rreq->netfs_ops->prepare_read(subreq);
- if (ret < 0) {
- netfs_cancel_read(subreq, ret);
- break;
- }
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
- }
- goto issue;
- }
-
- fill_with_zeroes:
- if (source == NETFS_FILL_WITH_ZEROES) {
+ } else if (subreq->start >= zlimit && size > 0) {
+ /* If this range lies beyond the zero-point, that part
+ * can just be cleared locally.
+ */
+ _debug("zero %llx-%llx", start, start + size);
+ subreq->len = size;
subreq->source = NETFS_FILL_WITH_ZEROES;
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+ if (rreq->cache_resources.ops)
+ __set_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags);
netfs_stat(&netfs_n_rh_zero);
- goto issue;
+ } else {
+ /* Read a cache hole from the server. If any part of
+ * this range lies beyond the zero-point or the EOF,
+ * that part can just be cleared locally.
+ */
+ uoff_t limit = min3(zlimit, start + size, hole_to);
+
+ _debug("limit %llx %llx", rreq->i_size, zero_point);
+ _debug("download %llx-%llx", start, start + size);
+ subreq->len = umin(limit - subreq->start, ULONG_MAX);
+ subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
+ if (rreq->cache_resources.ops)
+ __set_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags);
+ netfs_stat(&netfs_n_rh_download);
}
- if (source == NETFS_READ_FROM_CACHE) {
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- goto issue;
+ if (size == 0) {
+ pr_err("ZERO-LEN READ: R=%08x[%x] l=%zx/%zx s=%llx z=%llx i=%llx",
+ rreq->debug_id, subreq->debug_index,
+ subreq->len, size,
+ subreq->start, zero_point, rreq->i_size);
+ netfs_cancel_read(subreq, ret);
+ break;
}
- pr_err("Unexpected read source %u\n", source);
- WARN_ON_ONCE(1);
- netfs_cancel_read(subreq, ret);
- break;
+ rreq->io_streams[0].sreq_max_len = MAX_RW_COUNT;
+ rreq->io_streams[0].sreq_max_segs = INT_MAX;
+
+ if (prepare_read) {
+ ret = prepare_read(subreq);
+ if (ret < 0) {
+ netfs_cancel_read(subreq, ret);
+ break;
+ }
+ trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
+ }
- issue:
slice = netfs_prepare_read_iterator(subreq);
if (slice < 0) {
ret = slice;
@@ -306,6 +355,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
if (size <= 0)
netfs_all_subreqs_queued(rreq);
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
netfs_issue_read(rreq, subreq);
netfs_maybe_bulk_drop_ra_refs(rreq);
diff --git a/fs/netfs/buffered_write.c b/fs/netfs/buffered_write.c
index ead22980075f..49b47252f675 100644
--- a/fs/netfs/buffered_write.c
+++ b/fs/netfs/buffered_write.c
@@ -54,9 +54,6 @@ void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode,
i_size = i_size_read(inode);
if (end > i_size) {
i_size_write(inode, end);
-#if IS_ENABLED(CONFIG_FSCACHE)
- fscache_update_cookie(ctx->cache, NULL, &end);
-#endif
gap = SECTOR_SIZE - (i_size & (SECTOR_SIZE - 1));
if (copied > gap) {
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index c50750b72262..5c05b0dcb143 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -23,6 +23,8 @@
/*
* buffered_read.c
*/
+int netfs_read_query_cache(struct netfs_io_request *rreq,
+ struct fscache_occupancy *occ);
void netfs_queue_read(struct netfs_io_request *rreq,
struct netfs_io_subrequest *subreq);
void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error);
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index 3460aa1c4af1..13816a02a798 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -44,6 +44,7 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
rreq->gfp = gfp;
rreq->start = start;
rreq->collected_to = start;
+ rreq->cache_coll_to = start;
rreq->cleaned_to = start;
rreq->len = len;
rreq->progress_at = 0;
diff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c
index 61dbd644d6bb..96223f62516a 100644
--- a/fs/netfs/read_retry.c
+++ b/fs/netfs/read_retry.c
@@ -272,6 +272,7 @@ void netfs_retry_reads(struct netfs_io_request *rreq)
struct netfs_io_stream *stream = &rreq->io_streams[0];
netfs_stat(&netfs_n_rh_retry_read_req);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_retry_begin);
/* Wait for all outstanding I/O to quiesce before performing retries as
* we may need to renegotiate the I/O sizes.
@@ -282,6 +283,7 @@ void netfs_retry_reads(struct netfs_io_request *rreq)
trace_netfs_rreq(rreq, netfs_rreq_trace_resubmit);
netfs_retry_read_subrequests(rreq);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_retry_end);
}
/*
diff --git a/fs/netfs/read_single.c b/fs/netfs/read_single.c
index ccb5fc809d99..b370a8534324 100644
--- a/fs/netfs/read_single.c
+++ b/fs/netfs/read_single.c
@@ -58,20 +58,6 @@ static int netfs_single_begin_cache_read(struct netfs_io_request *rreq, struct n
return fscache_begin_read_operation(&rreq->cache_resources, netfs_i_cookie(ctx));
}
-static void netfs_single_cache_prepare_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
-{
- struct netfs_cache_resources *cres = &rreq->cache_resources;
-
- if (!cres->ops) {
- subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- return;
- }
- subreq->source = cres->ops->prepare_read(subreq, rreq->i_size);
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
-
-}
-
static void netfs_single_read_cache(struct netfs_io_request *rreq,
struct netfs_io_subrequest *subreq)
{
@@ -89,6 +75,14 @@ static void netfs_single_read_cache(struct netfs_io_request *rreq,
*/
static int netfs_single_dispatch_read(struct netfs_io_request *rreq)
{
+ struct fscache_occupancy occ = {
+ .query_from = 0,
+ .query_to = rreq->len,
+ .cached_from[0] = ULLONG_MAX,
+ .cached_to[0] = ULLONG_MAX,
+ .cached_from[1] = ULLONG_MAX,
+ .cached_to[1] = ULLONG_MAX,
+ };
struct netfs_io_subrequest *subreq;
int ret = 0;
@@ -96,14 +90,21 @@ static int netfs_single_dispatch_read(struct netfs_io_request *rreq)
if (!subreq)
return -ENOMEM;
- subreq->source = NETFS_SOURCE_UNKNOWN;
+ subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
subreq->start = 0;
subreq->len = rreq->len;
subreq->io_iter = rreq->buffer.iter;
netfs_queue_read(rreq, subreq);
- netfs_single_cache_prepare_read(rreq, subreq);
+ /* Try to use the cache if the cache content matches the size of the
+ * remote file.
+ */
+ netfs_read_query_cache(rreq, &occ);
+ if (occ.cached_from[0] == 0 &&
+ occ.cached_to[0] >= rreq->len)
+ subreq->source = NETFS_READ_FROM_CACHE;
+
switch (subreq->source) {
case NETFS_DOWNLOAD_FROM_SERVER:
netfs_stat(&netfs_n_rh_download);
@@ -118,6 +119,12 @@ static int netfs_single_dispatch_read(struct netfs_io_request *rreq)
rreq->submitted += subreq->len;
break;
case NETFS_READ_FROM_CACHE:
+ if (rreq->cache_resources.ops->prepare_read) {
+ ret = rreq->cache_resources.ops->prepare_read(subreq);
+ if (ret < 0)
+ goto cancel;
+ }
+
netfs_all_subreqs_queued(rreq);
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
netfs_single_read_cache(rreq, subreq);
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index 6d99d4a6f780..6e8ea534230d 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -188,6 +188,26 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
wreq->buffer.first_tail_slot = slot;
}
+/*
+ * Collect cache results.
+ */
+static void netfs_cache_collect(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ enum netfs_cache_collect block_type)
+{
+ struct netfs_cache_resources *cres = &wreq->cache_resources;
+
+ if (stream->source != NETFS_WRITE_TO_CACHE ||
+ wreq->cache_coll_to >= stream->collected_to)
+ return;
+
+ if (cres->ops && cres->ops->collect_write)
+ cres->ops->collect_write(wreq, wreq->cache_coll_to,
+ stream->collected_to - wreq->cache_coll_to,
+ block_type);
+ wreq->cache_coll_to = stream->collected_to;
+}
+
/*
* Collect and assess the results of various write subrequests. We may need to
* retry some of the results - or even do an RMW cycle for content crypto.
@@ -235,13 +255,19 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
/* Read first subreq pointer before IN_PROGRESS flag. */
while (front) {
+ enum netfs_cache_collect cache_collect;
+
trace_netfs_collect_sreq(wreq, front);
//_debug("sreq [%x] %llx %zx/%zx",
// front->debug_index, front->start, front->transferred, front->len);
if (stream->collected_to < front->start) {
trace_netfs_collect_gap(wreq, stream, issued_to, 'F');
+ if (stream->cache_collect != NETFS_CACHE_COLLECT_WRITE_GAP)
+ netfs_cache_collect(wreq, stream, stream->cache_collect);
stream->collected_to = front->start;
+ netfs_cache_collect(wreq, stream, NETFS_CACHE_COLLECT_WRITE_GAP);
+ stream->cache_collect = NETFS_CACHE_COLLECT_WRITE_GAP;
}
/* Stall if the front is still undergoing I/O. */
@@ -249,7 +275,6 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
notes |= HIT_PENDING;
break;
}
- smp_rmb(); /* Read counters after I-P flag. */
if (stream->failed) {
stream->collected_to = front->start + front->len;
@@ -262,15 +287,44 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
stream->transferred_valid = true;
notes |= MADE_PROGRESS;
}
- if (test_bit(NETFS_SREQ_FAILED, &front->flags)) {
- stream->failed = true;
- stream->error = front->error;
- if (stream->source == NETFS_UPLOAD_TO_SERVER)
- mapping_set_error(wreq->mapping, front->error);
- notes |= NEED_REASSESS | SAW_FAILURE;
+
+ /* Handle failed or cancelled subreqs. Failure of
+ * cache writes are handled differently to upload
+ * failures. Cache writes aren't fatal, provided we're
+ * not doing disconnected operation, and so we can kind
+ * of treat them as if they had succeeded - except that
+ * we need to log any holes they cause.
+ */
+ switch (stream->source) {
+ case NETFS_UPLOAD_TO_SERVER:
+ if (test_bit(NETFS_SREQ_FAILED, &front->flags)) {
+ if (!stream->failed) {
+ stream->failed = true;
+ stream->error = front->error;
+ mapping_set_error(wreq->mapping, front->error);
+ break;
+ }
+ notes |= NEED_REASSESS | SAW_FAILURE;
+ }
+ break;
+
+ case NETFS_WRITE_TO_CACHE:
+ cache_collect = test_bit(NETFS_SREQ_CANCELLED, &front->flags) ?
+ NETFS_CACHE_COLLECT_WRITE_CANCEL :
+ NETFS_CACHE_COLLECT_WRITE_DATA;
+ if (cache_collect != stream->cache_collect &&
+ stream->cache_collect != NETFS_CACHE_COLLECT_WRITE_GAP) {
+ trace_netfs_rreq(wreq, netfs_rreq_trace_cache_fail_collect);
+ netfs_cache_collect(wreq, stream, stream->cache_collect);
+ }
+ stream->cache_collect = cache_collect;
+ break;
+
+ default:
+ WARN_ON(1);
break;
}
- if (front->transferred < front->len) {
+ if (test_bit(NETFS_SREQ_NEED_RETRY, &front->flags)) {
stream->need_retry = true;
notes |= NEED_RETRY | MADE_PROGRESS;
break;
@@ -359,6 +413,7 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
*/
bool netfs_write_collection(struct netfs_io_request *wreq)
{
+ struct netfs_io_stream *cstream = &wreq->io_streams[1];
struct netfs_inode *ictx = netfs_inode(wreq->inode);
size_t transferred;
bool transferred_valid = false;
@@ -393,13 +448,19 @@ bool netfs_write_collection(struct netfs_io_request *wreq)
wreq->transferred = transferred;
trace_netfs_rreq(wreq, netfs_rreq_trace_write_done);
- if (wreq->io_streams[1].active &&
- wreq->io_streams[1].failed &&
- ictx->ops->invalidate_cache) {
- /* Cache write failure doesn't prevent writeback completion
- * unless we're in disconnected mode.
- */
- ictx->ops->invalidate_cache(wreq);
+ if (cstream->active) {
+ if (test_bit(NETFS_RREQ_CACHE_ERROR, &wreq->flags)) {
+ if (ictx->ops->invalidate_cache) {
+ /* Cache write failure doesn't prevent
+ * writeback completion unless we're in
+ * disconnected mode.
+ */
+ trace_netfs_rreq(wreq, netfs_rreq_trace_inval_cache);
+ ictx->ops->invalidate_cache(wreq);
+ }
+ } else if (!cstream->failed) {
+ netfs_cache_collect(wreq, cstream, cstream->cache_collect);
+ }
}
_debug("finished");
@@ -483,24 +544,51 @@ void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error)
if (IS_ERR_VALUE(transferred_or_error)) {
subreq->error = transferred_or_error;
- /* if need retry is set, error should not matter */
- if (!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
- set_bit(NETFS_SREQ_FAILED, &subreq->flags);
- trace_netfs_failure(wreq, subreq, transferred_or_error, netfs_fail_write);
- }
switch (subreq->source) {
case NETFS_WRITE_TO_CACHE:
+ /* We don't mark a cache-write subreq as failed.
+ * Instead we tell the issuer to produce dummy subreqs
+ * instead and make a note if we need to invalidate the
+ * cache at the end. We also don't pause the loop that
+ * grabs pages and launches upload subreqs.
+ *
+ * Note that we need to distinguish between -ENOBUFS
+ * (no space available in the cache) and other errors.
+ * In the former case, we can keep the data we have,
+ * though we might have to change the way the on-disk
+ * data is tracked.
+ */
netfs_stat(&netfs_n_wh_write_failed);
+ if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
+ break;
+
+ trace_netfs_failure(wreq, subreq, transferred_or_error, netfs_fail_write);
+ __set_bit(NETFS_SREQ_CANCELLED, &subreq->flags);
+ set_bit(NETFS_RREQ_CACHE_STOP, &wreq->flags);
+ if (transferred_or_error == -ENOBUFS)
+ trace_netfs_rreq(wreq, netfs_rreq_trace_cache_no_space);
+ else if (!test_and_set_bit(NETFS_RREQ_CACHE_ERROR, &wreq->flags))
+ trace_netfs_rreq(wreq, netfs_rreq_trace_cache_failed);
+ subreq->transferred = subreq->len;
break;
+
case NETFS_UPLOAD_TO_SERVER:
+ /* If need_retry is set, error should not matter */
+ if (!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
+ set_bit(NETFS_SREQ_FAILED, &subreq->flags);
+ trace_netfs_failure(wreq, subreq, transferred_or_error,
+ netfs_fail_upload);
+ }
+
+ set_bit(NETFS_RREQ_PAUSE, &wreq->flags);
+ trace_netfs_rreq(wreq, netfs_rreq_trace_set_pause);
netfs_stat(&netfs_n_wh_upload_failed);
break;
+
default:
break;
}
- trace_netfs_rreq(wreq, netfs_rreq_trace_set_pause);
- set_bit(NETFS_RREQ_PAUSE, &wreq->flags);
} else {
if (WARN(transferred_or_error > subreq->len - subreq->transferred,
"Subreq excess write: R=%x[%x] %zd > %zu - %zu",
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index 4ed50ac7c1f8..638f0122315c 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -111,6 +111,8 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
goto nomem;
wreq->cleaned_to = wreq->start;
+ if (wreq->cache_resources.dio_size > 1)
+ wreq->cache_coll_to = round_down(wreq->start, wreq->cache_resources.dio_size);
wreq->io_streams[0].stream_nr = 0;
wreq->io_streams[0].source = NETFS_UPLOAD_TO_SERVER;
@@ -230,6 +232,21 @@ static void netfs_do_issue_write(struct netfs_io_stream *stream,
_enter("R=%x[%x],%zx", wreq->debug_id, subreq->debug_index, subreq->len);
+ if (stream->source == NETFS_WRITE_TO_CACHE &&
+ unlikely(test_bit(NETFS_RREQ_CACHE_STOP, &wreq->flags))) {
+ size_t dio_size = wreq->cache_resources.dio_size;
+ size_t len, disp;
+
+ disp = subreq->start & (dio_size - 1);
+ len = round_up(subreq->len + disp, dio_size);
+
+ subreq->start -= disp;
+ subreq->len = len;
+
+ __set_bit(NETFS_SREQ_CANCELLED, &subreq->flags);
+ return netfs_write_subrequest_terminated(subreq, subreq->len);
+ }
+
if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
return netfs_write_subrequest_terminated(subreq, subreq->error);
@@ -263,6 +280,7 @@ void netfs_issue_write(struct netfs_io_request *wreq,
if (!subreq)
return;
+
stream->construct = NULL;
subreq->io_iter.count = subreq->len;
netfs_do_issue_write(stream, subreq);
diff --git a/fs/netfs/write_retry.c b/fs/netfs/write_retry.c
index 6cd584242af2..747d57467b4f 100644
--- a/fs/netfs/write_retry.c
+++ b/fs/netfs/write_retry.c
@@ -211,6 +211,7 @@ void netfs_retry_writes(struct netfs_io_request *wreq)
int s;
netfs_stat(&netfs_n_wh_retry_write_req);
+ trace_netfs_rreq(wreq, netfs_rreq_trace_retry_begin);
/* Wait for all outstanding I/O to quiesce before performing retries as
* we may need to renegotiate the I/O sizes.
@@ -235,4 +236,6 @@ void netfs_retry_writes(struct netfs_io_request *wreq)
netfs_retry_write_stream(wreq, stream);
}
}
+
+ trace_netfs_rreq(wreq, netfs_rreq_trace_retry_end);
}
diff --git a/include/linux/fscache.h b/include/linux/fscache.h
index e19fca38382b..f2d958bd1f48 100644
--- a/include/linux/fscache.h
+++ b/include/linux/fscache.h
@@ -147,6 +147,23 @@ struct fscache_cookie {
};
};
+enum fscache_extent_type {
+ FSCACHE_EXTENT_DATA,
+ FSCACHE_EXTENT_ZERO,
+} __mode(byte);
+
+/*
+ * Cache occupancy information.
+ */
+struct fscache_occupancy {
+ unsigned long long query_from; /* Point to query from */
+ unsigned long long query_to; /* Point to query to */
+ unsigned long long cached_from[2]; /* Point at which cache extents start */
+ unsigned long long cached_to[2]; /* Point at which cache extents end */
+ unsigned int granularity; /* Granularity desired */
+ enum fscache_extent_type cached_type[2]; /* Type of cache extent */
+};
+
/*
* slow-path functions for when there is actually caching available, and the
* netfs does actually have a valid token
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 7cdb14a130d9..3882b6c7aaf0 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -23,6 +23,7 @@
enum netfs_sreq_ref_trace;
typedef struct mempool mempool_t;
+struct fscache_occupancy;
struct folio_queue;
/**
@@ -126,6 +127,12 @@ static inline struct netfs_group *netfs_folio_group(struct folio *folio)
return priv;
}
+enum netfs_cache_collect {
+ NETFS_CACHE_COLLECT_WRITE_GAP, /* Gap in collection, no state either way */
+ NETFS_CACHE_COLLECT_WRITE_DATA, /* Currently collecting good writes */
+ NETFS_CACHE_COLLECT_WRITE_CANCEL, /* Currently collecting cancelled writes */
+};
+
/*
* Stream of I/O subrequests going to a particular destination, such as the
* server or the local cache. This is mainly intended for writing where we may
@@ -153,6 +160,7 @@ struct netfs_io_stream {
bool need_retry; /* T if this stream needs retrying */
bool failed; /* T if this stream failed */
bool transferred_valid; /* T is ->transferred is valid */
+ enum netfs_cache_collect cache_collect; /* Current writeback cache collect state */
};
/*
@@ -162,8 +170,10 @@ struct netfs_cache_resources {
const struct netfs_cache_ops *ops;
void *cache_priv;
void *cache_priv2;
+ uoff_t cache_i_size; /* Initial size of cache file */
unsigned int debug_id; /* Cookie debug ID */
unsigned int inval_counter; /* object->inval_counter at begin_op */
+ unsigned int dio_size; /* DIO block size */
};
/*
@@ -197,6 +207,7 @@ struct netfs_io_subrequest {
#define NETFS_SREQ_IN_PROGRESS 8 /* Unlocked when the subrequest completes */
#define NETFS_SREQ_NEED_RETRY 9 /* Set if the filesystem requests a retry */
#define NETFS_SREQ_FAILED 10 /* Set if the subreq failed unretryably */
+#define NETFS_SREQ_CANCELLED 11 /* Set if the subreq was cancelled by netfslib */
};
enum netfs_io_origin {
@@ -252,6 +263,7 @@ struct netfs_io_request {
uoff_t start; /* Start position */
atomic64_t issued_to; /* Write issuer folio cursor */
uoff_t collected_to; /* Point we've collected to */
+ uoff_t cache_coll_to; /* Point the cache has collected to */
uoff_t cleaned_to; /* Position we've cleaned folios to */
uoff_t abandon_to; /* Position to abandon folios to */
const struct folio *no_unlock_folio; /* Don't unlock this folio after read */
@@ -273,11 +285,13 @@ struct netfs_io_request {
#define NETFS_RREQ_FAILED 3 /* The request failed */
#define NETFS_RREQ_RETRYING 4 /* Set if we're in the retry path */
#define NETFS_RREQ_SHORT_TRANSFER 5 /* Set if we have a short transfer */
-#define NETFS_RREQ_OFFLOAD_COLLECTION 8 /* Offload collection to workqueue */
-#define NETFS_RREQ_NO_UNLOCK_FOLIO 9 /* Don't unlock no_unlock_folio on completion */
-#define NETFS_RREQ_FOLIO_COPY_TO_CACHE 10 /* Copy current folio to cache from read */
-#define NETFS_RREQ_UPLOAD_TO_SERVER 11 /* Need to write to the server */
-#define NETFS_RREQ_USE_IO_ITER 12 /* Use ->io_iter rather than ->i_pages */
+#define NETFS_RREQ_CACHE_STOP 8 /* Set to stop caching (ENOBUFS or error) */
+#define NETFS_RREQ_CACHE_ERROR 9 /* Set if we got an error from the cache */
+#define NETFS_RREQ_OFFLOAD_COLLECTION 12 /* Offload collection to workqueue */
+#define NETFS_RREQ_NO_UNLOCK_FOLIO 13 /* Don't unlock no_unlock_folio on completion */
+#define NETFS_RREQ_FOLIO_COPY_TO_CACHE 14 /* Copy current folio to cache from read */
+#define NETFS_RREQ_UPLOAD_TO_SERVER 15 /* Need to write to the server */
+#define NETFS_RREQ_USE_IO_ITER 16 /* Use ->io_iter rather than ->i_pages */
#define NETFS_RREQ_NEED_PUT_RA_REFS 17 /* Need to put the folio refs RA gave us */
#ifdef CONFIG_NETFS_PGPRIV2
#define NETFS_RREQ_USE_PGPRIV2 31 /* [DEPRECATED] Use PG_private_2 to mark
@@ -359,8 +373,7 @@ struct netfs_cache_ops {
/* Prepare a read operation, shortening it to a cached/uncached
* boundary as appropriate.
*/
- enum netfs_io_source (*prepare_read)(struct netfs_io_subrequest *subreq,
- uoff_t i_size);
+ int (*prepare_read)(struct netfs_io_subrequest *subreq);
/* Prepare a write subrequest, working out if we're allowed to do it
* and finding out the maximum amount of data to gather before
@@ -380,8 +393,17 @@ struct netfs_cache_ops {
* next chunk of data starts and how long it is.
*/
int (*query_occupancy)(struct netfs_cache_resources *cres,
- uoff_t start, size_t len, size_t granularity,
- uoff_t *_data_start, size_t *_data_len);
+ struct fscache_occupancy *occ);
+
+ /* Collect the result of buffered writeback to the cache. This
+ * includes copying a read to the cache. block_type is one of:
+ * - NETFS_CACHE_COLLECT_WRITE_DATA for a block of data
+ * - NETFS_CACHE_COLLECT_WRITE_GAP if a discontiguity was skipped
+ * - NETFS_CACHE_COLLECT_WRITE_CANCEL for a cancellation gap
+ */
+ void (*collect_write)(struct netfs_io_request *wreq,
+ uoff_t start, size_t len,
+ enum netfs_cache_collect block_type);
};
/* High-level read API. */
diff --git a/include/trace/events/cachefiles.h b/include/trace/events/cachefiles.h
index 4412d8344182..6146483300c2 100644
--- a/include/trace/events/cachefiles.h
+++ b/include/trace/events/cachefiles.h
@@ -52,6 +52,8 @@ enum cachefiles_coherency_trace {
cachefiles_coherency_check_ok,
cachefiles_coherency_check_type,
cachefiles_coherency_check_xattr,
+ cachefiles_coherency_discontiguous,
+ cachefiles_coherency_remove,
cachefiles_coherency_set_fail,
cachefiles_coherency_set_ok,
cachefiles_coherency_vol_check_cmp,
@@ -63,9 +65,11 @@ enum cachefiles_coherency_trace {
};
enum cachefiles_trunc_trace {
+ cachefiles_trunc_clear_padding,
cachefiles_trunc_dio_adjust,
cachefiles_trunc_expand_tmpfile,
cachefiles_trunc_shrink,
+ cachefiles_trunc_zap,
};
enum cachefiles_prepare_read_trace {
@@ -80,6 +84,7 @@ enum cachefiles_prepare_read_trace {
};
enum cachefiles_error_trace {
+ cachefiles_trace_alignment_error,
cachefiles_trace_fallocate_error,
cachefiles_trace_getxattr_error,
cachefiles_trace_link_error,
@@ -136,6 +141,8 @@ enum cachefiles_error_trace {
EM(cachefiles_coherency_check_ok, "OK ") \
EM(cachefiles_coherency_check_type, "BAD type") \
EM(cachefiles_coherency_check_xattr, "BAD xatt") \
+ EM(cachefiles_coherency_discontiguous, "--- gap ") \
+ EM(cachefiles_coherency_remove, "REMOVE ") \
EM(cachefiles_coherency_set_fail, "SET fail") \
EM(cachefiles_coherency_set_ok, "SET ok ") \
EM(cachefiles_coherency_vol_check_cmp, "VOL BAD cmp ") \
@@ -146,9 +153,11 @@ enum cachefiles_error_trace {
E_(cachefiles_coherency_vol_set_ok, "VOL SET ok ")
#define cachefiles_trunc_traces \
+ EM(cachefiles_trunc_clear_padding, "CLRPAD") \
EM(cachefiles_trunc_dio_adjust, "DIOADJ") \
EM(cachefiles_trunc_expand_tmpfile, "EXPTMP") \
- E_(cachefiles_trunc_shrink, "SHRINK")
+ EM(cachefiles_trunc_shrink, "SHRINK") \
+ E_(cachefiles_trunc_zap, "ZAP ")
#define cachefiles_prepare_read_traces \
EM(cachefiles_trace_read_after_eof, "after-eof ") \
@@ -161,6 +170,7 @@ enum cachefiles_error_trace {
E_(cachefiles_trace_read_seek_nxio, "seek-enxio")
#define cachefiles_error_traces \
+ EM(cachefiles_trace_alignment_error, "align") \
EM(cachefiles_trace_fallocate_error, "fallocate") \
EM(cachefiles_trace_getxattr_error, "getxattr") \
EM(cachefiles_trace_link_error, "link") \
@@ -371,12 +381,12 @@ TRACE_EVENT(cachefiles_rename,
TRACE_EVENT(cachefiles_coherency,
TP_PROTO(struct cachefiles_object *obj,
- ino_t ino,
+ ino_t ino, uoff_t obj_size,
const void *disk_aux,
enum cachefiles_content content,
enum cachefiles_coherency_trace why),
- TP_ARGS(obj, ino, disk_aux, content, why),
+ TP_ARGS(obj, ino, obj_size, disk_aux, content, why),
/* Note that obj may be NULL */
TP_STRUCT__entry(
@@ -384,6 +394,7 @@ TRACE_EVENT(cachefiles_coherency,
__field(enum cachefiles_coherency_trace, why)
__field(enum cachefiles_content, content)
__field(u64, ino)
+ __field(u64, obj_size)
__field(u64, aux)
__field(u64, disk_aux)
),
@@ -398,6 +409,7 @@ TRACE_EVENT(cachefiles_coherency,
__entry->why = why;
__entry->content = content;
__entry->ino = ino;
+ __entry->obj_size = obj_size;
__entry->aux = be64_to_cpup((__be64 *)obj->cookie->inline_aux);
/* cachefiles_xattr::data is not 64-byte aligned. */
@@ -412,10 +424,11 @@ TRACE_EVENT(cachefiles_coherency,
}
),
- TP_printk("o=%08x %s B=%llx c=%u aux=%llx dsk=%llx",
+ TP_printk("o=%08x %s B=%llx oz=%llx c=%u aux=%llx dsk=%llx",
__entry->obj,
__print_symbolic(__entry->why, cachefiles_coherency_traces),
__entry->ino,
+ __entry->obj_size,
__entry->content,
__entry->aux,
__entry->disk_aux)
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 1687b974900c..c395d3f5ed61 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -49,6 +49,10 @@
#define netfs_rreq_traces \
EM(netfs_rreq_trace_all_queued, "ALL-Q ") \
EM(netfs_rreq_trace_assess, "ASSESS ") \
+ EM(netfs_rreq_trace_cache_cancelled, "CA-CNCL") \
+ EM(netfs_rreq_trace_cache_failed, "CA-FAIL") \
+ EM(netfs_rreq_trace_cache_fail_collect, "CA-F-CO") \
+ EM(netfs_rreq_trace_cache_no_space, "CA-NOSP") \
EM(netfs_rreq_trace_collect, "COLLECT") \
EM(netfs_rreq_trace_complete, "COMPLET") \
EM(netfs_rreq_trace_copy, "COPY ") \
@@ -57,11 +61,14 @@
EM(netfs_rreq_trace_end_copy_to_cache, "END-C2C") \
EM(netfs_rreq_trace_free, "FREE ") \
EM(netfs_rreq_trace_intr, "INTR ") \
+ EM(netfs_rreq_trace_inval_cache, "INVL-CA") \
EM(netfs_rreq_trace_ki_complete, "KI-CMPL") \
EM(netfs_rreq_trace_ra_put_ref, "RA-PUT ") \
EM(netfs_rreq_trace_recollect, "RECLLCT") \
EM(netfs_rreq_trace_redirty, "REDIRTY") \
EM(netfs_rreq_trace_resubmit, "RESUBMT") \
+ EM(netfs_rreq_trace_retry_begin, "RETRY-BEGIN") \
+ EM(netfs_rreq_trace_retry_end, "RETRY-END") \
EM(netfs_rreq_trace_set_abandon, "S-ABNDN") \
EM(netfs_rreq_trace_set_pause, "PAUSE ") \
EM(netfs_rreq_trace_unlock, "UNLOCK ") \
@@ -133,12 +140,12 @@
#define netfs_failures \
EM(netfs_fail_check_write_begin, "check-write-begin") \
- EM(netfs_fail_copy_to_cache, "copy-to-cache") \
EM(netfs_fail_dio_read_short, "dio-read-short") \
EM(netfs_fail_dio_read_zero, "dio-read-zero") \
EM(netfs_fail_read, "read") \
EM(netfs_fail_short_read, "short-read") \
EM(netfs_fail_prepare_write, "prep-write") \
+ EM(netfs_fail_upload, "upload") \
E_(netfs_fail_write, "write")
#define netfs_rreq_ref_traces \
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 23/35] netfs: Add the cache object ID to netfs_read/write tracepoints
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (21 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 22/35] cachefiles: Don't rely on backing fs storage map for most use cases David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 24/35] netfs: Switch to using bvecq rather than folio_queue and rolling_buffer David Howells
` (12 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Add the cache object debug ID to netfs_read/write tracepoints to make
debugging easier as there's now a direct cross-reference with the
cachefiles tracepoints that only log that debug ID.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/cachefiles/io.c | 1 +
fs/netfs/fscache_io.c | 2 +-
include/linux/netfs.h | 3 ++-
include/trace/events/netfs.h | 27 +++++++++++++++------------
4 files changed, 19 insertions(+), 14 deletions(-)
diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c
index b33ad1d21d44..a43fe168127a 100644
--- a/fs/cachefiles/io.c
+++ b/fs/cachefiles/io.c
@@ -927,6 +927,7 @@ bool cachefiles_begin_operation(struct netfs_cache_resources *cres,
if (!cachefiles_cres_file(cres)) {
cres->ops = &cachefiles_netfs_cache_ops;
+ cres->object_id = object->debug_id;
if (object->file) {
spin_lock(&object->lock);
file = object->file;
diff --git a/fs/netfs/fscache_io.c b/fs/netfs/fscache_io.c
index 8bca63721eeb..056a2bae5d99 100644
--- a/fs/netfs/fscache_io.c
+++ b/fs/netfs/fscache_io.c
@@ -79,7 +79,7 @@ static int fscache_begin_operation(struct netfs_cache_resources *cres,
cres->ops = NULL;
cres->cache_priv = cookie;
cres->cache_priv2 = NULL;
- cres->debug_id = cookie->debug_id;
+ cres->cookie_id = cookie->debug_id;
cres->inval_counter = cookie->inval_counter;
if (!fscache_begin_cookie_access(cookie, why)) {
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 3882b6c7aaf0..ef1068a274a8 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -171,7 +171,8 @@ struct netfs_cache_resources {
void *cache_priv;
void *cache_priv2;
uoff_t cache_i_size; /* Initial size of cache file */
- unsigned int debug_id; /* Cookie debug ID */
+ unsigned int cookie_id; /* Cache cookie debug ID */
+ unsigned int object_id; /* Cache object debug ID */
unsigned int inval_counter; /* object->inval_counter at begin_op */
unsigned int dio_size; /* DIO block size */
};
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index c395d3f5ed61..927287a7290c 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -311,6 +311,7 @@ TRACE_EVENT(netfs_read,
TP_STRUCT__entry(
__field(unsigned int, rreq)
__field(unsigned int, cookie)
+ __field(unsigned int, object)
__field(uoff_t, i_size)
__field(uoff_t, start)
__field(size_t, len)
@@ -320,7 +321,8 @@ TRACE_EVENT(netfs_read,
TP_fast_assign(
__entry->rreq = rreq->debug_id;
- __entry->cookie = rreq->cache_resources.debug_id;
+ __entry->cookie = rreq->cache_resources.cookie_id;
+ __entry->object = rreq->cache_resources.object_id;
__entry->i_size = rreq->i_size;
__entry->start = start;
__entry->len = len;
@@ -328,10 +330,10 @@ TRACE_EVENT(netfs_read,
__entry->netfs_inode = rreq->inode->i_ino;
),
- TP_printk("R=%08x %s c=%08x ni=%llx s=%llx l=%zx sz=%llx",
+ TP_printk("R=%08x %s c=%08x o=%08x ni=%llx s=%llx l=%zx sz=%llx",
__entry->rreq,
__print_symbolic(__entry->what, netfs_read_traces),
- __entry->cookie,
+ __entry->cookie, __entry->object,
__entry->netfs_inode,
__entry->start, __entry->len, __entry->i_size)
);
@@ -555,6 +557,7 @@ TRACE_EVENT(netfs_write,
TP_STRUCT__entry(
__field(unsigned int, wreq)
__field(unsigned int, cookie)
+ __field(unsigned int, object)
__field(unsigned int, ino)
__field(enum netfs_write_trace, what)
__field(uoff_t, start)
@@ -562,20 +565,19 @@ TRACE_EVENT(netfs_write,
),
TP_fast_assign(
- struct netfs_inode *__ctx = netfs_inode(wreq->inode);
- struct fscache_cookie *__cookie = netfs_i_cookie(__ctx);
__entry->wreq = wreq->debug_id;
- __entry->cookie = __cookie ? __cookie->debug_id : 0;
+ __entry->cookie = wreq->cache_resources.cookie_id;
+ __entry->object = wreq->cache_resources.object_id;
__entry->ino = wreq->inode->i_ino;
__entry->what = what;
__entry->start = wreq->start;
__entry->len = wreq->len;
),
- TP_printk("R=%08x %s c=%08x i=%x by=%llx-%llx",
+ TP_printk("R=%08x %s c=%08x o=%08x i=%x by=%llx-%llx",
__entry->wreq,
__print_symbolic(__entry->what, netfs_write_traces),
- __entry->cookie,
+ __entry->cookie, __entry->object,
__entry->ino,
__entry->start, __entry->start + __entry->len - 1)
);
@@ -590,22 +592,23 @@ TRACE_EVENT(netfs_copy2cache,
__field(unsigned int, rreq)
__field(unsigned int, creq)
__field(unsigned int, cookie)
+ __field(unsigned int, object)
__field(unsigned int, ino)
),
TP_fast_assign(
- struct netfs_inode *__ctx = netfs_inode(rreq->inode);
- struct fscache_cookie *__cookie = netfs_i_cookie(__ctx);
__entry->rreq = rreq->debug_id;
__entry->creq = creq->debug_id;
- __entry->cookie = __cookie ? __cookie->debug_id : 0;
+ __entry->cookie = rreq->cache_resources.cookie_id;
+ __entry->object = rreq->cache_resources.object_id;
__entry->ino = rreq->inode->i_ino;
),
- TP_printk("R=%08x CR=%08x c=%08x i=%x ",
+ TP_printk("R=%08x CR=%08x c=%08x o=%08x i=%x ",
__entry->rreq,
__entry->creq,
__entry->cookie,
+ __entry->object,
__entry->ino)
);
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 24/35] netfs: Switch to using bvecq rather than folio_queue and rolling_buffer
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (22 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 23/35] netfs: Add the cache object ID to netfs_read/write tracepoints David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 25/35] smbdirect: Remove support for ITER_FOLIOQ from smbdirect_map_sges_from_iter() David Howells
` (11 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, Shyam Prasad N, Tom Talpey
Switch netfslib to using bvecq, a segmented bio_vec[] queue, instead of the
folio_queue and rolling_buffer constructs, to keep track of the regions of
memory it is performing I/O upon.
For unbuffered/direct I/O, the iterator is extracted into the queue up
front. For buffered I/O, the folios are added to the queue as the
operation proceeds, much as it does now with folio_queues. For buffered
reads and writes through the pagecache, bv_len is set to the length of the
entire folio and then slices of the relevant parts are passed to
subrequests.
The bvecq structs are marked with information as to how the regions
contained therein should be disposed of (unlock-only, free, unpin).
When setting up a subrequest, netfslib will furnish it with a slice of the
main buffer queue as a pointer to starting bvecq, slot and offset and, for
the moment, an ITER_BVECQ iterator is set to cover the slice in
subreq->io_iter.
Notes on the implementation:
(1) This patch uses the concept of a 'bvecq position', which is a tuple of
{ bvecq, slot, offset }. This is lighter weight than using a full
iov_iter, though that would also suffice. If not NULL, the position
also holds a reference on the bvecq it is pointing to. This is
probably overkill as only the hindmost position (that of collection)
needs to hold a reference.
(2) There are three positions on the netfs_io_request struct. Not all are
used by every request type.
Firstly, there's ->load_cursor, which is used by buffered read and
write to point to the next slot to have a folio inserted into it
(either loaded from the readahead_control or from writeback_iter()).
Secondly, there's ->dispatch_cursor, which is used to provide the
position in the buffer from which we start dispatching a subrequest.
Thirdly, there's the ->collect_cursor, which is used by the collection
routines to point to the next memory region to be cleaned up.
(3) There are two positions on the netfs_io_subrequest struct.
Firstly, there's ->dispatch_pos, which indicates the position from
which a subrequest's buffer begins. This is used as the base of the
position from which to retry (advanced by ->transfer).
Secondly, there's ->content, which is normally the same as
->dispatch_pos but if the bvecq chain got duplicated or the content
got copied, then this will point to that and will that will be
disposed of on retry. The filesystem may modify ->content, but not
->dispatch_pos.
(4) Maintenance of the position structs is done with helper functions,
such as bvecq_pos_attach() to hide the refcounting.
(5) When sending a write to the cache, the ends of the slice will be
rounded up/down to the backing file's DIO block alignment.
(6) bvecq_slice() is used to select a slice of the source buffer and
assign it to a subrequest. The source buffer position is advanced.
(7) netfs_extract_iter() is used by unbuffered/direct I/O API functions to
decant a chunk of the iov_iter supplied by the VFS into a bvecq chain
- and to label the bvecqs with appropriate disposal information
(e.g. unpin, free, nothing).
There are further options that can be explored in the future:
(1) Allow the provision of a duplicated bvecq chain for a specific slice
so that the filesystem can add bits on either end (such as adding
protocol headers and trailers and gluing several things together into
a compound operation).
(2) If a filesystem supports vectored/sparse read and write ops, it can be
given a chain with discontiguities in it to perform in a single op
(Ceph, for example, can do this).
(3) Make MSG_SPLICE_PAGES handling read the disposal info in the bvecq and
use that to indicate how it should get rid of the stuff it pasted into
a sk_buff.
(4) If a bounce buffer is needed (encryption, for example), the bounce
buffer can be held in a bvecq and sliced up instead of the main buffer
queue.
(5) Get rid of subreq->io_iter and move the iov_iter stuff down into the
filesystem. The I/O iterators are normally only needed transitorily,
and the one currently in netfs_io_subrequest is unnecessary most of
the time.
folio_queue and rolling_buffer will be removed in a follow up patch.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Namjae Jeon <linkinjeon@kernel.org>
cc: Shyam Prasad N <sprasad@microsoft.com>
cc: Tom Talpey <tom@talpey.com>
cc: linux-cifs@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/afs/dir.c | 5 +-
fs/afs/symlink.c | 6 +-
fs/cachefiles/io.c | 12 ---
fs/netfs/Makefile | 1 -
fs/netfs/buffered_read.c | 123 +++++++++++++++-----------
fs/netfs/direct_read.c | 78 +++++++----------
fs/netfs/direct_write.c | 69 ++++++++-------
fs/netfs/internal.h | 10 +--
fs/netfs/iterator.c | 2 +
fs/netfs/main.c | 6 --
fs/netfs/misc.c | 20 +----
fs/netfs/objects.c | 17 ++--
fs/netfs/read_collect.c | 116 +++++++++++++------------
fs/netfs/read_pgpriv2.c | 91 +++++++++++++-------
fs/netfs/read_retry.c | 102 +++++++++++++---------
fs/netfs/read_single.c | 17 +++-
fs/netfs/stats.c | 4 +-
fs/netfs/write_collect.c | 60 ++++++-------
fs/netfs/write_issue.c | 162 ++++++++++++++++++++++++-----------
fs/netfs/write_retry.c | 62 ++++++++------
include/linux/netfs.h | 27 +++---
include/trace/events/netfs.h | 46 +++++-----
22 files changed, 575 insertions(+), 461 deletions(-)
diff --git a/fs/afs/dir.c b/fs/afs/dir.c
index 1dff50323b56..b2eb5bce3e9f 100644
--- a/fs/afs/dir.c
+++ b/fs/afs/dir.c
@@ -2228,8 +2228,9 @@ static int afs_dir_writepages(struct address_space *mapping,
if (test_bit(AFS_VNODE_DIR_VALID, &dvnode->flags)) {
iov_iter_bvec_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0,
- i_size_read(&dvnode->netfs.inode));
- ret = netfs_writeback_single(mapping, wbc, &iter);
+ dvnode->directory_size);
+ ret = netfs_writeback_single(mapping, wbc, &iter,
+ i_size_read(&dvnode->netfs.inode));
if (ret == 1)
ret = 0; /* Skipped write due to lock conflict. */
}
diff --git a/fs/afs/symlink.c b/fs/afs/symlink.c
index 7fd4f985c441..71de59173ea5 100644
--- a/fs/afs/symlink.c
+++ b/fs/afs/symlink.c
@@ -245,9 +245,9 @@ int afs_symlink_writepages(struct address_space *mapping,
if (vnode->directory &&
atomic64_read(&vnode->cb_expires_at) != AFS_NO_CB_PROMISE) {
- iov_iter_bvec_queue(&iter, ITER_SOURCE, vnode->directory, 0, 0,
- i_size_read(&vnode->netfs.inode));
- ret = netfs_writeback_single(mapping, wbc, &iter);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, vnode->directory, 0, 0, PAGE_SIZE);
+ ret = netfs_writeback_single(mapping, wbc, &iter,
+ i_size_read(&vnode->netfs.inode));
}
if (ret == 0) {
diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c
index a43fe168127a..23007f450ef4 100644
--- a/fs/cachefiles/io.c
+++ b/fs/cachefiles/io.c
@@ -636,7 +636,6 @@ static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)
struct netfs_cache_resources *cres = &wreq->cache_resources;
struct cachefiles_object *object = cachefiles_cres_object(cres);
struct cachefiles_cache *cache = object->volume->cache;
- struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
const struct cred *saved_cred;
size_t off, pre, post, len = subreq->len;
uoff_t start = subreq->start;
@@ -661,17 +660,6 @@ static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)
}
/* We also need to end on the cache granularity boundary */
- if (start + len == wreq->i_size) {
- size_t part = len & (cache->bsize - 1);
- size_t need = cache->bsize - part;
-
- if (part && stream->submit_extendable_to >= need) {
- len += need;
- subreq->len += need;
- subreq->io_iter.count += need;
- }
- }
-
post = len & (cache->bsize - 1);
if (post) {
len -= post;
diff --git a/fs/netfs/Makefile b/fs/netfs/Makefile
index b1ea4439c1bb..421dd0be413b 100644
--- a/fs/netfs/Makefile
+++ b/fs/netfs/Makefile
@@ -14,7 +14,6 @@ netfs-y := \
read_collect.o \
read_retry.o \
read_single.o \
- rolling_buffer.o \
write_collect.o \
write_issue.o \
write_retry.o
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 5af849b2f283..8a843822d47b 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -114,26 +114,21 @@ static int netfs_begin_cache_read(struct netfs_io_request *rreq, struct netfs_in
static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+ ssize_t extracted;
size_t rsize = subreq->len;
if (subreq->source == NETFS_DOWNLOAD_FROM_SERVER)
- rsize = umin(rsize, rreq->io_streams[0].sreq_max_len);
-
- subreq->len = rsize;
- if (unlikely(rreq->io_streams[0].sreq_max_segs)) {
- size_t limit = netfs_limit_iter(&rreq->buffer.iter, 0, rsize,
- rreq->io_streams[0].sreq_max_segs);
-
- if (limit < rsize) {
- subreq->len = limit;
- trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
- }
+ rsize = umin(rsize, stream->sreq_max_len);
+
+ bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor);
+ extracted = bvecq_slice(&rreq->dispatch_cursor, rsize,
+ stream->sreq_max_segs, &subreq->nr_segs);
+ if (extracted < rsize) {
+ subreq->len = extracted;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
}
- subreq->io_iter = rreq->buffer.iter;
-
- iov_iter_truncate(&subreq->io_iter, subreq->len);
- rolling_buffer_advance(&rreq->buffer, subreq->len);
return subreq->len;
}
@@ -192,6 +187,10 @@ void netfs_queue_read(struct netfs_io_request *rreq,
static void netfs_issue_read(struct netfs_io_request *rreq,
struct netfs_io_subrequest *subreq)
{
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+ iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
switch (subreq->source) {
case NETFS_DOWNLOAD_FROM_SERVER:
rreq->netfs_ops->issue_read(subreq);
@@ -200,7 +199,8 @@ static void netfs_issue_read(struct netfs_io_request *rreq,
netfs_read_cache_to_pagecache(rreq, subreq);
break;
default:
- __set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags);
+ bvecq_zero(&subreq->content, subreq->len);
+ subreq->transferred = subreq->len;
subreq->error = 0;
iov_iter_zero(subreq->len, &subreq->io_iter);
subreq->transferred = subreq->len;
@@ -229,6 +229,10 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
uoff_t start = rreq->start;
int ret = 0;
+ _enter("R=%08x", rreq->debug_id);
+
+ bvecq_pos_set(&rreq->dispatch_cursor, &rreq->load_cursor);
+
do {
int (*prepare_read)(struct netfs_io_subrequest *subreq) = NULL;
struct netfs_io_subrequest *subreq;
@@ -373,6 +377,9 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
/* Defer error return as we may need to wait for outstanding I/O. */
cmpxchg(&rreq->error, 0, ret);
+
+ bvecq_pos_unset(&rreq->load_cursor);
+ bvecq_pos_unset(&rreq->dispatch_cursor);
}
/**
@@ -420,8 +427,7 @@ void netfs_readahead(struct readahead_control *ractl)
* acquires a ref on each folio that we will need to release later -
* but we don't want to do that until after we've started the I/O.
*/
- added = rolling_buffer_bulk_load_from_ra(&rreq->buffer, ractl,
- rreq->debug_id, rreq->gfp);
+ added = bvecq_load_from_ra(&rreq->load_cursor, ractl);
if (added < 0) {
ret = added;
goto cleanup_free;
@@ -430,6 +436,7 @@ void netfs_readahead(struct readahead_control *ractl)
rreq->submitted = rreq->start + added;
rreq->cleaned_to = rreq->start;
+ bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
netfs_read_set_unlock_at(rreq);
netfs_read_to_pagecache(rreq);
@@ -442,21 +449,21 @@ void netfs_readahead(struct readahead_control *ractl)
EXPORT_SYMBOL(netfs_readahead);
/*
- * Create a rolling buffer with a single occupying folio.
+ * Create a buffer queue with a single occupying folio.
*/
-static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio,
- unsigned int rollbuf_flags)
+static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio)
{
- ssize_t added;
+ struct bvecq *bq;
+ size_t fsize = folio_size(folio);
- if (rolling_buffer_init(&rreq->buffer, rreq->debug_id, ITER_DEST, rreq->gfp) < 0)
+ if (bvecq_buffer_init(&rreq->load_cursor, rreq->gfp, false) < 0)
return -ENOMEM;
- added = rolling_buffer_append(&rreq->buffer, folio, rollbuf_flags, rreq->gfp);
- if (added < 0)
- return added;
- rreq->submitted = rreq->start + added;
- rreq->progress_at = added;
+ bq = rreq->load_cursor.bvecq;
+ bvec_set_folio(&bq->bv[0], folio, fsize, 0);
+ bvecq_filled_to(bq, 1);
+ rreq->submitted = rreq->start + fsize;
+ rreq->progress_at = fsize;
return 0;
}
@@ -470,11 +477,11 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
struct netfs_group *group = netfs_folio_group(folio);
struct netfs_folio *finfo = netfs_folio_info(folio);
struct netfs_inode *ctx = netfs_inode(mapping->host);
- struct folio *sink = NULL;
- struct bio_vec *bvec;
+ struct bvecq *bq = NULL;
+ struct page *sink = NULL;
unsigned int from = finfo->dirty_offset;
unsigned int to = from + finfo->dirty_len;
- unsigned int off = 0, i = 0;
+ unsigned int off = 0, slot = 0;
size_t flen = folio_size(folio);
size_t nr_bvec = flen / PAGE_SIZE + 2;
size_t part;
@@ -499,32 +506,47 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
* end get copied to, but the middle is discarded.
*/
ret = -ENOMEM;
- bvec = kmalloc_objs(*bvec, nr_bvec);
- if (!bvec)
+ bq = bvecq_alloc_chain(nr_bvec, rreq->gfp, false);
+ if (!bq)
goto discard;
+ rreq->load_cursor.bvecq = bq;
- sink = folio_alloc(GFP_KERNEL, 0);
- if (!sink) {
- kfree(bvec);
+ sink = alloc_page(rreq->gfp);
+ if (!sink)
goto discard;
- }
trace_netfs_folio(folio, netfs_folio_trace_read_gaps);
- rreq->direct_bv = bvec;
- rreq->direct_bv_count = nr_bvec;
+ for (struct bvecq *p = bq; p; p = p->next)
+ p->mem_type = BVECQ_MEM_PAGECACHE;
+
if (from > 0) {
- bvec_set_folio(&bvec[i++], folio, from, 0);
+ folio_get(folio);
+ bvec_set_folio(&bq->bv[slot++], folio, from, 0);
off = from;
}
while (off < to) {
- part = min_t(size_t, to - off, PAGE_SIZE);
- bvec_set_folio(&bvec[i++], sink, part, 0);
+ if (slot >= bq->max_slots) {
+ bvecq_filled_to(bq, slot);
+ bq = bq->next;
+ slot = 0;
+ }
+ part = min(to - off, PAGE_SIZE);
+ get_page(sink);
+ bvec_set_page(&bq->bv[slot++], sink, part, 0);
off += part;
}
- if (to < flen)
- bvec_set_folio(&bvec[i++], folio, flen - to, to);
- iov_iter_bvec(&rreq->buffer.iter, ITER_DEST, bvec, i, rreq->len);
+ if (to < flen) {
+ if (slot >= bq->max_slots) {
+ bvecq_filled_to(bq, slot);
+ bq = bq->next;
+ slot = 0;
+ }
+ folio_get(folio);
+ bvec_set_folio(&bq->bv[slot++], folio, flen - to, to);
+ }
+ bvecq_filled_to(bq, slot);
+
rreq->submitted = rreq->start + flen;
netfs_read_to_pagecache(rreq);
@@ -541,13 +563,14 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
folio_mark_uptodate(folio);
}
- if (sink)
- folio_put(sink);
+ put_page(sink);
folio_unlock(folio);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret < 0 ? ret : 0;
discard:
+ if (sink)
+ put_page(sink);
netfs_put_failed_request(rreq);
alloc_error:
folio_unlock(folio);
@@ -598,7 +621,7 @@ int netfs_read_folio(struct file *file, struct folio *folio)
trace_netfs_read(rreq, rreq->start, rreq->len, netfs_read_trace_readpage);
/* Set up the output buffer */
- ret = netfs_create_singular_buffer(rreq, folio, 0);
+ ret = netfs_create_singular_buffer(rreq, folio);
if (ret < 0)
goto discard;
@@ -755,7 +778,7 @@ int netfs_write_begin(struct netfs_inode *ctx,
trace_netfs_read(rreq, pos, len, netfs_read_trace_write_begin);
/* Set up the output buffer */
- ret = netfs_create_singular_buffer(rreq, folio, 0);
+ ret = netfs_create_singular_buffer(rreq, folio);
if (ret < 0)
goto error_put;
@@ -820,7 +843,7 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,
trace_netfs_read(rreq, start, flen, netfs_read_trace_prefetch_for_write);
/* Set up the output buffer */
- ret = netfs_create_singular_buffer(rreq, folio, NETFS_ROLLBUF_PAGECACHE_MARK);
+ ret = netfs_create_singular_buffer(rreq, folio);
if (ret < 0)
goto error_put;
diff --git a/fs/netfs/direct_read.c b/fs/netfs/direct_read.c
index 5405e108b7a3..058cc6bb7124 100644
--- a/fs/netfs/direct_read.c
+++ b/fs/netfs/direct_read.c
@@ -16,44 +16,22 @@
#include <linux/netfs.h>
#include "internal.h"
-static void netfs_prepare_dio_read_iterator(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_request *rreq = subreq->rreq;
- size_t rsize;
-
- rsize = umin(subreq->len, rreq->io_streams[0].sreq_max_len);
- subreq->len = rsize;
-
- if (unlikely(rreq->io_streams[0].sreq_max_segs)) {
- size_t limit = netfs_limit_iter(&rreq->buffer.iter, 0, rsize,
- rreq->io_streams[0].sreq_max_segs);
-
- if (limit < rsize) {
- subreq->len = limit;
- trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
- }
- }
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
-
- subreq->io_iter = rreq->buffer.iter;
- iov_iter_truncate(&subreq->io_iter, subreq->len);
- iov_iter_advance(&rreq->buffer.iter, subreq->len);
-}
-
/*
* Perform a read to a buffer from the server, slicing up the region to be read
* according to the network rsize.
*/
static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
{
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
ssize_t size = rreq->len;
uoff_t start = rreq->start;
int ret;
+ bvecq_pos_set(&rreq->dispatch_cursor, &rreq->load_cursor);
+ bvecq_pos_transfer(&rreq->collect_cursor, &rreq->load_cursor);
+
do {
struct netfs_io_subrequest *subreq;
- ssize_t slice;
subreq = netfs_alloc_subrequest(rreq);
if (!subreq) {
@@ -79,14 +57,22 @@ static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
}
}
- netfs_prepare_dio_read_iterator(subreq);
- slice = subreq->len;
- size -= slice;
- start += slice;
- rreq->submitted += slice;
+ bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &rreq->dispatch_cursor);
+ subreq->len = bvecq_slice(&rreq->dispatch_cursor,
+ umin(size, stream->sreq_max_len),
+ stream->sreq_max_segs,
+ &subreq->nr_segs);
+
+ size -= subreq->len;
+ start += subreq->len;
+ rreq->submitted += subreq->len;
if (size <= 0)
netfs_all_subreqs_queued(rreq);
+ iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
rreq->netfs_ops->issue_read(subreq);
if (test_bit(NETFS_RREQ_PAUSE, &rreq->flags))
@@ -100,6 +86,8 @@ static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
netfs_all_subreqs_queued(rreq);
netfs_wake_collector(rreq);
}
+
+ bvecq_pos_unset(&rreq->dispatch_cursor);
}
/*
@@ -178,25 +166,17 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i
* buffer for ourselves as the caller's iterator will be trashed when
* we return.
*
- * In such a case, extract an iterator to represent as much of the the
- * output buffer as we can manage. Note that the extraction might not
- * be able to allocate a sufficiently large bvec array and may shorten
- * the request.
+ * Extract a buffer queue to represent as much of the output buffer as
+ * we can manage. The fragments are extracted into a bvecq which will
+ * have sufficient nodes allocated to hold all the data, though this
+ * may end up truncated if ENOMEM is encountered.
*/
- if (user_backed_iter(iter)) {
- ret = netfs_extract_user_iter(iter, rreq->len, &rreq->buffer.iter, 0);
- if (ret < 0)
- goto error_put;
- rreq->direct_bv = (struct bio_vec *)rreq->buffer.iter.bvec;
- rreq->direct_bv_count = ret;
- rreq->direct_bv_unpin = iov_iter_extract_will_pin(iter);
- rreq->len = iov_iter_count(&rreq->buffer.iter);
- } else {
- rreq->buffer.iter = *iter;
- rreq->len = orig_count;
- rreq->direct_bv_unpin = false;
- iov_iter_advance(iter, orig_count);
- }
+ ret = netfs_extract_iter(iter, rreq->len, INT_MAX,
+ &rreq->load_cursor.bvecq, 0, rreq->gfp);
+ if (ret < 0)
+ goto error_put;
+
+ rreq->len = ret;
// TODO: Set up bounce buffer if needed
diff --git a/fs/netfs/direct_write.c b/fs/netfs/direct_write.c
index 47ec47e3a95b..5bc3ec261f6c 100644
--- a/fs/netfs/direct_write.c
+++ b/fs/netfs/direct_write.c
@@ -73,7 +73,11 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq,
spin_unlock(&wreq->lock);
wreq->transferred += subreq->transferred;
- iov_iter_advance(&wreq->buffer.iter, subreq->transferred);
+ if (subreq->transferred < subreq->len) {
+ bvecq_pos_unset(&wreq->dispatch_cursor);
+ bvecq_pos_transfer(&wreq->dispatch_cursor, &subreq->dispatch_pos);
+ bvecq_pos_advance(&wreq->dispatch_cursor, subreq->transferred);
+ }
stream->collected_to = subreq->start + subreq->transferred;
wreq->collected_to = stream->collected_to;
@@ -99,6 +103,9 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
_enter("%llx", wreq->len);
+ bvecq_pos_set(&wreq->dispatch_cursor, &wreq->load_cursor);
+ bvecq_pos_set(&wreq->collect_cursor, &wreq->dispatch_cursor);
+
if (wreq->origin == NETFS_DIO_WRITE)
inode_dio_begin(wreq->inode);
@@ -111,6 +118,8 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
netfs_prepare_write(wreq, stream, wreq->start + wreq->transferred);
subreq = stream->construct;
stream->construct = NULL;
+ } else {
+ bvecq_pos_set(&subreq->dispatch_pos, &wreq->dispatch_cursor);
}
/* Check if (re-)preparation failed. */
@@ -120,16 +129,18 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
break;
}
- iov_iter_truncate(&subreq->io_iter, wreq->len - wreq->transferred);
+ subreq->len = bvecq_slice(&wreq->dispatch_cursor, stream->sreq_max_len,
+ stream->sreq_max_segs, &subreq->nr_segs);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+
+ iov_iter_bvec_queue(&subreq->io_iter, ITER_SOURCE,
+ subreq->content.bvecq, subreq->content.slot,
+ subreq->content.offset,
+ subreq->len);
+
if (!iov_iter_count(&subreq->io_iter))
break;
- subreq->len = netfs_limit_iter(&subreq->io_iter, 0,
- stream->sreq_max_len,
- stream->sreq_max_segs);
- iov_iter_truncate(&subreq->io_iter, subreq->len);
- stream->submit_extendable_to = subreq->len;
-
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
stream->issue_write(subreq);
@@ -166,9 +177,14 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
*/
subreq->error = -EAGAIN;
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+
+ bvecq_pos_unset(&subreq->content);
+ bvecq_pos_unset(&wreq->dispatch_cursor);
+ bvecq_pos_transfer(&wreq->dispatch_cursor, &subreq->dispatch_pos);
+
if (subreq->transferred > 0) {
- iov_iter_advance(&wreq->buffer.iter, subreq->transferred);
wreq->transferred += subreq->transferred;
+ bvecq_pos_advance(&wreq->dispatch_cursor, subreq->transferred);
}
if (stream->source == NETFS_UPLOAD_TO_SERVER &&
@@ -179,7 +195,6 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
__clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
__clear_bit(NETFS_SREQ_BOUNDARY, &subreq->flags);
__clear_bit(NETFS_SREQ_FAILED, &subreq->flags);
- subreq->io_iter = wreq->buffer.iter;
subreq->start = wreq->start + wreq->transferred;
subreq->len = wreq->len - wreq->transferred;
subreq->transferred = 0;
@@ -195,6 +210,8 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
netfs_stat(&netfs_n_wh_retry_write_subreq);
}
+ bvecq_pos_unset(&wreq->dispatch_cursor);
+ bvecq_pos_unset(&wreq->load_cursor);
netfs_unbuffered_write_done(wreq);
_leave(" = %d", ret);
return ret;
@@ -213,10 +230,10 @@ static void netfs_unbuffered_write_async(struct work_struct *work)
* encrypted file. This can also be used for direct I/O writes.
*/
ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *iter,
- struct netfs_group *netfs_group)
+ struct netfs_group *netfs_group)
{
struct netfs_io_request *wreq;
- ssize_t ret, n;
+ ssize_t ret;
uoff_t start = iocb->ki_pos;
uoff_t end = start + iov_iter_count(iter);
size_t len = iov_iter_count(iter);
@@ -252,25 +269,17 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
* allocate a sufficiently large bvec array and may shorten the
* request.
*/
- if (user_backed_iter(iter)) {
- n = netfs_extract_user_iter(iter, len, &wreq->buffer.iter, 0);
- if (n < 0) {
- ret = n;
- goto error_put;
- }
- wreq->direct_bv = (struct bio_vec *)wreq->buffer.iter.bvec;
- wreq->direct_bv_count = n;
- wreq->direct_bv_unpin = iov_iter_extract_will_pin(iter);
- } else {
- /* If this is a kernel-generated async DIO request,
- * assume that any resources the iterator points to
- * (eg. a bio_vec array) will persist till the end of
- * the op.
- */
- wreq->buffer.iter = *iter;
- }
+ ssize_t n = netfs_extract_iter(iter, len, INT_MAX,
+ &wreq->load_cursor.bvecq, 0, wreq->gfp);
- wreq->len = iov_iter_count(&wreq->buffer.iter);
+ if (n < 0) {
+ ret = n;
+ goto error_put;
+ }
+ wreq->len = n;
+ _debug("dio-write %zx/%zx %u/%u",
+ n, len, wreq->load_cursor.bvecq->nr_slots,
+ wreq->load_cursor.bvecq->max_slots);
}
__set_bit(NETFS_RREQ_USE_IO_ITER, &wreq->flags);
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 5c05b0dcb143..b402b504a370 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -7,7 +7,6 @@
#include <linux/slab.h>
#include <linux/seq_file.h>
-#include <linux/folio_queue.h>
#include <linux/netfs.h>
#include <linux/fscache.h>
#include <linux/fscache-cache.h>
@@ -71,9 +70,8 @@ static inline void netfs_proc_del_rreq(struct netfs_io_request *rreq) {}
/*
* misc.c
*/
-struct folio_queue *netfs_buffer_make_space(struct netfs_io_request *rreq,
- enum netfs_folioq_trace trace);
-void netfs_reset_iter(struct netfs_io_subrequest *subreq);
+struct bvecq *netfs_buffer_make_space(struct netfs_io_request *rreq,
+ enum netfs_bvecq_trace trace);
void netfs_wake_collector(struct netfs_io_request *rreq);
void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq);
void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,
@@ -201,7 +199,6 @@ extern atomic_t netfs_n_wh_retry_write_req;
extern atomic_t netfs_n_wh_retry_write_subreq;
extern atomic_t netfs_n_wb_lock_skip;
extern atomic_t netfs_n_wb_lock_wait;
-extern atomic_t netfs_n_folioq;
extern atomic_t netfs_n_bvecq;
int netfs_stats_show(struct seq_file *m, void *v);
@@ -239,8 +236,7 @@ void netfs_prepare_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
uoff_t start);
void netfs_reissue_write(struct netfs_io_stream *stream,
- struct netfs_io_subrequest *subreq,
- struct iov_iter *source);
+ struct netfs_io_subrequest *subreq);
void netfs_issue_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream);
size_t netfs_advance_write(struct netfs_io_request *wreq,
diff --git a/fs/netfs/iterator.c b/fs/netfs/iterator.c
index e66c589ca3b0..c207eacd5c5d 100644
--- a/fs/netfs/iterator.c
+++ b/fs/netfs/iterator.c
@@ -156,6 +156,7 @@ ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pag
}
EXPORT_SYMBOL_GPL(netfs_extract_iter);
+#if 0
/**
* netfs_extract_user_iter - Extract the pages from a user iterator into a bvec
* @orig: The original iterator
@@ -449,3 +450,4 @@ size_t netfs_limit_iter(const struct iov_iter *iter, size_t start_offset,
BUG();
}
EXPORT_SYMBOL(netfs_limit_iter);
+#endif
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index 3908ea2422c8..041810707eb4 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -109,9 +109,6 @@ static int __init netfs_init(void)
{
int ret = -ENOMEM;
- if (mempool_init_kmalloc_pool(&netfs_folioq_pool, 100, sizeof(struct folio_queue)) < 0)
- goto error_folioq_pool;
-
if (mempool_init_kmalloc_pool(&netfs_bvecq_pool, 100, BVECQ_STD_SIZE) < 0)
goto error_bvecq_pool;
@@ -169,8 +166,6 @@ static int __init netfs_init(void)
error_req:
mempool_exit(&netfs_bvecq_pool);
error_bvecq_pool:
- mempool_exit(&netfs_folioq_pool);
-error_folioq_pool:
return ret;
}
fs_initcall(netfs_init);
@@ -184,6 +179,5 @@ static void __exit netfs_exit(void)
mempool_exit(&netfs_request_pool);
kmem_cache_destroy(netfs_request_slab);
mempool_exit(&netfs_bvecq_pool);
- mempool_exit(&netfs_folioq_pool);
}
module_exit(netfs_exit);
diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c
index a3cd76d584b8..a2a2aeb1bb0d 100644
--- a/fs/netfs/misc.c
+++ b/fs/netfs/misc.c
@@ -8,6 +8,7 @@
#include <linux/swap.h>
#include "internal.h"
+#if 0
/**
* netfs_alloc_folioq_buffer - Allocate buffer space into a folio queue
* @mapping: Address space to set on the folio (or NULL).
@@ -103,24 +104,7 @@ void netfs_free_folioq_buffer(struct folio_queue *fq)
folio_batch_release(&fbatch);
}
EXPORT_SYMBOL(netfs_free_folioq_buffer);
-
-/*
- * Reset the subrequest iterator to refer just to the region remaining to be
- * read. The iterator may or may not have been advanced by socket ops or
- * extraction ops to an extent that may or may not match the amount actually
- * read.
- */
-void netfs_reset_iter(struct netfs_io_subrequest *subreq)
-{
- struct iov_iter *io_iter = &subreq->io_iter;
- size_t remain = subreq->len - subreq->transferred;
-
- if (io_iter->count > remain)
- iov_iter_advance(io_iter, io_iter->count - remain);
- else if (io_iter->count < remain)
- iov_iter_revert(io_iter, remain - io_iter->count);
- iov_iter_truncate(&subreq->io_iter, remain);
-}
+#endif
/**
* netfs_dirty_folio - Mark folio dirty and pin a cache object for writeback
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index 13816a02a798..740971955198 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -133,7 +133,6 @@ static void netfs_free_request_rcu(struct rcu_head *rcu)
static void netfs_deinit_request(struct netfs_io_request *rreq)
{
struct netfs_inode *ictx = netfs_inode(rreq->inode);
- unsigned int i;
trace_netfs_rreq(rreq, netfs_rreq_trace_free);
@@ -148,16 +147,10 @@ static void netfs_deinit_request(struct netfs_io_request *rreq)
rreq->netfs_ops->free_request(rreq);
if (rreq->cache_resources.ops)
rreq->cache_resources.ops->end_operation(&rreq->cache_resources);
- if (rreq->direct_bv) {
- for (i = 0; i < rreq->direct_bv_count; i++) {
- if (rreq->direct_bv[i].bv_page) {
- if (rreq->direct_bv_unpin)
- unpin_user_page(rreq->direct_bv[i].bv_page);
- }
- }
- kvfree(rreq->direct_bv);
- }
- rolling_buffer_clear(&rreq->buffer);
+ bvecq_pos_unset(&rreq->load_cursor);
+ bvecq_pos_unset(&rreq->dispatch_cursor);
+ bvecq_pos_unset(&rreq->collect_cursor);
+ bvecq_put(rreq->spare);
if (atomic_dec_and_test(&ictx->io_count))
wake_up_var(&ictx->io_count);
@@ -249,6 +242,8 @@ static void netfs_free_subrequest(struct netfs_io_subrequest *subreq)
trace_netfs_sreq(subreq, netfs_sreq_trace_free);
if (rreq->netfs_ops->free_subrequest)
rreq->netfs_ops->free_subrequest(subreq);
+ bvecq_pos_unset(&subreq->dispatch_pos);
+ bvecq_pos_unset(&subreq->content);
mempool_free(subreq, rreq->netfs_ops->subrequest_pool ?: &netfs_subrequest_pool);
netfs_stat_d(&netfs_n_rh_sreq);
netfs_put_request(rreq, netfs_rreq_trace_put_subreq);
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index f773579692ab..ea01b2f1f1f8 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -27,9 +27,13 @@
*/
static void netfs_clear_unread(struct netfs_io_subrequest *subreq)
{
- netfs_reset_iter(subreq);
- WARN_ON_ONCE(subreq->len - subreq->transferred != iov_iter_count(&subreq->io_iter));
- iov_iter_zero(iov_iter_count(&subreq->io_iter), &subreq->io_iter);
+ struct iov_iter iter;
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+ iov_iter_advance(&iter, subreq->transferred);
+ iov_iter_zero(subreq->len, &iter);
+
if (subreq->start + subreq->transferred >= subreq->rreq->i_size)
__set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags);
}
@@ -40,11 +44,11 @@ static void netfs_clear_unread(struct netfs_io_subrequest *subreq)
* dirty and let writeback handle it.
*/
static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
- struct folio_queue *folioq,
+ struct bvecq *bvecq,
int slot)
{
struct netfs_folio *finfo;
- struct folio *folio = folioq_folio(folioq, slot);
+ struct folio *folio = page_folio(bvecq->bv[slot].bv_page);
if (unlikely(folio_pos(folio) < rreq->abandon_to)) {
trace_netfs_folio(folio, netfs_folio_trace_abandon);
@@ -75,7 +79,7 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
trace_netfs_folio(folio, netfs_folio_trace_read_done);
}
- folioq_clear(folioq, slot);
+ bvecq->bv[slot].bv_page = NULL;
} else {
// TODO: Use of PG_private_2 is deprecated.
if (test_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags))
@@ -91,7 +95,7 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
folio_unlock(folio);
}
- folioq_clear(folioq, slot);
+ bvecq->bv[slot].bv_page = NULL;
}
/*
@@ -99,21 +103,25 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
*/
void netfs_read_set_unlock_at(struct netfs_io_request *rreq)
{
- struct folio_queue *folioq = rreq->buffer.tail;
- unsigned int slot = rreq->buffer.first_tail_slot;
+ const struct bvecq *bq = rreq->collect_cursor.bvecq;
+ unsigned int slot = rreq->collect_cursor.slot;
size_t cleaned_to = rreq->cleaned_to - rreq->start;
size_t progress_at = cleaned_to;
size_t minimum = 256 * 1024;
while (progress_at < rreq->len) {
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- if (!folioq)
+ if (!bvecq_acquire_slot(bq, slot)) {
+ const struct bvecq *next = bvecq_next(bq);
+
+ if (!next)
break;
+ if (bvecq_acquire_slot(bq, slot))
+ continue; /* More slots got added. */
+ bq = next;
slot = 0;
}
- progress_at += folioq_folio_size(folioq, slot);
+ progress_at += bq->bv[slot].bv_len;
if (progress_at - cleaned_to >= minimum)
break;
slot++;
@@ -129,8 +137,8 @@ void netfs_read_set_unlock_at(struct netfs_io_request *rreq)
static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
unsigned int *notes)
{
- struct folio_queue *folioq = rreq->buffer.tail;
- unsigned int slot = rreq->buffer.first_tail_slot;
+ struct bvecq *bvecq = rreq->collect_cursor.bvecq;
+ unsigned int slot = rreq->collect_cursor.slot;
uoff_t collected_to = rreq->collected_to;
if (rreq->cleaned_to >= rreq->collected_to)
@@ -138,15 +146,6 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
// TODO: Begin decryption
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&rreq->buffer);
- if (!folioq) {
- WRITE_ONCE(rreq->progress_at, ULONG_MAX);
- return;
- }
- slot = 0;
- }
-
/* We have to wait for readahead refs to have been released before we
* can unlock any folios as the ref-dropper walks i_pages and the only
* thing preventing these folios from being removed is the folio lock.
@@ -159,17 +158,30 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
uoff_t fpos = rreq->cleaned_to, fend;
size_t fsize;
+ /* Clean up the head bvecq segment. If we clear an entire
+ * segment, then we can get rid of it provided it's not also
+ * the tail segment being filled by the issuer.
+ */
+ if (!bvecq_acquire_slot(bvecq, slot)) {
+ rreq->collect_cursor.slot = slot;
+ if (!bvecq_delete_spent(&rreq->collect_cursor)) {
+ WRITE_ONCE(rreq->progress_at, ULONG_MAX);
+ return;
+ }
+ bvecq = rreq->collect_cursor.bvecq;
+ slot = rreq->collect_cursor.slot;
+ }
+
if (*notes & COPY_TO_CACHE)
set_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags);
- folio = folioq_folio(folioq, slot);
+ folio = page_folio(bvecq->bv[slot].bv_page);
if (WARN_ONCE(!folio_test_locked(folio),
"R=%08x: folio %lx is not locked\n",
rreq->debug_id, folio->index))
trace_netfs_folio(folio, netfs_folio_trace_not_locked);
- fsize = folioq_folio_size(folioq, slot);
- fpos = folio_pos(folio);
+ fsize = bvecq->bv[slot].bv_len;
fend = fpos + fsize;
trace_netfs_collect_folio(rreq, folio);
@@ -178,34 +190,19 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
if (collected_to < fend)
break;
- netfs_unlock_read_folio(rreq, folioq, slot);
+ netfs_unlock_read_folio(rreq, bvecq, slot);
+ slot++;
WRITE_ONCE(rreq->cleaned_to, fend);
*notes |= MADE_PROGRESS;
clear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags);
- /* Clean up the head folioq. If we clear an entire folioq, then
- * we can get rid of it provided it's not also the tail folioq
- * being filled by the issuer.
- */
- folioq_clear(folioq, slot);
- slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&rreq->buffer);
- if (!folioq)
- goto done;
- slot = 0;
- trace_netfs_folioq(folioq, netfs_trace_folioq_read_progress);
- }
-
if (fpos + fsize >= collected_to)
break;
}
- rreq->buffer.tail = folioq;
-done:
- rreq->buffer.first_tail_slot = slot;
-
+ bvecq_pos_move(&rreq->collect_cursor, bvecq);
+ rreq->collect_cursor.slot = slot;
netfs_read_set_unlock_at(rreq);
}
@@ -383,12 +380,17 @@ static void netfs_rreq_assess_dio(struct netfs_io_request *rreq)
if (rreq->origin == NETFS_UNBUFFERED_READ ||
rreq->origin == NETFS_DIO_READ) {
- for (i = 0; i < rreq->direct_bv_count; i++) {
- flush_dcache_page(rreq->direct_bv[i].bv_page);
- // TODO: cifs marks pages in the destination buffer
- // dirty under some circumstances after a read. Do we
- // need to do that too?
- set_page_dirty(rreq->direct_bv[i].bv_page);
+ for (struct bvecq *bq = rreq->collect_cursor.bvecq; bq; bq = bvecq_next(bq)) {
+ unsigned int nr_slots = bvecq_nr_slots_acquire(bq);
+ /* Read the slot count before the slots. */
+
+ for (i = 0; i < nr_slots; i++) {
+ flush_dcache_page(bq->bv[i].bv_page);
+ // TODO: cifs marks pages in the destination buffer
+ // dirty under some circumstances after a read. Do we
+ // need to do that too?
+ set_page_dirty(bq->bv[i].bv_page);
+ }
}
}
@@ -477,7 +479,15 @@ bool netfs_read_collection(struct netfs_io_request *rreq)
trace_netfs_rreq(rreq, netfs_rreq_trace_done);
netfs_clear_subrequests(rreq);
- netfs_unlock_abandoned_read_pages(rreq);
+ switch (rreq->origin) {
+ case NETFS_READAHEAD:
+ case NETFS_READPAGE:
+ case NETFS_READ_FOR_WRITE:
+ netfs_unlock_abandoned_read_pages(rreq);
+ break;
+ default:
+ break;
+ }
if (unlikely(rreq->copy_to_cache))
netfs_pgpriv2_end_copy_to_cache(rreq);
return true;
diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c
index aab156d00031..2e4eab1e327f 100644
--- a/fs/netfs/read_pgpriv2.c
+++ b/fs/netfs/read_pgpriv2.c
@@ -19,6 +19,9 @@
static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio *folio)
{
struct netfs_io_stream *cache = &creq->io_streams[1];
+ struct bvecq *queue;
+ unsigned int slot;
+ size_t dio_size = PAGE_SIZE;
size_t fsize = folio_size(folio), flen = fsize;
uoff_t fpos = folio_pos(folio), i_size;
bool to_eof = false;
@@ -48,18 +51,37 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
to_eof = true;
}
+ flen = round_up(flen, dio_size);
+
_debug("folio %zx %zx", flen, fsize);
trace_netfs_folio(folio, netfs_folio_trace_store_copy);
- /* Attach the folio to the rolling buffer. */
- if (rolling_buffer_append(&creq->buffer, folio, 0, creq->gfp) < 0) {
- folio_end_private_2(folio);
- clear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &creq->flags);
- return;
+ /* Institute a new bvec queue segment if the current one is full or if
+ * we encounter a discontiguity. The discontiguity break is important
+ * when it comes to bulk unlocking folios by file range.
+ */
+ queue = creq->load_cursor.bvecq;
+ if (bvecq_is_full(queue) ||
+ (fpos != creq->last_end && creq->last_end > 0 && queue->nr_slots > 0)) {
+ bvecq_buffer_append(&creq->load_cursor, creq->spare);
+ creq->spare = NULL;
+
+ queue = creq->load_cursor.bvecq;
}
- cache->submit_extendable_to = fsize;
+ /* Attach the folio to the rolling buffer. */
+ slot = queue->nr_slots;
+ bvec_set_folio(&queue->bv[slot], folio, fsize, 0);
+ trace_netfs_bv_slot(queue, slot);
+ slot++;
+ bvecq_filled_to(queue, slot);
+ creq->load_cursor.slot = slot;
+ creq->load_cursor.offset = 0;
+ creq->last_end = fpos + flen;
+
+ bvecq_pos_nudge(&creq->dispatch_cursor);
+
cache->submit_off = 0;
cache->submit_len = flen;
@@ -71,10 +93,9 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
do {
ssize_t part;
- creq->buffer.iter.iov_offset = cache->submit_off;
+ creq->dispatch_cursor.offset = cache->submit_off;
atomic64_set(&creq->issued_to, fpos + cache->submit_off);
- cache->submit_extendable_to = fsize - cache->submit_off;
part = netfs_advance_write(creq, cache, fpos + cache->submit_off,
cache->submit_len, to_eof);
cache->submit_off += part;
@@ -84,8 +105,7 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
cache->submit_len -= part;
} while (cache->submit_len > 0);
- creq->buffer.iter.iov_offset = 0;
- rolling_buffer_advance(&creq->buffer, fsize);
+ bvecq_pos_step(&creq->dispatch_cursor);
atomic64_set(&creq->issued_to, fpos + fsize);
if (flen < fsize)
@@ -111,6 +131,11 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache(
if (!creq->io_streams[1].avail)
goto cancel_put;
+ if (bvecq_buffer_init(&creq->load_cursor, creq->gfp, false) < 0)
+ goto cancel_put;
+ bvecq_pos_set(&creq->dispatch_cursor, &creq->load_cursor);
+ bvecq_pos_set(&creq->collect_cursor, &creq->dispatch_cursor);
+
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &creq->flags);
trace_netfs_copy2cache(rreq, creq);
trace_netfs_write(creq, netfs_write_trace_copy_to_cache);
@@ -139,6 +164,14 @@ void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *fo
if (IS_ERR(creq))
return;
+ if (!creq->spare) {
+ creq->spare = bvecq_alloc_one(BVECQ_STD_SLOTS, creq->gfp, false);
+ if (!creq->spare) {
+ clear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &creq->flags);
+ return;
+ }
+ }
+
trace_netfs_folio(folio, netfs_folio_trace_copy_to_cache);
folio_start_private_2(folio);
netfs_pgpriv2_copy_folio(creq, folio);
@@ -170,22 +203,27 @@ void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)
*/
bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *creq)
{
- struct folio_queue *folioq = creq->buffer.tail;
- unsigned int slot = creq->buffer.first_tail_slot;
+ struct bvecq *bq = creq->collect_cursor.bvecq;
+ unsigned int slot;
uoff_t collected_to = creq->collected_to;
bool made_progress = false;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&creq->buffer);
- slot = 0;
- }
+ slot = creq->collect_cursor.slot;
for (;;) {
struct folio *folio;
uoff_t fpos, fend;
size_t fsize, flen;
- folio = folioq_folio(folioq, slot);
+ if (!bvecq_acquire_slot(bq, slot)) {
+ creq->collect_cursor.slot = slot;
+ if (!bvecq_delete_spent(&creq->collect_cursor))
+ goto out;
+ bq = creq->collect_cursor.bvecq;
+ slot = creq->collect_cursor.slot;
+ }
+
+ folio = page_folio(bq->bv[slot].bv_page);
if (WARN_ONCE(!folio_test_private_2(folio),
"R=%08x: folio %lx is not marked private_2\n",
creq->debug_id, folio->index))
@@ -208,25 +246,18 @@ bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *creq)
creq->cleaned_to = fpos + fsize;
made_progress = true;
- /* Clean up the head folioq. If we clear an entire folioq, then
- * we can get rid of it provided it's not also the tail folioq
- * being filled by the issuer.
+ /* Clean up the head segment. If we clear an entire segment,
+ * then we can get rid of it provided it's not also the tail
+ * segment being filled by the issuer.
*/
- folioq_clear(folioq, slot);
+ bq->bv[slot].bv_page = NULL;
slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&creq->buffer);
- if (!folioq)
- goto done;
- slot = 0;
- }
if (fpos + fsize >= collected_to)
break;
}
- creq->buffer.tail = folioq;
-done:
- creq->buffer.first_tail_slot = slot;
+ creq->collect_cursor.slot = slot;
+out:
return made_progress;
}
diff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c
index 96223f62516a..733d9ff452c7 100644
--- a/fs/netfs/read_retry.c
+++ b/fs/netfs/read_retry.c
@@ -12,6 +12,12 @@
static void netfs_reissue_read(struct netfs_io_request *rreq,
struct netfs_io_subrequest *subreq)
{
+ bvecq_pos_unset(&subreq->content);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+ iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+ iov_iter_advance(&subreq->io_iter, subreq->transferred);
+
subreq->error = 0;
__clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
@@ -27,6 +33,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
{
struct netfs_io_subrequest *subreq;
struct netfs_io_stream *stream = &rreq->io_streams[0];
+ struct bvecq_pos dispatch_cursor = {};
struct list_head *next;
_enter("R=%x", rreq->debug_id);
@@ -46,9 +53,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
break;
if (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
- __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
subreq->retry_count++;
- netfs_reset_iter(subreq);
netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
netfs_reissue_read(rreq, subreq);
}
@@ -74,11 +79,12 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
do {
struct netfs_io_subrequest *from, *to, *tmp;
- struct iov_iter source;
uoff_t start, len;
size_t part;
bool boundary = false, subreq_superfluous = false;
+ bvecq_pos_unset(&dispatch_cursor);
+
/* Go through the subreqs and find the next span of contiguous
* buffer that we then rejig (cifs, for example, needs the
* rsize renegotiating) and reissue.
@@ -105,7 +111,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
break;
subreq = list_entry(next, struct netfs_io_subrequest, rreq_link);
- if (subreq->start + subreq->transferred != start + len ||
+ if (subreq->start != start + len ||
+ subreq->transferred > 0 ||
test_bit(NETFS_SREQ_BOUNDARY, &subreq->flags) ||
!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
break;
@@ -118,11 +125,14 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
/* Determine the set of buffers we're going to use. Each
* subreq gets a subset of a single overall contiguous buffer.
*/
- netfs_reset_iter(from);
- source = from->io_iter;
- source.count = len;
+ bvecq_pos_transfer(&dispatch_cursor, &from->dispatch_pos);
+ bvecq_pos_advance(&dispatch_cursor, from->transferred);
+ from->transferred = 0;
- /* Work through the sublist. */
+ /* Work through the sublist. The chain of buffers we're going
+ * to fill is attached to dispatch_cursor and we need to read
+ * 'len' amount of data from 'start'.
+ */
subreq = from;
list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) {
if (!len) {
@@ -130,16 +140,22 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
break;
}
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- subreq->start = start - subreq->transferred;
- subreq->len = len + subreq->transferred;
+ subreq->start = start;
+ subreq->len = len;
__clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
__clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
subreq->retry_count++;
+ subreq->transferred = 0;
+
+ bvecq_pos_unset(&subreq->content);
+ bvecq_pos_unset(&subreq->dispatch_pos);
+ bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
/* Renegotiate max_len (rsize) */
- stream->sreq_max_len = subreq->len;
+ stream->sreq_max_len = len;
+ stream->sreq_max_segs = INT_MAX;
if (rreq->netfs_ops->prepare_read &&
rreq->netfs_ops->prepare_read(subreq) < 0) {
trace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed);
@@ -147,13 +163,12 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
goto abandon;
}
- part = umin(len, stream->sreq_max_len);
- if (unlikely(stream->sreq_max_segs))
- part = netfs_limit_iter(&source, 0, part, stream->sreq_max_segs);
- subreq->len = subreq->transferred + part;
- subreq->io_iter = source;
- iov_iter_truncate(&subreq->io_iter, part);
- iov_iter_advance(&source, part);
+ part = bvecq_slice(&dispatch_cursor,
+ umin(len, stream->sreq_max_len),
+ stream->sreq_max_segs,
+ &subreq->nr_segs);
+ subreq->len = part;
+
len -= part;
start += part;
if (!len) {
@@ -217,9 +232,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
stream->sreq_max_len = umin(len, rreq->rsize);
- stream->sreq_max_segs = 0;
- if (unlikely(stream->sreq_max_segs))
- part = netfs_limit_iter(&source, 0, part, stream->sreq_max_segs);
+ stream->sreq_max_segs = INT_MAX;
netfs_stat(&netfs_n_rh_download);
if (rreq->netfs_ops->prepare_read(subreq) < 0) {
@@ -228,11 +241,12 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
goto abandon;
}
- part = umin(len, stream->sreq_max_len);
- subreq->len = subreq->transferred + part;
- subreq->io_iter = source;
- iov_iter_truncate(&subreq->io_iter, part);
- iov_iter_advance(&source, part);
+ bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
+ part = bvecq_slice(&dispatch_cursor,
+ umin(len, stream->sreq_max_len),
+ stream->sreq_max_segs,
+ &subreq->nr_segs);
+ subreq->len = part;
len -= part;
start += part;
@@ -246,12 +260,14 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
} while (!list_is_head(next, &stream->subrequests));
+out:
+ bvecq_pos_unset(&dispatch_cursor);
return;
/* If we hit an error, fail all remaining incomplete subrequests */
abandon_after:
if (list_is_last(&subreq->rreq_link, &stream->subrequests))
- return;
+ goto out;
subreq = list_next_entry(subreq, rreq_link);
abandon:
list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) {
@@ -262,6 +278,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
__set_bit(NETFS_SREQ_FAILED, &subreq->flags);
__clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
}
+ goto out;
}
/*
@@ -292,7 +309,7 @@ void netfs_retry_reads(struct netfs_io_request *rreq)
*/
void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq)
{
- struct folio_queue *p;
+ struct bvecq *p;
/* We have to wait for readahead refs to have been released before we
* can unlock any folios as the ref-dropper walks i_pages and the only
@@ -301,21 +318,22 @@ void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq)
if (test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags))
netfs_wait_for_put_ra_refs(rreq);
- for (p = rreq->buffer.tail; p; p = p->next) {
- for (int slot = 0; slot < folioq_count(p); slot++) {
- struct folio *folio = folioq_folio(p, slot);
-
- if (folio && !folioq_is_marked2(p, slot)) {
- if (folio == rreq->no_unlock_folio &&
- test_bit(NETFS_RREQ_NO_UNLOCK_FOLIO,
- &rreq->flags)) {
- _debug("no unlock");
- } else {
- trace_netfs_folio(folio,
- netfs_folio_trace_abandon);
- folio_unlock(folio);
- }
+ for (p = rreq->collect_cursor.bvecq; p; p = bvecq_next(p)) {
+ unsigned int nr_slots = bvecq_nr_slots_acquire(p);
+
+ for (int slot = 0; slot < nr_slots; slot++) {
+ if (!p->bv[slot].bv_page)
+ continue;
+
+ struct folio *folio = page_folio(p->bv[slot].bv_page);
+
+ if (folio == rreq->no_unlock_folio &&
+ test_bit(NETFS_RREQ_NO_UNLOCK_FOLIO, &rreq->flags)) {
+ _debug("no unlock");
+ continue;
}
+ trace_netfs_folio(folio, netfs_folio_trace_abandon);
+ folio_unlock(folio);
}
}
}
diff --git a/fs/netfs/read_single.c b/fs/netfs/read_single.c
index b370a8534324..9fd07dbb08b5 100644
--- a/fs/netfs/read_single.c
+++ b/fs/netfs/read_single.c
@@ -93,7 +93,12 @@ static int netfs_single_dispatch_read(struct netfs_io_request *rreq)
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
subreq->start = 0;
subreq->len = rreq->len;
- subreq->io_iter = rreq->buffer.iter;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &rreq->dispatch_cursor);
+
+ iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
netfs_queue_read(rreq, subreq);
@@ -174,6 +179,15 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite
if (IS_ERR(rreq))
return PTR_ERR(rreq);
+ ret = netfs_extract_iter(iter, rreq->len, INT_MAX, &rreq->dispatch_cursor.bvecq,
+ 0, rreq->gfp);
+ if (ret < 0)
+ goto cleanup_free;
+ if (ret < rreq->len) {
+ ret = -EIO;
+ goto cleanup_free;
+ }
+
rreq->progress_at = rreq->len;
ret = netfs_single_begin_cache_read(rreq, ictx);
@@ -183,7 +197,6 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite
netfs_stat(&netfs_n_rh_read_single);
trace_netfs_read(rreq, 0, rreq->len, netfs_read_trace_read_single);
- rreq->buffer.iter = *iter;
netfs_single_dispatch_read(rreq);
ret = netfs_wait_for_read(rreq);
diff --git a/fs/netfs/stats.c b/fs/netfs/stats.c
index a10d34f88597..0ba6ce9295b2 100644
--- a/fs/netfs/stats.c
+++ b/fs/netfs/stats.c
@@ -46,7 +46,6 @@ atomic_t netfs_n_wh_retry_write_req;
atomic_t netfs_n_wh_retry_write_subreq;
atomic_t netfs_n_wb_lock_skip;
atomic_t netfs_n_wb_lock_wait;
-atomic_t netfs_n_folioq;
atomic_t netfs_n_bvecq;
int netfs_stats_show(struct seq_file *m, void *v)
@@ -89,11 +88,10 @@ int netfs_stats_show(struct seq_file *m, void *v)
atomic_read(&netfs_n_rh_retry_read_subreq),
atomic_read(&netfs_n_wh_retry_write_req),
atomic_read(&netfs_n_wh_retry_write_subreq));
- seq_printf(m, "Objs : rr=%u sr=%u bq=%u foq=%u wsc=%u\n",
+ seq_printf(m, "Objs : rr=%u sr=%u bq=%u wsc=%u\n",
atomic_read(&netfs_n_rh_rreq),
atomic_read(&netfs_n_rh_sreq),
atomic_read(&netfs_n_bvecq),
- atomic_read(&netfs_n_folioq),
atomic_read(&netfs_n_wh_wstream_conflict));
seq_printf(m, "WbLock : skip=%u wait=%u\n",
atomic_read(&netfs_n_wb_lock_skip),
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index 6e8ea534230d..b980a65441dc 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -114,12 +114,12 @@ int netfs_folio_written_back(struct folio *folio)
static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
unsigned int *notes)
{
- struct folio_queue *folioq = wreq->buffer.tail;
- unsigned int slot = wreq->buffer.first_tail_slot;
+ struct bvecq *bvecq = wreq->collect_cursor.bvecq;
+ unsigned int slot = wreq->collect_cursor.slot;
uoff_t collected_to = wreq->collected_to;
- if (WARN_ON_ONCE(!folioq)) {
- pr_err("[!] Writeback unlock found empty rolling buffer!\n");
+ if (WARN_ON_ONCE(!bvecq)) {
+ pr_err("[!] Writeback unlock found empty buffer!\n");
netfs_dump_request(wreq);
return;
}
@@ -130,20 +130,28 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
return;
}
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&wreq->buffer);
- if (!folioq)
- return;
- slot = 0;
- }
-
for (;;) {
struct folio *folio;
struct netfs_folio *finfo;
uoff_t fpos, fend;
size_t fsize, flen;
- folio = folioq_folio(folioq, slot);
+ /* Try to clean up the head of the queue if it appears to be
+ * used up, but we need to be very careful - the cleanup can
+ * catch the dispatcher, which could lead to us having nothing
+ * left in the queue, causing the front and back pointers to
+ * end up on different tracks. To avoid this, we must always
+ * keep at least one segment in the queue.
+ */
+ if (!bvecq_acquire_slot(bvecq, slot)) {
+ wreq->collect_cursor.slot = slot;
+ if (!bvecq_delete_spent(&wreq->collect_cursor))
+ return;
+ bvecq = wreq->collect_cursor.bvecq;
+ slot = wreq->collect_cursor.slot;
+ }
+
+ folio = page_folio(bvecq->bv[slot].bv_page);
if (WARN_ONCE(!folio_test_writeback(folio),
"R=%08x: folio %lx is not under writeback\n",
wreq->debug_id, folio->index))
@@ -166,26 +174,13 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
wreq->cleaned_to = fpos + fsize;
*notes |= MADE_PROGRESS;
- /* Clean up the head folioq. If we clear an entire folioq, then
- * we can get rid of it provided it's not also the tail folioq
- * being filled by the issuer.
- */
- folioq_clear(folioq, slot);
+ bvecq->bv[slot].bv_page = NULL;
slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&wreq->buffer);
- if (!folioq)
- goto done;
- slot = 0;
- }
-
if (fpos + fsize >= collected_to)
break;
}
- wreq->buffer.tail = folioq;
-done:
- wreq->buffer.first_tail_slot = slot;
+ wreq->collect_cursor.slot = slot;
}
/*
@@ -230,7 +225,8 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
trace_netfs_rreq(wreq, netfs_rreq_trace_collect);
reassess_streams:
- issued_to = atomic64_read(&wreq->issued_to);
+ /* Order reading the issued_to point before reading the queue it refers to. */
+ issued_to = atomic64_read_acquire(&wreq->issued_to);
smp_rmb();
collected_to = ULLONG_MAX;
if (wreq->origin == NETFS_WRITEBACK ||
@@ -560,8 +556,12 @@ void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error)
* data is tracked.
*/
netfs_stat(&netfs_n_wh_write_failed);
- if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
- break;
+ if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
+ /* We don't retry failed cache writes. */
+ __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ if (!subreq->error)
+ subreq->error = -ENOBUFS;
+ }
trace_netfs_failure(wreq, subreq, transferred_or_error, netfs_fail_write);
__set_bit(NETFS_SREQ_CANCELLED, &subreq->flags);
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index 638f0122315c..a5c5ba5deb43 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -107,8 +107,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
ictx = netfs_inode(wreq->inode);
if (is_cacheable)
fscache_begin_write_operation(&wreq->cache_resources, netfs_i_cookie(ictx));
- if (rolling_buffer_init(&wreq->buffer, wreq->debug_id, ITER_SOURCE, wreq->gfp) < 0)
- goto nomem;
wreq->cleaned_to = wreq->start;
if (wreq->cache_resources.dio_size > 1)
@@ -133,9 +131,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
}
return wreq;
-nomem:
- netfs_put_failed_request(wreq);
- return ERR_PTR(-ENOMEM);
}
/**
@@ -160,21 +155,13 @@ void netfs_prepare_write(struct netfs_io_request *wreq,
uoff_t start)
{
struct netfs_io_subrequest *subreq;
- struct iov_iter *wreq_iter = &wreq->buffer.iter;
-
- /* Make sure we don't point the iterator at a used-up folio_queue
- * struct being used as a placeholder to prevent the queue from
- * collapsing. In such a case, extend the queue.
- */
- if (iov_iter_is_folioq(wreq_iter) &&
- wreq_iter->folioq_slot >= folioq_nr_slots(wreq_iter->folioq))
- rolling_buffer_make_space(&wreq->buffer, wreq->gfp);
subreq = netfs_alloc_subrequest(wreq);
subreq->source = stream->source;
subreq->start = start;
subreq->stream_nr = stream->stream_nr;
- subreq->io_iter = *wreq_iter;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &wreq->dispatch_cursor);
_enter("R=%x[%x]", wreq->debug_id, subreq->debug_index);
@@ -255,15 +242,16 @@ static void netfs_do_issue_write(struct netfs_io_stream *stream,
}
void netfs_reissue_write(struct netfs_io_stream *stream,
- struct netfs_io_subrequest *subreq,
- struct iov_iter *source)
+ struct netfs_io_subrequest *subreq)
{
- size_t size = subreq->len - subreq->transferred;
-
// TODO: Use encrypted buffer
- subreq->io_iter = *source;
- iov_iter_advance(source, size);
- iov_iter_truncate(&subreq->io_iter, size);
+ bvecq_pos_unset(&subreq->content);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+ iov_iter_bvec_queue(&subreq->io_iter, ITER_SOURCE,
+ subreq->content.bvecq, subreq->content.slot,
+ subreq->content.offset,
+ subreq->len);
+ iov_iter_advance(&subreq->io_iter, subreq->transferred);
subreq->retry_count++;
subreq->error = 0;
@@ -281,8 +269,13 @@ void netfs_issue_write(struct netfs_io_request *wreq,
if (!subreq)
return;
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+ iov_iter_bvec_queue(&subreq->io_iter, ITER_SOURCE,
+ subreq->content.bvecq, subreq->content.slot,
+ subreq->content.offset,
+ subreq->len);
+
stream->construct = NULL;
- subreq->io_iter.count = subreq->len;
netfs_do_issue_write(stream, subreq);
}
@@ -319,7 +312,6 @@ size_t netfs_advance_write(struct netfs_io_request *wreq,
_debug("part %zx/%zx %zx/%zx", subreq->len, stream->sreq_max_len, part, len);
subreq->len += part;
subreq->nr_segs++;
- stream->submit_extendable_to -= part;
if (subreq->len >= stream->sreq_max_len ||
subreq->nr_segs >= stream->sreq_max_segs ||
@@ -343,7 +335,8 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
struct netfs_io_stream *stream;
struct netfs_group *fgroup; /* TODO: Use this with ceph */
struct netfs_folio *finfo;
- size_t iter_off = 0;
+ struct bvecq *queue = wreq->load_cursor.bvecq;
+ unsigned int slot;
size_t fsize = folio_size(folio), flen = fsize, foff = 0;
uoff_t fpos = folio_pos(folio), i_size;
bool to_eof = false, streamw = false;
@@ -351,12 +344,20 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
_enter("");
- if (rolling_buffer_make_space(&wreq->buffer, wreq->gfp) < 0)
- return -ENOMEM;
+ if (!wreq->spare) {
+ wreq->spare = bvecq_alloc_one(BVECQ_STD_SLOTS, wreq->gfp, true);
+ if (!wreq->spare)
+ return -ENOMEM;
+ }
- /* netfs_perform_write() may shift i_size around the page or from out
- * of the page to beyond it, but cannot move i_size into or through the
- * page since we have it locked.
+ /* netfs_perform_write() may shift i_size around the folio or from out
+ * of the folio to beyond it, but cannot move i_size into or through
+ * the folio since we have it locked.
+ *
+ * Truncate could in theory move i_size into or before the folio, but
+ * it should take steps to prevent writeback from happening
+ * concurrently and should wait for any in-progress writebacks before
+ * proceeding.
*/
i_size = i_size_read(wreq->inode);
@@ -448,8 +449,29 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
trace_netfs_folio(folio, netfs_folio_trace_store_plus);
}
+ /* Institute a new bvec queue segment if the current one is full or if
+ * we encounter a discontiguity. The discontiguity break is important
+ * when it comes to bulk unlocking folios by file range.
+ */
+ if (bvecq_is_full(queue) ||
+ (fpos != wreq->last_end && wreq->last_end > 0)) {
+ bvecq_buffer_append(&wreq->load_cursor, wreq->spare);
+ wreq->spare = NULL;
+
+ queue = wreq->load_cursor.bvecq;
+ bvecq_pos_move(&wreq->dispatch_cursor, queue);
+ wreq->dispatch_cursor.slot = 0;
+ }
+
/* Attach the folio to the rolling buffer. */
- rolling_buffer_append(&wreq->buffer, folio, 0, wreq->gfp);
+ slot = queue->nr_slots;
+ bvec_set_folio(&queue->bv[slot], folio, fsize, 0);
+ trace_netfs_bv_slot(queue, slot);
+ slot++;
+ bvecq_filled_to(queue, slot);
+ wreq->load_cursor.slot = slot;
+ wreq->load_cursor.offset = 0;
+ wreq->last_end = fpos + fsize;
/* Move the submission point forward to allow for write-streaming data
* not starting at the front of the page. We don't do write-streaming
@@ -458,10 +480,18 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
* Also skip uploading for data that's been read and just needs copying
* to the cache.
*/
+ bvecq_pos_nudge(&wreq->dispatch_cursor);
+
for (int s = 0; s < NR_IO_STREAMS; s++) {
+ size_t soff = foff, slen = flen, alignment = 1;
+
+ if (stream->source == NETFS_WRITE_TO_CACHE)
+ alignment = wreq->cache_resources.dio_size;
stream = &wreq->io_streams[s];
- stream->submit_off = foff;
- stream->submit_len = flen;
+ stream->submit_off = round_down(soff, alignment);
+ slen += foff - stream->submit_off;
+ stream->submit_len = round_up(slen, alignment);
+
if (!stream->avail ||
(stream->source == NETFS_WRITE_TO_CACHE && streamw) ||
(stream->source == NETFS_UPLOAD_TO_SERVER &&
@@ -495,14 +525,10 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
break;
stream = &wreq->io_streams[choose_s];
- /* Advance the iterator(s). */
- if (stream->submit_off > iter_off) {
- rolling_buffer_advance(&wreq->buffer, stream->submit_off - iter_off);
- iter_off = stream->submit_off;
- }
+ /* Advance the cursor. */
+ wreq->dispatch_cursor.offset = stream->submit_off;
atomic64_set(&wreq->issued_to, fpos + stream->submit_off);
- stream->submit_extendable_to = fsize - stream->submit_off;
part = netfs_advance_write(wreq, stream, fpos + stream->submit_off,
stream->submit_len, to_eof);
stream->submit_off += part;
@@ -514,9 +540,9 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
debug = true;
}
- if (fsize > iter_off)
- rolling_buffer_advance(&wreq->buffer, fsize - iter_off);
- atomic64_set(&wreq->issued_to, fpos + fsize);
+ bvecq_pos_step(&wreq->dispatch_cursor);
+ /* Order loading the queue before updating the issue_to point */
+ atomic64_set_release(&wreq->issued_to, fpos + fsize);
if (!debug)
kdebug("R=%x: No submit", wreq->debug_id);
@@ -577,6 +603,11 @@ int netfs_writepages(struct address_space *mapping,
goto couldnt_start;
}
+ if (bvecq_buffer_init(&wreq->load_cursor, wreq->gfp, true) < 0)
+ goto nomem;
+ bvecq_pos_set(&wreq->dispatch_cursor, &wreq->load_cursor);
+ bvecq_pos_set(&wreq->collect_cursor, &wreq->dispatch_cursor);
+
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags);
trace_netfs_write(wreq, netfs_write_trace_writeback);
netfs_stat(&netfs_n_wh_writepages);
@@ -601,12 +632,17 @@ int netfs_writepages(struct address_space *mapping,
} while ((folio = writeback_iter(mapping, wbc, folio, &error)));
netfs_end_issue_write(wreq);
+ bvecq_pos_unset(&wreq->load_cursor);
+ bvecq_pos_unset(&wreq->dispatch_cursor);
netfs_wake_collector(wreq);
netfs_put_request(wreq, netfs_rreq_trace_put_return);
_leave(" = %d", error);
return error;
+nomem:
+ error = -ENOMEM;
+ netfs_put_failed_request(wreq);
couldnt_start:
if (error == -ENOMEM) {
folio_redirty_for_writepage(wbc, folio);
@@ -627,20 +663,24 @@ EXPORT_SYMBOL(netfs_writepages);
* netfs_writeback_single - Write back a monolithic payload
* @mapping: The mapping to write from
* @wbc: Hints from the VM
- * @iter: Data to write.
+ * @iter: Buffer to write from
+ * @len: Amount to write from buffer
*
- * Write a monolithic, non-pagecache object back to the server and/or
- * the cache.
+ * Write a monolithic, non-pagecache object back to the server and/or the
+ * cache. There's a maximum of one subrequest per stream. The buffer should
+ * be rounded out sufficiently that it can accommodate cache DIO rounding.
*
* Return: 0 if successful; 1 if skipped due to lock conflict and WB_SYNC_NONE;
* or a negative error code.
+ * the cache. There's a maximum of one subrequest per stream.
*/
int netfs_writeback_single(struct address_space *mapping,
struct writeback_control *wbc,
- struct iov_iter *iter)
+ struct iov_iter *iter, size_t len)
{
struct netfs_io_request *wreq;
struct netfs_inode *ictx = netfs_inode(mapping->host);
+ size_t clen;
int ret;
if (!netfs_wb_begin(ictx, wbc->sync_mode == WB_SYNC_NONE)) {
@@ -654,9 +694,27 @@ int netfs_writeback_single(struct address_space *mapping,
ret = PTR_ERR(wreq);
goto couldnt_start;
}
+ wreq->len = len;
+ clen = len;
+
+ if (wreq->cache_resources.dio_size > 1) {
+ clen = round_up(len, wreq->cache_resources.dio_size);
+ if (clen > iov_iter_count(iter)) {
+ ret = -EIO;
+ goto cleanup_free;
+ }
+ }
- wreq->buffer.iter = *iter;
- wreq->len = iov_iter_count(iter);
+ ret = netfs_extract_iter(iter, clen, INT_MAX, &wreq->dispatch_cursor.bvecq,
+ 0, wreq->gfp);
+ if (ret < 0)
+ goto cleanup_free;
+ if (ret < clen) {
+ ret = -EIO;
+ goto cleanup_free;
+ }
+
+ bvecq_pos_set(&wreq->collect_cursor, &wreq->dispatch_cursor);
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags);
trace_netfs_write(wreq, netfs_write_trace_writeback_single);
@@ -676,12 +734,14 @@ int netfs_writeback_single(struct address_space *mapping,
subreq = stream->construct;
subreq->len = wreq->len;
+ if (stream->source == NETFS_WRITE_TO_CACHE)
+ subreq->len = clen;
stream->submit_len = subreq->len;
- stream->submit_extendable_to = round_up(wreq->len, PAGE_SIZE);
netfs_issue_write(wreq, stream);
}
+ wreq->submitted = wreq->len;
netfs_all_subreqs_queued(wreq);
netfs_wake_collector(wreq);
@@ -689,11 +749,15 @@ int netfs_writeback_single(struct address_space *mapping,
* wait before modifying.
*/
ret = netfs_wait_for_write(wreq);
+ if (ret > 0)
+ ret = 0;
netfs_put_request(wreq, netfs_rreq_trace_put_return);
_leave(" = %d", ret);
return ret;
+cleanup_free:
+ netfs_put_failed_request(wreq);
couldnt_start:
netfs_wb_end(ictx);
_leave(" = %d", ret);
diff --git a/fs/netfs/write_retry.c b/fs/netfs/write_retry.c
index 747d57467b4f..c61bed687244 100644
--- a/fs/netfs/write_retry.c
+++ b/fs/netfs/write_retry.c
@@ -17,15 +17,17 @@
static void netfs_retry_write_stream(struct netfs_io_request *wreq,
struct netfs_io_stream *stream)
{
+ struct bvecq_pos dispatch_cursor = {};
struct list_head *next;
_enter("R=%x[%x:]", wreq->debug_id, stream->stream_nr);
if (list_empty(&stream->subrequests))
return;
+ if (WARN_ON_ONCE(stream->source != NETFS_UPLOAD_TO_SERVER))
+ return; /* Shouldn't be retrying cache writes. */
- if (stream->source == NETFS_UPLOAD_TO_SERVER &&
- wreq->netfs_ops->retry_request)
+ if (wreq->netfs_ops->retry_request)
wreq->netfs_ops->retry_request(wreq, stream);
if (unlikely(stream->failed))
@@ -39,12 +41,8 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
break;
if (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
- struct iov_iter source;
-
- netfs_reset_iter(subreq);
- source = subreq->io_iter;
netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_write(stream, subreq, &source);
+ netfs_reissue_write(stream, subreq);
}
}
return;
@@ -54,11 +52,12 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
do {
struct netfs_io_subrequest *subreq = NULL, *from, *to, *tmp;
- struct iov_iter source;
uoff_t start, len;
size_t part;
bool boundary = false;
+ bvecq_pos_unset(&dispatch_cursor);
+
/* Go through the stream and find the next span of contiguous
* data that we then rejig (cifs, for example, needs the wsize
* renegotiating) and reissue.
@@ -70,7 +69,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
if (test_bit(NETFS_SREQ_FAILED, &from->flags) ||
!test_bit(NETFS_SREQ_NEED_RETRY, &from->flags))
- return;
+ goto out;
for (;;) {
/* Read pointer to subreq before reading subreq state. */
@@ -79,7 +78,8 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
break;
subreq = list_entry(next, struct netfs_io_subrequest, rreq_link);
- if (subreq->start + subreq->transferred != start + len ||
+ if (subreq->start != start + len ||
+ subreq->transferred > 0 ||
test_bit(NETFS_SREQ_BOUNDARY, &subreq->flags) ||
!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
break;
@@ -90,11 +90,13 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
/* Determine the set of buffers we're going to use. Each
* subreq gets a subset of a single overall contiguous buffer.
*/
- netfs_reset_iter(from);
- source = from->io_iter;
- source.count = len;
+ bvecq_pos_transfer(&dispatch_cursor, &from->dispatch_pos);
+ bvecq_pos_advance(&dispatch_cursor, from->transferred);
- /* Work through the sublist. */
+ /* Work through the sublist. The chain of buffers we're going
+ * to fill is attached to dispatch_cursor and we need to read
+ * 'len' amount of data from 'start'.
+ */
subreq = from;
list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) {
if (!len)
@@ -104,16 +106,23 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
subreq->len = len;
__clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+ subreq->transferred = 0;
+
+ bvecq_pos_unset(&subreq->content);
+ bvecq_pos_unset(&subreq->dispatch_pos);
/* Renegotiate max_len (wsize) */
stream->sreq_max_len = len;
+ stream->sreq_max_segs = INT_MAX;
stream->prepare_write(subreq);
- part = umin(len, stream->sreq_max_len);
- if (unlikely(stream->sreq_max_segs))
- part = netfs_limit_iter(&source, 0, part, stream->sreq_max_segs);
+ bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
+ part = bvecq_slice(&dispatch_cursor,
+ umin(len, stream->sreq_max_len),
+ stream->sreq_max_segs,
+ &subreq->nr_segs);
subreq->len = part;
- subreq->transferred = 0;
+
len -= part;
start += part;
if (len && subreq == to &&
@@ -121,7 +130,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
boundary = true;
netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_write(stream, subreq, &source);
+ netfs_reissue_write(stream, subreq);
if (subreq == to)
break;
}
@@ -173,17 +182,19 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
netfs_stat(&netfs_n_wh_upload);
stream->sreq_max_len = umin(len, wreq->wsize);
break;
- case NETFS_WRITE_TO_CACHE:
- netfs_stat(&netfs_n_wh_write);
- break;
default:
WARN_ON_ONCE(1);
}
stream->prepare_write(subreq);
- part = umin(len, stream->sreq_max_len);
+ bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
+ part = bvecq_slice(&dispatch_cursor,
+ umin(len, stream->sreq_max_len),
+ stream->sreq_max_segs,
+ &subreq->nr_segs);
subreq->len = subreq->transferred + part;
+
len -= part;
start += part;
if (!len && boundary) {
@@ -191,13 +202,16 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
boundary = false;
}
- netfs_reissue_write(stream, subreq, &source);
+ netfs_reissue_write(stream, subreq);
if (!len)
break;
} while (len);
} while (!list_is_head(next, &stream->subrequests));
+
+out:
+ bvecq_pos_unset(&dispatch_cursor);
}
/*
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index ef1068a274a8..a58d2b6914ea 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -19,12 +19,13 @@
#include <linux/pagemap.h>
#include <linux/bvecq.h>
#include <linux/uio.h>
-#include <linux/rolling_buffer.h>
enum netfs_sreq_ref_trace;
typedef struct mempool mempool_t;
+struct readahead_control;
+struct netfs_io_request;
+struct netfs_io_subrequest;
struct fscache_occupancy;
-struct folio_queue;
/**
* folio_start_private_2 - Start an fscache write on a folio. [DEPRECATED]
@@ -145,7 +146,6 @@ struct netfs_io_stream {
unsigned int sreq_max_segs; /* 0 or max number of segments in an iterator */
unsigned int submit_off; /* Folio offset we're submitting from */
unsigned int submit_len; /* Amount of data left to submit */
- unsigned int submit_extendable_to; /* Amount I/O can be rounded up to */
void (*prepare_write)(struct netfs_io_subrequest *subreq);
void (*issue_write)(struct netfs_io_subrequest *subreq);
/* Collection tracking */
@@ -188,6 +188,8 @@ struct netfs_io_subrequest {
struct netfs_io_request *rreq; /* Supervising I/O request */
struct work_struct work;
struct list_head rreq_link; /* Link in rreq->subrequests */
+ struct bvecq_pos dispatch_pos; /* Bookmark in the combined queue of the start */
+ struct bvecq_pos content; /* The (copied) content of the subrequest */
struct iov_iter io_iter; /* Iterator for this subrequest */
uoff_t start; /* Where to start the I/O */
size_t len; /* Size of the I/O */
@@ -248,13 +250,14 @@ struct netfs_io_request {
struct netfs_io_stream io_streams[2]; /* Streams of parallel I/O operations */
#define NR_IO_STREAMS 2 //wreq->nr_io_streams
struct netfs_group *group; /* Writeback group being written back */
- struct rolling_buffer buffer; /* Unencrypted buffer */
-#define NETFS_ROLLBUF_PUT_MARK ROLLBUF_MARK_1
-#define NETFS_ROLLBUF_PAGECACHE_MARK ROLLBUF_MARK_2
+ struct bvecq *spare; /* Advance allocation of bvecq */
+ struct bvecq_pos load_cursor; /* Point at which new folios are loaded in */
+ struct bvecq_pos dispatch_cursor; /* Point from which buffers are dispatched */
+ struct bvecq_pos collect_cursor; /* Clear-up point of I/O buffer */
wait_queue_head_t waitq; /* Processor waiter */
void *netfs_priv; /* Private data for the netfs */
void *netfs_priv2; /* Private data for the netfs */
- struct bio_vec *direct_bv; /* DIO buffer list (when handling iovec-iter) */
+ uoff_t last_end; /* End pos of last folio submitted */
uoff_t submitted; /* Amount submitted for I/O so far */
uoff_t len; /* Length of the request */
size_t transferred; /* Amount to be indicated as transferred */
@@ -269,7 +272,6 @@ struct netfs_io_request {
uoff_t abandon_to; /* Position to abandon folios to */
const struct folio *no_unlock_folio; /* Don't unlock this folio after read */
gfp_t gfp; /* GFP flags to use */
- unsigned int direct_bv_count; /* Number of elements in direct_bv[] */
unsigned int debug_id;
unsigned int rsize; /* Maximum read size (0 for none) */
unsigned int wsize; /* Maximum write size (0 for none) */
@@ -277,7 +279,6 @@ struct netfs_io_request {
unsigned int nr_group_rel; /* Number of refs to release on ->group */
spinlock_t lock; /* Lock for queuing subreqs */
enum netfs_io_origin origin; /* Origin of the request */
- bool direct_bv_unpin; /* T if direct_bv[] must be unpinned */
refcount_t ref;
unsigned long flags;
#define NETFS_RREQ_IN_PROGRESS 0 /* Unlocked when the request completes (has ref) */
@@ -428,7 +429,7 @@ void netfs_single_mark_inode_dirty(struct inode *inode);
ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_iter *iter);
int netfs_writeback_single(struct address_space *mapping,
struct writeback_control *wbc,
- struct iov_iter *iter);
+ struct iov_iter *iter, size_t len);
/* Address operations API */
struct readahead_control;
@@ -473,12 +474,6 @@ void netfs_end_io_write(struct inode *inode);
int netfs_start_io_direct(struct inode *inode);
void netfs_end_io_direct(struct inode *inode);
-/* Miscellaneous APIs. */
-struct folio_queue *netfs_folioq_alloc(unsigned int rreq_id, gfp_t gfp,
- unsigned int trace /*enum netfs_folioq_trace*/);
-void netfs_folioq_free(struct folio_queue *folioq,
- unsigned int trace /*enum netfs_trace_folioq*/);
-
/* Buffer wrangling helpers API. */
int netfs_alloc_folioq_buffer(struct address_space *mapping,
struct folio_queue **_buffer,
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 927287a7290c..4b2ffd14a1ca 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -227,7 +227,9 @@
EM(netfs_folio_trace_redirtied, "redirtied") \
EM(netfs_folio_trace_store, "store") \
EM(netfs_folio_trace_store_copy, "store-copy") \
- E_(netfs_folio_trace_store_plus, "store+")
+ EM(netfs_folio_trace_store_plus, "store+") \
+ EM(netfs_folio_trace_zero, "zero") \
+ E_(netfs_folio_trace_zero_ra, "zero-ra")
#define netfs_collect_contig_traces \
EM(netfs_contig_trace_collect, "Collect") \
@@ -240,13 +242,13 @@
EM(netfs_trace_donate_to_next, "to-next") \
E_(netfs_trace_donate_to_deferred_next, "defer-next")
-#define netfs_folioq_traces \
- EM(netfs_trace_folioq_alloc_buffer, "alloc-buf") \
- EM(netfs_trace_folioq_clear, "clear") \
- EM(netfs_trace_folioq_delete, "delete") \
- EM(netfs_trace_folioq_make_space, "make-space") \
- EM(netfs_trace_folioq_rollbuf_init, "roll-init") \
- E_(netfs_trace_folioq_read_progress, "r-progress")
+#define netfs_bvecq_traces \
+ EM(netfs_trace_bvecq_alloc_buffer, "alloc-buf") \
+ EM(netfs_trace_bvecq_clear, "clear") \
+ EM(netfs_trace_bvecq_delete, "delete") \
+ EM(netfs_trace_bvecq_make_space, "make-space") \
+ EM(netfs_trace_bvecq_rollbuf_init, "roll-init") \
+ E_(netfs_trace_bvecq_read_progress, "r-progress")
#ifndef __NETFS_DECLARE_TRACE_ENUMS_ONCE_ONLY
#define __NETFS_DECLARE_TRACE_ENUMS_ONCE_ONLY
@@ -266,7 +268,7 @@ enum netfs_sreq_ref_trace { netfs_sreq_ref_traces } __mode(byte);
enum netfs_folio_trace { netfs_folio_traces } __mode(byte);
enum netfs_collect_contig_trace { netfs_collect_contig_traces } __mode(byte);
enum netfs_donate_trace { netfs_donate_traces } __mode(byte);
-enum netfs_folioq_trace { netfs_folioq_traces } __mode(byte);
+enum netfs_bvecq_trace { netfs_bvecq_traces } __mode(byte);
#endif
@@ -290,7 +292,7 @@ netfs_sreq_ref_traces;
netfs_folio_traces;
netfs_collect_contig_traces;
netfs_donate_traces;
-netfs_folioq_traces;
+netfs_bvecq_traces;
/*
* Now redefine the EM() and E_() macros to map the enums to the strings that
@@ -394,10 +396,10 @@ TRACE_EVENT(netfs_sreq,
__entry->len = sreq->len;
__entry->transferred = sreq->transferred;
__entry->start = sreq->start;
- __entry->slot = sreq->io_iter.folioq_slot;
+ __entry->slot = sreq->content.slot;
),
- TP_printk("R=%08x[%x] %s %s f=%03x s=%llx %zx/%zx s=%u e=%d",
+ TP_printk("R=%08x[%x] %s %s f=%03x s=%llx %zx/%zx bv=%u e=%d",
__entry->rreq, __entry->index,
__print_symbolic(__entry->source, netfs_sreq_sources),
__print_symbolic(__entry->what, netfs_sreq_traces),
@@ -769,27 +771,25 @@ TRACE_EVENT(netfs_collect_stream,
__entry->collected_to, __entry->issued_to)
);
-TRACE_EVENT(netfs_folioq,
- TP_PROTO(const struct folio_queue *fq,
- enum netfs_folioq_trace trace),
+TRACE_EVENT(netfs_bvecq,
+ TP_PROTO(const struct bvecq *bq,
+ enum netfs_bvecq_trace trace),
- TP_ARGS(fq, trace),
+ TP_ARGS(bq, trace),
TP_STRUCT__entry(
- __field(unsigned int, rreq)
__field(unsigned int, id)
- __field(enum netfs_folioq_trace, trace)
+ __field(enum netfs_bvecq_trace, trace)
),
TP_fast_assign(
- __entry->rreq = fq ? fq->rreq_id : 0;
- __entry->id = fq ? fq->debug_id : 0;
+ __entry->id = bq ? bq->priv : 0;
__entry->trace = trace;
),
- TP_printk("R=%08x fq=%x %s",
- __entry->rreq, __entry->id,
- __print_symbolic(__entry->trace, netfs_folioq_traces))
+ TP_printk("bq=%x %s",
+ __entry->id,
+ __print_symbolic(__entry->trace, netfs_bvecq_traces))
);
TRACE_EVENT(netfs_read_progress_at,
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 25/35] smbdirect: Remove support for ITER_FOLIOQ from smbdirect_map_sges_from_iter()
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (23 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 24/35] netfs: Switch to using bvecq rather than folio_queue and rolling_buffer David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 26/35] netfs: Remove netfs_alloc/free_folioq_buffer() David Howells
` (10 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, Shyam Prasad N, Tom Talpey
netfslib now only presents an bvecq queue and an associated ITER_BVECQ
iterator to the filesystem, so it isn't going to see the ITER_FOLIOQ
iterator. So remove that code.
Netfslib also won't supply ITER_BVEC/KVEC iterators, though smbdirect
might; further in future, it won't supply iterators at all, but rather a
bvecq slice (that can be used to construct an iterator).
Signed-off-by: David Howells <dhowells@redhat.com>
Acked-by: Stefan Metzmacher <metze@samba.org>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Namjae Jeon <linkinjeon@kernel.org>
cc: Stefan Metzmacher <metze@samba.org>
cc: Shyam Prasad N <sprasad@microsoft.com>
cc: Tom Talpey <tom@talpey.com>
cc: linux-cifs@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/smb/smbdirect/connection.c | 68 -----------------------------------
1 file changed, 68 deletions(-)
diff --git a/fs/smb/smbdirect/connection.c b/fs/smb/smbdirect/connection.c
index f5e7e75dd809..f217cf127cf9 100644
--- a/fs/smb/smbdirect/connection.c
+++ b/fs/smb/smbdirect/connection.c
@@ -6,7 +6,6 @@
#include "internal.h"
#include <linux/bvecq.h>
-#include <linux/folio_queue.h>
struct smbdirect_map_sges {
struct ib_sge *sge;
@@ -2131,70 +2130,6 @@ static ssize_t smbdirect_map_sges_from_kvec(struct iov_iter *iter,
return ret;
}
-/*
- * Extract folio fragments from a FOLIOQ-class iterator and add them to an
- * ib_sge list. The folios are not pinned.
- */
-static ssize_t smbdirect_map_sges_from_folioq(struct iov_iter *iter,
- struct smbdirect_map_sges *state,
- ssize_t maxsize)
-{
- const struct folio_queue *folioq = iter->folioq;
- unsigned int slot = iter->folioq_slot;
- ssize_t ret = 0;
- size_t offset = iter->iov_offset;
-
- if (WARN_ON_ONCE(!folioq))
- return -EIO;
-
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- if (WARN_ON_ONCE(!folioq))
- return -EIO;
- slot = 0;
- }
-
- do {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t fsize = folioq_folio_size(folioq, slot);
-
- if (offset < fsize) {
- size_t part = umin(maxsize, fsize - offset);
- bool ok;
-
- ok = smbdirect_map_sges_single_page(state,
- folio_page(folio, 0),
- offset,
- part);
- if (!ok)
- return -EIO;
-
- offset += part;
- ret += part;
- maxsize -= part;
- }
-
- if (offset >= fsize) {
- offset = 0;
- slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- if (!folioq->next) {
- WARN_ON_ONCE(ret < iter->count);
- break;
- }
- folioq = folioq->next;
- slot = 0;
- }
- }
- } while (state->num_sge < state->max_sge && maxsize > 0);
-
- iter->folioq = folioq;
- iter->folioq_slot = slot;
- iter->iov_offset = offset;
- iter->count -= ret;
- return ret;
-}
-
/*
* Extract page fragments from up to the given amount of the source iterator
* and build up an ib_sge list that refers to all of those bits. The ib_sge list
@@ -2225,9 +2160,6 @@ static ssize_t smbdirect_map_sges_from_iter(struct iov_iter *iter, size_t len,
case ITER_KVEC:
ret = smbdirect_map_sges_from_kvec(iter, state, len);
break;
- case ITER_FOLIOQ:
- ret = smbdirect_map_sges_from_folioq(iter, state, len);
- break;
default:
WARN_ONCE(1, "iov_iter_type[%u]\n", iov_iter_type(iter));
return -EIO;
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 26/35] netfs: Remove netfs_alloc/free_folioq_buffer()
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (24 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 25/35] smbdirect: Remove support for ITER_FOLIOQ from smbdirect_map_sges_from_iter() David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 27/35] netfs: Remove netfs_extract_user_iter() David Howells
` (9 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Remove netfs_alloc/free_folioq_buffer() as bvecq_alloc_*() and bvecq_put()
should now be used instead.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Namjae Jeon <linkinjeon@kernel.org>
cc: linux-cifs@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/afs/dir_edit.c | 1 -
fs/netfs/misc.c | 98 -----------------------------------------
fs/smb/client/smb2ops.c | 1 -
include/linux/netfs.h | 6 ---
4 files changed, 106 deletions(-)
diff --git a/fs/afs/dir_edit.c b/fs/afs/dir_edit.c
index 01c6265e8865..31fdef23c6b6 100644
--- a/fs/afs/dir_edit.c
+++ b/fs/afs/dir_edit.c
@@ -10,7 +10,6 @@
#include <linux/namei.h>
#include <linux/pagemap.h>
#include <linux/iversion.h>
-#include <linux/folio_queue.h>
#include "internal.h"
#include "xdr_fs.h"
diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c
index a2a2aeb1bb0d..4fa09560fb4b 100644
--- a/fs/netfs/misc.c
+++ b/fs/netfs/misc.c
@@ -8,104 +8,6 @@
#include <linux/swap.h>
#include "internal.h"
-#if 0
-/**
- * netfs_alloc_folioq_buffer - Allocate buffer space into a folio queue
- * @mapping: Address space to set on the folio (or NULL).
- * @_buffer: Pointer to the folio queue to add to (may point to a NULL; updated).
- * @_cur_size: Current size of the buffer (updated).
- * @size: Target size of the buffer.
- * @gfp: The allocation constraints.
- */
-int netfs_alloc_folioq_buffer(struct address_space *mapping,
- struct folio_queue **_buffer,
- size_t *_cur_size, ssize_t size, gfp_t gfp)
-{
- struct folio_queue *tail = *_buffer, *p;
-
- size = round_up(size, PAGE_SIZE);
- if (*_cur_size >= size)
- return 0;
-
- if (tail)
- while (tail->next)
- tail = tail->next;
-
- do {
- struct folio *folio;
- int order = 0, slot;
-
- if (!tail || folioq_full(tail)) {
- p = netfs_folioq_alloc(0, GFP_NOFS, netfs_trace_folioq_alloc_buffer);
- if (!p)
- return -ENOMEM;
- if (tail) {
- tail->next = p;
- p->prev = tail;
- } else {
- *_buffer = p;
- }
- tail = p;
- }
-
- if (size - *_cur_size > PAGE_SIZE)
- order = umin(ilog2(size - *_cur_size) - PAGE_SHIFT,
- MAX_PAGECACHE_ORDER);
-
- folio = folio_alloc(gfp, order);
- if (!folio && order > 0)
- folio = folio_alloc(gfp, 0);
- if (!folio)
- return -ENOMEM;
-
- folio->mapping = mapping;
- folio->index = *_cur_size / PAGE_SIZE;
- trace_netfs_folio(folio, netfs_folio_trace_alloc_buffer);
- slot = folioq_append_mark(tail, folio);
- *_cur_size += folioq_folio_size(tail, slot);
- } while (*_cur_size < size);
-
- return 0;
-}
-EXPORT_SYMBOL(netfs_alloc_folioq_buffer);
-
-/**
- * netfs_free_folioq_buffer - Free a folio queue.
- * @fq: The start of the folio queue to free
- *
- * Free up a chain of folio_queues and, if marked, the marked folios they point
- * to.
- */
-void netfs_free_folioq_buffer(struct folio_queue *fq)
-{
- struct folio_queue *next;
- struct folio_batch fbatch;
-
- folio_batch_init(&fbatch);
-
- for (; fq; fq = next) {
- for (int slot = 0; slot < folioq_count(fq); slot++) {
- struct folio *folio = folioq_folio(fq, slot);
-
- if (!folio ||
- !folioq_is_marked(fq, slot))
- continue;
-
- trace_netfs_folio(folio, netfs_folio_trace_put);
- if (folio_batch_add(&fbatch, folio))
- folio_batch_release(&fbatch);
- }
-
- netfs_stat_d(&netfs_n_folioq);
- next = fq->next;
- kfree(fq);
- }
-
- folio_batch_release(&fbatch);
-}
-EXPORT_SYMBOL(netfs_free_folioq_buffer);
-#endif
-
/**
* netfs_dirty_folio - Mark folio dirty and pin a cache object for writeback
* @mapping: The mapping the folio belongs to.
diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c
index 885f0cc861ed..97823bcf198c 100644
--- a/fs/smb/client/smb2ops.c
+++ b/fs/smb/client/smb2ops.c
@@ -13,7 +13,6 @@
#include <linux/sort.h>
#include <crypto/aead.h>
#include <linux/fiemap.h>
-#include <linux/folio_queue.h>
#include <uapi/linux/magic.h>
#include "cifsfs.h"
#include "cifsglob.h"
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index a58d2b6914ea..926c3c6f311b 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -474,12 +474,6 @@ void netfs_end_io_write(struct inode *inode);
int netfs_start_io_direct(struct inode *inode);
void netfs_end_io_direct(struct inode *inode);
-/* Buffer wrangling helpers API. */
-int netfs_alloc_folioq_buffer(struct address_space *mapping,
- struct folio_queue **_buffer,
- size_t *_cur_size, ssize_t size, gfp_t gfp);
-void netfs_free_folioq_buffer(struct folio_queue *fq);
-
/* Writeback exclusion API. */
bool netfs_wb_begin(struct netfs_inode *ictx, bool nowait);
void netfs_wb_end(struct netfs_inode *ictx);
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 27/35] netfs: Remove netfs_extract_user_iter()
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (25 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 26/35] netfs: Remove netfs_alloc/free_folioq_buffer() David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 28/35] iov_iter: Remove ITER_FOLIOQ David Howells
` (8 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Remove netfs_extract_user_iter() as it has been replaced with
netfs_extract_iter().
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/iterator.c | 104 ------------------------------------------
include/linux/netfs.h | 3 --
2 files changed, 107 deletions(-)
diff --git a/fs/netfs/iterator.c b/fs/netfs/iterator.c
index c207eacd5c5d..65164a1761e3 100644
--- a/fs/netfs/iterator.c
+++ b/fs/netfs/iterator.c
@@ -157,110 +157,6 @@ ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pag
EXPORT_SYMBOL_GPL(netfs_extract_iter);
#if 0
-/**
- * netfs_extract_user_iter - Extract the pages from a user iterator into a bvec
- * @orig: The original iterator
- * @orig_len: The amount of iterator to copy
- * @new: The iterator to be set up
- * @extraction_flags: Flags to qualify the request
- *
- * Extract the page fragments from the given amount of the source iterator and
- * build up a second iterator that refers to all of those bits. This allows
- * the original iterator to be disposed of.
- *
- * @extraction_flags can have ITER_ALLOW_P2PDMA set to request peer-to-peer DMA be
- * allowed on the pages extracted.
- *
- * On success, the number of elements in the bvec is returned, the original
- * iterator will have been advanced by the amount extracted.
- *
- * The iov_iter_extract_mode() function should be used to query how cleanup
- * should be performed.
- */
-ssize_t netfs_extract_user_iter(struct iov_iter *orig, size_t orig_len,
- struct iov_iter *new,
- iov_iter_extraction_t extraction_flags)
-{
- struct bio_vec *bv = NULL;
- struct page **pages;
- unsigned int cur_npages;
- unsigned int max_pages;
- unsigned int npages = 0;
- unsigned int i;
- ssize_t ret = 0;
- size_t count = orig_len, offset, len;
- size_t bv_size, pg_size;
-
- if (WARN_ON_ONCE(!iter_is_ubuf(orig) && !iter_is_iovec(orig)))
- return -EIO;
-
- max_pages = iov_iter_npages(orig, INT_MAX);
- bv_size = array_size(max_pages, sizeof(*bv));
- bv = kvmalloc(bv_size, GFP_KERNEL);
- if (!bv)
- return -ENOMEM;
-
- /* Put the page list at the end of the bvec list storage. bvec
- * elements are larger than page pointers, so as long as we work
- * 0->last, we should be fine.
- */
- pg_size = array_size(max_pages, sizeof(*pages));
- pages = (void *)bv + bv_size - pg_size;
-
- while (count && npages < max_pages) {
- ret = iov_iter_extract_pages(orig, &pages, count,
- max_pages - npages, extraction_flags,
- &offset);
- if (unlikely(ret <= 0)) {
- ret = ret ?: -EIO;
- break;
- }
-
- if (WARN(ret > count,
- "%s: extract_pages overrun %zd > %zu bytes\n",
- __func__, ret, count)) {
- ret = -EIO;
- break;
- }
-
- cur_npages = DIV_ROUND_UP(offset + ret, PAGE_SIZE);
- if (WARN(cur_npages > max_pages - npages,
- "%s: extract_pages overrun %u > %u pages\n",
- __func__, npages + cur_npages, max_pages)) {
- ret = -EIO;
- break;
- }
-
- count -= ret;
- ret += offset;
-
- for (i = 0; i < cur_npages; i++) {
- len = ret > PAGE_SIZE ? PAGE_SIZE : ret;
- bvec_set_page(bv + npages + i, *pages++, len - offset, offset);
- ret -= len;
- offset = 0;
- }
-
- npages += cur_npages;
- }
-
- /* Note: Don't try to clean up after EIO. Either we got no pages, so
- * nothing to clean up, or we got a buffer overrun, memory corruption
- * and can't trust the stuff in the buffer (a WARN was emitted).
- */
-
- if (ret < 0 && (ret == -ENOMEM || npages == 0)) {
- for (i = 0; i < npages; i++)
- unpin_user_page(bv[i].bv_page);
- kvfree(bv);
- return ret;
- }
-
- iov_iter_bvec(new, orig->data_source, bv, npages, orig_len - count);
- return npages;
-}
-EXPORT_SYMBOL_GPL(netfs_extract_user_iter);
-
/*
* Select the span of a bvec iterator we're going to use. Limit it by both maximum
* size and maximum number of segments. Returns the size of the span in bytes.
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 926c3c6f311b..8de9ce35f70d 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -459,9 +459,6 @@ void netfs_put_subrequest(struct netfs_io_subrequest *subreq,
ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages,
struct bvecq **_bvecq_head,
iov_iter_extraction_t extraction_flags, gfp_t gfp);
-ssize_t netfs_extract_user_iter(struct iov_iter *orig, size_t orig_len,
- struct iov_iter *new,
- iov_iter_extraction_t extraction_flags);
size_t netfs_limit_iter(const struct iov_iter *iter, size_t start_offset,
size_t max_size, size_t max_segs);
void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq);
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 28/35] iov_iter: Remove ITER_FOLIOQ
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (26 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 27/35] netfs: Remove netfs_extract_user_iter() David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 29/35] netfs: Remove folio_queue and rolling_buffer David Howells
` (7 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Remove ITER_FOLIOQ as it's no longer used.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
include/linux/iov_iter.h | 65 +--------
include/linux/uio.h | 12 --
lib/iov_iter.c | 175 +-----------------------
lib/scatterlist.c | 69 +---------
lib/tests/kunit_iov_iter.c | 271 -------------------------------------
5 files changed, 7 insertions(+), 585 deletions(-)
diff --git a/include/linux/iov_iter.h b/include/linux/iov_iter.h
index ec0aa2893689..39ffd5878f50 100644
--- a/include/linux/iov_iter.h
+++ b/include/linux/iov_iter.h
@@ -11,7 +11,6 @@
#include <linux/uio.h>
#include <linux/bvec.h>
#include <linux/bvecq.h>
-#include <linux/folio_queue.h>
typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len,
void *priv, void *priv2);
@@ -207,62 +206,6 @@ size_t iterate_bvecq(struct iov_iter *iter, size_t len, void *priv, void *priv2,
return progress;
}
-/*
- * Handle ITER_FOLIOQ.
- */
-static __always_inline
-size_t iterate_folioq(struct iov_iter *iter, size_t len, void *priv, void *priv2,
- iov_step_f step)
-{
- const struct folio_queue *folioq = iter->folioq;
- unsigned int slot = iter->folioq_slot;
- size_t progress = 0, skip = iter->iov_offset;
-
- if (slot == folioq_nr_slots(folioq)) {
- /* The iterator may have been extended. */
- folioq = folioq->next;
- slot = 0;
- }
-
- do {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t part, remain = 0, consumed;
- size_t fsize;
- void *base;
-
- if (!folio)
- break;
-
- fsize = folioq_folio_size(folioq, slot);
- if (skip < fsize) {
- base = kmap_local_folio(folio, skip);
- part = umin(len, PAGE_SIZE - skip % PAGE_SIZE);
- remain = step(base, progress, part, priv, priv2);
- kunmap_local(base);
- consumed = part - remain;
- len -= consumed;
- progress += consumed;
- skip += consumed;
- }
- if (skip >= fsize) {
- skip = 0;
- slot++;
- if (slot == folioq_nr_slots(folioq) && folioq->next) {
- folioq = folioq->next;
- slot = 0;
- }
- }
- if (remain)
- break;
- } while (len);
-
- iter->folioq_slot = slot;
- iter->folioq = folioq;
- iter->iov_offset = skip;
- iter->count -= progress;
- return progress;
-}
-
/*
* Handle ITER_XARRAY.
*/
@@ -374,8 +317,6 @@ size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv,
return iterate_kvec(iter, len, priv, priv2, step);
if (iov_iter_is_bvecq(iter))
return iterate_bvecq(iter, len, priv, priv2, step);
- if (iov_iter_is_folioq(iter))
- return iterate_folioq(iter, len, priv, priv2, step);
if (iov_iter_is_xarray(iter))
return iterate_xarray(iter, len, priv, priv2, step);
return iterate_discard(iter, len, priv, priv2, step);
@@ -410,8 +351,8 @@ size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv,
* buffer is presented in segments, which for kernel iteration are broken up by
* physical pages and mapped, with the mapped address being presented.
*
- * [!] Note This will only handle BVEC, KVEC, BVECQ, FOLIOQ, XARRAY and
- * DISCARD-type iterators; it will not handle UBUF or IOVEC-type iterators.
+ * [!] Note This will only handle BVEC, KVEC, BVECQ, XARRAY and DISCARD-type
+ * iterators; it will not handle UBUF or IOVEC-type iterators.
*
* A step functions, @step, must be provided, one for handling mapped kernel
* addresses and the other is given user addresses which have the potential to
@@ -440,8 +381,6 @@ size_t iterate_and_advance_kernel(struct iov_iter *iter, size_t len, void *priv,
return iterate_kvec(iter, len, priv, priv2, step);
if (iov_iter_is_bvecq(iter))
return iterate_bvecq(iter, len, priv, priv2, step);
- if (iov_iter_is_folioq(iter))
- return iterate_folioq(iter, len, priv, priv2, step);
if (iov_iter_is_xarray(iter))
return iterate_xarray(iter, len, priv, priv2, step);
return iterate_discard(iter, len, priv, priv2, step);
diff --git a/include/linux/uio.h b/include/linux/uio.h
index f7cfa6ea8213..e84a0c4f28c6 100644
--- a/include/linux/uio.h
+++ b/include/linux/uio.h
@@ -11,7 +11,6 @@
#include <uapi/linux/uio.h>
struct page;
-struct folio_queue;
typedef unsigned int __bitwise iov_iter_extraction_t;
@@ -27,7 +26,6 @@ enum iter_type {
ITER_BVEC,
ITER_KVEC,
ITER_BVECQ,
- ITER_FOLIOQ,
ITER_XARRAY,
ITER_DISCARD,
};
@@ -70,7 +68,6 @@ struct iov_iter {
const struct kvec *kvec;
const struct bio_vec *bvec;
const struct bvecq *bvecq;
- const struct folio_queue *folioq;
struct xarray *xarray;
void __user *ubuf;
};
@@ -80,7 +77,6 @@ struct iov_iter {
union {
unsigned long nr_segs;
u16 bvecq_slot;
- u8 folioq_slot;
loff_t xarray_start;
};
};
@@ -153,11 +149,6 @@ static inline bool iov_iter_is_bvecq(const struct iov_iter *i)
return iov_iter_type(i) == ITER_BVECQ;
}
-static inline bool iov_iter_is_folioq(const struct iov_iter *i)
-{
- return iov_iter_type(i) == ITER_FOLIOQ;
-}
-
static inline bool iov_iter_is_xarray(const struct iov_iter *i)
{
return iov_iter_type(i) == ITER_XARRAY;
@@ -306,9 +297,6 @@ void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count);
void iov_iter_bvec_queue(struct iov_iter *i, unsigned int direction,
const struct bvecq *bvecq,
unsigned int first_slot, unsigned int offset, size_t count);
-void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,
- const struct folio_queue *folioq,
- unsigned int first_slot, unsigned int offset, size_t count);
void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray *xarray,
loff_t start, size_t count);
ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages,
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index c7f7ff9a9b37..fdb51396157d 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -572,39 +572,6 @@ static void iov_iter_bvecq_advance(struct iov_iter *i, size_t by)
i->bvecq = bq;
}
-static void iov_iter_folioq_advance(struct iov_iter *i, size_t size)
-{
- const struct folio_queue *folioq = i->folioq;
- unsigned int slot = i->folioq_slot;
-
- if (!i->count)
- return;
- i->count -= size;
-
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
- }
-
- size += i->iov_offset; /* From beginning of current segment. */
- do {
- size_t fsize = folioq_folio_size(folioq, slot);
-
- if (likely(size < fsize))
- break;
- size -= fsize;
- slot++;
- if (slot >= folioq_nr_slots(folioq) && folioq->next) {
- folioq = folioq->next;
- slot = 0;
- }
- } while (size);
-
- i->iov_offset = size;
- i->folioq_slot = slot;
- i->folioq = folioq;
-}
-
void iov_iter_advance(struct iov_iter *i, size_t size)
{
if (unlikely(i->count < size))
@@ -619,8 +586,6 @@ void iov_iter_advance(struct iov_iter *i, size_t size)
iov_iter_bvec_advance(i, size);
} else if (iov_iter_is_bvecq(i)) {
iov_iter_bvecq_advance(i, size);
- } else if (iov_iter_is_folioq(i)) {
- iov_iter_folioq_advance(i, size);
} else if (iov_iter_is_discard(i)) {
i->count -= size;
}
@@ -654,32 +619,6 @@ static void iov_iter_bvecq_revert(struct iov_iter *i, size_t unroll)
i->bvecq = bq;
}
-static void iov_iter_folioq_revert(struct iov_iter *i, size_t unroll)
-{
- const struct folio_queue *folioq = i->folioq;
- unsigned int slot = i->folioq_slot;
-
- for (;;) {
- size_t fsize;
-
- if (slot == 0) {
- folioq = folioq->prev;
- slot = folioq_nr_slots(folioq);
- }
- slot--;
-
- fsize = folioq_folio_size(folioq, slot);
- if (unroll <= fsize) {
- i->iov_offset = fsize - unroll;
- break;
- }
- unroll -= fsize;
- }
-
- i->folioq_slot = slot;
- i->folioq = folioq;
-}
-
void iov_iter_revert(struct iov_iter *i, size_t unroll)
{
if (!unroll)
@@ -714,9 +653,6 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll)
} else if (iov_iter_is_bvecq(i)) {
i->iov_offset = 0;
iov_iter_bvecq_revert(i, unroll);
- } else if (iov_iter_is_folioq(i)) {
- i->iov_offset = 0;
- iov_iter_folioq_revert(i, unroll);
} else { /* same logics for iovec and kvec */
const struct iovec *iov = iter_iov(i);
while (1) {
@@ -766,8 +702,6 @@ size_t iov_iter_single_seg_count(const struct iov_iter *i)
}
return min(i->count, bq->bv[slot].bv_len - offset);
}
- if (unlikely(iov_iter_is_folioq(i)))
- return umin(folioq_folio_size(i->folioq, i->folioq_slot), i->count);
return i->count;
}
EXPORT_SYMBOL(iov_iter_single_seg_count);
@@ -833,36 +767,6 @@ void iov_iter_bvec_queue(struct iov_iter *i, unsigned int direction,
}
EXPORT_SYMBOL(iov_iter_bvec_queue);
-/**
- * iov_iter_folio_queue - Initialise an I/O iterator to use the folios in a folio queue
- * @i: The iterator to initialise.
- * @direction: The direction of the transfer.
- * @folioq: The starting point in the folio queue.
- * @first_slot: The first slot in the folio queue to use
- * @offset: The offset into the folio in the first slot to start at
- * @count: The size of the I/O buffer in bytes.
- *
- * Set up an I/O iterator to either draw data out of the pages attached to an
- * inode or to inject data into those pages. The pages *must* be prevented
- * from evaporation, either by taking a ref on them or locking them by the
- * caller.
- */
-void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,
- const struct folio_queue *folioq, unsigned int first_slot,
- unsigned int offset, size_t count)
-{
- BUG_ON(direction & ~1);
- *i = (struct iov_iter) {
- .iter_type = ITER_FOLIOQ,
- .data_source = direction,
- .folioq = folioq,
- .folioq_slot = first_slot,
- .count = count,
- .iov_offset = offset,
- };
-}
-EXPORT_SYMBOL(iov_iter_folio_queue);
-
/**
* iov_iter_xarray - Initialise an I/O iterator to use the pages in an xarray
* @i: The iterator to initialise.
@@ -1006,9 +910,7 @@ unsigned long iov_iter_alignment(const struct iov_iter *i)
if (iov_iter_is_bvecq(i))
return iov_iter_alignment_bvecq(i);
- /* With both xarray and folioq types, we're dealing with whole folios. */
- if (iov_iter_is_folioq(i))
- return i->iov_offset | i->count;
+ /* With the xarray type, we're dealing with whole folios. */
if (iov_iter_is_xarray(i))
return (i->xarray_start + i->iov_offset) | i->count;
@@ -1265,11 +1167,6 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages)
return bvec_npages(i, maxpages);
if (iov_iter_is_bvecq(i))
return iov_npages_bvecq(i, maxpages);
- if (iov_iter_is_folioq(i)) {
- unsigned offset = i->iov_offset % PAGE_SIZE;
- int npages = DIV_ROUND_UP(offset + i->count, PAGE_SIZE);
- return min(npages, maxpages);
- }
if (iov_iter_is_xarray(i)) {
unsigned offset = (i->xarray_start + i->iov_offset) % PAGE_SIZE;
int npages = DIV_ROUND_UP(offset + i->count, PAGE_SIZE);
@@ -1691,68 +1588,6 @@ static ssize_t iov_iter_extract_bvecq_pages(struct iov_iter *iter,
return extracted;
}
-/*
- * Extract a list of contiguous pages from an ITER_FOLIOQ iterator. This does
- * not get references on the pages, nor does it get a pin on them.
- */
-static ssize_t iov_iter_extract_folioq_pages(struct iov_iter *i,
- struct page ***pages, size_t maxsize,
- unsigned int maxpages,
- iov_iter_extraction_t extraction_flags,
- size_t *offset0)
-{
- const struct folio_queue *folioq = i->folioq;
- struct page **p;
- unsigned int nr = 0;
- size_t extracted = 0, offset, slot = i->folioq_slot;
-
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
- if (WARN_ON(i->iov_offset != 0))
- return -EIO;
- }
-
- offset = i->iov_offset & ~PAGE_MASK;
- *offset0 = offset;
-
- maxpages = want_pages_array(pages, maxsize, offset, maxpages);
- if (!maxpages)
- return -ENOMEM;
- p = *pages;
-
- for (;;) {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t offset = i->iov_offset, fsize = folioq_folio_size(folioq, slot);
- size_t part = PAGE_SIZE - offset % PAGE_SIZE;
-
- if (offset < fsize) {
- part = umin(part, umin(maxsize - extracted, fsize - offset));
- i->count -= part;
- i->iov_offset += part;
- extracted += part;
-
- p[nr++] = folio_page(folio, offset / PAGE_SIZE);
- }
-
- if (nr >= maxpages || extracted >= maxsize)
- break;
-
- if (i->iov_offset >= fsize) {
- i->iov_offset = 0;
- slot++;
- if (slot == folioq_nr_slots(folioq) && folioq->next) {
- folioq = folioq->next;
- slot = 0;
- }
- }
- }
-
- i->folioq = folioq;
- i->folioq_slot = slot;
- return extracted;
-}
-
/*
* Extract a list of contiguous pages from an ITER_XARRAY iterator. This does not
* get references on the pages, nor does it get a pin on them.
@@ -2015,8 +1850,8 @@ static ssize_t iov_iter_extract_user_pages(struct iov_iter *i,
* added to the pages, but refs will not be taken.
* iov_iter_extract_will_pin() will return true.
*
- * (*) If the iterator is ITER_KVEC, ITER_BVEC, ITER_FOLIOQ or ITER_XARRAY, the
- * pages are merely listed; no extra refs or pins are obtained.
+ * (*) If the iterator is ITER_KVEC, ITER_BVEC, ITER_XARRAY, the pages are
+ * merely listed; no extra refs or pins are obtained.
* iov_iter_extract_will_pin() will return 0.
*
* Note also:
@@ -2055,10 +1890,6 @@ ssize_t iov_iter_extract_pages(struct iov_iter *i,
return iov_iter_extract_bvecq_pages(i, pages, maxsize,
maxpages, extraction_flags,
offset0);
- if (iov_iter_is_folioq(i))
- return iov_iter_extract_folioq_pages(i, pages, maxsize,
- maxpages, extraction_flags,
- offset0);
if (iov_iter_is_xarray(i))
return iov_iter_extract_xarray_pages(i, pages, maxsize,
maxpages, extraction_flags,
diff --git a/lib/scatterlist.c b/lib/scatterlist.c
index 23e5a180103b..b9a7298306d9 100644
--- a/lib/scatterlist.c
+++ b/lib/scatterlist.c
@@ -12,7 +12,6 @@
#include <linux/bvec.h>
#include <linux/bvecq.h>
#include <linux/uio.h>
-#include <linux/folio_queue.h>
/**
* sg_nents - return total count of entries in scatterlist
@@ -1327,67 +1326,6 @@ static ssize_t extract_bvecq_to_sg(struct iov_iter *iter,
return ret;
}
-/*
- * Extract up to sg_max folios from an FOLIOQ-type iterator and add them to
- * the scatterlist. The pages are not pinned.
- */
-static ssize_t extract_folioq_to_sg(struct iov_iter *iter,
- ssize_t maxsize,
- struct sg_table *sgtable,
- unsigned int sg_max,
- iov_iter_extraction_t extraction_flags)
-{
- const struct folio_queue *folioq = iter->folioq;
- struct scatterlist *sg = sgtable->sgl + sgtable->nents;
- unsigned int slot = iter->folioq_slot;
- ssize_t ret = 0;
- size_t offset = iter->iov_offset;
-
- BUG_ON(!folioq);
-
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- if (WARN_ON_ONCE(!folioq))
- return 0;
- slot = 0;
- }
-
- do {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t fsize = folioq_folio_size(folioq, slot);
-
- if (offset < fsize) {
- size_t part = umin(maxsize - ret, fsize - offset);
-
- sg_set_page(sg, folio_page(folio, 0), part, offset);
- sgtable->nents++;
- sg++;
- sg_max--;
- offset += part;
- ret += part;
- }
-
- if (offset >= fsize) {
- offset = 0;
- slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- if (!folioq->next) {
- WARN_ON_ONCE(ret < iter->count);
- break;
- }
- folioq = folioq->next;
- slot = 0;
- }
- }
- } while (sg_max > 0 && ret < maxsize);
-
- iter->folioq = folioq;
- iter->folioq_slot = slot;
- iter->iov_offset = offset;
- iter->count -= ret;
- return ret;
-}
-
/*
* Extract up to sg_max folios from an XARRAY-type iterator and add them to
* the scatterlist. The pages are not pinned.
@@ -1451,8 +1389,8 @@ static ssize_t extract_xarray_to_sg(struct iov_iter *iter,
* addition of @sg_max elements.
*
* The pages referred to by UBUF- and IOVEC-type iterators are extracted and
- * pinned; BVEC-, BVECQ-, KVEC-, FOLIOQ- and XARRAY-type are extracted but
- * aren't pinned; DISCARD-type is not supported.
+ * pinned; BVEC-, BVECQ-, KVEC-, XARRAY-type are extracted but aren't pinned;
+ * DISCARD-type is not supported.
*
* No end mark is placed on the scatterlist; that's left to the caller.
*
@@ -1487,9 +1425,6 @@ ssize_t extract_iter_to_sg(struct iov_iter *iter, size_t maxsize,
case ITER_BVECQ:
return extract_bvecq_to_sg(iter, maxsize, sgtable, sg_max,
extraction_flags);
- case ITER_FOLIOQ:
- return extract_folioq_to_sg(iter, maxsize, sgtable, sg_max,
- extraction_flags);
case ITER_XARRAY:
return extract_xarray_to_sg(iter, maxsize, sgtable, sg_max,
extraction_flags);
diff --git a/lib/tests/kunit_iov_iter.c b/lib/tests/kunit_iov_iter.c
index f6413e6b925a..1c7085234006 100644
--- a/lib/tests/kunit_iov_iter.c
+++ b/lib/tests/kunit_iov_iter.c
@@ -13,7 +13,6 @@
#include <linux/uio.h>
#include <linux/bvec.h>
#include <linux/bvecq.h>
-#include <linux/folio_queue.h>
#include <linux/scatterlist.h>
#include <linux/minmax.h>
#include <linux/mman.h>
@@ -383,176 +382,6 @@ static void __init iov_kunit_copy_from_bvec(struct kunit *test)
KUNIT_SUCCEED(test);
}
-static void iov_kunit_destroy_folioq(void *data)
-{
- struct folio_queue *folioq, *next;
-
- for (folioq = data; folioq; folioq = next) {
- next = folioq->next;
- kfree(folioq);
- }
-}
-
-static void __init iov_kunit_load_folioq(struct kunit *test,
- struct iov_iter *iter, int dir,
- struct folio_queue *folioq,
- struct page **pages, size_t npages)
-{
- struct folio_queue *p = folioq;
- size_t size = 0;
- int i;
-
- for (i = 0; i < npages; i++) {
- if (folioq_full(p)) {
- p->next = kzalloc_obj(struct folio_queue);
- KUNIT_ASSERT_NOT_ERR_OR_NULL(test, p->next);
- folioq_init(p->next, 0);
- p->next->prev = p;
- p = p->next;
- }
- folioq_append(p, page_folio(pages[i]));
- size += PAGE_SIZE;
- }
- iov_iter_folio_queue(iter, dir, folioq, 0, 0, size);
-}
-
-static struct folio_queue *iov_kunit_create_folioq(struct kunit *test)
-{
- struct folio_queue *folioq;
-
- folioq = kzalloc_obj(struct folio_queue);
- KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folioq);
- kunit_add_action_or_reset(test, iov_kunit_destroy_folioq, folioq);
- folioq_init(folioq, 0);
- return folioq;
-}
-
-/*
- * Test copying to a ITER_FOLIOQ-type iterator.
- */
-static void __init iov_kunit_copy_to_folioq(struct kunit *test)
-{
- const struct kvec_test_range *pr;
- struct iov_iter iter;
- struct folio_queue *folioq;
- struct page **spages, **bpages;
- u8 *scratch, *buffer;
- size_t bufsize, npages, size, copied;
- int i, patt;
-
- bufsize = 0x100000;
- npages = bufsize / PAGE_SIZE;
-
- folioq = iov_kunit_create_folioq(test);
-
- scratch = iov_kunit_create_buffer(test, &spages, npages);
- for (i = 0; i < bufsize; i++)
- scratch[i] = pattern(i);
-
- buffer = iov_kunit_create_buffer(test, &bpages, npages);
- memset(buffer, 0, bufsize);
-
- iov_kunit_load_folioq(test, &iter, READ, folioq, bpages, npages);
-
- i = 0;
- for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
- size = pr->to - pr->from;
- KUNIT_ASSERT_LE(test, pr->to, bufsize);
-
- iov_iter_folio_queue(&iter, READ, folioq, 0, 0, pr->to);
- iov_iter_advance(&iter, pr->from);
- copied = copy_to_iter(scratch + i, size, &iter);
-
- KUNIT_EXPECT_EQ(test, copied, size);
- KUNIT_EXPECT_EQ(test, iter.count, 0);
- KUNIT_EXPECT_EQ(test, iter.iov_offset, pr->to % PAGE_SIZE);
- i += size;
- if (test->status == KUNIT_FAILURE)
- goto stop;
- }
-
- /* Build the expected image in the scratch buffer. */
- patt = 0;
- memset(scratch, 0, bufsize);
- for (pr = kvec_test_ranges; pr->from >= 0; pr++)
- for (i = pr->from; i < pr->to; i++)
- scratch[i] = pattern(patt++);
-
- /* Compare the images */
- for (i = 0; i < bufsize; i++) {
- KUNIT_EXPECT_EQ_MSG(test, buffer[i], scratch[i], "at i=%x", i);
- if (buffer[i] != scratch[i])
- return;
- }
-
-stop:
- KUNIT_SUCCEED(test);
-}
-
-/*
- * Test copying from a ITER_FOLIOQ-type iterator.
- */
-static void __init iov_kunit_copy_from_folioq(struct kunit *test)
-{
- const struct kvec_test_range *pr;
- struct iov_iter iter;
- struct folio_queue *folioq;
- struct page **spages, **bpages;
- u8 *scratch, *buffer;
- size_t bufsize, npages, size, copied;
- int i, j;
-
- bufsize = 0x100000;
- npages = bufsize / PAGE_SIZE;
-
- folioq = iov_kunit_create_folioq(test);
-
- buffer = iov_kunit_create_buffer(test, &bpages, npages);
- for (i = 0; i < bufsize; i++)
- buffer[i] = pattern(i);
-
- scratch = iov_kunit_create_buffer(test, &spages, npages);
- memset(scratch, 0, bufsize);
-
- iov_kunit_load_folioq(test, &iter, READ, folioq, bpages, npages);
-
- i = 0;
- for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
- size = pr->to - pr->from;
- KUNIT_ASSERT_LE(test, pr->to, bufsize);
-
- iov_iter_folio_queue(&iter, WRITE, folioq, 0, 0, pr->to);
- iov_iter_advance(&iter, pr->from);
- copied = copy_from_iter(scratch + i, size, &iter);
-
- KUNIT_EXPECT_EQ(test, copied, size);
- KUNIT_EXPECT_EQ(test, iter.count, 0);
- KUNIT_EXPECT_EQ(test, iter.iov_offset, pr->to % PAGE_SIZE);
- i += size;
- }
-
- /* Build the expected image in the main buffer. */
- i = 0;
- memset(buffer, 0, bufsize);
- for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
- for (j = pr->from; j < pr->to; j++) {
- buffer[i++] = pattern(j);
- if (i >= bufsize)
- goto stop;
- }
- }
-stop:
-
- /* Compare the images */
- for (i = 0; i < bufsize; i++) {
- KUNIT_EXPECT_EQ_MSG(test, scratch[i], buffer[i], "at i=%x", i);
- if (scratch[i] != buffer[i])
- return;
- }
-
- KUNIT_SUCCEED(test);
-}
-
static void iov_kunit_destroy_bvecq(void *data)
{
struct bvecq *bq, *next;
@@ -1124,85 +953,6 @@ static void __init iov_kunit_extract_pages_bvecq(struct kunit *test)
KUNIT_SUCCEED(test);
}
-/*
- * Test the extraction of ITER_FOLIOQ-type iterators.
- */
-static void __init iov_kunit_extract_pages_folioq(struct kunit *test)
-{
- const struct kvec_test_range *pr;
- struct folio_queue *folioq;
- struct iov_iter iter;
- struct page **bpages, *pagelist[8], **pages = pagelist;
- ssize_t len;
- size_t bufsize, size = 0, npages;
- int i, from;
-
- bufsize = 0x100000;
- npages = bufsize / PAGE_SIZE;
-
- folioq = iov_kunit_create_folioq(test);
-
- iov_kunit_create_buffer(test, &bpages, npages);
- iov_kunit_load_folioq(test, &iter, READ, folioq, bpages, npages);
-
- for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
- from = pr->from;
- size = pr->to - from;
- KUNIT_ASSERT_LE(test, pr->to, bufsize);
-
- iov_iter_folio_queue(&iter, WRITE, folioq, 0, 0, pr->to);
- iov_iter_advance(&iter, from);
-
- do {
- size_t offset0 = LONG_MAX;
-
- for (i = 0; i < ARRAY_SIZE(pagelist); i++)
- pagelist[i] = (void *)(unsigned long)0xaa55aa55aa55aa55ULL;
-
- len = iov_iter_extract_pages(&iter, &pages, 100 * 1024,
- ARRAY_SIZE(pagelist), 0, &offset0);
- KUNIT_EXPECT_GE(test, len, 0);
- if (len < 0)
- break;
- KUNIT_EXPECT_LE(test, len, size);
- KUNIT_EXPECT_EQ(test, iter.count, size - len);
- if (len == 0)
- break;
- size -= len;
- KUNIT_EXPECT_GE(test, (ssize_t)offset0, 0);
- KUNIT_EXPECT_LT(test, offset0, PAGE_SIZE);
-
- for (i = 0; i < ARRAY_SIZE(pagelist); i++) {
- struct page *p;
- ssize_t part = min_t(ssize_t, len, PAGE_SIZE - offset0);
- int ix;
-
- KUNIT_ASSERT_GE(test, part, 0);
- ix = from / PAGE_SIZE;
- KUNIT_ASSERT_LT(test, ix, npages);
- p = bpages[ix];
- KUNIT_EXPECT_PTR_EQ(test, pagelist[i], p);
- KUNIT_EXPECT_EQ(test, offset0, from % PAGE_SIZE);
- from += part;
- len -= part;
- KUNIT_ASSERT_GE(test, len, 0);
- if (len == 0)
- break;
- offset0 = 0;
- }
-
- if (test->status == KUNIT_FAILURE)
- goto stop;
- } while (iov_iter_count(&iter) > 0);
-
- KUNIT_EXPECT_EQ(test, size, 0);
- KUNIT_EXPECT_EQ(test, iter.count, 0);
- }
-
-stop:
- KUNIT_SUCCEED(test);
-}
-
/*
* Test the extraction of ITER_XARRAY-type iterators.
*/
@@ -1430,23 +1180,6 @@ static void __init iov_kunit_iter_to_sg_bvec(struct kunit *test)
iov_kunit_iter_to_sg_check(test, &iter, bufsize, &data);
}
-static void __init iov_kunit_iter_to_sg_folioq(struct kunit *test)
-{
- struct iov_kunit_iter_to_sg_data data;
- struct folio_queue *folioq;
- struct iov_iter iter;
- size_t bufsize;
-
- bufsize = 0x200000;
- iov_kunit_iter_to_sg_init(test, bufsize, false, &data);
-
- folioq = iov_kunit_create_folioq(test);
- iov_kunit_load_folioq(test, &iter, READ, folioq, data.pages,
- data.npages);
-
- iov_kunit_iter_to_sg_check(test, &iter, bufsize, &data);
-}
-
static void __init iov_kunit_iter_to_sg_xarray(struct kunit *test)
{
struct iov_kunit_iter_to_sg_data data;
@@ -1485,18 +1218,14 @@ static struct kunit_case __refdata iov_kunit_cases[] = {
KUNIT_CASE(iov_kunit_copy_from_bvec),
KUNIT_CASE(iov_kunit_copy_to_bvecq),
KUNIT_CASE(iov_kunit_copy_from_bvecq),
- KUNIT_CASE(iov_kunit_copy_to_folioq),
- KUNIT_CASE(iov_kunit_copy_from_folioq),
KUNIT_CASE(iov_kunit_copy_to_xarray),
KUNIT_CASE(iov_kunit_copy_from_xarray),
KUNIT_CASE(iov_kunit_extract_pages_kvec),
KUNIT_CASE(iov_kunit_extract_pages_bvec),
KUNIT_CASE(iov_kunit_extract_pages_bvecq),
- KUNIT_CASE(iov_kunit_extract_pages_folioq),
KUNIT_CASE(iov_kunit_extract_pages_xarray),
KUNIT_CASE(iov_kunit_iter_to_sg_kvec),
KUNIT_CASE(iov_kunit_iter_to_sg_bvec),
- KUNIT_CASE(iov_kunit_iter_to_sg_folioq),
KUNIT_CASE(iov_kunit_iter_to_sg_xarray),
KUNIT_CASE(iov_kunit_iter_to_sg_ubuf),
{}
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 29/35] netfs: Remove folio_queue and rolling_buffer
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (27 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 28/35] iov_iter: Remove ITER_FOLIOQ David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 30/35] netfs: Simplify read abandonment David Howells
` (6 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Remove folio_queue and rolling_buffer as they're no longer used.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
Documentation/core-api/folio_queue.rst | 209 --------------
Documentation/core-api/index.rst | 1 -
Documentation/filesystems/netfs_library.rst | 2 +-
fs/netfs/internal.h | 1 -
fs/netfs/iterator.c | 192 -------------
fs/netfs/main.c | 1 -
fs/netfs/rolling_buffer.c | 301 --------------------
include/linux/folio_queue.h | 282 ------------------
include/linux/netfs.h | 2 -
include/linux/rolling_buffer.h | 64 -----
kernel/bpf/btf.c | 2 -
11 files changed, 1 insertion(+), 1056 deletions(-)
delete mode 100644 Documentation/core-api/folio_queue.rst
delete mode 100644 fs/netfs/rolling_buffer.c
delete mode 100644 include/linux/folio_queue.h
delete mode 100644 include/linux/rolling_buffer.h
diff --git a/Documentation/core-api/folio_queue.rst b/Documentation/core-api/folio_queue.rst
deleted file mode 100644
index b7628896d2b6..000000000000
--- a/Documentation/core-api/folio_queue.rst
+++ /dev/null
@@ -1,209 +0,0 @@
-.. SPDX-License-Identifier: GPL-2.0+
-
-===========
-Folio Queue
-===========
-
-:Author: David Howells <dhowells@redhat.com>
-
-.. Contents:
-
- * Overview
- * Initialisation
- * Adding and removing folios
- * Querying information about a folio
- * Querying information about a folio_queue
- * Folio queue iteration
- * Folio marks
- * Lockless simultaneous production/consumption issues
-
-
-Overview
-========
-
-The folio_queue struct forms a single segment in a segmented list of folios
-that can be used to form an I/O buffer. As such, the list can be iterated over
-using the ITER_FOLIOQ iov_iter type.
-
-The publicly accessible members of the structure are::
-
- struct folio_queue {
- struct folio_queue *next;
- struct folio_queue *prev;
- ...
- };
-
-A pair of pointers are provided, ``next`` and ``prev``, that point to the
-segments on either side of the segment being accessed. Whilst this is a
-doubly-linked list, it is intentionally not a circular list; the outward
-sibling pointers in terminal segments should be NULL.
-
-Each segment in the list also stores:
-
- * an ordered sequence of folio pointers,
- * the size of each folio and
- * three 1-bit marks per folio,
-
-but these should not be accessed directly as the underlying data structure may
-change, but rather the access functions outlined below should be used.
-
-The facility can be made accessible by::
-
- #include <linux/folio_queue.h>
-
-and to use the iterator::
-
- #include <linux/uio.h>
-
-
-Initialisation
-==============
-
-A segment should be initialised by calling::
-
- void folioq_init(struct folio_queue *folioq);
-
-with a pointer to the segment to be initialised. Note that this will not
-necessarily initialise all the folio pointers, so care must be taken to check
-the number of folios added.
-
-
-Adding and removing folios
-==========================
-
-Folios can be set in the next unused slot in a segment struct by calling one
-of::
-
- unsigned int folioq_append(struct folio_queue *folioq,
- struct folio *folio);
-
- unsigned int folioq_append_mark(struct folio_queue *folioq,
- struct folio *folio);
-
-Both functions update the stored folio count, store the folio and note its
-size. The second function also sets the first mark for the folio added. Both
-functions return the number of the slot used. [!] Note that no attempt is made
-to check that the capacity wasn't overrun and the list will not be extended
-automatically.
-
-A folio can be excised by calling::
-
- void folioq_clear(struct folio_queue *folioq, unsigned int slot);
-
-This clears the slot in the array and also clears all the marks for that folio,
-but doesn't change the folio count - so future accesses of that slot must check
-if the slot is occupied.
-
-
-Querying information about a folio
-==================================
-
-Information about the folio in a particular slot may be queried by the
-following function::
-
- struct folio *folioq_folio(const struct folio_queue *folioq,
- unsigned int slot);
-
-If a folio has not yet been set in that slot, this may yield an undefined
-pointer. The size of the folio in a slot may be queried with either of::
-
- unsigned int folioq_folio_order(const struct folio_queue *folioq,
- unsigned int slot);
-
- size_t folioq_folio_size(const struct folio_queue *folioq,
- unsigned int slot);
-
-The first function returns the size as an order and the second as a number of
-bytes.
-
-
-Querying information about a folio_queue
-========================================
-
-Information may be retrieved about a particular segment with the following
-functions::
-
- unsigned int folioq_nr_slots(const struct folio_queue *folioq);
-
- unsigned int folioq_count(struct folio_queue *folioq);
-
- bool folioq_full(struct folio_queue *folioq);
-
-The first function returns the maximum capacity of a segment. It must not be
-assumed that this won't vary between segments. The second returns the number
-of folios added to a segments and the third is a shorthand to indicate if the
-segment has been filled to capacity.
-
-Not that the count and fullness are not affected by clearing folios from the
-segment. These are more about indicating how many slots in the array have been
-initialised, and it assumed that slots won't get reused, but rather the segment
-will get discarded as the queue is consumed.
-
-
-Folio marks
-===========
-
-Folios within a queue can also have marks assigned to them. These marks can be
-used to note information such as if a folio needs folio_put() calling upon it.
-There are three marks available to be set for each folio.
-
-The marks can be set by::
-
- void folioq_mark(struct folio_queue *folioq, unsigned int slot);
- void folioq_mark2(struct folio_queue *folioq, unsigned int slot);
-
-Cleared by::
-
- void folioq_unmark(struct folio_queue *folioq, unsigned int slot);
- void folioq_unmark2(struct folio_queue *folioq, unsigned int slot);
-
-And the marks can be queried by::
-
- bool folioq_is_marked(const struct folio_queue *folioq, unsigned int slot);
- bool folioq_is_marked2(const struct folio_queue *folioq, unsigned int slot);
-
-The marks can be used for any purpose and are not interpreted by this API.
-
-
-Folio queue iteration
-=====================
-
-A list of segments may be iterated over using the I/O iterator facility using
-an ``iov_iter`` iterator of ``ITER_FOLIOQ`` type. The iterator may be
-initialised with::
-
- void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,
- const struct folio_queue *folioq,
- unsigned int first_slot, unsigned int offset,
- size_t count);
-
-This may be told to start at a particular segment, slot and offset within a
-queue. The iov iterator functions will follow the next pointers when advancing
-and prev pointers when reverting when needed.
-
-
-Lockless simultaneous production/consumption issues
-===================================================
-
-If properly managed, the list can be extended by the producer at the head end
-and shortened by the consumer at the tail end simultaneously without the need
-to take locks. The ITER_FOLIOQ iterator inserts appropriate barriers to aid
-with this.
-
-Care must be taken when simultaneously producing and consuming a list. If the
-last segment is reached and the folios it refers to are entirely consumed by
-the IOV iterators, an iov_iter struct will be left pointing to the last segment
-with a slot number equal to the capacity of that segment. The iterator will
-try to continue on from this if there's another segment available when it is
-used again, but care must be taken lest the segment got removed and freed by
-the consumer before the iterator was advanced.
-
-It is recommended that the queue always contain at least one segment, even if
-that segment has never been filled or is entirely spent. This prevents the
-head and tail pointers from collapsing.
-
-
-API Function Reference
-======================
-
-.. kernel-doc:: include/linux/folio_queue.h
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 13769d5c40bf..16c529a33ac4 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -39,7 +39,6 @@ Library functionality that is used throughout the kernel.
kref
cleanup
assoc_array
- folio_queue
xarray
maple_tree
idr
diff --git a/Documentation/filesystems/netfs_library.rst b/Documentation/filesystems/netfs_library.rst
index ddd799df6ce3..18e3c3aae57c 100644
--- a/Documentation/filesystems/netfs_library.rst
+++ b/Documentation/filesystems/netfs_library.rst
@@ -449,7 +449,7 @@ be called from the writeback code to write the data to the cache, if there is
one.
The inode should be marked ``NETFS_ICTX_SINGLE_NO_UPLOAD`` if this API is to be
-used. The writeback function requires the buffer to be of ITER_FOLIOQ type.
+used.
High-Level VM API
==================
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index b402b504a370..79260a13682b 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -45,7 +45,6 @@ extern spinlock_t netfs_proc_lock;
extern mempool_t netfs_request_pool;
extern mempool_t netfs_subrequest_pool;
extern mempool_t netfs_bvecq_pool;
-extern mempool_t netfs_folioq_pool;
#ifdef CONFIG_PROC_FS
static inline void netfs_proc_add_rreq(struct netfs_io_request *rreq)
diff --git a/fs/netfs/iterator.c b/fs/netfs/iterator.c
index 65164a1761e3..a557cd3dbb73 100644
--- a/fs/netfs/iterator.c
+++ b/fs/netfs/iterator.c
@@ -155,195 +155,3 @@ ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pag
}
EXPORT_SYMBOL_GPL(netfs_extract_iter);
-
-#if 0
-/*
- * Select the span of a bvec iterator we're going to use. Limit it by both maximum
- * size and maximum number of segments. Returns the size of the span in bytes.
- */
-static size_t netfs_limit_bvec(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs)
-{
- const struct bio_vec *bvecs = iter->bvec;
- unsigned int nbv = iter->nr_segs, ix = 0, nsegs = 0;
- size_t len, span = 0, n = iter->count;
- size_t skip = iter->iov_offset + start_offset;
-
- if (WARN_ON(!iov_iter_is_bvec(iter)) ||
- WARN_ON(start_offset > n) ||
- n == 0)
- return 0;
-
- while (n && ix < nbv && skip) {
- len = bvecs[ix].bv_len;
- if (skip < len)
- break;
- skip -= len;
- n -= len;
- ix++;
- }
-
- while (n && ix < nbv) {
- len = min3(n, bvecs[ix].bv_len - skip, max_size);
- span += len;
- nsegs++;
- ix++;
- if (span >= max_size || nsegs >= max_segs)
- break;
- skip = 0;
- n -= len;
- }
-
- return min(span, max_size);
-}
-
-/*
- * Select the span of a kvec iterator we're going to use. Limit it by both
- * maximum size and maximum number of segments. Returns the size of the span
- * in bytes.
- */
-static size_t netfs_limit_kvec(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs)
-{
- const struct kvec *kvecs = iter->kvec;
- unsigned int nkv = iter->nr_segs, ix = 0, nsegs = 0;
- size_t len, span = 0, n = iter->count;
- size_t skip = iter->iov_offset + start_offset;
-
- if (WARN_ON(!iov_iter_is_kvec(iter)) ||
- WARN_ON(start_offset > n) ||
- n == 0)
- return 0;
-
- while (n && ix < nkv && skip) {
- len = kvecs[ix].iov_len;
- if (skip < len)
- break;
- skip -= len;
- n -= len;
- ix++;
- }
-
- while (n && ix < nkv) {
- len = min3(n, kvecs[ix].iov_len - skip, max_size);
- span += len;
- nsegs++;
- ix++;
- if (span >= max_size || nsegs >= max_segs)
- break;
- skip = 0;
- n -= len;
- }
-
- return min(span, max_size);
-}
-
-/*
- * Select the span of an xarray iterator we're going to use. Limit it by both
- * maximum size and maximum number of segments. It is assumed that segments
- * can be larger than a page in size, provided they're physically contiguous.
- * Returns the size of the span in bytes.
- */
-static size_t netfs_limit_xarray(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs)
-{
- struct folio *folio;
- unsigned int nsegs = 0;
- uoff_t pos = iter->xarray_start + iter->iov_offset;
- pgoff_t index = pos / PAGE_SIZE;
- size_t span = 0, n = iter->count;
-
- XA_STATE(xas, iter->xarray, index);
-
- if (WARN_ON(!iov_iter_is_xarray(iter)) ||
- WARN_ON(start_offset > n) ||
- n == 0)
- return 0;
- max_size = min(max_size, n - start_offset);
-
- rcu_read_lock();
- xas_for_each(&xas, folio, ULONG_MAX) {
- size_t offset, flen, len;
- if (xas_retry(&xas, folio))
- continue;
- if (WARN_ON(xa_is_value(folio)))
- break;
- if (WARN_ON(folio_test_hugetlb(folio)))
- break;
-
- flen = folio_size(folio);
- offset = offset_in_folio(folio, pos);
- len = min(max_size, flen - offset);
- span += len;
- nsegs++;
- if (span >= max_size || nsegs >= max_segs)
- break;
- }
-
- rcu_read_unlock();
- return min(span, max_size);
-}
-
-/*
- * Select the span of a folio queue iterator we're going to use. Limit it by
- * both maximum size and maximum number of segments. Returns the size of the
- * span in bytes.
- */
-static size_t netfs_limit_folioq(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs)
-{
- const struct folio_queue *folioq = iter->folioq;
- unsigned int nsegs = 0;
- unsigned int slot = iter->folioq_slot;
- size_t span = 0, n = iter->count;
-
- if (WARN_ON(!iov_iter_is_folioq(iter)) ||
- WARN_ON(start_offset > n) ||
- n == 0)
- return 0;
- max_size = umin(max_size, n - start_offset);
-
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
- }
-
- start_offset += iter->iov_offset;
- do {
- size_t flen = folioq_folio_size(folioq, slot);
-
- if (start_offset < flen) {
- span += flen - start_offset;
- nsegs++;
- start_offset = 0;
- } else {
- start_offset -= flen;
- }
- if (span >= max_size || nsegs >= max_segs)
- break;
-
- slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
- }
- } while (folioq);
-
- return umin(span, max_size);
-}
-
-size_t netfs_limit_iter(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs)
-{
- if (iov_iter_is_folioq(iter))
- return netfs_limit_folioq(iter, start_offset, max_size, max_segs);
- if (iov_iter_is_bvec(iter))
- return netfs_limit_bvec(iter, start_offset, max_size, max_segs);
- if (iov_iter_is_xarray(iter))
- return netfs_limit_xarray(iter, start_offset, max_size, max_segs);
- if (iov_iter_is_kvec(iter))
- return netfs_limit_kvec(iter, start_offset, max_size, max_segs);
- BUG();
-}
-EXPORT_SYMBOL(netfs_limit_iter);
-#endif
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index 041810707eb4..5d8b87f71888 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -29,7 +29,6 @@ static struct kmem_cache *netfs_subrequest_slab;
mempool_t netfs_request_pool;
mempool_t netfs_subrequest_pool;
mempool_t netfs_bvecq_pool;
-mempool_t netfs_folioq_pool;
#ifdef CONFIG_PROC_FS
LIST_HEAD(netfs_io_requests);
diff --git a/fs/netfs/rolling_buffer.c b/fs/netfs/rolling_buffer.c
deleted file mode 100644
index ff6c7ee23f4d..000000000000
--- a/fs/netfs/rolling_buffer.c
+++ /dev/null
@@ -1,301 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/* Rolling buffer helpers
- *
- * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.
- * Written by David Howells (dhowells@redhat.com)
- */
-
-#include <linux/bitops.h>
-#include <linux/mempool.h>
-#include <linux/pagemap.h>
-#include <linux/rolling_buffer.h>
-#include <linux/slab.h>
-#include "internal.h"
-
-static atomic_t debug_ids;
-
-/**
- * netfs_folioq_alloc - Allocate a folio_queue struct
- * @rreq_id: Associated debugging ID for tracing purposes
- * @gfp: Allocation constraints
- * @trace: Trace tag to indicate the purpose of the allocation
- *
- * Allocate, initialise and account the folio_queue struct and log a trace line
- * to mark the allocation.
- */
-struct folio_queue *netfs_folioq_alloc(unsigned int rreq_id, gfp_t gfp,
- unsigned int /*enum netfs_folioq_trace*/ trace)
-{
- struct folio_queue *fq;
-
- if (gfp == GFP_KERNEL)
- fq = netfs_folioq_pool.alloc(gfp, netfs_folioq_pool.pool_data);
- else
- fq = mempool_alloc(&netfs_folioq_pool, gfp);
- if (fq) {
- netfs_stat(&netfs_n_folioq);
- folioq_init(fq, rreq_id);
- fq->debug_id = atomic_inc_return(&debug_ids);
- trace_netfs_folioq(fq, trace);
- }
- return fq;
-}
-EXPORT_SYMBOL(netfs_folioq_alloc);
-
-/**
- * netfs_folioq_free - Free a folio_queue struct
- * @folioq: The object to free
- * @trace: Trace tag to indicate which free
- *
- * Free and unaccount the folio_queue struct.
- */
-void netfs_folioq_free(struct folio_queue *folioq,
- unsigned int /*enum netfs_trace_folioq*/ trace)
-{
- trace_netfs_folioq(folioq, trace);
- netfs_stat_d(&netfs_n_folioq);
- mempool_free(folioq, &netfs_folioq_pool);
-}
-EXPORT_SYMBOL(netfs_folioq_free);
-
-/*
- * Initialise a rolling buffer. We allocate an empty folio queue struct to so
- * that the pointers can be independently driven by the producer and the
- * consumer.
- */
-int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,
- unsigned int direction, gfp_t gfp)
-{
- struct folio_queue *fq;
-
- fq = netfs_folioq_alloc(rreq_id, gfp, netfs_trace_folioq_rollbuf_init);
- if (!fq)
- return -ENOMEM;
-
- roll->head = fq;
- roll->tail = fq;
- iov_iter_folio_queue(&roll->iter, direction, fq, 0, 0, 0);
- return 0;
-}
-
-/*
- * Add another folio_queue to a rolling buffer if there's no space left.
- */
-int rolling_buffer_make_space(struct rolling_buffer *roll, gfp_t gfp)
-{
- struct folio_queue *fq, *head = roll->head;
-
- if (!folioq_full(head))
- return 0;
-
- fq = netfs_folioq_alloc(head->rreq_id, gfp, netfs_trace_folioq_make_space);
- if (!fq)
- return -ENOMEM;
- fq->prev = head;
-
- roll->head = fq;
- if (folioq_full(head)) {
- /* Make sure we don't leave the master iterator pointing to a
- * block that might get immediately consumed.
- */
- if (roll->iter.folioq == head &&
- roll->iter.folioq_slot == folioq_nr_slots(head)) {
- roll->iter.folioq = fq;
- roll->iter.folioq_slot = 0;
- }
- }
-
- /* Make sure the initialisation is stored before the next pointer.
- *
- * [!] NOTE: After we set head->next, the consumer is at liberty to
- * immediately delete the old head.
- */
- smp_store_release(&head->next, fq);
- return 0;
-}
-
-/*
- * Decant the list of folios to read into a rolling buffer.
- */
-ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
- struct readahead_control *ractl,
- struct folio_batch *put_batch)
-{
- struct folio_queue *fq;
- struct page **vec;
- int nr, ix, to;
- ssize_t size = 0;
-
- if (rolling_buffer_make_space(roll, GFP_KERNEL) < 0)
- return -ENOMEM;
-
- fq = roll->head;
- vec = (struct page **)fq->vec.folios;
- nr = __readahead_batch(ractl, vec + folio_batch_count(&fq->vec),
- folio_batch_space(&fq->vec));
- ix = fq->vec.nr;
- to = ix + nr;
- fq->vec.nr = to;
- for (; ix < to; ix++) {
- struct folio *folio = folioq_folio(fq, ix);
- unsigned int order = folio_order(folio);
-
- fq->orders[ix] = order;
- size += PAGE_SIZE << order;
- trace_netfs_folio(folio, netfs_folio_trace_read);
- if (!folio_batch_add(put_batch, folio))
- folio_batch_release(put_batch);
- }
- WRITE_ONCE(roll->iter.count, roll->iter.count + size);
-
- /* Store the counter after setting the slot. */
- smp_store_release(&roll->next_head_slot, to);
- return size;
-}
-
-/*
- * Decant the entire list of folios to read into a rolling buffer.
- */
-ssize_t rolling_buffer_bulk_load_from_ra(struct rolling_buffer *roll,
- struct readahead_control *ractl,
- unsigned int rreq_id, gfp_t gfp)
-{
- XA_STATE(xas, &ractl->mapping->i_pages, ractl->_index);
- struct folio_queue *fq;
- struct folio *folio;
- ssize_t loaded = 0;
- int nr, slot = 0, npages = 0;
-
- /* First allocate all the folioqs we're going to need to avoid having
- * to deal with ENOMEM later.
- */
- nr = ractl->_nr_folios;
- do {
- fq = netfs_folioq_alloc(rreq_id, gfp,
- netfs_trace_folioq_make_space);
- if (!fq) {
- rolling_buffer_clear(roll);
- return -ENOMEM;
- }
- fq->prev = roll->head;
- if (!roll->tail)
- roll->tail = fq;
- else
- roll->head->next = fq;
- roll->head = fq;
-
- nr -= folioq_nr_slots(fq);
- } while (nr > 0);
-
- rcu_read_lock();
-
- fq = roll->tail;
- xas_for_each(&xas, folio, ractl->_index + ractl->_nr_pages - 1) {
- unsigned int order;
-
- if (xas_retry(&xas, folio))
- continue;
- VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
-
- order = folio_order(folio);
- fq->orders[slot] = order;
- fq->vec.folios[slot] = folio;
- loaded += PAGE_SIZE << order;
- npages += 1 << order;
- trace_netfs_folio(folio, netfs_folio_trace_read);
-
- slot++;
- if (slot >= folioq_nr_slots(fq)) {
- fq->vec.nr = slot;
- fq = fq->next;
- if (!fq) {
- WARN_ON_ONCE(npages < readahead_count(ractl));
- break;
- }
- slot = 0;
- }
- }
-
- rcu_read_unlock();
-
- if (fq)
- fq->vec.nr = slot;
-
- WRITE_ONCE(roll->iter.count, loaded);
- iov_iter_folio_queue(&roll->iter, ITER_DEST, roll->tail, 0, 0, loaded);
- ractl->_index += npages;
- ractl->_nr_pages -= npages;
- return loaded;
-}
-
-/*
- * Append a folio to the rolling buffer.
- */
-ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,
- unsigned int flags, gfp_t gfp)
-{
- ssize_t size = folio_size(folio);
- int slot;
-
- if (rolling_buffer_make_space(roll, gfp) < 0)
- return -ENOMEM;
-
- slot = folioq_append(roll->head, folio);
- if (flags & ROLLBUF_MARK_1)
- folioq_mark(roll->head, slot);
- if (flags & ROLLBUF_MARK_2)
- folioq_mark2(roll->head, slot);
-
- WRITE_ONCE(roll->iter.count, roll->iter.count + size);
-
- /* Store the counter after setting the slot. */
- smp_store_release(&roll->next_head_slot, slot);
- return size;
-}
-
-/*
- * Delete a spent buffer from a rolling queue and return the next in line. We
- * don't return the last buffer to keep the pointers independent, but return
- * NULL instead.
- */
-struct folio_queue *rolling_buffer_delete_spent(struct rolling_buffer *roll)
-{
- struct folio_queue *spent = roll->tail, *next = READ_ONCE(spent->next);
-
- if (!next)
- return NULL;
- next->prev = NULL;
- netfs_folioq_free(spent, netfs_trace_folioq_delete);
- roll->tail = next;
- return next;
-}
-
-/*
- * Clear out a rolling queue. Folios that have mark 1 set are put.
- */
-void rolling_buffer_clear(struct rolling_buffer *roll)
-{
- struct folio_batch fbatch;
- struct folio_queue *p;
-
- folio_batch_init(&fbatch);
-
- while ((p = roll->tail)) {
- roll->tail = p->next;
- for (int slot = 0; slot < folioq_count(p); slot++) {
- struct folio *folio = folioq_folio(p, slot);
-
- if (!folio)
- continue;
- if (folioq_is_marked(p, slot)) {
- trace_netfs_folio(folio, netfs_folio_trace_put);
- if (!folio_batch_add(&fbatch, folio))
- folio_batch_release(&fbatch);
- }
- }
-
- netfs_folioq_free(p, netfs_trace_folioq_clear);
- }
-
- folio_batch_release(&fbatch);
-}
diff --git a/include/linux/folio_queue.h b/include/linux/folio_queue.h
deleted file mode 100644
index f6d5f1f127c9..000000000000
--- a/include/linux/folio_queue.h
+++ /dev/null
@@ -1,282 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/* Queue of folios definitions
- *
- * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.
- * Written by David Howells (dhowells@redhat.com)
- *
- * See:
- *
- * Documentation/core-api/folio_queue.rst
- *
- * for a description of the API.
- */
-
-#ifndef _LINUX_FOLIO_QUEUE_H
-#define _LINUX_FOLIO_QUEUE_H
-
-#include <linux/folio_batch.h>
-#include <linux/mm.h>
-
-/*
- * Segment in a queue of running buffers. Each segment can hold a number of
- * folios and a portion of the queue can be referenced with the ITER_FOLIOQ
- * iterator. The possibility exists of inserting non-folio elements into the
- * queue (such as gaps).
- *
- * Explicit prev and next pointers are used instead of a list_head to make it
- * easier to add segments to tail and remove them from the head without the
- * need for a lock.
- */
-struct folio_queue {
- struct folio_batch vec; /* Folios in the queue segment */
- u8 orders[FOLIO_BATCH_SIZE]; /* Order of each folio */
- struct folio_queue *next; /* Next queue segment or NULL */
- struct folio_queue *prev; /* Previous queue segment of NULL */
- unsigned long marks; /* 1-bit mark per folio */
- unsigned long marks2; /* Second 1-bit mark per folio */
-#if FOLIO_BATCH_SIZE > BITS_PER_LONG
-#error marks is not big enough
-#endif
- unsigned int rreq_id;
- unsigned int debug_id;
-};
-
-/**
- * folioq_init - Initialise a folio queue segment
- * @folioq: The segment to initialise
- * @rreq_id: The request identifier to use in tracelines.
- *
- * Initialise a folio queue segment and set an identifier to be used in traces.
- *
- * Note that the folio pointers are left uninitialised.
- */
-static inline void folioq_init(struct folio_queue *folioq, unsigned int rreq_id)
-{
- folio_batch_init(&folioq->vec);
- folioq->next = NULL;
- folioq->prev = NULL;
- folioq->marks = 0;
- folioq->marks2 = 0;
- folioq->rreq_id = rreq_id;
- folioq->debug_id = 0;
-}
-
-/**
- * folioq_nr_slots: Query the capacity of a folio queue segment
- * @folioq: The segment to query
- *
- * Query the number of folios that a particular folio queue segment might hold.
- * [!] NOTE: This must not be assumed to be the same for every segment!
- */
-static inline unsigned int folioq_nr_slots(const struct folio_queue *folioq)
-{
- return FOLIO_BATCH_SIZE;
-}
-
-/**
- * folioq_count: Query the occupancy of a folio queue segment
- * @folioq: The segment to query
- *
- * Query the number of folios that have been added to a folio queue segment.
- * Note that this is not decreased as folios are removed from a segment.
- */
-static inline unsigned int folioq_count(struct folio_queue *folioq)
-{
- return folio_batch_count(&folioq->vec);
-}
-
-/**
- * folioq_full: Query if a folio queue segment is full
- * @folioq: The segment to query
- *
- * Query if a folio queue segment is fully occupied. Note that this does not
- * change if folios are removed from a segment.
- */
-static inline bool folioq_full(struct folio_queue *folioq)
-{
- //return !folio_batch_space(&folioq->vec);
- return folioq_count(folioq) >= folioq_nr_slots(folioq);
-}
-
-/**
- * folioq_is_marked: Check first folio mark in a folio queue segment
- * @folioq: The segment to query
- * @slot: The slot number of the folio to query
- *
- * Determine if the first mark is set for the folio in the specified slot in a
- * folio queue segment.
- */
-static inline bool folioq_is_marked(const struct folio_queue *folioq, unsigned int slot)
-{
- return test_bit(slot, &folioq->marks);
-}
-
-/**
- * folioq_mark: Set the first mark on a folio in a folio queue segment
- * @folioq: The segment to modify
- * @slot: The slot number of the folio to modify
- *
- * Set the first mark for the folio in the specified slot in a folio queue
- * segment.
- */
-static inline void folioq_mark(struct folio_queue *folioq, unsigned int slot)
-{
- set_bit(slot, &folioq->marks);
-}
-
-/**
- * folioq_unmark: Clear the first mark on a folio in a folio queue segment
- * @folioq: The segment to modify
- * @slot: The slot number of the folio to modify
- *
- * Clear the first mark for the folio in the specified slot in a folio queue
- * segment.
- */
-static inline void folioq_unmark(struct folio_queue *folioq, unsigned int slot)
-{
- clear_bit(slot, &folioq->marks);
-}
-
-/**
- * folioq_is_marked2: Check second folio mark in a folio queue segment
- * @folioq: The segment to query
- * @slot: The slot number of the folio to query
- *
- * Determine if the second mark is set for the folio in the specified slot in a
- * folio queue segment.
- */
-static inline bool folioq_is_marked2(const struct folio_queue *folioq, unsigned int slot)
-{
- return test_bit(slot, &folioq->marks2);
-}
-
-/**
- * folioq_mark2: Set the second mark on a folio in a folio queue segment
- * @folioq: The segment to modify
- * @slot: The slot number of the folio to modify
- *
- * Set the second mark for the folio in the specified slot in a folio queue
- * segment.
- */
-static inline void folioq_mark2(struct folio_queue *folioq, unsigned int slot)
-{
- set_bit(slot, &folioq->marks2);
-}
-
-/**
- * folioq_unmark2: Clear the second mark on a folio in a folio queue segment
- * @folioq: The segment to modify
- * @slot: The slot number of the folio to modify
- *
- * Clear the second mark for the folio in the specified slot in a folio queue
- * segment.
- */
-static inline void folioq_unmark2(struct folio_queue *folioq, unsigned int slot)
-{
- clear_bit(slot, &folioq->marks2);
-}
-
-/**
- * folioq_append: Add a folio to a folio queue segment
- * @folioq: The segment to add to
- * @folio: The folio to add
- *
- * Add a folio to the tail of the sequence in a folio queue segment, increasing
- * the occupancy count and returning the slot number for the folio just added.
- * The folio size is extracted and stored in the queue and the marks are left
- * unmodified.
- *
- * Note that it's left up to the caller to check that the segment capacity will
- * not be exceeded and to extend the queue.
- */
-static inline unsigned int folioq_append(struct folio_queue *folioq, struct folio *folio)
-{
- unsigned int slot = folioq->vec.nr++;
-
- folioq->vec.folios[slot] = folio;
- folioq->orders[slot] = folio_order(folio);
- return slot;
-}
-
-/**
- * folioq_append_mark: Add a folio to a folio queue segment
- * @folioq: The segment to add to
- * @folio: The folio to add
- *
- * Add a folio to the tail of the sequence in a folio queue segment, increasing
- * the occupancy count and returning the slot number for the folio just added.
- * The folio size is extracted and stored in the queue, the first mark is set
- * and and the second and third marks are left unmodified.
- *
- * Note that it's left up to the caller to check that the segment capacity will
- * not be exceeded and to extend the queue.
- */
-static inline unsigned int folioq_append_mark(struct folio_queue *folioq, struct folio *folio)
-{
- unsigned int slot = folioq->vec.nr++;
-
- folioq->vec.folios[slot] = folio;
- folioq->orders[slot] = folio_order(folio);
- folioq_mark(folioq, slot);
- return slot;
-}
-
-/**
- * folioq_folio: Get a folio from a folio queue segment
- * @folioq: The segment to access
- * @slot: The folio slot to access
- *
- * Retrieve the folio in the specified slot from a folio queue segment. Note
- * that no bounds check is made and if the slot hasn't been added into yet, the
- * pointer will be undefined. If the slot has been cleared, NULL will be
- * returned.
- */
-static inline struct folio *folioq_folio(const struct folio_queue *folioq, unsigned int slot)
-{
- return folioq->vec.folios[slot];
-}
-
-/**
- * folioq_folio_order: Get the order of a folio from a folio queue segment
- * @folioq: The segment to access
- * @slot: The folio slot to access
- *
- * Retrieve the order of the folio in the specified slot from a folio queue
- * segment. Note that no bounds check is made and if the slot hasn't been
- * added into yet, the order returned will be 0.
- */
-static inline unsigned int folioq_folio_order(const struct folio_queue *folioq, unsigned int slot)
-{
- return folioq->orders[slot];
-}
-
-/**
- * folioq_folio_size: Get the size of a folio from a folio queue segment
- * @folioq: The segment to access
- * @slot: The folio slot to access
- *
- * Retrieve the size of the folio in the specified slot from a folio queue
- * segment. Note that no bounds check is made and if the slot hasn't been
- * added into yet, the size returned will be PAGE_SIZE.
- */
-static inline size_t folioq_folio_size(const struct folio_queue *folioq, unsigned int slot)
-{
- return PAGE_SIZE << folioq_folio_order(folioq, slot);
-}
-
-/**
- * folioq_clear: Clear a folio from a folio queue segment
- * @folioq: The segment to clear
- * @slot: The folio slot to clear
- *
- * Clear a folio from a sequence in a folio queue segment and clear its marks.
- * The occupancy count is left unchanged.
- */
-static inline void folioq_clear(struct folio_queue *folioq, unsigned int slot)
-{
- folioq->vec.folios[slot] = NULL;
- folioq_unmark(folioq, slot);
- folioq_unmark2(folioq, slot);
-}
-
-#endif /* _LINUX_FOLIO_QUEUE_H */
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 8de9ce35f70d..af111c589a50 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -459,8 +459,6 @@ void netfs_put_subrequest(struct netfs_io_subrequest *subreq,
ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages,
struct bvecq **_bvecq_head,
iov_iter_extraction_t extraction_flags, gfp_t gfp);
-size_t netfs_limit_iter(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs);
void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq);
void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error);
diff --git a/include/linux/rolling_buffer.h b/include/linux/rolling_buffer.h
deleted file mode 100644
index 761735bdf3ab..000000000000
--- a/include/linux/rolling_buffer.h
+++ /dev/null
@@ -1,64 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/* Rolling buffer of folios
- *
- * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.
- * Written by David Howells (dhowells@redhat.com)
- */
-
-#ifndef _ROLLING_BUFFER_H
-#define _ROLLING_BUFFER_H
-
-#include <linux/folio_queue.h>
-#include <linux/uio.h>
-
-/*
- * Rolling buffer. Whilst the buffer is live and in use, folios and folio
- * queue segments can be added to one end by one thread and removed from the
- * other end by another thread. The buffer isn't allowed to be empty; it must
- * always have at least one folio_queue in it so that neither side has to
- * modify both queue pointers.
- *
- * The iterator in the buffer is extended as buffers are inserted. It can be
- * snapshotted to use a segment of the buffer.
- */
-struct rolling_buffer {
- struct folio_queue *head; /* Producer's insertion point */
- struct folio_queue *tail; /* Consumer's removal point */
- struct iov_iter iter; /* Iterator tracking what's left in the buffer */
- u8 next_head_slot; /* Next slot in ->head */
- u8 first_tail_slot; /* First slot in ->tail */
-};
-
-/*
- * Snapshot of a rolling buffer.
- */
-struct rolling_buffer_snapshot {
- struct folio_queue *curr_folioq; /* Queue segment in which current folio resides */
- unsigned char curr_slot; /* Folio currently being read */
- unsigned char curr_order; /* Order of folio */
-};
-
-/* Marks to store per-folio in the internal folio_queue structs. */
-#define ROLLBUF_MARK_1 BIT(0)
-#define ROLLBUF_MARK_2 BIT(1)
-
-int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,
- unsigned int direction, gfp_t gfp);
-int rolling_buffer_make_space(struct rolling_buffer *roll, gfp_t gfp);
-ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
- struct readahead_control *ractl,
- struct folio_batch *put_batch);
-ssize_t rolling_buffer_bulk_load_from_ra(struct rolling_buffer *roll,
- struct readahead_control *ractl,
- unsigned int rreq_id, gfp_t gfp);
-ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,
- unsigned int flags, gfp_t gfp);
-struct folio_queue *rolling_buffer_delete_spent(struct rolling_buffer *roll);
-void rolling_buffer_clear(struct rolling_buffer *roll);
-
-static inline void rolling_buffer_advance(struct rolling_buffer *roll, size_t amount)
-{
- iov_iter_advance(&roll->iter, amount);
-}
-
-#endif /* _ROLLING_BUFFER_H */
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 7e25051dc1ea..b9558a0c2c1e 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -6742,8 +6742,6 @@ static const struct bpf_raw_tp_null_args raw_tp_null_args[] = {
/* amdgpu */
{ "amdgpu_vm_bo_map", 0x1 },
{ "amdgpu_vm_bo_unmap", 0x1 },
- /* netfs */
- { "netfs_folioq", 0x1 },
/* xfs from xfs_defer_pending_class */
{ "xfs_defer_create_intent", 0x1 },
{ "xfs_defer_cancel_list", 0x1 },
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 30/35] netfs: Simplify read abandonment
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (28 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 29/35] netfs: Remove folio_queue and rolling_buffer David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 15:39 ` READ_PLUS in NetFS? " Aurélien Couderc
2026-08-24 19:47 ` David Howells
2026-08-24 14:41 ` [PATCH v10 31/35] netfs: Check for too much data being read David Howells
` (5 subsequent siblings)
35 siblings, 2 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Currently, if one or more of the subrequests in a read request fails, the
read collection algorithm will attempt to salvage folios that are fully
downloaded but that span multiple subreqs, provided all of the contributory
subreqs succeeded, but this makes the algorithm quite complicated as a
subreq can contribute to multiple folios and a folio can be contributed to
by multiple subreqs.
Simplify this by just abandoning the rest of a read request once a
permanently failed subreq is hit. This is what read_folio, DIO/unbuffered
read, gap filling, RMW and write preload all need to do; readahead is the
odd one out as it doesn't have any return other than unlocking folios.
With this change, even for readahead, the rest of the read is simply
abandoned; read() and suchlike will call ->read_folio() on each
non-uptodate folio to retry and retrieve the error.
Note that retryable failures still get retried by netfslib as part of the
request as before.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/read_collect.c | 190 ++++++++++++++++++++++++++++------------
include/linux/netfs.h | 1 +
2 files changed, 137 insertions(+), 54 deletions(-)
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index ea01b2f1f1f8..a648e404ed04 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -19,8 +19,8 @@
#define MADE_PROGRESS 0x04 /* Made progress cleaning up a stream or the folio set */
#define BUFFERED 0x08 /* The pagecache needs cleaning up */
#define NEED_RETRY 0x10 /* A front op requests retrying */
-#define COPY_TO_CACHE 0x40 /* Need to copy subrequest to cache */
-#define ABANDON_SREQ 0x80 /* Need to abandon untransferred part of subrequest */
+#define COPY_TO_CACHE 0x20 /* Need to copy subrequest to cache */
+#define ABANDON_RREQ 0x40 /* Need to abandon the rest of a request */
/*
* Clear the unread part of an I/O request.
@@ -206,6 +206,78 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
netfs_read_set_unlock_at(rreq);
}
+/*
+ * Abandon all remaining read results. Once we've hit a permanent failure, we
+ * assume that the file is probably unusable. In the event of readahead, in
+ * theory we might manage to read some data later in the file, that we could
+ * still unlock, but ->read_folio() will be called again.
+ */
+static void netfs_abandon_read_results(struct netfs_io_request *rreq)
+{
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+ unsigned int notes = ABANDON_RREQ;
+
+ _enter("%llx-%llx", rreq->start, rreq->start + rreq->len);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_collect);
+ trace_netfs_collect(rreq);
+
+ if (rreq->origin == NETFS_READAHEAD ||
+ rreq->origin == NETFS_READPAGE ||
+ rreq->origin == NETFS_READ_FOR_WRITE)
+ notes |= BUFFERED;
+
+ /* Remove completed subrequests from the front of the stream and
+ * advance the completion point. We stop when we hit something that's
+ * in progress. The issuer thread may be adding stuff to the tail
+ * whilst we're doing this.
+ */
+ for (;;) {
+ struct netfs_io_subrequest *front;
+ unsigned long front_flags;
+
+ front = list_first_entry_or_null_acquire(&stream->subrequests,
+ struct netfs_io_subrequest, rreq_link);
+ /* Read first subreq pointer before IN_PROGRESS flag. */
+ if (!front)
+ break;
+
+ front_flags = smp_load_acquire(&front->flags);
+ /* Order read of flags before read of anything else, such as error. */
+
+ /* Wait for each subreq to complete. */
+ if (test_bit(NETFS_SREQ_IN_PROGRESS, &front_flags)) {
+ notes |= HIT_PENDING;
+ break;
+ }
+
+ /* The subreq now belongs to us. */
+ if (!stream->failed) {
+ stream->failed = true;
+ stream->error = front->error;
+ rreq->error = front->error;
+ trace_netfs_rreq(rreq, netfs_rreq_trace_set_abandon);
+ }
+
+ stream->collected_to = front->start + front->len;
+ trace_netfs_sreq(front, netfs_sreq_trace_abandoned);
+
+ spin_lock(&rreq->lock);
+ list_del_init(&front->rreq_link);
+ spin_unlock(&rreq->lock);
+ netfs_put_subrequest(front, netfs_sreq_trace_put_abandon);
+ }
+
+ rreq->collected_to = stream->collected_to;
+ rreq->abandon_to = rreq->collected_to;
+ if (notes & BUFFERED)
+ netfs_read_unlock_folios(rreq, ¬es);
+ else
+ rreq->cleaned_to = rreq->collected_to;
+
+ trace_netfs_collect_stream(rreq, stream);
+ trace_netfs_collect_state(rreq, rreq->collected_to, notes);
+}
+
/*
* Collect and assess the results of various read subrequests. We may need to
* retry some of the results.
@@ -225,6 +297,9 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
trace_netfs_collect(rreq);
reassess:
+ if (test_bit(NETFS_RREQ_ABANDON_REQ, &rreq->flags))
+ goto abandon_request;
+
if (rreq->origin == NETFS_READAHEAD ||
rreq->origin == NETFS_READPAGE ||
rreq->origin == NETFS_READ_FOR_WRITE)
@@ -242,7 +317,9 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
/* Read first subreq pointer before IN_PROGRESS flag. */
while (front) {
+ unsigned long front_flags;
size_t transferred;
+ uoff_t unlock_at = rreq->start + rreq->progress_at;
trace_netfs_collect_sreq(rreq, front);
_debug("sreq [%x] %llx %zx/%zx",
@@ -253,25 +330,63 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
stream->collected_to = front->start;
}
- if (netfs_check_subreq_in_progress(front))
+ front_flags = smp_load_acquire(&front->flags);
+ /* Order read of flags before read of anything else, such as error. */
+
+ if (test_bit(NETFS_SREQ_FAILED, &front_flags))
+ goto abandon_request;
+ if (test_bit(NETFS_SREQ_IN_PROGRESS, &front_flags))
notes |= HIT_PENDING;
- smp_rmb(); /* Read counters after IN_PROGRESS flag. */
+
transferred = READ_ONCE(front->transferred);
+ /* If we can collect the next folio from a pending op, do so,
+ * but we should only do it if we don't otherwise need to wait
+ * for completion.
+ */
+ if ((notes & HIT_PENDING) &&
+ (notes & BUFFERED) &&
+ !test_bit(NETFS_SREQ_HIT_EOF, &front_flags) &&
+ front->error == 0 &&
+ transferred < front->len
+ ) {
+ stream->collected_to = front->start + transferred;
+ rreq->collected_to = stream->collected_to;
+
+ if (test_bit(NETFS_SREQ_COPY_TO_CACHE, &front_flags))
+ notes |= COPY_TO_CACHE;
+ if (front->start + transferred >= unlock_at)
+ netfs_read_unlock_folios(rreq, ¬es);
+ }
+
+ /* Stall if the front is still undergoing I/O. */
+ if (notes & HIT_PENDING)
+ break;
+
+ if (test_bit(NETFS_SREQ_NEED_RETRY, &front_flags)) {
+ stream->need_retry = true;
+ notes |= NEED_RETRY | MADE_PROGRESS;
+ break;
+ } else if (test_bit(NETFS_RREQ_SHORT_TRANSFER, &rreq->flags)) {
+ notes |= MADE_PROGRESS;
+ } else {
+ stream->transferred += transferred;
+ stream->transferred_valid = true;
+ if (front->transferred < front->len)
+ set_bit(NETFS_RREQ_SHORT_TRANSFER, &rreq->flags);
+ notes |= MADE_PROGRESS;
+ }
+
/* If we can now collect the next folio, do so. We don't want
* to defer this as we have to decide whether we need to copy
* to the cache or not, and that may differ between adjacent
* subreqs.
*/
if (notes & BUFFERED) {
- uoff_t unlock_at = rreq->start + rreq->progress_at;
-
/* Clear the tail of a short read. */
- if (!(notes & HIT_PENDING) &&
- front->error == 0 &&
- transferred < front->len &&
- (test_bit(NETFS_SREQ_HIT_EOF, &front->flags) ||
- test_bit(NETFS_SREQ_CLEAR_TAIL, &front->flags))) {
+ if (transferred < front->len &&
+ (test_bit(NETFS_SREQ_HIT_EOF, &front_flags) ||
+ test_bit(NETFS_SREQ_CLEAR_TAIL, &front_flags))) {
netfs_clear_unread(front);
transferred = front->transferred = front->len;
trace_netfs_sreq(front, netfs_sreq_trace_clear);
@@ -280,67 +395,28 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
stream->collected_to = front->start + transferred;
rreq->collected_to = stream->collected_to;
- if (test_bit(NETFS_SREQ_COPY_TO_CACHE, &front->flags))
+ if (test_bit(NETFS_SREQ_COPY_TO_CACHE, &front_flags))
notes |= COPY_TO_CACHE;
- if (test_bit(NETFS_SREQ_FAILED, &front->flags)) {
- rreq->abandon_to = front->start + front->len;
- front->transferred = front->len;
- transferred = front->len;
- trace_netfs_rreq(rreq, netfs_rreq_trace_set_abandon);
- }
if (front->start + transferred >= unlock_at ||
- test_bit(NETFS_SREQ_HIT_EOF, &front->flags))
+ test_bit(NETFS_SREQ_HIT_EOF, &front_flags))
netfs_read_unlock_folios(rreq, ¬es);
} else {
stream->collected_to = front->start + transferred;
rreq->collected_to = stream->collected_to;
}
- /* Stall if the front is still undergoing I/O. */
- if (notes & HIT_PENDING)
- break;
-
- if (test_bit(NETFS_SREQ_FAILED, &front->flags)) {
- if (!stream->failed) {
- stream->error = front->error;
- rreq->error = front->error;
- set_bit(NETFS_RREQ_FAILED, &rreq->flags);
- stream->failed = true;
- }
- notes |= MADE_PROGRESS | ABANDON_SREQ;
- } else if (test_bit(NETFS_SREQ_NEED_RETRY, &front->flags)) {
- stream->need_retry = true;
- notes |= NEED_RETRY | MADE_PROGRESS;
- break;
- } else if (test_bit(NETFS_RREQ_SHORT_TRANSFER, &rreq->flags)) {
- notes |= MADE_PROGRESS;
- } else {
- if (!stream->failed) {
- stream->transferred += transferred;
- stream->transferred_valid = true;
- }
- if (front->transferred < front->len)
- set_bit(NETFS_RREQ_SHORT_TRANSFER, &rreq->flags);
- notes |= MADE_PROGRESS;
- }
-
/* Remove if completely consumed. */
stream->source = front->source;
spin_lock(&rreq->lock);
remove = front;
- trace_netfs_sreq(front,
- notes & ABANDON_SREQ ?
- netfs_sreq_trace_abandoned : netfs_sreq_trace_consumed);
+ trace_netfs_sreq(front, netfs_sreq_trace_consumed);
list_del_init(&front->rreq_link);
front = list_first_entry_or_null(&stream->subrequests,
struct netfs_io_subrequest, rreq_link);
spin_unlock(&rreq->lock);
- netfs_put_subrequest(remove,
- notes & ABANDON_SREQ ?
- netfs_sreq_trace_put_abandon :
- netfs_sreq_trace_put_done);
+ netfs_put_subrequest(remove, netfs_sreq_trace_put_done);
}
trace_netfs_collect_stream(rreq, stream);
@@ -369,6 +445,12 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
_debug("retry");
netfs_retry_reads(rreq);
goto out;
+
+abandon_request:
+ set_bit(NETFS_RREQ_FAILED, &rreq->flags);
+ set_bit(NETFS_RREQ_ABANDON_REQ, &rreq->flags);
+ netfs_wake_rreq_flag(rreq, NETFS_RREQ_PAUSE, netfs_rreq_trace_unpause);
+ return netfs_abandon_read_results(rreq);
}
/*
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index af111c589a50..6ee3891845e9 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -287,6 +287,7 @@ struct netfs_io_request {
#define NETFS_RREQ_FAILED 3 /* The request failed */
#define NETFS_RREQ_RETRYING 4 /* Set if we're in the retry path */
#define NETFS_RREQ_SHORT_TRANSFER 5 /* Set if we have a short transfer */
+#define NETFS_RREQ_ABANDON_REQ 6 /* Set if the request is to be abandoned */
#define NETFS_RREQ_CACHE_STOP 8 /* Set to stop caching (ENOBUFS or error) */
#define NETFS_RREQ_CACHE_ERROR 9 /* Set if we got an error from the cache */
#define NETFS_RREQ_OFFLOAD_COLLECTION 12 /* Offload collection to workqueue */
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 31/35] netfs: Check for too much data being read
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (29 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 30/35] netfs: Simplify read abandonment David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 32/35] netfs: Add a method to get an estimate of the amount that can be written David Howells
` (4 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Put in a check in read subreq termination to detect more data being read
for a subrequest than was requested. In the event that this happens, abort
the rest of the read request on the basis that some of the read buffer may
have been corrupted and return -EIO.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/read_collect.c | 24 ++++++++++++++++++++++++
include/trace/events/netfs.h | 1 +
2 files changed, 25 insertions(+)
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index a648e404ed04..e91f7962df04 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -339,6 +339,15 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
notes |= HIT_PENDING;
transferred = READ_ONCE(front->transferred);
+ if (unlikely(transferred > front->len)) {
+ /* Ugh... A subreq overran its allotted length. It
+ * may have corrupted the read buffer.
+ */
+ stream->failed = true;
+ stream->error = -EIO;
+ rreq->error = -EIO;
+ goto abandon_request;
+ }
/* If we can collect the next folio from a pending op, do so,
* but we should only do it if we don't otherwise need to wait
@@ -655,6 +664,21 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)
break;
}
+ /* If the subrequest read more than it was supposed to, abort
+ * the request with EIO as we may have clobbered other parts
+ * of the buffer that are already read.
+ */
+ if (subreq->transferred > subreq->len) {
+ trace_netfs_sreq(subreq, netfs_sreq_trace_too_much);
+ __set_bit(NETFS_SREQ_FAILED, &subreq->flags);
+ __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ subreq->error = -EIO;
+ trace_netfs_failure(rreq, subreq, subreq->error, netfs_fail_read);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_set_pause);
+ set_bit(NETFS_RREQ_PAUSE, &rreq->flags);
+ goto skip_error_checks;
+ }
+
/* Deal with retry requests, short reads and errors. If we retry
* but don't make progress, we abandon the attempt.
*/
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 4b2ffd14a1ca..312dc2214d92 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -133,6 +133,7 @@
EM(netfs_sreq_trace_submit, "SUBMT") \
EM(netfs_sreq_trace_superfluous, "SPRFL") \
EM(netfs_sreq_trace_terminated, "TERM ") \
+ EM(netfs_sreq_trace_too_much, "!TOOM") \
EM(netfs_sreq_trace_wait_for, "_WAIT") \
EM(netfs_sreq_trace_write, "WRITE") \
EM(netfs_sreq_trace_write_skip, "SKIP ") \
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 32/35] netfs: Add a method to get an estimate of the amount that can be written
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (30 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 31/35] netfs: Check for too much data being read David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 33/35] netfs: Rework writeback to use a separate list of regions to be unlocked David Howells
` (3 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Add a method, both to the netfs cache API and the netfs filesystem API,
that allows netfslib to query how much can be written to a stream in one
go.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/9p/vfs_addr.c | 17 +++++++++++++++++
fs/afs/file.c | 2 +-
fs/afs/internal.h | 3 +++
fs/afs/write.c | 17 +++++++++++++++++
fs/cachefiles/io.c | 15 ++++++++++++++-
fs/netfs/write_issue.c | 2 ++
fs/smb/client/file.c | 14 ++++++++++++++
include/linux/netfs.h | 22 ++++++++++++++++++++++
8 files changed, 90 insertions(+), 2 deletions(-)
diff --git a/fs/9p/vfs_addr.c b/fs/9p/vfs_addr.c
index 2129fcb0f65c..9b163a13ec3f 100644
--- a/fs/9p/vfs_addr.c
+++ b/fs/9p/vfs_addr.c
@@ -48,6 +48,22 @@ static void v9fs_begin_writeback(struct netfs_io_request *wreq)
wreq->io_streams[0].avail = true;
}
+/*
+ * Estimate how much data should be accumulated before we start issuing
+ * write subrequests.
+ */
+static int v9fs_estimate_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate)
+{
+ struct p9_fid *fid = wreq->netfs_priv;
+ unsigned long long limit = ULLONG_MAX - stream->issue_from;
+ unsigned long long max_len = fid->clnt->msize - P9_IOHDRSZ;
+
+ estimate->issue_at = stream->issue_from + umin(max_len, limit);
+ return 0;
+}
+
/*
* Issue a subrequest to write to the server.
*/
@@ -185,6 +201,7 @@ const struct netfs_request_ops v9fs_req_ops = {
.free_request = v9fs_free_request,
.issue_read = v9fs_issue_read,
.begin_writeback = v9fs_begin_writeback,
+ .estimate_write = v9fs_estimate_write,
.issue_write = v9fs_issue_write,
};
diff --git a/fs/afs/file.c b/fs/afs/file.c
index 11695be8c4a5..4a412e0472ec 100644
--- a/fs/afs/file.c
+++ b/fs/afs/file.c
@@ -469,7 +469,7 @@ const struct netfs_request_ops afs_req_ops = {
.update_i_size = afs_update_i_size,
.invalidate_cache = afs_netfs_invalidate_cache,
.begin_writeback = afs_begin_writeback,
- .prepare_write = afs_prepare_write,
+ .estimate_write = afs_estimate_write,
.issue_write = afs_issue_write,
.retry_request = afs_retry_request,
};
diff --git a/fs/afs/internal.h b/fs/afs/internal.h
index 556e7e7777f2..ae2bc699bfde 100644
--- a/fs/afs/internal.h
+++ b/fs/afs/internal.h
@@ -1698,6 +1698,9 @@ extern int afs_check_volume_status(struct afs_volume *, struct afs_operation *);
/*
* write.c
*/
+int afs_estimate_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate);
void afs_prepare_write(struct netfs_io_subrequest *subreq);
void afs_issue_write(struct netfs_io_subrequest *subreq);
void afs_begin_writeback(struct netfs_io_request *wreq);
diff --git a/fs/afs/write.c b/fs/afs/write.c
index 7f34b939706a..a0c9916f594b 100644
--- a/fs/afs/write.c
+++ b/fs/afs/write.c
@@ -82,6 +82,23 @@ static const struct afs_operation_ops afs_store_data_operation = {
.success = afs_store_data_success,
};
+/*
+ * Estimate the maximum size of a write we can send to the server.
+ */
+int afs_estimate_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate)
+{
+ unsigned long long limit = ULLONG_MAX - stream->issue_from;
+ unsigned long long max_len = 256 * 1024 * 1024;
+
+ //if (test_bit(NETFS_SREQ_RETRYING, &subreq->flags))
+ // max_len = 512 * 1024;
+
+ estimate->issue_at = stream->issue_from + umin(max_len, limit);
+ return 0;
+}
+
/*
* Prepare a subrequest to write to the server. This sets the max_len
* parameter.
diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c
index 23007f450ef4..91b69be78c6b 100644
--- a/fs/cachefiles/io.c
+++ b/fs/cachefiles/io.c
@@ -26,7 +26,10 @@ struct cachefiles_kiocb {
};
struct cachefiles_object *object;
netfs_io_terminated_t term_func;
- void *term_func_priv;
+ union {
+ struct netfs_io_subrequest *subreq;
+ void *term_func_priv;
+ };
bool was_async;
unsigned int inval_counter; /* Copy of cookie->inval_counter */
u64 b_writing;
@@ -611,6 +614,15 @@ static int cachefiles_prepare_write(struct netfs_cache_resources *cres,
return ret;
}
+static int cachefiles_estimate_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate)
+{
+ estimate->issue_at = stream->issue_from + MAX_RW_COUNT;
+ estimate->max_segs = BIO_MAX_VECS;
+ return 0;
+}
+
static void cachefiles_prepare_write_subreq(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *wreq = subreq->rreq;
@@ -898,6 +910,7 @@ static const struct netfs_cache_ops cachefiles_netfs_cache_ops = {
.issue_write = cachefiles_issue_write,
.prepare_write = cachefiles_prepare_write,
.prepare_write_subreq = cachefiles_prepare_write_subreq,
+ .estimate_write = cachefiles_estimate_write,
.query_occupancy = cachefiles_query_occupancy,
.collect_write = cachefiles_collect_write,
};
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index a5c5ba5deb43..025ea5763fda 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -114,6 +114,7 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
wreq->io_streams[0].stream_nr = 0;
wreq->io_streams[0].source = NETFS_UPLOAD_TO_SERVER;
+ wreq->io_streams[0].estimate_write = ictx->ops->estimate_write;
wreq->io_streams[0].prepare_write = ictx->ops->prepare_write;
wreq->io_streams[0].issue_write = ictx->ops->issue_write;
wreq->io_streams[0].collected_to = start;
@@ -126,6 +127,7 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
if (fscache_resources_valid(&wreq->cache_resources)) {
wreq->io_streams[1].avail = true;
wreq->io_streams[1].active = true;
+ wreq->io_streams[1].estimate_write = wreq->cache_resources.ops->estimate_write;
wreq->io_streams[1].prepare_write = wreq->cache_resources.ops->prepare_write_subreq;
wreq->io_streams[1].issue_write = wreq->cache_resources.ops->issue_write;
}
diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c
index ac89c1ba56b1..7ad464782439 100644
--- a/fs/smb/client/file.c
+++ b/fs/smb/client/file.c
@@ -39,6 +39,19 @@
static int cifs_reopen_file(struct cifsFileInfo *cfile, bool can_flush);
+/*
+ * Estimate the amount of data that can be written in one RPC op.
+ */
+static int cifs_estimate_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate)
+{
+ struct cifs_sb_info *cifs_sb = CIFS_SB(wreq->inode->i_sb);
+
+ estimate->issue_at = stream->issue_from + cifs_sb->ctx->wsize;
+ return 0;
+}
+
/*
* Prepare a subrequest to upload to the server. We need to allocate credits
* so that we know the maximum amount of data that we can include in it.
@@ -362,6 +375,7 @@ const struct netfs_request_ops cifs_req_ops = {
.issue_read = cifs_issue_read,
.done = cifs_rreq_done,
.begin_writeback = cifs_begin_writeback,
+ .estimate_write = cifs_estimate_write,
.prepare_write = cifs_prepare_write,
.issue_write = cifs_issue_write,
.invalidate_cache = cifs_netfs_invalidate_cache,
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 6ee3891845e9..656f4f3f80b6 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -134,6 +134,16 @@ enum netfs_cache_collect {
NETFS_CACHE_COLLECT_WRITE_CANCEL, /* Currently collecting cancelled writes */
};
+/*
+ * Estimate of maximum write subrequest for writeback. The filesystem is
+ * responsible for filling this in when called from ->estimate_write(), though
+ * netfslib will preset infinite defaults.
+ */
+struct netfs_write_estimate {
+ uoff_t issue_at; /* Point at which we must submit */
+ int max_segs; /* Max number of segments in a single RPC */
+};
+
/*
* Stream of I/O subrequests going to a particular destination, such as the
* server or the local cache. This is mainly intended for writing where we may
@@ -142,10 +152,14 @@ enum netfs_cache_collect {
struct netfs_io_stream {
/* Submission tracking */
struct netfs_io_subrequest *construct; /* Op being constructed */
+ uoff_t issue_from; /* Current issue point */
size_t sreq_max_len; /* Maximum size of a subrequest */
unsigned int sreq_max_segs; /* 0 or max number of segments in an iterator */
unsigned int submit_off; /* Folio offset we're submitting from */
unsigned int submit_len; /* Amount of data left to submit */
+ int (*estimate_write)(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate);
void (*prepare_write)(struct netfs_io_subrequest *subreq);
void (*issue_write)(struct netfs_io_subrequest *subreq);
/* Collection tracking */
@@ -328,6 +342,9 @@ struct netfs_request_ops {
/* Write request handling */
void (*begin_writeback)(struct netfs_io_request *wreq);
+ int (*estimate_write)(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate);
void (*prepare_write)(struct netfs_io_subrequest *subreq);
void (*issue_write)(struct netfs_io_subrequest *subreq);
void (*retry_request)(struct netfs_io_request *wreq, struct netfs_io_stream *stream);
@@ -364,6 +381,11 @@ struct netfs_cache_ops {
netfs_io_terminated_t term_func,
void *term_func_priv);
+ /* Estimate the amount of data that can be written in an op. */
+ int (*estimate_write)(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate);
+
/* Write data to the cache from a netfs subrequest. */
void (*issue_write)(struct netfs_io_subrequest *subreq);
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 33/35] netfs: Rework writeback to use a separate list of regions to be unlocked
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (31 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 32/35] netfs: Add a method to get an estimate of the amount that can be written David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 34/35] netfs: Combine prepare and issue ops and grab the buffers on request David Howells
` (2 subsequent siblings)
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Currently, the netfslib buffered writeback algorithm walks the list of
folios, using that to determine the folios that need to be unlocked. This
is tricky, however, as different streams really want different folios or
different parts of folios (e.g. data that's read from the server will be
written to the cache, but not written back to the server, and a small
region that can be written to the server may need to be rounded out for DIO
write to the cache).
This also may require the collector thread to be walking the folio list at
the same time that the application thread is filling it - and at the same
time as things are doing I/O to or from it.
Also, it requires very careful cleanup during collection, such that there's
always at least one link in the chain so that the consumer never gets
disconnected from the consumer.
Instead, rework the buffered writeback collector such that the list of
regions to be unlocked is kept separate from the bvecq chain that is used
to pass list of folio or other memory fragments to the filesystem with each
subrequest.
With this, the collector walks the xarray to find the folios to unlock
rather than scanning the list of folios and then deriving a folio pointer
from the bio_vec. This has the future potential to be combined with
folio_end_writeback() as both need to do the xarray lookup, and there are
also stats jugging and suchlike that can be done in bulk for several
folios.
The way that folios are passed to subrequests is also modified so that a
piece of a folio at a time is attached to the bvecq in such a way that
should bounce buffering be needed (e.g. for content encryption), the
individual crypto blocks can be passed instead of having to assemble a
large folio's worth at a time (which might be a problem under memory
pressure).
Note that this can also be applied to buffered read collection, but not
unbuffered/DIO collection as in the latter case, the list of fragments is
the only stable reference available to perform the GUP cleanup.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/buffered_read.c | 1 +
fs/netfs/internal.h | 5 +-
fs/netfs/main.c | 7 +
fs/netfs/objects.c | 11 +
fs/netfs/read_pgpriv2.c | 4 +-
fs/netfs/write_collect.c | 196 +++++++----
fs/netfs/write_issue.c | 641 +++++++++++++++++++++++++----------
include/linux/netfs.h | 30 +-
include/trace/events/netfs.h | 32 +-
9 files changed, 675 insertions(+), 252 deletions(-)
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 8a843822d47b..88e5404080e8 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -464,6 +464,7 @@ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct fo
bvecq_filled_to(bq, 1);
rreq->submitted = rreq->start + fsize;
rreq->progress_at = fsize;
+ bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
return 0;
}
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 79260a13682b..3aa9c7ef85de 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -44,6 +44,7 @@ extern struct list_head netfs_io_requests;
extern spinlock_t netfs_proc_lock;
extern mempool_t netfs_request_pool;
extern mempool_t netfs_subrequest_pool;
+extern mempool_t netfs_writeback_pool;
extern mempool_t netfs_bvecq_pool;
#ifdef CONFIG_PROC_FS
@@ -220,7 +221,7 @@ static inline void netfs_stat_d(atomic_t *stat)
/*
* write_collect.c
*/
-int netfs_folio_written_back(struct folio *folio);
+void netfs_folio_written_back(struct folio *folio, void *cleaner_priv);
bool netfs_write_collection(struct netfs_io_request *wreq);
void netfs_write_collection_worker(struct work_struct *work);
@@ -231,6 +232,8 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
struct file *file,
uoff_t start,
enum netfs_io_origin origin);
+struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream);
void netfs_prepare_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
uoff_t start);
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index 5d8b87f71888..b8da5e85cc67 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -28,6 +28,7 @@ static struct kmem_cache *netfs_request_slab;
static struct kmem_cache *netfs_subrequest_slab;
mempool_t netfs_request_pool;
mempool_t netfs_subrequest_pool;
+mempool_t netfs_writeback_pool;
mempool_t netfs_bvecq_pool;
#ifdef CONFIG_PROC_FS
@@ -110,6 +111,9 @@ static int __init netfs_init(void)
if (mempool_init_kmalloc_pool(&netfs_bvecq_pool, 100, BVECQ_STD_SIZE) < 0)
goto error_bvecq_pool;
+ if (mempool_init_kmalloc_pool(&netfs_writeback_pool, 100,
+ sizeof(struct netfs_writeback)) < 0)
+ goto error_writeback_pool;
netfs_request_slab = kmem_cache_create("netfs_request",
sizeof(struct netfs_io_request), 0,
@@ -163,6 +167,8 @@ static int __init netfs_init(void)
error_reqpool:
kmem_cache_destroy(netfs_request_slab);
error_req:
+ mempool_exit(&netfs_writeback_pool);
+error_writeback_pool:
mempool_exit(&netfs_bvecq_pool);
error_bvecq_pool:
return ret;
@@ -177,6 +183,7 @@ static void __exit netfs_exit(void)
kmem_cache_destroy(netfs_subrequest_slab);
mempool_exit(&netfs_request_pool);
kmem_cache_destroy(netfs_request_slab);
+ mempool_exit(&netfs_writeback_pool);
mempool_exit(&netfs_bvecq_pool);
}
module_exit(netfs_exit);
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index 740971955198..a1d0de65484e 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -66,6 +66,8 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
INIT_LIST_HEAD(&stream->subrequests);
stream->collected_to = rreq->start;
+ stream->issue_from = rreq->start;
+ stream->alignment = 1;
}
if (origin == NETFS_READAHEAD ||
@@ -151,6 +153,15 @@ static void netfs_deinit_request(struct netfs_io_request *rreq)
bvecq_pos_unset(&rreq->dispatch_cursor);
bvecq_pos_unset(&rreq->collect_cursor);
bvecq_put(rreq->spare);
+ while (rreq->writebacks) {
+ struct netfs_writeback *wback = rreq->writebacks;
+
+ rreq->writebacks = wback->next;
+ mempool_free(wback, &netfs_bvecq_pool);
+
+ }
+ for (int i = 0; i < NR_IO_STREAMS; i++)
+ bvecq_pos_unset(&rreq->io_streams[i].dispatch_cursor);
if (atomic_dec_and_test(&ictx->io_count))
wake_up_var(&ictx->io_count);
diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c
index 2e4eab1e327f..14fe90751b5b 100644
--- a/fs/netfs/read_pgpriv2.c
+++ b/fs/netfs/read_pgpriv2.c
@@ -95,7 +95,7 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
creq->dispatch_cursor.offset = cache->submit_off;
- atomic64_set(&creq->issued_to, fpos + cache->submit_off);
+ atomic64_set(&cache->issued_to, fpos + cache->submit_off);
part = netfs_advance_write(creq, cache, fpos + cache->submit_off,
cache->submit_len, to_eof);
cache->submit_off += part;
@@ -106,7 +106,7 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
} while (cache->submit_len > 0);
bvecq_pos_step(&creq->dispatch_cursor);
- atomic64_set(&creq->issued_to, fpos + fsize);
+ atomic64_set(&cache->issued_to, fpos + fsize);
if (flen < fsize)
netfs_issue_write(creq, cache);
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index b980a65441dc..ccc9fa5d1883 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -21,6 +21,51 @@
#define NEED_RETRY 0x10 /* A front op requests retrying */
#define SAW_FAILURE 0x20 /* One stream or hit a permanent failure */
+/**
+ * folio_end_writeback_range - End writeback for the folios within the range
+ * @mapping: The pagecache to modify
+ * @from: Pointer to the starting position (updated)
+ * @to: The end position (exclusive)
+ * @cleaner_func: Function to clean up the folios in the range
+ * @cleaner_priv: Private data for the cleaner func
+ *
+ * Unlock folios that are entirely within in the given range, where @from is
+ * included in the range, but @to is excluded from the range.
+ *
+ * Return: True if at least one folio got cleaned, false otherwise. @from will
+ * be updated to point past the last folio cleaned.
+ */
+static inline
+bool folio_end_writeback_range(struct address_space *mapping,
+ uoff_t *from, uoff_t to,
+ void (*cleaner_func)(struct folio *folio,
+ void *cleaner_priv),
+ void *cleaner_priv)
+{
+ struct folio *folio;
+ XA_STATE(xas, &mapping->i_pages, *from / PAGE_SIZE);
+ bool cleaned = false;
+
+ rcu_read_lock();
+ xas_for_each(&xas, folio, (to - 1) / PAGE_SIZE) {
+ uoff_t fend;
+
+ if (xas_retry(&xas, folio))
+ continue;
+
+ fend = folio_next_pos(folio);
+ if (fend > to)
+ break;
+
+ cleaner_func(folio, cleaner_priv);
+ folio_end_writeback(folio);
+ *from = fend;
+ cleaned = true;
+ }
+ rcu_read_unlock();
+ return cleaned;
+}
+
static void netfs_dump_request(const struct netfs_io_request *rreq)
{
pr_err("Request R=%08x r=%d fl=%lx or=%x e=%ld\n",
@@ -28,8 +73,8 @@ static void netfs_dump_request(const struct netfs_io_request *rreq)
rreq->origin, rreq->error);
pr_err(" st=%llx tsl=%zx/%llx/%llx\n",
rreq->start, rreq->transferred, rreq->submitted, rreq->len);
- pr_err(" cci=%llx/%llx/%llx\n",
- rreq->cleaned_to, rreq->collected_to, atomic64_read(&rreq->issued_to));
+ pr_err(" cci=%llx/%llx\n",
+ rreq->cleaned_to, rreq->collected_to);
pr_err(" iw=%pSR\n", rreq->netfs_ops->issue_write);
for (int i = 0; i < NR_IO_STREAMS; i++) {
const struct netfs_io_subrequest *sreq;
@@ -38,8 +83,9 @@ static void netfs_dump_request(const struct netfs_io_request *rreq)
pr_err(" str[%x] s=%x e=%d acnf=%u,%u,%u,%u\n",
s->stream_nr, s->source, s->error,
s->avail, s->active, s->need_retry, s->failed);
- pr_err(" str[%x] ct=%llx t=%zx\n",
- s->stream_nr, s->collected_to, s->transferred);
+ pr_err(" str[%x] it=%llx ct=%llx t=%zx\n",
+ s->stream_nr, atomic64_read(&s->issued_to),
+ s->collected_to, s->transferred);
list_for_each_entry(sreq, &s->subrequests, rreq_link) {
pr_err(" sreq[%x:%x] sc=%u s=%llx t=%zx/%zx r=%d f=%lx\n",
sreq->stream_nr, sreq->debug_index, sreq->source,
@@ -54,14 +100,23 @@ static void netfs_dump_request(const struct netfs_io_request *rreq)
* that we are not allowed to lock the folio here on pain of deadlocking with
* truncate.
*/
-int netfs_folio_written_back(struct folio *folio)
+void netfs_folio_written_back(struct folio *folio, void *cleaner_priv)
{
enum netfs_folio_trace why = netfs_folio_trace_endwb;
+ struct netfs_io_request *wreq = cleaner_priv;
struct inode *inode = folio_inode(folio);
struct netfs_inode *ictx = netfs_inode(inode);
struct netfs_folio *finfo;
struct netfs_group *group = NULL;
- int gcount = 0;
+
+ if (WARN_ONCE(!folio_test_writeback(folio),
+ "R=%08x: folio %lx is not under writeback\n",
+ wreq->debug_id, folio->index)) {
+ trace_netfs_folio(folio, netfs_folio_trace_not_under_wback);
+ netfs_dump_request(wreq);
+ }
+
+ trace_netfs_collect_folio(wreq, folio);
if ((finfo = netfs_folio_info(folio))) {
/* Streaming writes cannot be redirtied whilst under writeback,
@@ -77,7 +132,7 @@ int netfs_folio_written_back(struct folio *folio)
folio_detach_private(folio);
group = finfo->netfs_group;
- gcount++;
+ wreq->nr_group_rel++;
kfree(finfo);
why = netfs_folio_trace_endwb_s;
goto end_wb;
@@ -97,15 +152,13 @@ int netfs_folio_written_back(struct folio *folio)
why = netfs_folio_trace_redirtied;
if (!folio_test_dirty(folio)) {
folio_detach_private(folio);
- gcount++;
+ wreq->nr_group_rel++;
why = netfs_folio_trace_endwb_g;
}
}
end_wb:
trace_netfs_folio(folio, why);
- folio_end_writeback(folio);
- return gcount;
}
/*
@@ -114,15 +167,7 @@ int netfs_folio_written_back(struct folio *folio)
static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
unsigned int *notes)
{
- struct bvecq *bvecq = wreq->collect_cursor.bvecq;
- unsigned int slot = wreq->collect_cursor.slot;
- uoff_t collected_to = wreq->collected_to;
-
- if (WARN_ON_ONCE(!bvecq)) {
- pr_err("[!] Writeback unlock found empty buffer!\n");
- netfs_dump_request(wreq);
- return;
- }
+ struct netfs_writeback *wback, *next;
if (wreq->origin == NETFS_PGPRIV2_COPY_TO_CACHE) {
if (netfs_pgpriv2_unlock_copied_folios(wreq))
@@ -130,57 +175,46 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
return;
}
+ wback = wreq->writebacks;
+
for (;;) {
- struct folio *folio;
- struct netfs_folio *finfo;
- uoff_t fpos, fend;
- size_t fsize, flen;
-
- /* Try to clean up the head of the queue if it appears to be
- * used up, but we need to be very careful - the cleanup can
- * catch the dispatcher, which could lead to us having nothing
- * left in the queue, causing the front and back pointers to
- * end up on different tracks. To avoid this, we must always
- * keep at least one segment in the queue.
- */
- if (!bvecq_acquire_slot(bvecq, slot)) {
- wreq->collect_cursor.slot = slot;
- if (!bvecq_delete_spent(&wreq->collect_cursor))
- return;
- bvecq = wreq->collect_cursor.bvecq;
- slot = wreq->collect_cursor.slot;
- }
+ uoff_t stop_at;
+ size_t len;
- folio = page_folio(bvecq->bv[slot].bv_page);
- if (WARN_ONCE(!folio_test_writeback(folio),
- "R=%08x: folio %lx is not under writeback\n",
- wreq->debug_id, folio->index))
- trace_netfs_folio(folio, netfs_folio_trace_not_under_wback);
+ /* Jump over discontiguities. */
+ if (wreq->cleaned_to < wback->start)
+ wreq->cleaned_to = wback->start;
- fpos = folio_pos(folio);
- fsize = folio_size(folio);
- finfo = netfs_folio_info(folio);
- flen = finfo ? finfo->dirty_offset + finfo->dirty_len : fsize;
+ if (wreq->collected_to <= wreq->cleaned_to)
+ break;
- fend = min_t(uoff_t, fpos + flen, wreq->i_size);
+ /* Order read of region length before reading folios. */
+ len = smp_load_acquire(&wback->len);
- trace_netfs_collect_folio(wreq, folio);
+ if (wreq->cleaned_to >= wback->start + len) {
+ /* Order read of next before recheck length. */
+ next = smp_load_acquire(&wback->next);
+ if (!next)
+ break; /* We don't remove the tail writeback. */
- /* Unlock any folio we've transferred all of. */
- if (collected_to < fend)
- break;
+ /* Order read of region length before reading folios. */
+ if (len != smp_load_acquire(&wback->len))
+ continue; /* len/next update race. */
- wreq->nr_group_rel += netfs_folio_written_back(folio);
- wreq->cleaned_to = fpos + fsize;
- *notes |= MADE_PROGRESS;
+ mempool_free(wback, &netfs_bvecq_pool);
+ wreq->writebacks = next;
+ wback = next;
+ continue;
+ }
- bvecq->bv[slot].bv_page = NULL;
- slot++;
- if (fpos + fsize >= collected_to)
+ stop_at = min(wreq->collected_to, wback->start + len);
+
+ trace_netfs_collect_folios(wreq, wback->start, len);
+ if (!folio_end_writeback_range(wreq->mapping, &wreq->cleaned_to, stop_at,
+ netfs_folio_written_back, wreq))
break;
+ *notes |= MADE_PROGRESS;
}
-
- wreq->collect_cursor.slot = slot;
}
/*
@@ -225,9 +259,7 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
trace_netfs_rreq(wreq, netfs_rreq_trace_collect);
reassess_streams:
- /* Order reading the issued_to point before reading the queue it refers to. */
- issued_to = atomic64_read_acquire(&wreq->issued_to);
- smp_rmb();
+ issued_to = ULLONG_MAX;
collected_to = ULLONG_MAX;
if (wreq->origin == NETFS_WRITEBACK ||
wreq->origin == NETFS_PGPRIV2_COPY_TO_CACHE)
@@ -241,19 +273,34 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
* to the tail whilst we're doing this.
*/
for (s = 0; s < NR_IO_STREAMS; s++) {
+ uoff_t s_issued_to;
+
stream = &wreq->io_streams[s];
- /* Read active flag before list pointers */
+ /* Read active flag before issued_to */
if (!smp_load_acquire(&stream->active))
continue;
- front = list_first_entry_or_null_acquire(&stream->subrequests,
- struct netfs_io_subrequest, rreq_link);
- /* Read first subreq pointer before IN_PROGRESS flag. */
-
- while (front) {
+ for (;;) {
enum netfs_cache_collect cache_collect;
- trace_netfs_collect_sreq(wreq, front);
+ /* Order reading the issued_to point before reading the
+ * queue it refers to.
+ */
+ s_issued_to = atomic64_read_acquire(&stream->issued_to);
+ if (s_issued_to < issued_to)
+ issued_to = s_issued_to;
+
+ front = list_first_entry_or_null_acquire(&stream->subrequests,
+ struct netfs_io_subrequest,
+ rreq_link);
+ /* Read first subreq pointer before IN_PROGRESS flag. */
+ if (!front) {
+ if (stream->source == NETFS_UPLOAD_TO_SERVER &&
+ test_bit(NETFS_RREQ_PAUSE, &wreq->flags))
+ notes |= MADE_PROGRESS;
+ break;
+ }
+
//_debug("sreq [%x] %llx %zx/%zx",
// front->debug_index, front->start, front->transferred, front->len);
@@ -272,13 +319,17 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
break;
}
+ trace_netfs_collect_sreq(wreq, front);
+
if (stream->failed) {
- stream->collected_to = front->start + front->len;
+ stream->collected_to = front->start + front->len + front->post_gap;
notes |= MADE_PROGRESS | SAW_FAILURE;
goto cancel;
}
if (front->start + front->transferred > stream->collected_to) {
stream->collected_to = front->start + front->transferred;
+ if (front->transferred == front->len)
+ stream->collected_to += front->post_gap;
stream->transferred = stream->collected_to - wreq->start;
stream->transferred_valid = true;
notes |= MADE_PROGRESS;
@@ -328,6 +379,7 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
cancel:
/* Remove if completely consumed. */
+ stream->collected_to = front->start + front->len + front->post_gap;
spin_lock(&wreq->lock);
remove = front;
@@ -540,6 +592,8 @@ void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error)
if (IS_ERR_VALUE(transferred_or_error)) {
subreq->error = transferred_or_error;
+ if (transferred_or_error == -ENOMEM)
+ set_bit(NETFS_RREQ_SAW_ENOMEM, &wreq->flags);
switch (subreq->source) {
case NETFS_WRITE_TO_CACHE:
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index 025ea5763fda..0c71f8c151c2 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -36,6 +36,38 @@
#include <linux/pagemap.h>
#include "internal.h"
+#define NOTE_UPLOAD_AVAIL 0x001 /* Upload is available */
+#define NOTE_CACHE_AVAIL 0x002 /* Local cache is available */
+#define NOTE_CACHE_COPY 0x004 /* Copy folio to cache */
+#define NOTE_UPLOAD 0x008 /* Upload folio to server */
+#define NOTE_UPLOAD_STARTED 0x010 /* Upload started */
+#define NOTE_STREAMW 0x020 /* Folio is from a streaming write */
+#define NOTE_FLUSH_ANYWAY 0x040 /* Flush data, even if not hit estimated limit */
+
+#define NOTES__KEEP_MASK (NOTE_UPLOAD_AVAIL | NOTE_CACHE_AVAIL | NOTE_UPLOAD_STARTED)
+
+struct netfs_wb_params {
+ uoff_t fpos;
+ unsigned int notes; /* Notes on applicability */
+
+ /* When we're using a bounce buffer, the outer data window is all of
+ * the data we encrypted, rounded out to the largest alignment; the
+ * inner data window is all the data that got changed, rounded out to
+ * the smallest alignment.
+ *
+ * We have two alignments at play: the size of chunk which we encrypt
+ * in one go (typically 4KiB) and the local cache DIO size.
+ */
+ unsigned int inner_align; /* Smallest alignment */
+ unsigned int inner_off; /* Start of inner data window */
+ unsigned int inner_end; /* End of inner data window */
+ unsigned int outer_align; /* Largest alignment */
+ unsigned int outer_off; /* Start of outer data window */
+ unsigned int outer_end; /* End of outer data window */
+
+ struct netfs_write_estimate estimates[NR_IO_STREAMS];
+};
+
/*
* Kill all dirty folios in the event of an unrecoverable error, starting with
* a locked folio we've already obtained from writeback_iter().
@@ -114,6 +146,7 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
wreq->io_streams[0].stream_nr = 0;
wreq->io_streams[0].source = NETFS_UPLOAD_TO_SERVER;
+ wreq->io_streams[0].applicable = NOTE_UPLOAD;
wreq->io_streams[0].estimate_write = ictx->ops->estimate_write;
wreq->io_streams[0].prepare_write = ictx->ops->prepare_write;
wreq->io_streams[0].issue_write = ictx->ops->issue_write;
@@ -122,6 +155,7 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
wreq->io_streams[1].stream_nr = 1;
wreq->io_streams[1].source = NETFS_WRITE_TO_CACHE;
+ wreq->io_streams[1].applicable = NOTE_CACHE_COPY;
wreq->io_streams[1].collected_to = start;
wreq->io_streams[1].transferred = 0;
if (fscache_resources_valid(&wreq->cache_resources)) {
@@ -130,6 +164,7 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
wreq->io_streams[1].estimate_write = wreq->cache_resources.ops->estimate_write;
wreq->io_streams[1].prepare_write = wreq->cache_resources.ops->prepare_write_subreq;
wreq->io_streams[1].issue_write = wreq->cache_resources.ops->issue_write;
+ wreq->io_streams[1].alignment = wreq->cache_resources.dio_size;
}
return wreq;
@@ -148,6 +183,58 @@ void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq)
}
EXPORT_SYMBOL(netfs_prepare_write_failed);
+/*
+ * Allocate and prepare a write subrequest. Will only return NULL if not
+ * performing writeback; if performing writeback, mempools may be accessed and
+ * the allocator may wait forever.
+ */
+struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream)
+{
+ struct netfs_io_subrequest *subreq;
+
+ subreq = netfs_alloc_subrequest(wreq);
+ if (!subreq)
+ return subreq;
+
+ subreq->source = stream->source;
+ subreq->start = stream->issue_from;
+ subreq->len = stream->buffered;
+ subreq->stream_nr = stream->stream_nr;
+
+ _enter("R=%x[%x]", wreq->debug_id, subreq->debug_index);
+
+ trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
+
+ switch (stream->source) {
+ case NETFS_UPLOAD_TO_SERVER:
+ netfs_stat(&netfs_n_wh_upload);
+ break;
+ case NETFS_WRITE_TO_CACHE:
+ netfs_stat(&netfs_n_wh_write);
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
+
+ __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
+
+ /* We add to the end of the list whilst the collector may be walking
+ * the list. The collector only goes nextwards and uses the lock to
+ * remove entries off of the front.
+ */
+ spin_lock(&wreq->lock);
+ /* Write IN_PROGRESS before pointer to new subreq */
+ list_add_tail_release(&subreq->rreq_link, &stream->subrequests);
+ if (list_is_first(&subreq->rreq_link, &stream->subrequests) &&
+ stream->collected_to == 0)
+ stream->collected_to = subreq->start;
+
+ spin_unlock(&wreq->lock);
+ return subreq;
+}
+
/*
* Prepare a write subrequest. We need to allocate a new subrequest
* if we don't have one.
@@ -209,6 +296,51 @@ void netfs_prepare_write(struct netfs_io_request *wreq,
stream->construct = subreq;
}
+/*
+ * Advance the state of the amount of data buffered on a stream.
+ */
+static void netfs_advance_stream(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_io_subrequest *subreq)
+{
+ stream->issue_from += subreq->len;
+ stream->buffered -= subreq->len;
+ if (stream->buffered == 0) {
+ subreq->post_gap = stream->post_gap;
+ stream->post_gap = 0;
+ stream->buffering = false;
+ bvecq_pos_unset(&stream->dispatch_cursor);
+ }
+ /* Order loading the queue before updating the issue_to point */
+ atomic64_set_release(&stream->issued_to, stream->issue_from);
+}
+
+/*
+ * Prepare the buffer for a buffered write.
+ */
+static int netfs_prepare_buffered_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *wreq = subreq->rreq;
+ struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
+ ssize_t len;
+
+ _enter("%zx,{,%u,%u},%u",
+ subreq->len, stream->dispatch_cursor.slot, stream->dispatch_cursor.offset, max_segs);
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs, &subreq->nr_segs);
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ netfs_advance_stream(wreq, stream, subreq);
+ return 0;
+}
+
/*
* Set the I/O iterator for the filesystem/cache to use and dispatch the I/O
* operation. The operation may be asynchronous and should call
@@ -326,31 +458,242 @@ size_t netfs_advance_write(struct netfs_io_request *wreq,
}
/*
- * Write some of a pending folio data back to the server.
+ * Prepare and issue a subrequest.
+ * TODO: Replace with combined ->prepare/->issue call().
+ */
+static int netfs_prep_and_issue_subreq(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_io_subrequest *subreq)
+{
+ stream->sreq_max_len = UINT_MAX;
+ stream->sreq_max_segs = INT_MAX;
+ switch (stream->source) {
+ case NETFS_UPLOAD_TO_SERVER:
+ netfs_stat(&netfs_n_wh_upload);
+ stream->sreq_max_len = wreq->wsize;
+ break;
+ case NETFS_WRITE_TO_CACHE:
+ netfs_stat(&netfs_n_wh_write);
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
+
+ if (stream->prepare_write)
+ stream->prepare_write(subreq);
+ netfs_prepare_buffered_write_buffer(subreq, stream->sreq_max_segs);
+ iov_iter_bvec_queue(&subreq->io_iter, ITER_SOURCE,
+ subreq->content.bvecq, subreq->content.slot,
+ subreq->content.offset,
+ subreq->len);
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+ stream->issue_write(subreq);
+ return 0;
+}
+
+/*
+ * Issue writes for a stream.
*/
-static int netfs_write_folio(struct netfs_io_request *wreq,
- struct writeback_control *wbc,
- struct folio *folio)
+static void netfs_writeback_flush(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_wb_params *params)
{
- struct netfs_io_stream *upload = &wreq->io_streams[0];
- struct netfs_io_stream *cache = &wreq->io_streams[1];
- struct netfs_io_stream *stream;
+ struct netfs_write_estimate *estimate = ¶ms->estimates[stream->stream_nr];
+
+ for (;;) {
+ struct netfs_io_subrequest *subreq;
+ int ret;
+
+ if (test_bit(NETFS_RREQ_PAUSE, &wreq->flags))
+ netfs_wait_for_paused_write(wreq);
+
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ /* subreq allocation in a writeback is backed by a mempool and
+ * will wait for an new one to come available.
+ */
+
+ if (stream->source == NETFS_WRITE_TO_CACHE &&
+ unlikely(test_bit(NETFS_RREQ_CACHE_STOP, &wreq->flags))) {
+ estimate->issue_at = ULLONG_MAX;
+ estimate->max_segs = INT_MAX;
+ __set_bit(NETFS_SREQ_CANCELLED, &subreq->flags);
+ netfs_advance_stream(wreq, stream, subreq);
+ netfs_write_subrequest_terminated(subreq, subreq->len);
+ return;
+ }
+
+ ret = netfs_prep_and_issue_subreq(wreq, stream, subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ bvecq_pos_advance(&stream->dispatch_cursor, subreq->len);
+ netfs_advance_stream(wreq, stream, subreq);
+ netfs_write_subrequest_terminated(subreq, ret);
+ }
+ /* We no longer own subreq. */
+
+ if (stream->buffered == 0) {
+ if (stream->stream_nr == 0)
+ params->notes &= ~NOTE_UPLOAD_STARTED;
+ return;
+ }
+
+ if (!(params->notes & NOTE_FLUSH_ANYWAY)) {
+ estimate->issue_at = ULLONG_MAX;
+ estimate->max_segs = INT_MAX;
+ stream->estimate_write(wreq, stream, estimate);
+ if (stream->issue_from + stream->buffered < estimate->issue_at &&
+ estimate->max_segs > 0)
+ return;
+ }
+ }
+}
+
+/*
+ * End the issuing of writes, let the collector know we're done.
+ */
+static void netfs_writeback_end(struct netfs_io_request *wreq,
+ struct netfs_wb_params *params)
+{
+ bool needs_poke = true;
+
+ params->notes |= NOTE_FLUSH_ANYWAY;
+
+ for (int s = 0; s < NR_IO_STREAMS; s++) {
+ struct netfs_io_stream *stream = &wreq->io_streams[s];
+
+ if (stream->buffering) {
+ netfs_writeback_flush(wreq, stream, params);
+ stream->buffering = false;
+ }
+ }
+
+ netfs_all_subreqs_queued(wreq);
+
+ for (int s = 0; s < NR_IO_STREAMS; s++) {
+ struct netfs_io_stream *stream = &wreq->io_streams[s];
+
+ if (!stream->active)
+ continue;
+ if (!list_empty(&stream->subrequests))
+ needs_poke = false;
+ }
+
+ if (needs_poke)
+ netfs_wake_collector(wreq);
+}
+
+/*
+ * Add a single, physically contiguous segment of data to a writeback stream
+ * and dispatch subrequests when we hit a discontiguity or have accumulated
+ * sufficient data to hit the estimated dispatch point.
+ */
+static void netfs_writeback_add_seg_to_stream(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_wb_params *params,
+ uoff_t start, size_t len)
+{
+ struct netfs_write_estimate *estimate = ¶ms->estimates[stream->stream_nr];
+
+ _enter("%llx,%zx", start, len);
+
+ params->notes &= ~NOTE_FLUSH_ANYWAY;
+
+ /* Flush if not contiguous with the previous slice. */
+ if (stream->buffering && start != stream->last_end) {
+ params->notes |= NOTE_FLUSH_ANYWAY;
+ netfs_writeback_flush(wreq, stream, params);
+ params->notes &= ~NOTE_FLUSH_ANYWAY;
+ }
+
+ /* Begin the assembly of a slice and get an estimate of how much we can
+ * accumulate before we have to flush.
+ */
+ if (!stream->buffering) {
+ stream->issue_from = start;
+ bvecq_pos_set(&stream->dispatch_cursor, &wreq->load_cursor);
+ stream->buffering = true;
+ stream->buffered = 0;
+ estimate->issue_at = ULLONG_MAX;
+ estimate->max_segs = INT_MAX;
+ stream->estimate_write(wreq, stream, estimate);
+ }
+
+ stream->buffered += len;
+ stream->last_end = start + len;
+ estimate->max_segs--;
+
+ _debug("[%u] %llx + %zx >= %llx, %u %x",
+ stream->stream_nr, stream->issue_from, stream->buffered,
+ estimate->issue_at, estimate->max_segs, params->notes);
+
+ if (stream->issue_from + stream->buffered >= estimate->issue_at ||
+ estimate->max_segs <= 0)
+ netfs_writeback_flush(wreq, stream, params);
+}
+
+/*
+ * Add a folio directly to the writeback streams and dispatch subrequests as
+ * needed.
+ */
+static void netfs_writeback_add_folio_to_stream(struct netfs_io_request *wreq,
+ struct netfs_wb_params *params,
+ struct folio *folio)
+{
+ size_t fsize = folio_size(folio);
+ uoff_t fpos = params->fpos;
+
+ /* Attach the folio to the rolling buffer. */
+ bvecq_append_page(&wreq->load_cursor, &folio->page, 0, fsize, wreq->gfp, true);
+ wreq->load_cursor.slot--;
+
+ trace_netfs_bv_slot(wreq->load_cursor.bvecq, wreq->load_cursor.slot - 1);
+
+ for (int s = 0; s < NR_IO_STREAMS; s++) {
+ struct netfs_io_stream *stream = &wreq->io_streams[s];
+ size_t off, end;
+
+ if (!stream->active || !(params->notes & stream->applicable))
+ continue;
+
+ /* Select the appropriately sized chunk. */
+ if (stream->source == NETFS_WRITE_TO_CACHE) {
+ off = params->outer_off;
+ end = params->outer_end;
+ } else {
+ off = params->inner_off;
+ end = params->inner_end;
+ }
+
+ if (end < fsize)
+ stream->post_gap = fsize - end;
+
+ wreq->load_cursor.offset = off;
+ netfs_writeback_add_seg_to_stream(wreq, stream, params, fpos + off, end - off);
+ }
+
+
+ /* Advance the load cursor after copying to the dispatch cursor. */
+ wreq->load_cursor.slot++;
+ wreq->load_cursor.offset = 0;
+}
+
+/*
+ * Queue a folio for writeback.
+ */
+static void netfs_writeback_folio(struct netfs_io_request *wreq,
+ struct writeback_control *wbc,
+ struct folio *folio,
+ struct netfs_wb_params *params)
+{
+ struct netfs_writeback *wback;
struct netfs_group *fgroup; /* TODO: Use this with ceph */
struct netfs_folio *finfo;
- struct bvecq *queue = wreq->load_cursor.bvecq;
- unsigned int slot;
- size_t fsize = folio_size(folio), flen = fsize, foff = 0;
+ size_t fsize = folio_size(folio), fend = fsize, foff = 0;
uoff_t fpos = folio_pos(folio), i_size;
- bool to_eof = false, streamw = false;
- bool debug = false;
- _enter("");
-
- if (!wreq->spare) {
- wreq->spare = bvecq_alloc_one(BVECQ_STD_SLOTS, wreq->gfp, true);
- if (!wreq->spare)
- return -ENOMEM;
- }
+ _enter("%x", params->notes);
/* netfs_perform_write() may shift i_size around the folio or from out
* of the folio to beyond it, but cannot move i_size into or through
@@ -363,15 +706,16 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
*/
i_size = i_size_read(wreq->inode);
+ params->fpos = fpos;
if (fpos >= i_size) {
/* mmap beyond eof. */
_debug("beyond eof");
folio_start_writeback(folio);
folio_unlock(folio);
- wreq->nr_group_rel += netfs_folio_written_back(folio);
+ netfs_folio_written_back(folio, wreq);
netfs_put_group_many(wreq->group, wreq->nr_group_rel);
wreq->nr_group_rel = 0;
- return 0;
+ return;
}
if (fpos + fsize > wreq->i_size)
@@ -381,21 +725,23 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
finfo = netfs_folio_info(folio);
if (finfo) {
foff = finfo->dirty_offset;
- flen = foff + finfo->dirty_len;
- streamw = true;
+ fend = foff + finfo->dirty_len;
+ params->notes |= NOTE_STREAMW;
}
- if (flen > i_size - fpos) {
- flen = i_size - fpos;
- if (!streamw)
- folio_zero_segment(folio, flen, fsize);
- to_eof = true;
- } else if (flen == i_size - fpos) {
- to_eof = true;
+ if (fend > i_size - fpos) {
+ fend = i_size - fpos;
+ if (!(params->notes & NOTE_STREAMW))
+ folio_zero_segment(folio, fend, fsize);
}
- flen -= foff;
- _debug("folio %zx %zx %zx", foff, flen, fsize);
+ /* Account for cache and crypto alignments. */
+ params->inner_off = round_down(foff, params->inner_align);
+ params->inner_end = round_up (fend, params->inner_align);
+ params->outer_off = round_down(foff, params->outer_align);
+ params->outer_end = round_up (fend, params->outer_align);
+
+ _debug("folio %zx %zx %zx", foff, fend - foff, fsize);
/* Deal with discontinuities in the stream of dirty pages. These can
* arise from a number of sources:
@@ -414,149 +760,93 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
* write-back group.
*/
if (fgroup == NETFS_FOLIO_COPY_TO_CACHE) {
- netfs_issue_write(wreq, upload);
+ if (!(params->notes & NOTE_CACHE_AVAIL)) {
+ trace_netfs_folio(folio, netfs_folio_trace_cancel_copy);
+ goto cancel_folio;
+ }
+ params->notes |= NOTE_CACHE_COPY;
+ trace_netfs_folio(folio, netfs_folio_trace_store_copy);
} else if (fgroup != wreq->group) {
/* We can't write this page to the server yet. */
kdebug("wrong group");
- folio_redirty_for_writepage(wbc, folio);
- folio_unlock(folio);
- netfs_issue_write(wreq, upload);
- netfs_issue_write(wreq, cache);
- return 0;
+ goto skip_folio;
+ } else if (!(params->notes & (NOTE_UPLOAD_AVAIL | NOTE_CACHE_AVAIL))) {
+ trace_netfs_folio(folio, netfs_folio_trace_cancel_store);
+ goto cancel_folio_discard;
+ } else {
+ if (params->notes & NOTE_UPLOAD_STARTED) {
+ params->notes |= NOTE_UPLOAD;
+ trace_netfs_folio(folio, netfs_folio_trace_store_plus);
+ } else {
+ params->notes |= NOTE_UPLOAD | NOTE_UPLOAD_STARTED;
+ trace_netfs_folio(folio, netfs_folio_trace_store);
+ }
+ if ((params->notes & NOTE_CACHE_AVAIL) &&
+ !(params->notes & NOTE_STREAMW))
+ params->notes |= NOTE_CACHE_COPY;
}
- if (foff > 0)
- netfs_issue_write(wreq, upload);
- if (streamw)
- netfs_issue_write(wreq, cache);
-
folio_start_writeback(folio);
folio_unlock(folio);
- if (fgroup == NETFS_FOLIO_COPY_TO_CACHE) {
- if (!cache->avail) {
- trace_netfs_folio(folio, netfs_folio_trace_cancel_copy);
- netfs_issue_write(wreq, upload);
- netfs_folio_written_back(folio);
- return 0;
- }
- trace_netfs_folio(folio, netfs_folio_trace_store_copy);
- } else if (!upload->avail && !cache->avail) {
- trace_netfs_folio(folio, netfs_folio_trace_cancel_store);
- netfs_folio_written_back(folio);
- return 0;
- } else if (!upload->construct) {
- trace_netfs_folio(folio, netfs_folio_trace_store);
+ /* Keep track of what we will need to unlock. */
+ wback = wreq->writebacks_tail;
+ if (!wback || fpos != wback->start + wback->len || wback->len > LONG_MAX) {
+ wback = mempool_alloc(&netfs_writeback_pool, wreq->gfp);
+ wback->next = NULL;
+ wback->start = fpos;
+ wback->len = fsize;
+
+ if (wreq->writebacks)
+ /* Order write of next after last write of len in old tail. */
+ smp_store_release(&wreq->writebacks_tail->next, wback);
+ else
+ wreq->writebacks = wback;
+ wreq->writebacks_tail = wback;
} else {
- trace_netfs_folio(folio, netfs_folio_trace_store_plus);
- }
-
- /* Institute a new bvec queue segment if the current one is full or if
- * we encounter a discontiguity. The discontiguity break is important
- * when it comes to bulk unlocking folios by file range.
- */
- if (bvecq_is_full(queue) ||
- (fpos != wreq->last_end && wreq->last_end > 0)) {
- bvecq_buffer_append(&wreq->load_cursor, wreq->spare);
- wreq->spare = NULL;
-
- queue = wreq->load_cursor.bvecq;
- bvecq_pos_move(&wreq->dispatch_cursor, queue);
- wreq->dispatch_cursor.slot = 0;
+ /* Order update of len after setting pointer. */
+ smp_store_release(&wback->len, wback->len + fsize);
}
- /* Attach the folio to the rolling buffer. */
- slot = queue->nr_slots;
- bvec_set_folio(&queue->bv[slot], folio, fsize, 0);
- trace_netfs_bv_slot(queue, slot);
- slot++;
- bvecq_filled_to(queue, slot);
- wreq->load_cursor.slot = slot;
- wreq->load_cursor.offset = 0;
- wreq->last_end = fpos + fsize;
-
- /* Move the submission point forward to allow for write-streaming data
- * not starting at the front of the page. We don't do write-streaming
- * with the cache as the cache requires DIO alignment.
- *
- * Also skip uploading for data that's been read and just needs copying
- * to the cache.
- */
- bvecq_pos_nudge(&wreq->dispatch_cursor);
-
+ /* Flush any streams not being used for this folio. */
for (int s = 0; s < NR_IO_STREAMS; s++) {
- size_t soff = foff, slen = flen, alignment = 1;
-
- if (stream->source == NETFS_WRITE_TO_CACHE)
- alignment = wreq->cache_resources.dio_size;
- stream = &wreq->io_streams[s];
- stream->submit_off = round_down(soff, alignment);
- slen += foff - stream->submit_off;
- stream->submit_len = round_up(slen, alignment);
-
- if (!stream->avail ||
- (stream->source == NETFS_WRITE_TO_CACHE && streamw) ||
- (stream->source == NETFS_UPLOAD_TO_SERVER &&
- fgroup == NETFS_FOLIO_COPY_TO_CACHE)) {
- stream->submit_off = UINT_MAX;
- stream->submit_len = 0;
- }
- }
+ struct netfs_io_stream *stream = &wreq->io_streams[s];
- /* Attach the folio to one or more subrequests. For a big folio, we
- * could end up with thousands of subrequests if the wsize is small -
- * but we might need to wait during the creation of subrequests for
- * network resources (eg. SMB credits).
- */
- for (;;) {
- ssize_t part;
- size_t lowest_off = ULONG_MAX;
- int choose_s = -1;
-
- /* Always add to the lowest-submitted stream first. */
- for (int s = 0; s < NR_IO_STREAMS; s++) {
- stream = &wreq->io_streams[s];
- if (stream->submit_len > 0 &&
- stream->submit_off < lowest_off) {
- lowest_off = stream->submit_off;
- choose_s = s;
+ if (!stream->active || !(params->notes & stream->applicable)) {
+ if (stream->buffering) {
+ params->notes |= NOTE_FLUSH_ANYWAY;
+ netfs_writeback_flush(wreq, stream, params);
}
+ atomic64_set_release(&stream->issued_to, fpos + params->outer_end);
}
-
- if (choose_s < 0)
- break;
- stream = &wreq->io_streams[choose_s];
-
- /* Advance the cursor. */
- wreq->dispatch_cursor.offset = stream->submit_off;
-
- atomic64_set(&wreq->issued_to, fpos + stream->submit_off);
- part = netfs_advance_write(wreq, stream, fpos + stream->submit_off,
- stream->submit_len, to_eof);
- stream->submit_off += part;
- if (part > stream->submit_len)
- stream->submit_len = 0;
- else
- stream->submit_len -= part;
- if (part > 0)
- debug = true;
}
- bvecq_pos_step(&wreq->dispatch_cursor);
- /* Order loading the queue before updating the issue_to point */
- atomic64_set_release(&wreq->issued_to, fpos + fsize);
-
- if (!debug)
- kdebug("R=%x: No submit", wreq->debug_id);
+ /* Initiate or extend the dispatch of each selected stream. At this
+ * point we may need to copy the data to a bounce buffer and push the
+ * bounce bits instead.
+ */
+ // TODO: Do bouncing if selected.
+ netfs_writeback_add_folio_to_stream(wreq, params, folio);
- if (foff + flen < fsize)
- for (int s = 0; s < NR_IO_STREAMS; s++)
- netfs_issue_write(wreq, &wreq->io_streams[s]);
+out:
+ _leave(" = %x", params->notes);
+ return;
- _leave(" = 0");
- return 0;
+skip_folio:
+ folio_redirty_for_writepage(wbc, folio);
+ folio_unlock(folio);
+ goto out;
+cancel_folio_discard:
+ netfs_put_group(fgroup);
+cancel_folio:
+ folio_detach_private(folio);
+ kfree(finfo);
+ folio_unlock(folio);
+ folio_cancel_dirty(folio);
+ goto out;
}
+#if 0 // TODO: Remove
/*
* End the issuing of writes, letting the collector know we're done.
*/
@@ -579,6 +869,7 @@ static void netfs_end_issue_write(struct netfs_io_request *wreq)
if (needs_poke)
netfs_wake_collector(wreq);
}
+#endif
/*
* Write some of the pending data back to the server
@@ -588,6 +879,7 @@ int netfs_writepages(struct address_space *mapping,
{
struct netfs_inode *ictx = netfs_inode(mapping->host);
struct netfs_io_request *wreq = NULL;
+ struct netfs_wb_params params = {};
struct folio *folio;
int error = 0;
@@ -605,46 +897,50 @@ int netfs_writepages(struct address_space *mapping,
goto couldnt_start;
}
- if (bvecq_buffer_init(&wreq->load_cursor, wreq->gfp, true) < 0)
- goto nomem;
- bvecq_pos_set(&wreq->dispatch_cursor, &wreq->load_cursor);
- bvecq_pos_set(&wreq->collect_cursor, &wreq->dispatch_cursor);
+ bvecq_buffer_init(&wreq->load_cursor, GFP_NOFS, true);
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags);
trace_netfs_write(wreq, netfs_write_trace_writeback);
netfs_stat(&netfs_n_wh_writepages);
- do {
- _debug("wbiter %lx %llx", folio->index, atomic64_read(&wreq->issued_to));
+ params.inner_align = 1;
+ params.outer_align = 1;
- /* It appears we don't have to handle cyclic writeback wrapping. */
- WARN_ON_ONCE(wreq && folio_pos(folio) < atomic64_read(&wreq->issued_to));
+ if (wreq->io_streams[1].avail) {
+ params.notes |= NOTE_CACHE_AVAIL;
+ params.outer_align = wreq->cache_resources.dio_size;
+ }
+ // TODO: Adjust alignments for crypto
+
+ do {
+ _debug("wbiter %lx", folio->index);
if (netfs_folio_group(folio) != NETFS_FOLIO_COPY_TO_CACHE &&
unlikely(!test_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags))) {
set_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags);
wreq->netfs_ops->begin_writeback(wreq);
+ if (wreq->io_streams[0].avail) {
+ params.notes |= NOTE_UPLOAD_AVAIL;
+ /* Order setting the active flag after other fields. */
+ smp_store_release(&wreq->io_streams[0].active, true);
+ }
}
- error = netfs_write_folio(wreq, wbc, folio);
- if (error == -ENOMEM) {
- folio_redirty_for_writepage(wbc, folio);
- folio_unlock(folio);
- }
+ params.notes &= NOTES__KEEP_MASK;
+ netfs_writeback_folio(wreq, wbc, folio, ¶ms);
} while ((folio = writeback_iter(mapping, wbc, folio, &error)));
- netfs_end_issue_write(wreq);
+ netfs_writeback_end(wreq, ¶ms);
+
bvecq_pos_unset(&wreq->load_cursor);
- bvecq_pos_unset(&wreq->dispatch_cursor);
+ for (int i = 0; i < NR_IO_STREAMS; i++)
+ bvecq_pos_unset(&wreq->io_streams[i].dispatch_cursor);
netfs_wake_collector(wreq);
netfs_put_request(wreq, netfs_rreq_trace_put_return);
_leave(" = %d", error);
return error;
-nomem:
- error = -ENOMEM;
- netfs_put_failed_request(wreq);
couldnt_start:
if (error == -ENOMEM) {
folio_redirty_for_writepage(wbc, folio);
@@ -738,7 +1034,6 @@ int netfs_writeback_single(struct address_space *mapping,
subreq->len = wreq->len;
if (stream->source == NETFS_WRITE_TO_CACHE)
subreq->len = clen;
- stream->submit_len = subreq->len;
netfs_issue_write(wreq, stream);
}
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 656f4f3f80b6..2c5b7f3a48b7 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -134,6 +134,16 @@ enum netfs_cache_collect {
NETFS_CACHE_COLLECT_WRITE_CANCEL, /* Currently collecting cancelled writes */
};
+/*
+ * Record of a contiguous region undergoing writeback. The tail region (ie. if
+ * next is NULL) may be extended dynamically.
+ */
+struct netfs_writeback {
+ struct netfs_writeback *next; /* Next extent in list */
+ uoff_t start; /* Start position */
+ size_t len; /* Total size (can increase) */
+};
+
/*
* Estimate of maximum write subrequest for writeback. The filesystem is
* responsible for filling this in when called from ->estimate_write(), though
@@ -150,23 +160,32 @@ struct netfs_write_estimate {
* have to write to multiple destinations concurrently.
*/
struct netfs_io_stream {
- /* Submission tracking */
+ /* Submission tracking (main dispatch only; not retry) */
+ struct bvecq_pos dispatch_cursor; /* Point from which buffers are dispatched */
struct netfs_io_subrequest *construct; /* Op being constructed */
uoff_t issue_from; /* Current issue point */
+ uoff_t last_end; /* End file pos of last folio added */
+ size_t buffered; /* Amount in buffer */
+ size_t post_gap; /* Length of partial folio tail */
size_t sreq_max_len; /* Maximum size of a subrequest */
unsigned int sreq_max_segs; /* 0 or max number of segments in an iterator */
unsigned int submit_off; /* Folio offset we're submitting from */
unsigned int submit_len; /* Amount of data left to submit */
+ unsigned int alignment; /* Required alignment */
+ u8 applicable; /* What sources are applicable (NOTE_* mask) */
+ bool buffering; /* T if buffering on this stream */
int (*estimate_write)(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
struct netfs_write_estimate *estimate);
void (*prepare_write)(struct netfs_io_subrequest *subreq);
void (*issue_write)(struct netfs_io_subrequest *subreq);
+ atomic64_t issued_to; /* Point to which can be considered issued */
+
/* Collection tracking */
struct list_head subrequests; /* Contributory I/O operations */
uoff_t collected_to; /* Position we've collected results to */
size_t transferred; /* The amount transferred from this stream */
- unsigned short error; /* Aggregate error for the stream */
+ short error; /* Aggregate error for the stream */
enum netfs_io_source source; /* Where to read from/write to */
unsigned char stream_nr; /* Index of stream in parent table */
bool avail; /* T if stream is available */
@@ -207,11 +226,12 @@ struct netfs_io_subrequest {
struct iov_iter io_iter; /* Iterator for this subrequest */
uoff_t start; /* Where to start the I/O */
size_t len; /* Size of the I/O */
+ size_t post_gap; /* Length of partial folio tail */
size_t transferred; /* Amount of data transferred */
refcount_t ref;
short error; /* 0 or error that occurred */
unsigned short debug_index; /* Index in list (for debugging output) */
- unsigned int nr_segs; /* Number of segs in io_iter */
+ unsigned int nr_segs; /* Number of segments in content */
u8 retry_count; /* The number of retries (0 on initial pass) */
enum netfs_io_source source; /* Where to read from/write to */
unsigned char stream_nr; /* I/O stream this belongs to */
@@ -263,6 +283,8 @@ struct netfs_io_request {
#endif
struct netfs_io_stream io_streams[2]; /* Streams of parallel I/O operations */
#define NR_IO_STREAMS 2 //wreq->nr_io_streams
+ struct netfs_writeback *writebacks; /* List of regions undergoing writeback */
+ struct netfs_writeback *writebacks_tail; /* Tail of region list */
struct netfs_group *group; /* Writeback group being written back */
struct bvecq *spare; /* Advance allocation of bvecq */
struct bvecq_pos load_cursor; /* Point at which new folios are loaded in */
@@ -279,7 +301,6 @@ struct netfs_io_request {
long error; /* 0 or error that occurred */
uoff_t i_size; /* Size of the file */
uoff_t start; /* Start position */
- atomic64_t issued_to; /* Write issuer folio cursor */
uoff_t collected_to; /* Point we've collected to */
uoff_t cache_coll_to; /* Point the cache has collected to */
uoff_t cleaned_to; /* Position we've cleaned folios to */
@@ -302,6 +323,7 @@ struct netfs_io_request {
#define NETFS_RREQ_RETRYING 4 /* Set if we're in the retry path */
#define NETFS_RREQ_SHORT_TRANSFER 5 /* Set if we have a short transfer */
#define NETFS_RREQ_ABANDON_REQ 6 /* Set if the request is to be abandoned */
+#define NETFS_RREQ_SAW_ENOMEM 7 /* Set if we encounted ENOMEM */
#define NETFS_RREQ_CACHE_STOP 8 /* Set to stop caching (ENOBUFS or error) */
#define NETFS_RREQ_CACHE_ERROR 9 /* Set if we got an error from the cache */
#define NETFS_RREQ_OFFLOAD_COLLECTION 12 /* Offload collection to workqueue */
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 312dc2214d92..6ccf9de42858 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -764,7 +764,7 @@ TRACE_EVENT(netfs_collect_stream,
__entry->wreq = wreq->debug_id;
__entry->stream = stream->stream_nr;
__entry->collected_to = stream->collected_to;
- __entry->issued_to = atomic64_read(&wreq->issued_to);
+ __entry->issued_to = atomic64_read(&stream->issued_to);
),
TP_printk("R=%08x[%x:] cto=%llx ito=%llx",
@@ -772,6 +772,36 @@ TRACE_EVENT(netfs_collect_stream,
__entry->collected_to, __entry->issued_to)
);
+TRACE_EVENT(netfs_collect_folios,
+ TP_PROTO(const struct netfs_io_request *wreq,
+ uoff_t range_start, size_t range_len),
+
+ TP_ARGS(wreq, range_start, range_len),
+
+ TP_STRUCT__entry(
+ __field(unsigned int, wreq)
+ __field(size_t, range_len)
+ __field(uoff_t, range_start)
+ __field(uoff_t, cleaned_to)
+ __field(uoff_t, collected_to)
+ ),
+
+ TP_fast_assign(
+ __entry->wreq = wreq->debug_id;
+ __entry->range_len = range_len;
+ __entry->range_start = range_start;
+ __entry->cleaned_to = wreq->cleaned_to;
+ __entry->collected_to = wreq->collected_to;
+ ),
+
+ TP_printk("R=%08x r=%llx-%llx cln=%llx col=%llx",
+ __entry->wreq,
+ __entry->range_start,
+ __entry->range_start + __entry->range_len,
+ __entry->cleaned_to,
+ __entry->collected_to)
+ );
+
TRACE_EVENT(netfs_bvecq,
TP_PROTO(const struct bvecq *bq,
enum netfs_bvecq_trace trace),
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 34/35] netfs: Combine prepare and issue ops and grab the buffers on request
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (32 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 33/35] netfs: Rework writeback to use a separate list of regions to be unlocked David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 14:41 ` [PATCH v10 35/35] netfs: Clean up now-unused code David Howells
2026-08-24 15:01 ` [PATCH v10 36/35] cachefiles: Preset the state xattr when creating a new file David Howells
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Modify the way subrequests are generated in netfslib to try and simplify
the code. The issue, primarily, is in writeback: the code has to create
multiple streams of write requests to disparate targets with different
properties (e.g. server and fscache), where not every folio needs to go to
every target (e.g. data just read from the server may only need writing to
the cache).
The current model in writeback, at least, is to go carefully through every
folio, preparing a subrequest for each stream when it was detected that
part of the current folio needed to go to that stream, and repeating this
within and across contiguous folios; then to issue subrequests as they
become full or hit boundaries after first setting up the buffer. However,
this is quite difficult to follow - and makes it tricky to handle
discontiguous folios in a request.
This is changed such that netfs now accumulates buffers and attaches them
to each stream when they become valid for that stream, then flushes the
stream when a limit or a boundary is hit. The issuing code in netfs then
loops around creating and issuing subrequests without calling a separate
prepare stage (though a function is provided to get an estimate of when
flushing should occur). The filesystem (or cache) then gets to take a
slice of the master bvec chain as its I/O buffer for each subrequest,
including discontiguities if it can support a sparse/vectored RPC (as Ceph
can).
Similar-ish changes also apply to buffered read and unbuffered read and
write, though in each of those cases there is only a single contiguous
stream. Though for buffered read this consists of interwoven requests from
multiple sources (server or cache).
To this end, netfslib is changed in the following ways:
(1) ->prepare_xxx(), buffer selection and ->issue_xxx() are now collapsed
together such that one ->issue_xxx() call is made with the subrequest
defined to the maximum extent; the filesystem/cache then reduces the
length of the subrequest and calls back to netfslib to grab a slice of
the buffer, which may reduce the subrequest further if a maximum
segment limit is set. The filesystem/cache then dispatches the
operation.
(2) Retry buffer tracking is added to the netfs_io_request struct. This
is then selected by the subrequest retry counter being non-zero.
(3) The use of iov_iter is pushed down to the filesystem. Netfslib now
provides the filesystem with a bvecq holding the buffer rather than an
iov_iter. The bvecq can be duplicated and headers/trailers attached
to hold protocol and several duplicated bvecqs can be linked together
to create a compound operation.
(4) If the ->issue_xxx() functions terminate with -ENOMEM, a flag is set
on the request to abort further subrequest generation/retrying. If
it fails early with any other error, generally the subreq will be
extended as much as possible and failed with the error returned.
(5) During writeback, netfslib now builds up an accumulation of buffered
data before issuing writes on each stream (one server, one cache). It
asks each stream for an estimate of how much data to accumulate before
it next generates subrequests on the stream. The filesystem or cache
is not required to use up all the data accumulated on a stream at that
time unless the end of the pagecache is hit.
(6) During read-gaps, in which there are two gaps on either end of a dirty
streaming write page that need to be filled, a buffer is constructed
consisting of the two ends plus a sink page repeated to cover the
middle portion. This is passed to the server as a single write. For
something like Ceph, this should probably be done either as a
vectored/sparse read or as two separate reads (if different Ceph
objects are involved).
(7) During unbuffered/DIO read/write, there is a single contiguous file
region to be read or written as a single stream. The dispatching
function just creates subrequests and calls ->issue_xxx() repeatedly
to eat through the bufferage.
(8) At the start of buffered read, the entire set of folios allocated by
VM readahead is loaded into a bvecq chain, rather than trying to do it
piecemeal as-needed. As the pages were already added and locked by
the VM, this is slightly more efficient than loading piecemeal as only
a single iteration of the xarray is required.
(9) During buffered read, there is a single contiguous file region, to
read as a single stream - however, this stream may be stitched
together from subrequests to multiple sources. Which sources are used
where is now determined by querying the cache to find the next couple
of extents in which it has data; netfslib uses this to direct the
subrequests towards the appropriate sources.
Each subrequest is given the maximum length in the current extent and
then ->issue_read() is called. The filesystem then limits the size
and slices off a piece of the buffer for that extent.
(10) Cachefiles now provides an estimation function that indicates the
standard maxima for doing DIO (MAX_RW_COUNT and BIO_MAX_VECS).
Note that sparse cachefiles still rely on the backing filesystem for
content mapping. That will need to be addressed in a future patch and is
not trivial to fix.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/9p/vfs_addr.c | 43 ++++--
fs/afs/file.c | 27 ++--
fs/afs/fsclient.c | 8 +-
fs/afs/internal.h | 5 +-
fs/afs/write.c | 33 ++---
fs/afs/yfsclient.c | 6 +-
fs/cachefiles/io.c | 230 ++++++++++++++++++++++---------
fs/ceph/addr.c | 123 ++++++++++-------
fs/netfs/buffered_read.c | 257 +++++++++++++++++++----------------
fs/netfs/direct_read.c | 84 +++++++-----
fs/netfs/direct_write.c | 141 ++++++++++---------
fs/netfs/fscache_io.c | 2 +-
fs/netfs/internal.h | 43 ++++--
fs/netfs/main.c | 1 +
fs/netfs/misc.c | 33 ++++-
fs/netfs/objects.c | 4 +-
fs/netfs/read_collect.c | 38 ++----
fs/netfs/read_pgpriv2.c | 125 +++++++++++------
fs/netfs/read_retry.c | 208 +++++++++++++---------------
fs/netfs/read_single.c | 155 +++++++++++++--------
fs/netfs/write_issue.c | 157 +++++++++++++--------
fs/netfs/write_retry.c | 168 ++++++++++++-----------
fs/nfs/fscache.c | 24 +++-
fs/smb/client/cifssmb.c | 13 +-
fs/smb/client/file.c | 134 ++++++++----------
fs/smb/client/smb2ops.c | 9 +-
fs/smb/client/smb2pdu.c | 28 ++--
fs/smb/client/transport.c | 15 +-
include/linux/netfs.h | 54 +++-----
include/trace/events/netfs.h | 30 +++-
net/9p/client.c | 8 +-
31 files changed, 1301 insertions(+), 905 deletions(-)
diff --git a/fs/9p/vfs_addr.c b/fs/9p/vfs_addr.c
index 9b163a13ec3f..fc5b9c677986 100644
--- a/fs/9p/vfs_addr.c
+++ b/fs/9p/vfs_addr.c
@@ -67,28 +67,51 @@ static int v9fs_estimate_write(struct netfs_io_request *wreq,
/*
* Issue a subrequest to write to the server.
*/
-static void v9fs_issue_write(struct netfs_io_subrequest *subreq)
+static int v9fs_issue_write(struct netfs_io_subrequest *subreq)
{
+ struct iov_iter iter;
struct p9_fid *fid = subreq->rreq->netfs_priv;
- int err, len;
+ int err, len = 0;
- len = p9_client_write(fid, subreq->start, &subreq->io_iter, &err);
+ subreq->len = umin(subreq->len, fid->clnt->msize - P9_IOHDRSZ);
+
+ err = netfs_prepare_write_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
+ len = p9_client_write(fid, subreq->start, &iter, &err);
if (len > 0)
__set_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+
netfs_write_subrequest_terminated(subreq, len ?: err);
+ return 0;
}
/**
* v9fs_issue_read - Issue a read from 9P
* @subreq: The read to make
*/
-static void v9fs_issue_read(struct netfs_io_subrequest *subreq)
+static int v9fs_issue_read(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
+ struct iov_iter iter;
struct p9_fid *fid = rreq->netfs_priv;
char *target;
unsigned long long pos = subreq->start + subreq->transferred;
- int total = 0, err, len, n;
+ size_t len;
+ int total = 0, err;
+
+ err = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
if (S_ISLNK(rreq->inode->i_mode)) {
/* p9_client_readlink() must not be called for legacy protocols
@@ -105,12 +128,13 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)
err = p9_client_readlink(fid, &target);
if (err != 0)
goto fill_subreq;
- len = strlen(target);
- n = copy_to_iter(target, len, &subreq->io_iter);
+ len = min(strlen(target), subreq->len);
+ total = copy_to_iter(target, len, &iter);
kfree(target);
- total = n;
} else {
- total = p9_client_read(fid, pos, &subreq->io_iter, &err);
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+
+ total = p9_client_read(fid, pos, &iter, &err);
}
fill_subreq:
@@ -128,6 +152,7 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)
subreq->error = err;
netfs_read_subreq_terminated(subreq);
+ return 0;
}
/**
diff --git a/fs/afs/file.c b/fs/afs/file.c
index 4a412e0472ec..a6b61d609f02 100644
--- a/fs/afs/file.c
+++ b/fs/afs/file.c
@@ -332,11 +332,12 @@ void afs_fetch_data_immediate_cancel(struct afs_call *call)
/*
* Fetch file data from the volume.
*/
-static void afs_issue_read(struct netfs_io_subrequest *subreq)
+static int afs_issue_read(struct netfs_io_subrequest *subreq)
{
struct afs_operation *op;
struct afs_vnode *vnode = AFS_FS_I(subreq->rreq->inode);
struct key *key = subreq->rreq->netfs_priv;
+ int ret;
_enter("%s{%llx:%llu.%u},%x,,,",
vnode->volume->name,
@@ -345,11 +346,15 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq)
vnode->fid.unique,
key_serial(key));
+ ret = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
+
op = afs_alloc_operation(key, vnode->volume);
if (IS_ERR(op)) {
- subreq->error = PTR_ERR(op);
- netfs_read_subreq_terminated(subreq);
- return;
+ ret = PTR_ERR(op);
+ goto failed;
}
afs_op_set_vnode(op, 0, vnode);
@@ -364,20 +369,24 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq)
op->flags |= AFS_OPERATION_ASYNC;
if (!afs_begin_vnode_operation(op)) {
- subreq->error = afs_put_operation(op);
- netfs_read_subreq_terminated(subreq);
- return;
+ ret = afs_put_operation(op);
+ goto failed;
}
if (!afs_select_fileserver(op)) {
- afs_end_read(op);
- return;
+ afs_end_read(op); /* Error recorded here. */
+ return 0;
}
afs_issue_read_call(op);
} else {
afs_do_sync_operation(op);
}
+ return 0;
+failed:
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ return 0;
}
static int afs_init_request(struct netfs_io_request *rreq, struct file *file)
diff --git a/fs/afs/fsclient.c b/fs/afs/fsclient.c
index 1a3f186a6a11..9713551dd842 100644
--- a/fs/afs/fsclient.c
+++ b/fs/afs/fsclient.c
@@ -339,7 +339,9 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call)
if (call->remaining == 0)
goto no_more_data;
- call->iter = &subreq->io_iter;
+ iov_iter_bvec_queue(&call->def_iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
call->iov_len = umin(call->remaining, subreq->len - subreq->transferred);
call->unmarshall++;
fallthrough;
@@ -1088,7 +1090,7 @@ static void afs_fs_store_data64(struct afs_operation *op)
if (!call)
return afs_op_nomem(op);
- call->write_iter = op->store.write_iter;
+ call->write_iter = &op->store.write_iter;
/* marshall the parameters */
bp = call->request;
@@ -1142,7 +1144,7 @@ void afs_fs_store_data(struct afs_operation *op)
if (!call)
return afs_op_nomem(op);
- call->write_iter = op->store.write_iter;
+ call->write_iter = &op->store.write_iter;
/* marshall the parameters */
bp = call->request;
diff --git a/fs/afs/internal.h b/fs/afs/internal.h
index ae2bc699bfde..5c536cc7b53c 100644
--- a/fs/afs/internal.h
+++ b/fs/afs/internal.h
@@ -914,7 +914,7 @@ struct afs_operation {
afs_lock_type_t type;
} lock;
struct {
- struct iov_iter *write_iter;
+ struct iov_iter write_iter;
loff_t pos;
loff_t size;
loff_t i_size;
@@ -1701,8 +1701,7 @@ extern int afs_check_volume_status(struct afs_volume *, struct afs_operation *);
int afs_estimate_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
struct netfs_write_estimate *estimate);
-void afs_prepare_write(struct netfs_io_subrequest *subreq);
-void afs_issue_write(struct netfs_io_subrequest *subreq);
+int afs_issue_write(struct netfs_io_subrequest *subreq);
void afs_begin_writeback(struct netfs_io_request *wreq);
void afs_retry_request(struct netfs_io_request *wreq, struct netfs_io_stream *stream);
extern int afs_writepages(struct address_space *, struct writeback_control *);
diff --git a/fs/afs/write.c b/fs/afs/write.c
index a0c9916f594b..46f81d9e7d7f 100644
--- a/fs/afs/write.c
+++ b/fs/afs/write.c
@@ -99,20 +99,6 @@ int afs_estimate_write(struct netfs_io_request *wreq,
return 0;
}
-/*
- * Prepare a subrequest to write to the server. This sets the max_len
- * parameter.
- */
-void afs_prepare_write(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_stream *stream = &subreq->rreq->io_streams[subreq->stream_nr];
-
- //if (test_bit(NETFS_SREQ_RETRYING, &subreq->flags))
- // subreq->max_len = 512 * 1024;
- //else
- stream->sreq_max_len = 256 * 1024 * 1024;
-}
-
/*
* Issue a subrequest to write to the server.
*/
@@ -156,12 +142,15 @@ static void afs_issue_write_worker(struct work_struct *work)
op->flags |= AFS_OPERATION_UNINTR;
op->ops = &afs_store_data_operation;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
afs_begin_vnode_operation(op);
- op->store.write_iter = &subreq->io_iter;
op->store.i_size = umax(pos + len, netfs_read_remote_i_size(&vnode->netfs.inode));
op->mtime = inode_get_mtime(&vnode->netfs.inode);
+ iov_iter_bvec_queue(&op->store.write_iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
afs_wait_for_operation(op);
ret = afs_put_operation(op);
switch (ret) {
@@ -185,11 +174,21 @@ static void afs_issue_write_worker(struct work_struct *work)
netfs_write_subrequest_terminated(subreq, ret < 0 ? ret : subreq->len);
}
-void afs_issue_write(struct netfs_io_subrequest *subreq)
+int afs_issue_write(struct netfs_io_subrequest *subreq)
{
+ int ret;
+
+ if (subreq->len > 256 * 1024 * 1024)
+ subreq->len = 256 * 1024 * 1024;
+ ret = netfs_prepare_write_buffer(subreq, INT_MAX);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
+
subreq->work.func = afs_issue_write_worker;
if (!queue_work(system_dfl_wq, &subreq->work))
WARN_ON_ONCE(1);
+ return 0;
}
/*
@@ -200,6 +199,8 @@ void afs_begin_writeback(struct netfs_io_request *wreq)
{
if (S_ISREG(wreq->inode->i_mode))
afs_get_writeback_key(wreq);
+
+ wreq->io_streams[0].avail = true;
}
/*
diff --git a/fs/afs/yfsclient.c b/fs/afs/yfsclient.c
index d941179730a9..52c588092050 100644
--- a/fs/afs/yfsclient.c
+++ b/fs/afs/yfsclient.c
@@ -385,7 +385,9 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call)
if (call->remaining == 0)
goto no_more_data;
- call->iter = &subreq->io_iter;
+ iov_iter_bvec_queue(&call->def_iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
call->iov_len = min(call->remaining, subreq->len - subreq->transferred);
call->unmarshall++;
fallthrough;
@@ -1357,7 +1359,7 @@ void yfs_fs_store_data(struct afs_operation *op)
if (!call)
return afs_op_nomem(op);
- call->write_iter = op->store.write_iter;
+ call->write_iter = &op->store.write_iter;
/* marshall the parameters */
bp = call->request;
diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c
index 91b69be78c6b..a1580385d5fb 100644
--- a/fs/cachefiles/io.c
+++ b/fs/cachefiles/io.c
@@ -197,12 +197,131 @@ static int cachefiles_read(struct netfs_cache_resources *cres,
return ret;
}
+/*
+ * Handle completion of a read from the cache issued by netfslib.
+ */
+static void cachefiles_issue_read_complete(struct kiocb *iocb, long ret)
+{
+ struct cachefiles_kiocb *ki = container_of(iocb, struct cachefiles_kiocb, iocb);
+ struct netfs_io_subrequest *subreq = ki->subreq;
+ struct inode *inode = file_inode(ki->iocb.ki_filp);
+
+ _enter("%ld", ret);
+
+ if (ret < 0) {
+ subreq->error = -ESTALE;
+ trace_cachefiles_io_error(ki->object, inode, ret,
+ cachefiles_trace_read_error);
+ }
+
+ if (ret >= 0) {
+ if (ki->object->cookie->inval_counter == ki->inval_counter) {
+ subreq->error = 0;
+ if (ret > 0) {
+ subreq->transferred += ret;
+ __set_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ }
+ } else {
+ subreq->error = -ESTALE;
+ }
+ }
+
+ netfs_read_subreq_terminated(subreq);
+ cachefiles_put_kiocb(ki);
+}
+
+/*
+ * Issue a read operation to the cache. We assume that the cache is readable
+ * as the caller should have called ->query_occupancy() first.
+ */
+static int cachefiles_issue_read(struct netfs_io_subrequest *subreq)
+{
+ struct netfs_cache_resources *cres = &subreq->rreq->cache_resources;
+ struct cachefiles_object *object = cachefiles_cres_object(cres);
+ struct cachefiles_kiocb *ki;
+ struct iov_iter iter;
+ struct file *file = cachefiles_cres_file(cres);
+ unsigned int old_nofs;
+ ssize_t ret;
+
+ fscache_count_read();
+
+ _enter("%pD,%lli,%llx,%zx/%llx",
+ file, file_inode(file)->i_ino, subreq->start, subreq->len,
+ i_size_read(file_inode(file)));
+
+ if (subreq->len > MAX_RW_COUNT)
+ subreq->len = MAX_RW_COUNT;
+
+ ret = netfs_prepare_read_buffer(subreq, BIO_MAX_VECS);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
+ ret = -ENOMEM;
+ ki = kzalloc_obj(struct cachefiles_kiocb);
+ if (!ki)
+ goto failed;
+
+ refcount_set(&ki->ki_refcnt, 2);
+ ki->iocb.ki_filp = file;
+ ki->iocb.ki_pos = subreq->start;
+ ki->iocb.ki_flags = IOCB_DIRECT;
+ ki->iocb.ki_ioprio = get_current_ioprio();
+ ki->iocb.ki_complete = cachefiles_issue_read_complete;
+ ki->object = object;
+ ki->inval_counter = cres->inval_counter;
+ ki->subreq = subreq;
+ ki->was_async = true;
+
+ get_file(ki->iocb.ki_filp);
+ cachefiles_grab_object(object, cachefiles_obj_get_ioreq);
+
+ trace_cachefiles_read(object, file_inode(file), ki->iocb.ki_pos, subreq->len);
+ old_nofs = memalloc_nofs_save();
+ ret = cachefiles_inject_read_error();
+ if (ret == 0)
+ ret = vfs_iocb_iter_read(file, &ki->iocb, &iter);
+ memalloc_nofs_restore(old_nofs);
+
+ switch (ret) {
+ case -EIOCBQUEUED:
+ break;
+
+ case -ERESTARTSYS:
+ case -ERESTARTNOINTR:
+ case -ERESTARTNOHAND:
+ case -ERESTART_RESTARTBLOCK:
+ /* There's no easy way to restart the syscall since other AIO's
+ * may be already running. Just fail this IO with EINTR.
+ */
+ ret = -EINTR;
+ fallthrough;
+ default:
+ ki->was_async = false;
+ cachefiles_issue_read_complete(&ki->iocb, ret);
+ break;
+ }
+
+ cachefiles_put_kiocb(ki);
+ _leave(" = %zd", ret);
+ return 0;
+failed:
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ return 0;
+}
+
/*
* Query the occupancy of the cache in a region, returning the extent of the
- * next two chunks of cached data and the next hole.
+ * next two chunks of cached data and the next hole. The occupancy map is
+ * preloaded to show just one giant hole.
*/
-static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
- struct fscache_occupancy *occ)
+static void cachefiles_query_occupancy(struct netfs_cache_resources *cres,
+ struct fscache_occupancy *occ)
{
struct cachefiles_object *object;
struct inode *inode;
@@ -212,7 +331,7 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
int i;
if (!fscache_wait_for_operation(cres, FSCACHE_WANT_READ))
- return -ENOBUFS;
+ return;
object = cachefiles_cres_object(cres);
file = cachefiles_cres_file(cres);
@@ -247,7 +366,7 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
ret = vfs_llseek(file, occ->query_from, SEEK_DATA);
if (IS_ERR_VALUE_LL(ret)) {
if (ret != -ENXIO)
- return ret;
+ goto done;
occ->query_from = ULLONG_MAX;
goto done;
}
@@ -260,7 +379,7 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
ret = vfs_llseek(file, occ->query_from, SEEK_HOLE);
if (IS_ERR_VALUE_LL(ret)) {
if (ret != -ENXIO)
- return ret;
+ goto done;
occ->query_from = ULLONG_MAX;
goto done;
}
@@ -273,7 +392,6 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
done:
_debug("query[0] %llx-%llx", occ->cached_from[0], occ->cached_to[0]);
_debug("query[1] %llx-%llx", occ->cached_from[1], occ->cached_to[1]);
- return 0;
}
/*
@@ -590,9 +708,9 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
cachefiles_has_space_for_write);
}
-static int cachefiles_prepare_write(struct netfs_cache_resources *cres,
- uoff_t *_start, size_t *_len, size_t upper_len,
- uoff_t i_size, bool no_space_allocated_yet)
+static int cachefiles_prepare_write_old(struct netfs_cache_resources *cres,
+ uoff_t *_start, size_t *_len, size_t upper_len,
+ uoff_t i_size, bool no_space_allocated_yet)
{
struct cachefiles_object *object = cachefiles_cres_object(cres);
struct cachefiles_cache *cache = object->volume->cache;
@@ -623,80 +741,63 @@ static int cachefiles_estimate_write(struct netfs_io_request *wreq,
return 0;
}
-static void cachefiles_prepare_write_subreq(struct netfs_io_subrequest *subreq)
+static int cachefiles_issue_write(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *wreq = subreq->rreq;
struct netfs_cache_resources *cres = &wreq->cache_resources;
- struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
-
- _enter("W=%x[%x] %llx", wreq->debug_id, subreq->debug_index, subreq->start);
+ struct cachefiles_object *object = cachefiles_cres_object(cres);
+ struct cachefiles_cache *cache = object->volume->cache;
+ struct iov_iter iter;
+ const struct cred *saved_cred;
+ uoff_t start = subreq->start;
+ size_t len = subreq->len;
+ int ret = -EINVAL;
- stream->sreq_max_len = MAX_RW_COUNT;
- stream->sreq_max_segs = BIO_MAX_VECS;
+ _enter("W=%x[%x] %llx-%llx",
+ wreq->debug_id, subreq->debug_index, start, start + len - 1);
if (!cachefiles_cres_file(cres)) {
if (!fscache_wait_for_operation(cres, FSCACHE_WANT_WRITE))
- return netfs_prepare_write_failed(subreq);
+ return -ENOBUFS;
if (!cachefiles_cres_file(cres))
- return netfs_prepare_write_failed(subreq);
+ return -ENOBUFS;
}
-}
-static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_request *wreq = subreq->rreq;
- struct netfs_cache_resources *cres = &wreq->cache_resources;
- struct cachefiles_object *object = cachefiles_cres_object(cres);
- struct cachefiles_cache *cache = object->volume->cache;
- const struct cred *saved_cred;
- size_t off, pre, post, len = subreq->len;
- uoff_t start = subreq->start;
- int ret;
+ ret = netfs_prepare_write_buffer(subreq, BIO_MAX_VECS);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
- _enter("W=%x[%x] %llx-%llx",
- wreq->debug_id, subreq->debug_index, start, start + len - 1);
+ /* The buffer extraction func may round out start and end. */
+ start = subreq->start;
+ len = subreq->len;
- /* We need to start on the cache granularity boundary */
- off = start & (cache->bsize - 1);
- if (off) {
- pre = cache->bsize - off;
- if (pre >= len) {
- fscache_count_dio_misfit();
- netfs_write_subrequest_terminated(subreq, len);
- return;
- }
- subreq->transferred += pre;
- start += pre;
- len -= pre;
- iov_iter_advance(&subreq->io_iter, pre);
- }
-
- /* We also need to end on the cache granularity boundary */
- post = len & (cache->bsize - 1);
- if (post) {
- len -= post;
- if (len == 0) {
- fscache_count_dio_misfit();
- netfs_write_subrequest_terminated(subreq, post);
- return;
- }
- iov_iter_truncate(&subreq->io_iter, len);
+ /* We need to start and end on cache granularity boundaries. */
+ if (WARN_ON_ONCE(start & (cache->bsize - 1)) ||
+ WARN_ON_ONCE(len & (cache->bsize - 1))) {
+ fscache_count_dio_misfit();
+ ret = -EIO;
+ goto failed;
}
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, len);
+
trace_netfs_sreq(subreq, netfs_sreq_trace_cache_prepare);
cachefiles_begin_secure(cache, &saved_cred);
ret = __cachefiles_prepare_write(object, cachefiles_cres_file(cres),
&start, &len, len, true);
cachefiles_end_secure(cache, saved_cred);
- if (ret < 0) {
- netfs_write_subrequest_terminated(subreq, ret);
- return;
- }
+ if (ret < 0)
+ goto failed;
trace_netfs_sreq(subreq, netfs_sreq_trace_cache_write);
- cachefiles_write(&subreq->rreq->cache_resources,
- subreq->start, &subreq->io_iter,
+ cachefiles_write(&subreq->rreq->cache_resources, subreq->start, &iter,
netfs_write_subrequest_terminated, subreq);
+ return 0;
+failed:
+ netfs_write_subrequest_terminated(subreq, ret);
+ return 0;
}
/*
@@ -753,6 +854,7 @@ static void cachefiles_collect_write(struct netfs_io_request *wreq,
WARN_ON(new_size & (cache->bsize - 1)) ||
WARN_ON(start & (cache->bsize - 1)) ||
WARN_ON(len & (cache->bsize - 1))) {
+ trace_netfs_rreq(wreq, netfs_rreq_trace_cache_align_error);
trace_cachefiles_io_error(object, inode, -EIO,
cachefiles_trace_alignment_error);
cachefiles_remove_object_xattr(cache, object, file->f_path.dentry);
@@ -907,10 +1009,10 @@ static const struct netfs_cache_ops cachefiles_netfs_cache_ops = {
.end_operation = cachefiles_end_operation,
.read = cachefiles_read,
.write = cachefiles_write,
+ .issue_read = cachefiles_issue_read,
.issue_write = cachefiles_issue_write,
- .prepare_write = cachefiles_prepare_write,
- .prepare_write_subreq = cachefiles_prepare_write_subreq,
.estimate_write = cachefiles_estimate_write,
+ .prepare_write_old = cachefiles_prepare_write_old,
.query_occupancy = cachefiles_query_occupancy,
.collect_write = cachefiles_collect_write,
};
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index 6586f6c1dc73..b13c071bd574 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -274,7 +274,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
ceph_dec_osd_stopping_blocker(fsc->mdsc);
}
-static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
+static int ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
struct inode *inode = rreq->inode;
@@ -283,7 +283,8 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
struct ceph_mds_request *req;
struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(inode->i_sb);
struct ceph_inode_info *ci = ceph_inode(inode);
- ssize_t err = 0;
+ struct iov_iter iter;
+ ssize_t err;
size_t len;
int mode;
@@ -292,8 +293,20 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
__set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags);
__clear_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags);
- if (subreq->start >= inode->i_size)
+ err = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ if (subreq->start >= inode->i_size) {
+ __set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags);
+ err = 0;
goto out;
+ }
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset,
+ subreq->len);
/* We need to fetch the inline data. */
mode = ceph_try_to_choose_auth_mds(inode, CEPH_STAT_CAP_INLINE_DATA);
@@ -302,11 +315,13 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
err = PTR_ERR(req);
goto out;
}
+
req->r_ino1 = ci->i_vino;
req->r_args.getattr.mask = cpu_to_le32(CEPH_STAT_CAP_INLINE_DATA);
req->r_num_caps = 2;
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+
err = ceph_mdsc_do_request(mdsc, NULL, req);
if (err < 0)
goto out;
@@ -316,11 +331,13 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
if (iinfo->inline_version == CEPH_INLINE_NONE) {
/* The data got uninlined */
ceph_mdsc_put_request(req);
- return false;
+ __set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ err = -EAGAIN;
+ goto out;
}
len = min_t(size_t, iinfo->inline_len - subreq->start, subreq->len);
- err = copy_to_iter(iinfo->inline_data + subreq->start, len, &subreq->io_iter);
+ err = copy_to_iter(iinfo->inline_data + subreq->start, len, &iter);
if (err == 0) {
err = -EFAULT;
} else {
@@ -333,26 +350,10 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
subreq->error = err;
trace_netfs_sreq(subreq, netfs_sreq_trace_io_progress);
netfs_read_subreq_terminated(subreq);
- return true;
-}
-
-static int ceph_netfs_prepare_read(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_request *rreq = subreq->rreq;
- struct inode *inode = rreq->inode;
- struct ceph_inode_info *ci = ceph_inode(inode);
- struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
- u64 objno, objoff;
- u32 xlen;
-
- /* Truncate the extent at the end of the current block */
- ceph_calc_file_object_mapping(&ci->i_layout, subreq->start, subreq->len,
- &objno, &objoff, &xlen);
- rreq->io_streams[0].sreq_max_len = umin(xlen, fsc->mount_options->rsize);
return 0;
}
-static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
+static int ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
struct inode *inode = rreq->inode;
@@ -361,19 +362,18 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
struct ceph_client *cl = fsc->client;
struct ceph_osd_request *req = NULL;
struct ceph_vino vino = ceph_vino(inode);
- int err;
- u64 len;
+ struct iov_iter iter;
+ u64 objno, objoff, len, off = subreq->start;
+ u32 maxlen;
+ int err = -EIO;
bool sparse = IS_ENCRYPTED(inode) || ceph_test_mount_opt(fsc, SPARSEREAD);
- u64 off = subreq->start;
int extent_cnt;
- if (ceph_inode_is_shutdown(inode)) {
- err = -EIO;
- goto out;
- }
+ if (ceph_inode_is_shutdown(inode))
+ return -EIO;
- if (ceph_has_inline_data(ci) && ceph_netfs_issue_op_inline(subreq))
- return;
+ if (ceph_has_inline_data(ci))
+ return ceph_netfs_issue_op_inline(subreq);
// TODO: This rounding here is slightly dodgy. It *should* work, for
// now, as the cache only deals in blocks that are a multiple of
@@ -383,26 +383,48 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
len = subreq->len;
ceph_fscrypt_adjust_off_and_len(inode, &off, &len);
+ /* Truncate the extent at the end of the current block */
+ ceph_calc_file_object_mapping(&ci->i_layout, subreq->start, len,
+ &objno, &objoff, &maxlen);
+ maxlen = min(maxlen, fsc->mount_options->rsize);
+ len = min(len, maxlen);
+ subreq->len = len;
+
+ /* Grab a slice of read buffer. This may shrink the subreq. */
+ err = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ /* Create a request. In theory, this may shrink the request again, but
+ * it shouldn't since we calculated the object size above and already
+ * shrank to that, but if it does, we'll just end up doing a short read
+ * and retrying to get the rest.
+ */
+ len = subreq->len;
req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout, vino,
off, &len, 0, 1, sparse ? CEPH_OSD_OP_SPARSE_READ : CEPH_OSD_OP_READ,
CEPH_OSD_FLAG_READ, NULL, ci->i_truncate_seq,
ci->i_truncate_size, false);
if (IS_ERR(req)) {
err = PTR_ERR(req);
- req = NULL;
- goto out;
+ goto failed_noput;
}
if (sparse) {
extent_cnt = __ceph_sparse_read_ext_count(inode, len);
err = ceph_alloc_sparse_ext_map(&req->r_ops[0], extent_cnt);
if (err)
- goto out;
+ goto failed;
}
doutc(cl, "%llx.%llx pos=%llu orig_len=%zu len=%llu\n",
ceph_vinop(inode), subreq->start, subreq->len, len);
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset,
+ subreq->len);
+
/*
* FIXME: For now, use CEPH_OSD_DATA_TYPE_PAGES instead of _ITER for
* encrypted inodes. We'd need infrastructure that handles an iov_iter
@@ -421,13 +443,11 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
* ceph_msg_data_cursor_init() triggers BUG_ON() in the case
* if msg->sparse_read_total > msg->data_length.
*/
- subreq->io_iter.count = len;
-
- err = iov_iter_get_pages_alloc2(&subreq->io_iter, &pages, len, &page_off);
+ err = iov_iter_get_pages_alloc2(&iter, &pages, len, &page_off);
if (err < 0) {
doutc(cl, "%llx.%llx failed to allocate pages, %d\n",
ceph_vinop(inode), err);
- goto out;
+ goto eio;
}
/* should always give us a page-aligned read */
@@ -438,12 +458,10 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
osd_req_op_extent_osd_data_pages(req, 0, pages, len, 0, false,
false);
} else {
- osd_req_op_extent_osd_iter(req, 0, &subreq->io_iter);
- }
- if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) {
- err = -EIO;
- goto out;
+ osd_req_op_extent_osd_iter(req, 0, &iter);
}
+ if (!ceph_inc_osd_stopping_blocker(fsc->mdsc))
+ goto eio;
req->r_callback = finish_netfs_read;
req->r_priv = subreq;
req->r_inode = inode;
@@ -451,19 +469,22 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
ceph_osdc_start_request(req->r_osdc, req);
-out:
ceph_osdc_put_request(req);
- if (err) {
- subreq->error = err;
- netfs_read_subreq_terminated(subreq);
- }
- doutc(cl, "%llx.%llx result %d\n", ceph_vinop(inode), err);
+ doutc(cl, "%llx.%llx result -EIOCBQUEUED\n", ceph_vinop(inode));
+ return 0;
+eio:
+ err = -EIO;
+failed:
+ ceph_osdc_put_request(req);
+failed_noput:
+ subreq->error = err;
+ netfs_read_subreq_terminated(subreq);
+ return 0;
}
static int ceph_init_request(struct netfs_io_request *rreq, struct file *file)
{
struct inode *inode = rreq->inode;
- struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
struct ceph_client *cl = ceph_inode_to_client(inode);
int got = 0, want = CEPH_CAP_FILE_CACHE;
struct ceph_netfs_request_data *priv;
@@ -515,7 +536,6 @@ static int ceph_init_request(struct netfs_io_request *rreq, struct file *file)
priv->caps = got;
rreq->netfs_priv = priv;
- rreq->io_streams[0].sreq_max_len = fsc->mount_options->rsize;
out:
if (ret < 0) {
@@ -543,7 +563,6 @@ static void ceph_netfs_free_request(struct netfs_io_request *rreq)
const struct netfs_request_ops ceph_netfs_ops = {
.init_request = ceph_init_request,
.free_request = ceph_netfs_free_request,
- .prepare_read = ceph_netfs_prepare_read,
.issue_read = ceph_netfs_issue_read,
.expand_readahead = ceph_netfs_expand_readahead,
.check_write_begin = ceph_netfs_check_write_begin,
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 88e5404080e8..fbb0c312b723 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -100,70 +100,98 @@ static int netfs_begin_cache_read(struct netfs_io_request *rreq, struct netfs_in
}
/*
- * netfs_prepare_read_iterator - Prepare the subreq iterator for I/O
- * @subreq: The subrequest to be set up
- *
- * Prepare the I/O iterator representing the read buffer on a subrequest for
- * the filesystem to use for I/O (it can be passed directly to a socket). This
- * is intended to be called from the ->issue_read() method once the filesystem
- * has trimmed the request to the size it wants.
- *
- * Returns the limited size if successful and -ENOMEM if insufficient memory
- * available.
+ * Prepare the I/O buffer on a buffered read subrequest for the filesystem to
+ * use as a bvec queue.
*/
-static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq)
+static int netfs_prepare_buffered_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
struct netfs_io_request *rreq = subreq->rreq;
struct netfs_io_stream *stream = &rreq->io_streams[0];
ssize_t extracted;
- size_t rsize = subreq->len;
- if (subreq->source == NETFS_DOWNLOAD_FROM_SERVER)
- rsize = umin(rsize, stream->sreq_max_len);
+ _enter("R=%08x[%x] l=%zx s=%u",
+ rreq->debug_id, subreq->debug_index, subreq->len, max_segs);
- bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor);
- extracted = bvecq_slice(&rreq->dispatch_cursor, rsize,
- stream->sreq_max_segs, &subreq->nr_segs);
- if (extracted < rsize) {
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+ extracted = bvecq_slice(&stream->dispatch_cursor, subreq->len,
+ max_segs, &subreq->nr_segs);
+
+ if (extracted < subreq->len) {
subreq->len = extracted;
trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
}
+ stream->buffered -= extracted;
+ stream->issue_from = subreq->start + subreq->len;
+ rreq->submitted = stream->issue_from;
- return subreq->len;
+ if (!stream->buffered)
+ netfs_all_subreqs_queued(rreq);
+ return 0;
}
-/*
- * Issue a read against the cache.
- * - Eats the caller's ref on subreq.
+/**
+ * netfs_prepare_read_buffer - Get the buffer for a subrequest
+ * @subreq: The subrequest to get the buffer for
+ * @max_segs: Maximum number of segments in buffer (or INT_MAX)
+ *
+ * Extract a slice of buffer from the stream and attach it to the subrequest as
+ * a bio_vec queue. The maximum amount of data attached is set by
+ * @subreq->len, but this may be shortened if @max_segs would be exceeded.
+ *
+ * [!] NOTE: This must be run in the same thread as ->issue_read() was called
+ * in as we access the readahead_control struct if there is one.
*/
-static void netfs_read_cache_to_pagecache(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
+int netfs_prepare_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
- struct netfs_cache_resources *cres = &rreq->cache_resources;
-
- netfs_stat(&netfs_n_rh_read);
- cres->ops->read(cres, subreq->start, &subreq->io_iter, NETFS_READ_HOLE_IGNORE,
- netfs_cache_read_terminated, subreq);
+ switch (subreq->rreq->origin) {
+ case NETFS_READAHEAD:
+ case NETFS_READPAGE:
+ case NETFS_READ_FOR_WRITE:
+ if (subreq->retry_count)
+ return netfs_prepare_buffered_read_retry_buffer(subreq, max_segs);
+ return netfs_prepare_buffered_read_buffer(subreq, max_segs);
+
+ case NETFS_UNBUFFERED_READ:
+ case NETFS_DIO_READ:
+ case NETFS_READ_GAPS:
+ return netfs_prepare_unbuffered_read_buffer(subreq, max_segs);
+ case NETFS_READ_SINGLE:
+ return netfs_prepare_read_single_buffer(subreq, max_segs);
+ default:
+ WARN_ON_ONCE(1);
+ return -EIO;
+ }
}
+EXPORT_SYMBOL(netfs_prepare_read_buffer);
-int netfs_read_query_cache(struct netfs_io_request *rreq, struct fscache_occupancy *occ)
+void netfs_read_query_cache(struct netfs_io_request *rreq, struct fscache_occupancy *occ)
{
struct netfs_cache_resources *cres = &rreq->cache_resources;
occ->granularity = PAGE_SIZE;
if (occ->query_from >= occ->query_to)
- return 0;
+ return;
if (!cres->ops)
- return 0;
+ return;
occ->query_from = round_up(occ->query_from, occ->granularity);
- return cres->ops->query_occupancy(cres, occ);
+ cres->ops->query_occupancy(cres, occ);
}
-void netfs_queue_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
+/*
+ * Allocate and prepare a read subrequest.
+ */
+struct netfs_io_subrequest *netfs_alloc_read_subrequest(struct netfs_io_request *rreq)
{
+ struct netfs_io_subrequest *subreq;
struct netfs_io_stream *stream = &rreq->io_streams[0];
+ subreq = netfs_alloc_subrequest(rreq);
+ if (!subreq)
+ return subreq;
+
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
/* We add to the end of the list whilst the collector may be walking
@@ -173,40 +201,56 @@ void netfs_queue_read(struct netfs_io_request *rreq,
spin_lock(&rreq->lock);
/* Write IN_PROGRESS before pointer to new subreq */
list_add_tail_release(&subreq->rreq_link, &stream->subrequests);
- if (list_is_first(&subreq->rreq_link, &stream->subrequests)) {
- if (!stream->active) {
- stream->collected_to = subreq->start;
- /* Store list pointers before active flag */
- smp_store_release(&stream->active, true);
- }
- }
+ if (!stream->active)
+ /* Store list pointers before active flag */
+ smp_store_release(&stream->active, true);
spin_unlock(&rreq->lock);
+ return subreq;
}
static void netfs_issue_read(struct netfs_io_request *rreq,
struct netfs_io_subrequest *subreq)
{
- bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
- iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
- subreq->content.slot, subreq->content.offset, subreq->len);
+ struct netfs_cache_resources *cres = &rreq->cache_resources;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+ int ret;
+
+ _enter("R=%08x[%x]", rreq->debug_id, subreq->debug_index);
switch (subreq->source) {
case NETFS_DOWNLOAD_FROM_SERVER:
- rreq->netfs_ops->issue_read(subreq);
- break;
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0)
+ goto fail;
+ return;
case NETFS_READ_FROM_CACHE:
- netfs_read_cache_to_pagecache(rreq, subreq);
- break;
+ netfs_stat(&netfs_n_rh_read);
+ ret = cres->ops->issue_read(subreq);
+ if (ret < 0)
+ goto fail;
+ return;
default:
- bvecq_zero(&subreq->content, subreq->len);
+ WARN_ON_ONCE(1);
+ fallthrough;
+ case NETFS_FILL_WITH_ZEROES:
+ stream->issue_from = subreq->start + subreq->len;
+ stream->buffered -= subreq->len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+ if (!stream->buffered)
+ netfs_all_subreqs_queued(rreq);
+ bvecq_zero(&stream->dispatch_cursor, subreq->len);
subreq->transferred = subreq->len;
subreq->error = 0;
- iov_iter_zero(subreq->len, &subreq->io_iter);
- subreq->transferred = subreq->len;
- netfs_read_subreq_terminated(subreq);
- break;
+ return netfs_read_subreq_terminated(subreq);
}
+
+fail:
+ /* Ownership of subreq was returned to us. */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ stream->buffered -= subreq->len;
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
}
/*
@@ -225,19 +269,16 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
.cached_to[1] = ULLONG_MAX,
};
struct fscache_occupancy *occ = &_occ;
- ssize_t size = rreq->len;
- uoff_t start = rreq->start;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
int ret = 0;
_enter("R=%08x", rreq->debug_id);
- bvecq_pos_set(&rreq->dispatch_cursor, &rreq->load_cursor);
+ bvecq_pos_set(&stream->dispatch_cursor, &rreq->load_cursor);
do {
- int (*prepare_read)(struct netfs_io_subrequest *subreq) = NULL;
struct netfs_io_subrequest *subreq;
- ssize_t slice;
- uoff_t hole_to, cache_to;
+ uoff_t hole_to, cache_to, stop;
/* If we don't have any, find out the next couple of data
* extents from the cache, containing of following the
@@ -246,7 +287,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
*/
hole_to = occ->cached_from[0];
cache_to = occ->cached_to[0];
- if (start >= cache_to) {
+ if (stream->issue_from >= cache_to) {
/* Extent exhausted; shuffle down. */
int i;
@@ -261,52 +302,50 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
if (occ->cached_from[0] != ULLONG_MAX)
continue;
- /* Get new extents */
- ret = netfs_read_query_cache(rreq, occ);
- if (ret < 0)
- break;
+ /* Wait for the cache to enter the readable state and
+ * get more extents. If the cache doesn't become
+ * readable, this leaves the default negative extent in
+ * place.
+ */
+ netfs_read_query_cache(rreq, occ);
continue;
}
- subreq = netfs_alloc_subrequest(rreq);
+ subreq = netfs_alloc_read_subrequest(rreq);
if (!subreq) {
ret = -ENOMEM;
break;
}
- subreq->start = start;
- subreq->len = size;
-
- netfs_queue_read(rreq, subreq);
+ subreq->start = stream->issue_from;
+ stop = stream->issue_from + stream->buffered;
uoff_t zero_point = netfs_read_zero_point(rreq->inode);
uoff_t zlimit = umin(zero_point, rreq->i_size);
_debug("rsub %llx %llx-%llx", subreq->start, hole_to, cache_to);
- if (start >= hole_to && start < cache_to) {
+ if (stream->issue_from >= hole_to && stream->issue_from < cache_to) {
/* Overlap with a cached region, where the cache may
* record a block of zeroes.
*/
- _debug("cached s=%llx c=%llx l=%zx", start, cache_to, size);
- subreq->len = umin(cache_to - start, size);
- subreq->len = round_up(subreq->len, occ->granularity);
+ _debug("cached s=%llx c=%llx l=%zx",
+ stream->issue_from, cache_to, stream->buffered);
+ subreq->len = umin(cache_to - stream->issue_from, stream->buffered);
if (occ->cached_type[0] == FSCACHE_EXTENT_ZERO) {
subreq->source = NETFS_FILL_WITH_ZEROES;
netfs_stat(&netfs_n_rh_zero);
} else {
+ subreq->len = round_up(subreq->len, occ->granularity);
subreq->source = NETFS_READ_FROM_CACHE;
- prepare_read = rreq->cache_resources.ops->prepare_read;
}
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
-
- } else if (subreq->start >= zlimit && size > 0) {
+ } else if (subreq->start >= zlimit &&
+ subreq->start < stop) {
/* If this range lies beyond the zero-point, that part
* can just be cleared locally.
*/
- _debug("zero %llx-%llx", start, start + size);
- subreq->len = size;
+ _debug("zero %llx-%llx", subreq->start, stop);
+ subreq->len = stream->buffered;
subreq->source = NETFS_FILL_WITH_ZEROES;
if (rreq->cache_resources.ops)
__set_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags);
@@ -316,10 +355,10 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
* this range lies beyond the zero-point or the EOF,
* that part can just be cleared locally.
*/
- uoff_t limit = min3(zlimit, start + size, hole_to);
+ uoff_t limit = min3(zlimit, stop, hole_to);
_debug("limit %llx %llx", rreq->i_size, zero_point);
- _debug("download %llx-%llx", start, start + size);
+ _debug("download %llx-%llx", subreq->start, stop);
subreq->len = umin(limit - subreq->start, ULONG_MAX);
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
if (rreq->cache_resources.ops)
@@ -327,39 +366,15 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
netfs_stat(&netfs_n_rh_download);
}
- if (size == 0) {
+ if (subreq->len == 0) {
pr_err("ZERO-LEN READ: R=%08x[%x] l=%zx/%zx s=%llx z=%llx i=%llx",
rreq->debug_id, subreq->debug_index,
- subreq->len, size,
+ subreq->len, stream->buffered,
subreq->start, zero_point, rreq->i_size);
netfs_cancel_read(subreq, ret);
break;
}
- rreq->io_streams[0].sreq_max_len = MAX_RW_COUNT;
- rreq->io_streams[0].sreq_max_segs = INT_MAX;
-
- if (prepare_read) {
- ret = prepare_read(subreq);
- if (ret < 0) {
- netfs_cancel_read(subreq, ret);
- break;
- }
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
- }
-
- slice = netfs_prepare_read_iterator(subreq);
- if (slice < 0) {
- ret = slice;
- netfs_cancel_read(subreq, ret);
- break;
- }
- start += slice;
- size -= slice;
- if (size <= 0)
- netfs_all_subreqs_queued(rreq);
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
netfs_issue_read(rreq, subreq);
netfs_maybe_bulk_drop_ra_refs(rreq);
@@ -368,18 +383,19 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
if (test_bit(NETFS_RREQ_FAILED, &rreq->flags))
break;
cond_resched();
- } while (size > 0);
+ } while (stream->buffered > 0);
- if (unlikely(size > 0)) {
+ if (unlikely(!netfs_are_all_subreqs_queued(rreq))) {
netfs_all_subreqs_queued(rreq);
netfs_wake_collector(rreq);
}
/* Defer error return as we may need to wait for outstanding I/O. */
- cmpxchg(&rreq->error, 0, ret);
+ if (ret < 0)
+ cmpxchg(&rreq->error, 0, ret);
bvecq_pos_unset(&rreq->load_cursor);
- bvecq_pos_unset(&rreq->dispatch_cursor);
+ bvecq_pos_unset(&stream->dispatch_cursor);
}
/**
@@ -400,17 +416,22 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
void netfs_readahead(struct readahead_control *ractl)
{
struct netfs_io_request *rreq;
+ struct netfs_io_stream *stream;
struct netfs_inode *ictx = netfs_inode(ractl->mapping->host);
ssize_t added;
uoff_t start = readahead_pos(ractl);
size_t size = readahead_length(ractl);
int ret;
+ _enter("");
+
rreq = netfs_alloc_request(ractl->mapping, ractl->file, start, size,
NETFS_READAHEAD);
if (IS_ERR(rreq))
return;
+ stream = &rreq->io_streams[0];
+
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &rreq->flags);
ret = netfs_begin_cache_read(rreq, ictx);
@@ -435,9 +456,9 @@ void netfs_readahead(struct readahead_control *ractl)
__set_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags);
rreq->submitted = rreq->start + added;
- rreq->cleaned_to = rreq->start;
bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
netfs_read_set_unlock_at(rreq);
+ stream->buffered = added;
netfs_read_to_pagecache(rreq);
netfs_maybe_bulk_drop_ra_refs(rreq);
@@ -453,18 +474,20 @@ EXPORT_SYMBOL(netfs_readahead);
*/
static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio)
{
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
struct bvecq *bq;
size_t fsize = folio_size(folio);
if (bvecq_buffer_init(&rreq->load_cursor, rreq->gfp, false) < 0)
return -ENOMEM;
+ bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
bq = rreq->load_cursor.bvecq;
bvec_set_folio(&bq->bv[0], folio, fsize, 0);
bvecq_filled_to(bq, 1);
rreq->submitted = rreq->start + fsize;
rreq->progress_at = fsize;
- bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
+ stream->buffered = fsize;
return 0;
}
@@ -474,6 +497,7 @@ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct fo
static int netfs_read_gaps(struct file *file, struct folio *folio)
{
struct netfs_io_request *rreq;
+ struct netfs_io_stream *stream;
struct address_space *mapping = folio->mapping;
struct netfs_group *group = netfs_folio_group(folio);
struct netfs_folio *finfo = netfs_folio_info(folio);
@@ -495,6 +519,7 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
ret = PTR_ERR(rreq);
goto alloc_error;
}
+ stream = &rreq->io_streams[0];
ret = netfs_begin_cache_read(rreq, ctx);
if (ret == -ENOMEM || ret == -EINTR || ret == -ERESTARTSYS)
@@ -549,6 +574,7 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
bvecq_filled_to(bq, slot);
rreq->submitted = rreq->start + flen;
+ stream->buffered = flen;
netfs_read_to_pagecache(rreq);
@@ -627,6 +653,7 @@ int netfs_read_folio(struct file *file, struct folio *folio)
goto discard;
netfs_read_to_pagecache(rreq);
+
ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret < 0 ? ret : 0;
diff --git a/fs/netfs/direct_read.c b/fs/netfs/direct_read.c
index 058cc6bb7124..8ded0197af91 100644
--- a/fs/netfs/direct_read.c
+++ b/fs/netfs/direct_read.c
@@ -16,6 +16,32 @@
#include <linux/netfs.h>
#include "internal.h"
+int netfs_prepare_unbuffered_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *rreq = subreq->rreq;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ stream->buffered -= subreq->len;
+ stream->issue_from += subreq->len;
+ rreq->submitted = stream->issue_from;
+
+ if (stream->buffered == 0)
+ netfs_all_subreqs_queued(rreq);
+ return 0;
+}
+
/*
* Perform a read to a buffer from the server, slicing up the region to be read
* according to the network rsize.
@@ -23,17 +49,15 @@
static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
{
struct netfs_io_stream *stream = &rreq->io_streams[0];
- ssize_t size = rreq->len;
- uoff_t start = rreq->start;
- int ret;
- bvecq_pos_set(&rreq->dispatch_cursor, &rreq->load_cursor);
- bvecq_pos_transfer(&rreq->collect_cursor, &rreq->load_cursor);
+ bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
+ bvecq_pos_transfer(&stream->dispatch_cursor, &rreq->load_cursor);
do {
struct netfs_io_subrequest *subreq;
+ int ret;
- subreq = netfs_alloc_subrequest(rreq);
+ subreq = netfs_alloc_read_subrequest(rreq);
if (!subreq) {
/* Stash the error in the request if there's not
* already an error set.
@@ -43,51 +67,32 @@ static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
}
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- subreq->start = start;
- subreq->len = size;
-
- netfs_queue_read(rreq, subreq);
+ subreq->start = stream->issue_from;
+ subreq->len = stream->buffered;
netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read) {
- ret = rreq->netfs_ops->prepare_read(subreq);
- if (ret < 0) {
- netfs_cancel_read(subreq, ret);
- break;
- }
- }
-
- bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor);
- bvecq_pos_set(&subreq->content, &rreq->dispatch_cursor);
- subreq->len = bvecq_slice(&rreq->dispatch_cursor,
- umin(size, stream->sreq_max_len),
- stream->sreq_max_segs,
- &subreq->nr_segs);
-
- size -= subreq->len;
- start += subreq->len;
- rreq->submitted += subreq->len;
- if (size <= 0)
- netfs_all_subreqs_queued(rreq);
- iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
- subreq->content.slot, subreq->content.offset, subreq->len);
-
- rreq->netfs_ops->issue_read(subreq);
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ stream->buffered -= subreq->len;
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ break;
+ }
if (test_bit(NETFS_RREQ_PAUSE, &rreq->flags))
netfs_wait_for_paused_read(rreq);
if (test_bit(NETFS_RREQ_FAILED, &rreq->flags))
break;
cond_resched();
- } while (size > 0);
+ } while (stream->buffered > 0);
- if (unlikely(size > 0)) {
+ if (unlikely(stream->buffered > 0)) {
netfs_all_subreqs_queued(rreq);
netfs_wake_collector(rreq);
}
- bvecq_pos_unset(&rreq->dispatch_cursor);
+ bvecq_pos_unset(&stream->dispatch_cursor);
}
/*
@@ -138,6 +143,7 @@ static ssize_t netfs_unbuffered_read(struct netfs_io_request *rreq, bool sync)
ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *iter)
{
struct netfs_io_request *rreq;
+ struct netfs_io_stream *stream;
ssize_t ret;
size_t orig_count = iov_iter_count(iter);
bool sync = is_sync_kiocb(iocb);
@@ -162,6 +168,8 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i
netfs_stat(&netfs_n_rh_dio_read);
trace_netfs_read(rreq, rreq->start, rreq->len, netfs_read_trace_dio_read);
+ stream = &rreq->io_streams[0];
+
/* If this is an async op, we have to keep track of the destination
* buffer for ourselves as the caller's iterator will be trashed when
* we return.
@@ -177,6 +185,8 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i
goto error_put;
rreq->len = ret;
+ stream->buffered = ret;
+ stream->issue_from = rreq->start;
// TODO: Set up bounce buffer if needed
diff --git a/fs/netfs/direct_write.c b/fs/netfs/direct_write.c
index 5bc3ec261f6c..40a2d34789ad 100644
--- a/fs/netfs/direct_write.c
+++ b/fs/netfs/direct_write.c
@@ -9,6 +9,34 @@
#include <linux/uio.h>
#include "internal.h"
+/*
+ * Prepare the buffer for an unbuffered/DIO write.
+ */
+int netfs_prepare_unbuffered_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_stream *stream = &subreq->rreq->io_streams[subreq->stream_nr];
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ // TODO: Wait here for completion of prev subreq
+
+ stream->issue_from += subreq->len;
+ stream->buffered -= subreq->len;
+ if (stream->buffered == 0)
+ netfs_all_subreqs_queued(subreq->rreq);
+ return 0;
+}
+
/*
* Perform the cleanup rituals after an unbuffered write is complete.
*/
@@ -74,9 +102,9 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq,
wreq->transferred += subreq->transferred;
if (subreq->transferred < subreq->len) {
- bvecq_pos_unset(&wreq->dispatch_cursor);
- bvecq_pos_transfer(&wreq->dispatch_cursor, &subreq->dispatch_pos);
- bvecq_pos_advance(&wreq->dispatch_cursor, subreq->transferred);
+ bvecq_pos_unset(&stream->dispatch_cursor);
+ bvecq_pos_transfer(&stream->dispatch_cursor, &subreq->dispatch_pos);
+ bvecq_pos_advance(&stream->dispatch_cursor, subreq->transferred);
}
stream->collected_to = subreq->start + subreq->transferred;
@@ -85,6 +113,7 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq,
trace_netfs_collect_stream(wreq, stream);
trace_netfs_collect_state(wreq, wreq->collected_to, 0);
+ /* TODO: Progressively clean up wreq->direct_bq */
}
/*
@@ -103,60 +132,50 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
_enter("%llx", wreq->len);
- bvecq_pos_set(&wreq->dispatch_cursor, &wreq->load_cursor);
- bvecq_pos_set(&wreq->collect_cursor, &wreq->dispatch_cursor);
+ stream->issue_from = wreq->start;
+ stream->buffered = wreq->len;
+ bvecq_pos_set(&stream->dispatch_cursor, &wreq->load_cursor);
if (wreq->origin == NETFS_DIO_WRITE)
inode_dio_begin(wreq->inode);
- stream->collected_to = wreq->start;
-
for (;;) {
bool retry = false;
if (!subreq) {
- netfs_prepare_write(wreq, stream, wreq->start + wreq->transferred);
- subreq = stream->construct;
- stream->construct = NULL;
- } else {
- bvecq_pos_set(&subreq->dispatch_pos, &wreq->dispatch_cursor);
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ if (!subreq) {
+ ret = -ENOMEM;
+ goto failed;
+ }
}
- /* Check if (re-)preparation failed. */
- if (unlikely(test_bit(NETFS_SREQ_FAILED, &subreq->flags))) {
- netfs_write_subrequest_terminated(subreq, subreq->error);
- wreq->error = subreq->error;
- break;
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. Note that
+ * ->dispatch_pos may or may not be initialised.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ netfs_write_subrequest_terminated(subreq, ret);
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed);
+ subreq = NULL;
+ goto failed;
}
- subreq->len = bvecq_slice(&wreq->dispatch_cursor, stream->sreq_max_len,
- stream->sreq_max_segs, &subreq->nr_segs);
- bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
-
- iov_iter_bvec_queue(&subreq->io_iter, ITER_SOURCE,
- subreq->content.bvecq, subreq->content.slot,
- subreq->content.offset,
- subreq->len);
-
- if (!iov_iter_count(&subreq->io_iter))
- break;
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- stream->issue_write(subreq);
-
- /* Async, need to wait. */
- netfs_wait_for_in_progress_stream(wreq, stream);
-
- if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
+ ret = netfs_wait_for_in_progress_subreq(wreq, subreq);
+ if (ret < 0) {
+ if (ret != -EAGAIN) {
+ /* Don't need to lock here as the collection is
+ * done in this thread.
+ */
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed);
+ subreq = NULL;
+ goto failed;
+ }
retry = true;
- } else if (test_bit(NETFS_SREQ_FAILED, &subreq->flags)) {
- ret = subreq->error;
- wreq->error = ret;
- netfs_see_subrequest(subreq, netfs_sreq_trace_see_failed);
- subreq = NULL;
- break;
}
- ret = 0;
if (!retry) {
netfs_unbuffered_write_collect(wreq, stream, subreq);
@@ -171,20 +190,21 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
continue;
}
- /* We need to retry the last subrequest, so first reset the
- * iterator, taking into account what, if anything, we managed
- * to transfer.
+ /* We need to retry the last subrequest, so first wind back the
+ * buffer position.
*/
subreq->error = -EAGAIN;
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
bvecq_pos_unset(&subreq->content);
- bvecq_pos_unset(&wreq->dispatch_cursor);
- bvecq_pos_transfer(&wreq->dispatch_cursor, &subreq->dispatch_pos);
+ bvecq_pos_unset(&stream->dispatch_cursor);
+ bvecq_pos_transfer(&stream->dispatch_cursor, &subreq->dispatch_pos);
+ stream->issue_from -= subreq->len - subreq->transferred;
+ stream->buffered += subreq->len - subreq->transferred;
if (subreq->transferred > 0) {
- wreq->transferred += subreq->transferred;
- bvecq_pos_advance(&wreq->dispatch_cursor, subreq->transferred);
+ wreq->transferred += subreq->transferred;
+ bvecq_pos_advance(&stream->dispatch_cursor, subreq->transferred);
}
if (stream->source == NETFS_UPLOAD_TO_SERVER &&
@@ -193,25 +213,21 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
__clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
__clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
- __clear_bit(NETFS_SREQ_BOUNDARY, &subreq->flags);
__clear_bit(NETFS_SREQ_FAILED, &subreq->flags);
- subreq->start = wreq->start + wreq->transferred;
- subreq->len = wreq->len - wreq->transferred;
+ __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ subreq->start = stream->issue_from;
+ subreq->len = stream->buffered;
subreq->transferred = 0;
subreq->retry_count += 1;
- stream->sreq_max_len = UINT_MAX;
- stream->sreq_max_segs = INT_MAX;
netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- if (stream->prepare_write)
- stream->prepare_write(subreq);
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
netfs_stat(&netfs_n_wh_retry_write_subreq);
}
- bvecq_pos_unset(&wreq->dispatch_cursor);
- bvecq_pos_unset(&wreq->load_cursor);
+failed:
+ bvecq_pos_unset(&stream->dispatch_cursor);
netfs_unbuffered_write_done(wreq);
_leave(" = %d", ret);
return ret;
@@ -255,6 +271,7 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
if (IS_ERR(wreq))
return PTR_ERR(wreq);
+ wreq->len = iov_iter_count(iter);
wreq->io_streams[0].avail = true;
trace_netfs_write(wreq, (iocb->ki_flags & IOCB_DIRECT ?
netfs_write_trace_dio_write :
@@ -265,9 +282,7 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
* we have to save the source buffer as the iterator is only
* good until we return. In such a case, extract an iterator
* to represent as much of the the output buffer as we can
- * manage. Note that the extraction might not be able to
- * allocate a sufficiently large bvec array and may shorten the
- * request.
+ * manage. Note that the extraction may shorten the request.
*/
ssize_t n = netfs_extract_iter(iter, len, INT_MAX,
&wreq->load_cursor.bvecq, 0, wreq->gfp);
@@ -282,8 +297,6 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
wreq->load_cursor.bvecq->max_slots);
}
- __set_bit(NETFS_RREQ_USE_IO_ITER, &wreq->flags);
-
/* Copy the data into the bounce buffer and encrypt it. */
// TODO
diff --git a/fs/netfs/fscache_io.c b/fs/netfs/fscache_io.c
index 056a2bae5d99..0b85bd33dfe5 100644
--- a/fs/netfs/fscache_io.c
+++ b/fs/netfs/fscache_io.c
@@ -239,7 +239,7 @@ void __fscache_write_to_cache(struct fscache_cookie *cookie,
fscache_access_io_write) < 0)
goto abandon_free;
- ret = cres->ops->prepare_write(cres, &start, &len, len, i_size, false);
+ ret = cres->ops->prepare_write_old(cres, &start, &len, len, i_size, false);
if (ret < 0)
goto abandon_end;
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 3aa9c7ef85de..3d345c9d6d5a 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -22,10 +22,9 @@
/*
* buffered_read.c
*/
-int netfs_read_query_cache(struct netfs_io_request *rreq,
- struct fscache_occupancy *occ);
-void netfs_queue_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq);
+void netfs_read_query_cache(struct netfs_io_request *rreq,
+ struct fscache_occupancy *occ);
+struct netfs_io_subrequest *netfs_alloc_read_subrequest(struct netfs_io_request *rreq);
void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error);
int netfs_prefetch_for_write(struct file *file, struct folio *folio,
size_t offset, size_t len);
@@ -36,6 +35,18 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,
void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode,
uoff_t pos, size_t copied);
+/*
+ * direct_read.c
+ */
+int netfs_prepare_unbuffered_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
+/*
+ * direct_write.c
+ */
+int netfs_prepare_unbuffered_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
/*
* main.c
*/
@@ -74,6 +85,8 @@ struct bvecq *netfs_buffer_make_space(struct netfs_io_request *rreq,
enum netfs_bvecq_trace trace);
void netfs_wake_collector(struct netfs_io_request *rreq);
void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq);
+int netfs_wait_for_in_progress_subreq(struct netfs_io_request *rreq,
+ struct netfs_io_subrequest *subreq);
void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,
struct netfs_io_stream *stream);
ssize_t netfs_wait_for_read(struct netfs_io_request *rreq);
@@ -155,9 +168,18 @@ static inline bool netfs_using_pgpriv2(const struct netfs_io_request *rreq)
/*
* read_retry.c
*/
+int netfs_prepare_buffered_read_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+int netfs_reset_for_read_retry(struct netfs_io_subrequest *subreq);
void netfs_retry_reads(struct netfs_io_request *rreq);
void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq);
+/*
+ * read_single.c
+ */
+int netfs_prepare_read_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
/*
* stats.c
*/
@@ -228,26 +250,19 @@ void netfs_write_collection_worker(struct work_struct *work);
/*
* write_issue.c
*/
+struct netfs_writethrough;
struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
struct file *file,
uoff_t start,
enum netfs_io_origin origin);
struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wreq,
struct netfs_io_stream *stream);
-void netfs_prepare_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- uoff_t start);
-void netfs_reissue_write(struct netfs_io_stream *stream,
- struct netfs_io_subrequest *subreq);
-void netfs_issue_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream);
-size_t netfs_advance_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- uoff_t start, size_t len, bool to_eof);
/*
* write_retry.c
*/
+int netfs_prepare_write_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
void netfs_retry_writes(struct netfs_io_request *wreq);
/*
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index b8da5e85cc67..3736a2a7662d 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -18,6 +18,7 @@ MODULE_DESCRIPTION("Network fs support");
MODULE_AUTHOR("Red Hat, Inc.");
MODULE_LICENSE("GPL");
+EXPORT_TRACEPOINT_SYMBOL(netfs_rreq);
EXPORT_TRACEPOINT_SYMBOL(netfs_sreq);
unsigned netfs_debug;
diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c
index 4fa09560fb4b..a729e51a8ab2 100644
--- a/fs/netfs/misc.c
+++ b/fs/netfs/misc.c
@@ -250,6 +250,37 @@ void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq)
netfs_wake_collector(rreq);
}
+/*
+ * Wait for a subrequest to come to completion.
+ */
+int netfs_wait_for_in_progress_subreq(struct netfs_io_request *rreq,
+ struct netfs_io_subrequest *subreq)
+{
+ if (netfs_check_subreq_in_progress(subreq)) {
+ DEFINE_WAIT(myself);
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_wait_quiesce);
+ for (;;) {
+ prepare_to_wait(&rreq->waitq, &myself, TASK_UNINTERRUPTIBLE);
+
+ if (!netfs_check_subreq_in_progress(subreq))
+ break;
+
+ trace_netfs_sreq(subreq, netfs_sreq_trace_wait_for);
+ schedule();
+ }
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_waited_quiesce);
+ finish_wait(&rreq->waitq, &myself);
+ }
+
+ if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
+ return -EAGAIN;
+ if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
+ return subreq->error;
+ return 0;
+}
+
/*
* Wait for all outstanding I/O in a stream to quiesce.
*/
@@ -380,7 +411,7 @@ static ssize_t netfs_wait_for_in_progress(struct netfs_io_request *rreq,
case NETFS_UNBUFFERED_WRITE:
break;
default:
- if (rreq->submitted < rreq->len) {
+ if (rreq->transferred < rreq->len) {
trace_netfs_failure(rreq, NULL, ret, netfs_fail_short_read);
ret = -EIO;
}
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index a1d0de65484e..3b22748474b9 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -55,8 +55,6 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
rreq->i_size = i_size_read(inode);
rreq->debug_id = atomic_inc_return(&debug_ids);
rreq->wsize = INT_MAX;
- rreq->io_streams[0].sreq_max_len = ULONG_MAX;
- rreq->io_streams[0].sreq_max_segs = 0;
spin_lock_init(&rreq->lock);
init_waitqueue_head(&rreq->waitq);
refcount_set(&rreq->ref, 2);
@@ -150,8 +148,8 @@ static void netfs_deinit_request(struct netfs_io_request *rreq)
if (rreq->cache_resources.ops)
rreq->cache_resources.ops->end_operation(&rreq->cache_resources);
bvecq_pos_unset(&rreq->load_cursor);
- bvecq_pos_unset(&rreq->dispatch_cursor);
bvecq_pos_unset(&rreq->collect_cursor);
+ bvecq_pos_unset(&rreq->retry_cursor);
bvecq_put(rreq->spare);
while (rreq->writebacks) {
struct netfs_writeback *wback = rreq->writebacks;
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index e91f7962df04..9e50759591a9 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -36,6 +36,7 @@ static void netfs_clear_unread(struct netfs_io_subrequest *subreq)
if (subreq->start + subreq->transferred >= subreq->rreq->i_size)
__set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags);
+ trace_netfs_rreq(subreq->rreq, netfs_rreq_trace_zero_unread);
}
/*
@@ -407,8 +408,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
if (test_bit(NETFS_SREQ_COPY_TO_CACHE, &front_flags))
notes |= COPY_TO_CACHE;
- if (front->start + transferred >= unlock_at ||
- test_bit(NETFS_SREQ_HIT_EOF, &front_flags))
+ if (front->start + transferred >= unlock_at)
netfs_read_unlock_folios(rreq, ¬es);
} else {
stream->collected_to = front->start + transferred;
@@ -500,31 +500,6 @@ static void netfs_rreq_assess_dio(struct netfs_io_request *rreq)
inode_dio_end(rreq->inode);
}
-/*
- * Do processing after reading a monolithic single object.
- */
-static void netfs_rreq_assess_single(struct netfs_io_request *rreq)
-{
- struct netfs_io_stream *stream = &rreq->io_streams[0];
-
- if (!rreq->error && stream->source == NETFS_DOWNLOAD_FROM_SERVER &&
- fscache_resources_valid(&rreq->cache_resources)) {
- trace_netfs_rreq(rreq, netfs_rreq_trace_dirty);
- netfs_single_mark_inode_dirty(rreq->inode);
- }
-
- if (rreq->iocb) {
- rreq->iocb->ki_pos += rreq->transferred;
- if (rreq->iocb->ki_complete) {
- trace_netfs_rreq(rreq, netfs_rreq_trace_ki_complete);
- rreq->iocb->ki_complete(
- rreq->iocb, rreq->error ? rreq->error : rreq->transferred);
- }
- }
- if (rreq->netfs_ops->done)
- rreq->netfs_ops->done(rreq);
-}
-
/*
* Perform the collection of subrequests and folios.
*
@@ -558,7 +533,7 @@ bool netfs_read_collection(struct netfs_io_request *rreq)
netfs_rreq_assess_dio(rreq);
break;
case NETFS_READ_SINGLE:
- netfs_rreq_assess_single(rreq);
+ WARN_ON_ONCE(1);
break;
default:
break;
@@ -692,6 +667,11 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)
} else if (test_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags)) {
__set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
trace_netfs_sreq(subreq, netfs_sreq_trace_partial_read);
+ } else if (subreq->source == NETFS_READ_FROM_CACHE) {
+ netfs_stat(&netfs_n_rh_read_failed);
+ __set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ subreq->error = -ENODATA;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_short);
} else {
__set_bit(NETFS_SREQ_FAILED, &subreq->flags);
subreq->error = -ENODATA;
@@ -710,6 +690,8 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)
if (unlikely(subreq->error < 0)) {
trace_netfs_failure(rreq, subreq, subreq->error, netfs_fail_read);
+ if (subreq->error == -ENOMEM)
+ set_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags);
if (subreq->source == NETFS_READ_FROM_CACHE) {
netfs_stat(&netfs_n_rh_read_failed);
__set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c
index 14fe90751b5b..3c239682bbaf 100644
--- a/fs/netfs/read_pgpriv2.c
+++ b/fs/netfs/read_pgpriv2.c
@@ -13,8 +13,37 @@
#include <linux/task_io_accounting_ops.h>
#include "internal.h"
+int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *creq = subreq->rreq;
+ struct netfs_io_stream *stream = &creq->io_streams[1];
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ // TODO: Wait here for completion of prev subreq
+
+ stream->issue_from += subreq->len;
+ stream->buffered -= subreq->len;
+ if (stream->buffered == 0)
+ netfs_all_subreqs_queued(creq);
+ return 0;
+}
+
/*
- * [DEPRECATED] Copy a folio to the cache with PG_private_2 set.
+ * [DEPRECATED] Copy a folio to the cache with PG_private_2 set. Note that the
+ * folio won't necessarily be contiguous with the previous one as there might
+ * be a mixture of folios read from the cache and downloaded from the server
+ * (or just zeroed).
*/
static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio *folio)
{
@@ -24,7 +53,6 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
size_t dio_size = PAGE_SIZE;
size_t fsize = folio_size(folio), flen = fsize;
uoff_t fpos = folio_pos(folio), i_size;
- bool to_eof = false;
_enter("");
@@ -44,12 +72,8 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
if (fpos + fsize > creq->i_size)
creq->i_size = i_size;
- if (flen > i_size - fpos) {
+ if (flen > i_size - fpos)
flen = i_size - fpos;
- to_eof = true;
- } else if (flen == i_size - fpos) {
- to_eof = true;
- }
flen = round_up(flen, dio_size);
@@ -63,7 +87,7 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
*/
queue = creq->load_cursor.bvecq;
if (bvecq_is_full(queue) ||
- (fpos != creq->last_end && creq->last_end > 0 && queue->nr_slots > 0)) {
+ (fpos != cache->last_end && cache->last_end > 0 && queue->nr_slots > 0)) {
bvecq_buffer_append(&creq->load_cursor, creq->spare);
creq->spare = NULL;
@@ -78,38 +102,10 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
bvecq_filled_to(queue, slot);
creq->load_cursor.slot = slot;
creq->load_cursor.offset = 0;
- creq->last_end = fpos + flen;
-
- bvecq_pos_nudge(&creq->dispatch_cursor);
-
- cache->submit_off = 0;
- cache->submit_len = flen;
+ cache->last_end = fpos + flen;
+ trace_netfs_wback(creq, folio, 0);
- /* Attach the folio to one or more subrequests. For a big folio, we
- * could end up with thousands of subrequests if the wsize is small -
- * but we might need to wait during the creation of subrequests for
- * network resources (eg. SMB credits).
- */
- do {
- ssize_t part;
-
- creq->dispatch_cursor.offset = cache->submit_off;
-
- atomic64_set(&cache->issued_to, fpos + cache->submit_off);
- part = netfs_advance_write(creq, cache, fpos + cache->submit_off,
- cache->submit_len, to_eof);
- cache->submit_off += part;
- if (part > cache->submit_len)
- cache->submit_len = 0;
- else
- cache->submit_len -= part;
- } while (cache->submit_len > 0);
-
- bvecq_pos_step(&creq->dispatch_cursor);
- atomic64_set(&cache->issued_to, fpos + fsize);
-
- if (flen < fsize)
- netfs_issue_write(creq, cache);
+ cache->buffered += flen;
}
/*
@@ -119,6 +115,7 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache(
struct netfs_io_request *rreq, struct folio *folio)
{
struct netfs_io_request *creq;
+ struct netfs_io_stream *cache;
if (!fscache_resources_valid(&rreq->cache_resources))
goto cancel;
@@ -128,13 +125,14 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache(
if (IS_ERR(creq))
goto cancel;
- if (!creq->io_streams[1].avail)
+ cache = &creq->io_streams[1];
+ if (!cache->avail)
goto cancel_put;
if (bvecq_buffer_init(&creq->load_cursor, creq->gfp, false) < 0)
goto cancel_put;
- bvecq_pos_set(&creq->dispatch_cursor, &creq->load_cursor);
- bvecq_pos_set(&creq->collect_cursor, &creq->dispatch_cursor);
+ bvecq_pos_set(&cache->dispatch_cursor, &creq->load_cursor);
+ bvecq_pos_set(&creq->collect_cursor, &creq->load_cursor);
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &creq->flags);
trace_netfs_copy2cache(rreq, creq);
@@ -177,24 +175,61 @@ void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *fo
netfs_pgpriv2_copy_folio(creq, folio);
}
+/*
+ * Issue all pending writes on the cache stream.
+ */
+static void netfs_pgpriv2_issue_stream(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream)
+{
+ atomic64_set_release(&stream->issued_to, wreq->start);
+
+ do {
+ struct netfs_io_subrequest *subreq;
+ int ret;
+
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ if (!subreq)
+ break;
+
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. Punt the
+ * error to the collector and stop the issuance of new
+ * subreqs.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ stream->buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
+ break;
+ }
+ /* We no longer own subreq. */
+
+ if (test_bit(NETFS_RREQ_SAW_ENOMEM, &wreq->flags))
+ break;
+
+ } while (stream->buffered > 0);
+
+ netfs_all_subreqs_queued(wreq);
+}
+
/*
* [DEPRECATED] End writing to the cache, flushing out any outstanding writes.
*/
void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)
{
struct netfs_io_request *creq = rreq->copy_to_cache;
+ struct netfs_io_stream *stream = &creq->io_streams[1];
if (IS_ERR_OR_NULL(creq))
return;
- netfs_issue_write(creq, &creq->io_streams[1]);
- netfs_all_subreqs_queued(creq);
+ netfs_pgpriv2_issue_stream(creq, stream);
trace_netfs_rreq(rreq, netfs_rreq_trace_end_copy_to_cache);
if (list_empty_careful(&creq->io_streams[1].subrequests))
netfs_wake_collector(creq);
netfs_put_request(creq, netfs_rreq_trace_put_return);
- creq->copy_to_cache = NULL;
+ rreq->copy_to_cache = NULL;
}
/*
diff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c
index 733d9ff452c7..51ee780b11ed 100644
--- a/fs/netfs/read_retry.c
+++ b/fs/netfs/read_retry.c
@@ -9,20 +9,55 @@
#include <linux/slab.h>
#include "internal.h"
-static void netfs_reissue_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
+/*
+ * Prepare the I/O buffer on a buffered read subrequest for the filesystem to
+ * use as a bvec queue.
+ */
+int netfs_prepare_buffered_read_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
- bvecq_pos_unset(&subreq->content);
+ struct netfs_io_request *rreq = subreq->rreq;
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &rreq->retry_cursor);
bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
- iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
- subreq->content.slot, subreq->content.offset, subreq->len);
- iov_iter_advance(&subreq->io_iter, subreq->transferred);
+ len = bvecq_slice(&rreq->retry_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+ rreq->retry_buffered -= subreq->len;
+ rreq->retry_start += subreq->len;
+ return 0;
+}
- subreq->error = 0;
+/*
+ * Reset the state of the subrequest and discard any buffering so that we can
+ * retry (where this may include sending it to the server instead of the
+ * cache).
+ */
+int netfs_reset_for_read_retry(struct netfs_io_subrequest *subreq)
+{
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+
+ if (subreq->retry_count > 3) {
+ trace_netfs_sreq(subreq, netfs_sreq_trace_too_many_retries);
+ return subreq->error;
+ }
+
+ subreq->retry_count++;
__clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ __clear_bit(NETFS_SREQ_FAILED, &subreq->flags);
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
- netfs_stat(&netfs_n_rh_retry_read_subreq);
- subreq->rreq->netfs_ops->issue_read(subreq);
+ bvecq_pos_unset(&subreq->content);
+ bvecq_pos_unset(&subreq->dispatch_pos);
+ subreq->error = 0;
+ subreq->transferred = 0;
+ netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
+ netfs_stat(&netfs_n_wh_retry_write_subreq);
+ return 0;
}
/*
@@ -33,8 +68,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
{
struct netfs_io_subrequest *subreq;
struct netfs_io_stream *stream = &rreq->io_streams[0];
- struct bvecq_pos dispatch_cursor = {};
struct list_head *next;
+ int ret;
_enter("R=%x", rreq->debug_id);
@@ -44,46 +79,19 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
if (rreq->netfs_ops->retry_request)
rreq->netfs_ops->retry_request(rreq, NULL);
- /* If there's no renegotiation to do, just resend each retryable subreq
- * up to the first permanently failed one.
- */
- if (!rreq->netfs_ops->prepare_read &&
- !rreq->cache_resources.ops) {
- list_for_each_entry(subreq, &stream->subrequests, rreq_link) {
- if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
- break;
- if (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
- subreq->retry_count++;
- netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_read(rreq, subreq);
- }
- }
- return;
- }
+ /* Read pointer to subreq before reading subreq state. */
+ next = smp_load_acquire(&stream->subrequests.next);
- /* Okay, we need to renegotiate all the download requests and flip any
- * failed cache reads over to being download requests and negotiate
- * those also. All fully successful subreqs have been removed from the
- * list and any spare data from those has been donated.
- *
- * What we do is decant the list and rebuild it one subreq at a time so
- * that we don't end up with donations jumping over a gap we're busy
- * populating with smaller subrequests. In the event that the subreq
- * we just launched finishes before we insert the next subreq, it'll
- * fill in rreq->prev_donated instead.
- *
- * Note: Alternatively, we could split the tail subrequest right before
- * we reissue it and fix up the donations under lock.
+ /* Renegotiate all the download requests and flip any failed cache
+ * reads over to being download requests and negotiate those also.
*/
- next = stream->subrequests.next;
-
do {
struct netfs_io_subrequest *from, *to, *tmp;
- uoff_t start, len;
- size_t part;
- bool boundary = false, subreq_superfluous = false;
+ uoff_t start;
+ size_t len;
+ bool subreq_superfluous = false;
- bvecq_pos_unset(&dispatch_cursor);
+ bvecq_pos_unset(&rreq->retry_cursor);
/* Go through the subreqs and find the next span of contiguous
* buffer that we then rejig (cifs, for example, needs the
@@ -98,8 +106,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
rreq->debug_id, from->debug_index,
from->start, from->transferred, from->len);
- if (test_bit(NETFS_SREQ_FAILED, &from->flags) ||
- !test_bit(NETFS_SREQ_NEED_RETRY, &from->flags)) {
+ if (!test_bit(NETFS_SREQ_NEED_RETRY, &from->flags)) {
subreq = from;
goto abandon;
}
@@ -113,20 +120,21 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
subreq = list_entry(next, struct netfs_io_subrequest, rreq_link);
if (subreq->start != start + len ||
subreq->transferred > 0 ||
- test_bit(NETFS_SREQ_BOUNDARY, &subreq->flags) ||
!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
break;
to = subreq;
len += to->len;
}
- _debug(" - range: %llx-%llx %llx", start, start + len - 1, len);
+ _debug(" - range: %llx-%llx %zx", start, start + len - 1, len);
/* Determine the set of buffers we're going to use. Each
- * subreq gets a subset of a single overall contiguous buffer.
+ * subreq takes a subset of a single overall contiguous buffer.
*/
- bvecq_pos_transfer(&dispatch_cursor, &from->dispatch_pos);
- bvecq_pos_advance(&dispatch_cursor, from->transferred);
+ bvecq_pos_transfer(&rreq->retry_cursor, &from->dispatch_pos);
+ bvecq_pos_advance(&rreq->retry_cursor, from->transferred);
+ rreq->retry_start = start;
+ rreq->retry_buffered = len;
from->transferred = 0;
/* Work through the sublist. The chain of buffers we're going
@@ -135,51 +143,30 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
*/
subreq = from;
list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) {
- if (!len) {
+ if (rreq->retry_buffered == 0) {
subreq_superfluous = true;
break;
}
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- subreq->start = start;
- subreq->len = len;
- __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
- __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
- subreq->retry_count++;
- subreq->transferred = 0;
+ subreq->start = rreq->retry_start;
+ subreq->len = rreq->retry_buffered;
- bvecq_pos_unset(&subreq->content);
- bvecq_pos_unset(&subreq->dispatch_pos);
- bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
-
- /* Renegotiate max_len (rsize) */
- stream->sreq_max_len = len;
- stream->sreq_max_segs = INT_MAX;
- if (rreq->netfs_ops->prepare_read &&
- rreq->netfs_ops->prepare_read(subreq) < 0) {
- trace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed);
+ ret = netfs_reset_for_read_retry(subreq);
+ if (ret < 0) {
__set_bit(NETFS_SREQ_FAILED, &subreq->flags);
+ rreq->error = ret;
goto abandon;
}
- part = bvecq_slice(&dispatch_cursor,
- umin(len, stream->sreq_max_len),
- stream->sreq_max_segs,
- &subreq->nr_segs);
- subreq->len = part;
-
- len -= part;
- start += part;
- if (!len) {
- if (boundary)
- __set_bit(NETFS_SREQ_BOUNDARY, &subreq->flags);
- } else {
- __clear_bit(NETFS_SREQ_BOUNDARY, &subreq->flags);
+ netfs_stat(&netfs_n_rh_download);
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ goto abandon_after;
}
-
- netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_read(rreq, subreq);
+ if (test_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags))
+ goto abandon_after;
if (subreq == to) {
subreq_superfluous = false;
break;
@@ -189,7 +176,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
/* If we managed to use fewer subreqs, we can discard the
* excess; if we used the same number, then we're done.
*/
- if (!len) {
+ if (rreq->retry_buffered == 0) {
if (!subreq_superfluous)
continue;
list_for_each_entry_safe_from(subreq, tmp,
@@ -207,7 +194,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
}
/* We ran out of subrequests, so we need to allocate some more
- * and insert them after.
+ * and insert them after. They must start with being marked
+ * for retry to switch to the retry cursor.
*/
do {
subreq = netfs_alloc_subrequest(rreq);
@@ -216,8 +204,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
goto abandon_after;
}
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- subreq->start = start;
- subreq->len = len;
+ subreq->start = rreq->retry_start;
+ subreq->len = rreq->retry_buffered;
subreq->stream_nr = stream->stream_nr;
subreq->retry_count = 1;
@@ -225,43 +213,32 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
refcount_read(&subreq->ref),
netfs_sreq_trace_new);
+ __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
+
spin_lock(&rreq->lock);
+ /* Write IN_PROGRESS before pointer to new subreq */
+ smp_wmb();
list_add(&subreq->rreq_link, &to->rreq_link);
spin_unlock(&rreq->lock);
to = subreq;
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- stream->sreq_max_len = umin(len, rreq->rsize);
- stream->sreq_max_segs = INT_MAX;
-
netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read(subreq) < 0) {
- trace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed);
- __set_bit(NETFS_SREQ_FAILED, &subreq->flags);
- goto abandon;
- }
-
- bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
- part = bvecq_slice(&dispatch_cursor,
- umin(len, stream->sreq_max_len),
- stream->sreq_max_segs,
- &subreq->nr_segs);
- subreq->len = part;
-
- len -= part;
- start += part;
- if (!len && boundary) {
- __set_bit(NETFS_SREQ_BOUNDARY, &to->flags);
- boundary = false;
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ goto abandon_after;
}
+ if (test_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags))
+ goto abandon_after;
- netfs_reissue_read(rreq, subreq);
- } while (len);
+ } while (rreq->retry_buffered > 0);
} while (!list_is_head(next, &stream->subrequests));
out:
- bvecq_pos_unset(&dispatch_cursor);
+ bvecq_pos_unset(&rreq->retry_cursor);
return;
/* If we hit an error, fail all remaining incomplete subrequests */
@@ -334,6 +311,7 @@ void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq)
}
trace_netfs_folio(folio, netfs_folio_trace_abandon);
folio_unlock(folio);
+ p->bv[slot].bv_page = NULL;
}
}
}
diff --git a/fs/netfs/read_single.c b/fs/netfs/read_single.c
index 9fd07dbb08b5..4d387db98668 100644
--- a/fs/netfs/read_single.c
+++ b/fs/netfs/read_single.c
@@ -16,6 +16,22 @@
#include <linux/netfs.h>
#include "internal.h"
+int netfs_prepare_read_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *rreq = subreq->rreq;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+
+ bvecq_pos_set(&subreq->dispatch_pos, &rreq->load_cursor);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+
+ stream->buffered = 0;
+ stream->issue_from += subreq->len;
+ rreq->submitted = stream->issue_from;
+ netfs_all_subreqs_queued(rreq);
+ return 0;
+}
+
/**
* netfs_single_mark_inode_dirty - Mark a single, monolithic object inode dirty
* @inode: The inode to mark
@@ -58,17 +74,6 @@ static int netfs_single_begin_cache_read(struct netfs_io_request *rreq, struct n
return fscache_begin_read_operation(&rreq->cache_resources, netfs_i_cookie(ctx));
}
-static void netfs_single_read_cache(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
-{
- struct netfs_cache_resources *cres = &rreq->cache_resources;
-
- _enter("R=%08x[%x]", rreq->debug_id, subreq->debug_index);
- netfs_stat(&netfs_n_rh_read);
- cres->ops->read(cres, subreq->start, &subreq->io_iter, NETFS_READ_HOLE_FAIL,
- netfs_cache_read_terminated, subreq);
-}
-
/*
* Perform a read to a buffer from the cache or the server. Only a single
* subreq is permitted as the object must be fetched in a single transaction.
@@ -84,70 +89,84 @@ static int netfs_single_dispatch_read(struct netfs_io_request *rreq)
.cached_to[1] = ULLONG_MAX,
};
struct netfs_io_subrequest *subreq;
- int ret = 0;
+ int ret;
+
+ netfs_read_query_cache(rreq, &occ);
- subreq = netfs_alloc_subrequest(rreq);
+ subreq = netfs_alloc_read_subrequest(rreq);
if (!subreq)
return -ENOMEM;
- subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
subreq->start = 0;
subreq->len = rreq->len;
- bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor);
- bvecq_pos_set(&subreq->content, &rreq->dispatch_cursor);
-
- iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
- subreq->content.slot, subreq->content.offset, subreq->len);
-
- netfs_queue_read(rreq, subreq);
+ trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
/* Try to use the cache if the cache content matches the size of the
* remote file.
*/
- netfs_read_query_cache(rreq, &occ);
if (occ.cached_from[0] == 0 &&
- occ.cached_to[0] >= rreq->len)
- subreq->source = NETFS_READ_FROM_CACHE;
+ occ.cached_to[0] >= rreq->len) {
+ struct netfs_cache_resources *cres = &rreq->cache_resources;
- switch (subreq->source) {
- case NETFS_DOWNLOAD_FROM_SERVER:
- netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read) {
- ret = rreq->netfs_ops->prepare_read(subreq);
- if (ret < 0)
- goto cancel;
+ subreq->source = NETFS_READ_FROM_CACHE;
+ netfs_stat(&netfs_n_rh_read);
+ ret = cres->ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
}
- netfs_all_subreqs_queued(rreq);
- rreq->netfs_ops->issue_read(subreq);
- rreq->submitted += subreq->len;
- break;
- case NETFS_READ_FROM_CACHE:
- if (rreq->cache_resources.ops->prepare_read) {
- ret = rreq->cache_resources.ops->prepare_read(subreq);
- if (ret < 0)
- goto cancel;
+ ret = netfs_wait_for_in_progress_subreq(rreq, subreq);
+ if (ret == 0)
+ goto success;
+ if (ret == -ENOMEM)
+ goto cancel;
+
+ /* Didn't manage to retrieve from the cache, so toss it to the
+ * server instead.
+ */
+ if (netfs_reset_for_read_retry(subreq) < 0)
+ goto cancel;
+ }
+
+ __set_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags);
+
+ /* Try to send it to the cache. */
+ for (;;) {
+ subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
+ netfs_stat(&netfs_n_rh_download);
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
}
- netfs_all_subreqs_queued(rreq);
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- netfs_single_read_cache(rreq, subreq);
- rreq->submitted += subreq->len;
- ret = 0;
- break;
- default:
- pr_warn("Unexpected single-read source %u\n", subreq->source);
- WARN_ON_ONCE(true);
- ret = -EIO;
- goto cancel;
+ ret = netfs_wait_for_in_progress_subreq(rreq, subreq);
+ if (ret == 0)
+ goto success;
+ if (ret == -ENOMEM)
+ goto cancel;
+ if (ret != -EAGAIN)
+ goto failed;
+ if (netfs_reset_for_read_retry(subreq) < 0)
+ goto cancel;
}
- return ret;
+success:
+ rreq->transferred = subreq->transferred;
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_consumed);
+ return 0;
cancel:
- netfs_cancel_read(subreq, ret);
- netfs_all_subreqs_queued(rreq);
- netfs_wake_collector(rreq);
+ rreq->error = ret;
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_cancel);
+ return ret;
+failed:
+ rreq->error = ret;
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed);
return ret;
}
@@ -179,7 +198,7 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite
if (IS_ERR(rreq))
return PTR_ERR(rreq);
- ret = netfs_extract_iter(iter, rreq->len, INT_MAX, &rreq->dispatch_cursor.bvecq,
+ ret = netfs_extract_iter(iter, rreq->len, INT_MAX, &rreq->load_cursor.bvecq,
0, rreq->gfp);
if (ret < 0)
goto cleanup_free;
@@ -197,9 +216,29 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite
netfs_stat(&netfs_n_rh_read_single);
trace_netfs_read(rreq, 0, rreq->len, netfs_read_trace_read_single);
- netfs_single_dispatch_read(rreq);
+ ret = netfs_single_dispatch_read(rreq);
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_complete);
+ if (ret == 0) {
+ task_io_account_read(rreq->transferred);
+
+ if (test_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags) &&
+ fscache_resources_valid(&rreq->cache_resources)) {
+ trace_netfs_rreq(rreq, netfs_rreq_trace_dirty);
+ netfs_single_mark_inode_dirty(rreq->inode);
+ }
+ ret = rreq->transferred;
+ }
+
+ if (rreq->netfs_ops->done)
+ rreq->netfs_ops->done(rreq);
+
+ netfs_wake_rreq_flag(rreq, NETFS_RREQ_IN_PROGRESS, netfs_rreq_trace_wake_ip);
+ /* As we cleared NETFS_RREQ_IN_PROGRESS, we acquired its ref. */
+ netfs_put_request(rreq, netfs_rreq_trace_put_work_ip);
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_done);
- ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret;
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index 0c71f8c151c2..e0d14d77b751 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -68,6 +68,9 @@ struct netfs_wb_params {
struct netfs_write_estimate estimates[NR_IO_STREAMS];
};
+static int netfs_prepare_write_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
/*
* Kill all dirty folios in the event of an unrecoverable error, starting with
* a locked folio we've already obtained from writeback_iter().
@@ -148,7 +151,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
wreq->io_streams[0].source = NETFS_UPLOAD_TO_SERVER;
wreq->io_streams[0].applicable = NOTE_UPLOAD;
wreq->io_streams[0].estimate_write = ictx->ops->estimate_write;
- wreq->io_streams[0].prepare_write = ictx->ops->prepare_write;
wreq->io_streams[0].issue_write = ictx->ops->issue_write;
wreq->io_streams[0].collected_to = start;
wreq->io_streams[0].transferred = 0;
@@ -162,7 +164,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
wreq->io_streams[1].avail = true;
wreq->io_streams[1].active = true;
wreq->io_streams[1].estimate_write = wreq->cache_resources.ops->estimate_write;
- wreq->io_streams[1].prepare_write = wreq->cache_resources.ops->prepare_write_subreq;
wreq->io_streams[1].issue_write = wreq->cache_resources.ops->issue_write;
wreq->io_streams[1].alignment = wreq->cache_resources.dio_size;
}
@@ -170,19 +171,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
return wreq;
}
-/**
- * netfs_prepare_write_failed - Note write preparation failed
- * @subreq: The subrequest to mark
- *
- * Mark a subrequest to note that preparation for write failed.
- */
-void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq)
-{
- __set_bit(NETFS_SREQ_FAILED, &subreq->flags);
- trace_netfs_sreq(subreq, netfs_sreq_trace_prep_failed);
-}
-EXPORT_SYMBOL(netfs_prepare_write_failed);
-
/*
* Allocate and prepare a write subrequest. Will only return NULL if not
* performing writeback; if performing writeback, mempools may be accessed and
@@ -235,6 +223,7 @@ struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wr
return subreq;
}
+#if 0 // TODO: Remove old stuff
/*
* Prepare a write subrequest. We need to allocate a new subrequest
* if we don't have one.
@@ -295,6 +284,7 @@ void netfs_prepare_write(struct netfs_io_request *wreq,
stream->construct = subreq;
}
+#endif
/*
* Advance the state of the amount of data buffered on a stream.
@@ -341,6 +331,44 @@ static int netfs_prepare_buffered_write_buffer(struct netfs_io_subrequest *subre
return 0;
}
+/**
+ * netfs_prepare_write_buffer - Get the buffer for a subrequest
+ * @subreq: The subrequest to get the buffer for
+ * @max_segs: Maximum number of segments in buffer (or INT_MAX)
+ *
+ * Extract a slice of buffer from the stream and attach it to the subrequest as
+ * a bio_vec queue. The maximum amount of data attached is set by
+ * @subreq->len, but this may be shortened if @max_segs would be exceeded.
+ */
+int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *rreq = subreq->rreq;
+
+ switch (rreq->origin) {
+ case NETFS_WRITEBACK:
+ if (test_bit(NETFS_RREQ_RETRYING, &rreq->flags))
+ return netfs_prepare_write_retry_buffer(subreq, max_segs);
+ return netfs_prepare_buffered_write_buffer(subreq, max_segs);
+
+ case NETFS_UNBUFFERED_WRITE:
+ case NETFS_DIO_WRITE:
+ return netfs_prepare_unbuffered_write_buffer(subreq, max_segs);
+
+ case NETFS_WRITEBACK_SINGLE:
+ return netfs_prepare_write_single_buffer(subreq, max_segs);
+
+ case NETFS_PGPRIV2_COPY_TO_CACHE:
+ return netfs_prepare_pgpriv2_write_buffer(subreq, max_segs);
+
+ default:
+ WARN_ON_ONCE(1);
+ return -EIO;
+ }
+}
+EXPORT_SYMBOL(netfs_prepare_write_buffer);
+
+#if 0 // TODO: Remove old stuff
/*
* Set the I/O iterator for the filesystem/cache to use and dispatch the I/O
* operation. The operation may be asynchronous and should call
@@ -395,8 +423,8 @@ void netfs_reissue_write(struct netfs_io_stream *stream,
netfs_do_issue_write(stream, subreq);
}
-void netfs_issue_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream)
+static void netfs_issue_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream)
{
struct netfs_io_subrequest *subreq = stream->construct;
@@ -491,6 +519,7 @@ static int netfs_prep_and_issue_subreq(struct netfs_io_request *wreq,
stream->issue_write(subreq);
return 0;
}
+#endif
/*
* Issue writes for a stream.
@@ -523,7 +552,7 @@ static void netfs_writeback_flush(struct netfs_io_request *wreq,
return;
}
- ret = netfs_prep_and_issue_subreq(wreq, stream, subreq);
+ ret = stream->issue_write(subreq);
if (ret < 0) {
/* Ownership of subreq was returned to us. */
trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
@@ -649,6 +678,7 @@ static void netfs_writeback_add_folio_to_stream(struct netfs_io_request *wreq,
wreq->load_cursor.slot--;
trace_netfs_bv_slot(wreq->load_cursor.bvecq, wreq->load_cursor.slot - 1);
+ trace_netfs_wback(wreq, folio, params->notes);
for (int s = 0; s < NR_IO_STREAMS; s++) {
struct netfs_io_stream *stream = &wreq->io_streams[s];
@@ -846,31 +876,6 @@ static void netfs_writeback_folio(struct netfs_io_request *wreq,
goto out;
}
-#if 0 // TODO: Remove
-/*
- * End the issuing of writes, letting the collector know we're done.
- */
-static void netfs_end_issue_write(struct netfs_io_request *wreq)
-{
- bool needs_poke = true;
-
- netfs_all_subreqs_queued(wreq);
-
- for (int s = 0; s < NR_IO_STREAMS; s++) {
- struct netfs_io_stream *stream = &wreq->io_streams[s];
-
- if (!stream->active)
- continue;
- if (!list_empty(&stream->subrequests))
- needs_poke = false;
- netfs_issue_write(wreq, stream);
- }
-
- if (needs_poke)
- netfs_wake_collector(wreq);
-}
-#endif
-
/*
* Write some of the pending data back to the server
*/
@@ -957,6 +962,25 @@ int netfs_writepages(struct address_space *mapping,
}
EXPORT_SYMBOL(netfs_writepages);
+/*
+ * Prepare a buffer for a single monolithic write.
+ */
+static int netfs_prepare_write_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *wreq = subreq->rreq;
+ struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+
+ stream->buffered = 0;
+ stream->issue_from = subreq->len;
+ wreq->submitted = subreq->len;
+ netfs_all_subreqs_queued(wreq);
+ return 0;
+}
+
/**
* netfs_writeback_single - Write back a monolithic payload
* @mapping: The mapping to write from
@@ -968,6 +992,11 @@ EXPORT_SYMBOL(netfs_writepages);
* cache. There's a maximum of one subrequest per stream. The buffer should
* be rounded out sufficiently that it can accommodate cache DIO rounding.
*
+ * This is normally only used to write to the cache (for AFS directories and
+ * symlinks); it doesn't normally write to the server as well. The filesystem
+ * can override that by setting NETFS_RREQ_UPLOAD_TO_SERVER when the request is
+ * initialised.
+ *
* Return: 0 if successful; 1 if skipped due to lock conflict and WB_SYNC_NONE;
* or a negative error code.
* the cache. There's a maximum of one subrequest per stream.
@@ -981,6 +1010,11 @@ int netfs_writeback_single(struct address_space *mapping,
size_t clen;
int ret;
+ _enter("%zx,%zx", iov_iter_count(iter), len);
+
+ if (!len)
+ return 0;
+
if (!netfs_wb_begin(ictx, wbc->sync_mode == WB_SYNC_NONE)) {
/* The VFS will have undirtied the inode. */
netfs_single_mark_inode_dirty(&ictx->inode);
@@ -992,9 +1026,9 @@ int netfs_writeback_single(struct address_space *mapping,
ret = PTR_ERR(wreq);
goto couldnt_start;
}
+
wreq->len = len;
clen = len;
-
if (wreq->cache_resources.dio_size > 1) {
clen = round_up(len, wreq->cache_resources.dio_size);
if (clen > iov_iter_count(iter)) {
@@ -1003,7 +1037,7 @@ int netfs_writeback_single(struct address_space *mapping,
}
}
- ret = netfs_extract_iter(iter, clen, INT_MAX, &wreq->dispatch_cursor.bvecq,
+ ret = netfs_extract_iter(iter, clen, INT_MAX, &wreq->load_cursor.bvecq,
0, wreq->gfp);
if (ret < 0)
goto cleanup_free;
@@ -1012,12 +1046,14 @@ int netfs_writeback_single(struct address_space *mapping,
goto cleanup_free;
}
- bvecq_pos_set(&wreq->collect_cursor, &wreq->dispatch_cursor);
-
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags);
trace_netfs_write(wreq, netfs_write_trace_writeback_single);
netfs_stat(&netfs_n_wh_writepages);
+ /* This normally just writes to the cache; if the filesystem wants to
+ * write to the server too, it must set UPLOAD_TO_SERVER in
+ * ->init_request().
+ */
if (test_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags))
wreq->netfs_ops->begin_writeback(wreq);
@@ -1028,14 +1064,29 @@ int netfs_writeback_single(struct address_space *mapping,
if (!stream->avail)
continue;
- netfs_prepare_write(wreq, stream, 0);
-
- subreq = stream->construct;
- subreq->len = wreq->len;
+ stream->issue_from = 0;
+ stream->buffered = len;
if (stream->source == NETFS_WRITE_TO_CACHE)
- subreq->len = clen;
+ stream->buffered = clen;
- netfs_issue_write(wreq, stream);
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ if (!subreq) {
+ ret = -ENOMEM;
+ break;
+ }
+
+ bvecq_pos_set(&stream->dispatch_cursor, &wreq->load_cursor);
+
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ stream->buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
+ /* Punt the error to the collector. */
+ }
+
+ bvecq_pos_unset(&stream->dispatch_cursor);
}
wreq->submitted = wreq->len;
diff --git a/fs/netfs/write_retry.c b/fs/netfs/write_retry.c
index c61bed687244..153c180f4fed 100644
--- a/fs/netfs/write_retry.c
+++ b/fs/netfs/write_retry.c
@@ -12,13 +12,44 @@
#include "internal.h"
/*
- * Perform retries on the streams that need it.
+ * Prepare the write buffer for a retry. We can't necessarily reuse the write
+ * buffer from the previous run of a subrequest because the filesystem is
+ * permitted to modify it (add headers/trailers, encrypt it). Further, the
+ * subrequest may now be a different size (e.g. cifs has to negotiate for
+ * maximum transfer size). Also, we can't look at *stream as that may still
+ * refer to the source material being broken up into original subrequests.
+ */
+int netfs_prepare_write_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *wreq = subreq->rreq;
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &wreq->retry_cursor);
+ bvecq_pos_set(&subreq->content, &wreq->retry_cursor);
+ len = bvecq_slice(&wreq->retry_cursor, subreq->len, max_segs, &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ wreq->retry_start += len;
+ wreq->retry_buffered -= len;
+ if (wreq->retry_buffered == 0)
+ bvecq_pos_unset(&wreq->retry_cursor);
+ return 0;
+}
+
+/*
+ * Perform retries on the streams that need it. This only has to deal with
+ * buffered writes; unbuffered write retry is handled in direct_write.c.
*/
static void netfs_retry_write_stream(struct netfs_io_request *wreq,
struct netfs_io_stream *stream)
{
- struct bvecq_pos dispatch_cursor = {};
struct list_head *next;
+ int ret;
_enter("R=%x[%x:]", wreq->debug_id, stream->stream_nr);
@@ -33,30 +64,15 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
if (unlikely(stream->failed))
return;
- /* If there's no renegotiation to do, just resend each failed subreq. */
- if (!stream->prepare_write) {
- struct netfs_io_subrequest *subreq;
-
- list_for_each_entry(subreq, &stream->subrequests, rreq_link) {
- if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
- break;
- if (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
- netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_write(stream, subreq);
- }
- }
- return;
- }
-
- next = stream->subrequests.next;
+ /* Read pointer to subreq before reading subreq state. */
+ next = smp_load_acquire(&stream->subrequests.next);
do {
struct netfs_io_subrequest *subreq = NULL, *from, *to, *tmp;
uoff_t start, len;
- size_t part;
- bool boundary = false;
+ bool subreq_superfluous = false;
- bvecq_pos_unset(&dispatch_cursor);
+ bvecq_pos_unset(&wreq->retry_cursor);
/* Go through the stream and find the next span of contiguous
* data that we then rejig (cifs, for example, needs the wsize
@@ -80,7 +96,6 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
subreq = list_entry(next, struct netfs_io_subrequest, rreq_link);
if (subreq->start != start + len ||
subreq->transferred > 0 ||
- test_bit(NETFS_SREQ_BOUNDARY, &subreq->flags) ||
!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
break;
to = subreq;
@@ -90,8 +105,10 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
/* Determine the set of buffers we're going to use. Each
* subreq gets a subset of a single overall contiguous buffer.
*/
- bvecq_pos_transfer(&dispatch_cursor, &from->dispatch_pos);
- bvecq_pos_advance(&dispatch_cursor, from->transferred);
+ bvecq_pos_transfer(&wreq->retry_cursor, &from->dispatch_pos);
+ bvecq_pos_advance(&wreq->retry_cursor, from->transferred);
+ wreq->retry_start = start;
+ wreq->retry_buffered = len;
/* Work through the sublist. The chain of buffers we're going
* to fill is attached to dispatch_cursor and we need to read
@@ -99,47 +116,50 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
*/
subreq = from;
list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) {
- if (!len)
+ if (!wreq->retry_buffered) {
+ subreq_superfluous = true;
break;
-
- subreq->start = start;
- subreq->len = len;
- __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
- trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- subreq->transferred = 0;
+ }
bvecq_pos_unset(&subreq->content);
bvecq_pos_unset(&subreq->dispatch_pos);
- /* Renegotiate max_len (wsize) */
- stream->sreq_max_len = len;
- stream->sreq_max_segs = INT_MAX;
- stream->prepare_write(subreq);
-
- bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
- part = bvecq_slice(&dispatch_cursor,
- umin(len, stream->sreq_max_len),
- stream->sreq_max_segs,
- &subreq->nr_segs);
- subreq->len = part;
-
- len -= part;
- start += part;
- if (len && subreq == to &&
- __test_and_clear_bit(NETFS_SREQ_BOUNDARY, &to->flags))
- boundary = true;
-
+ __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ __clear_bit(NETFS_SREQ_FAILED, &subreq->flags);
+ __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
+ subreq->start = wreq->retry_start;
+ subreq->len = wreq->retry_buffered;
+ subreq->transferred = 0;
+ subreq->retry_count += 1;
+ subreq->error = 0;
+
+ netfs_stat(&netfs_n_wh_retry_write_subreq);
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_write(stream, subreq);
- if (subreq == to)
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us.
+ * Expand the subreq to consume the entire
+ * remaining amount to be retried and fail it.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ wreq->retry_buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
+ break;
+ }
+
+ if (subreq == to) {
+ subreq_superfluous = false;
break;
+ }
}
/* If we managed to use fewer subreqs, we can discard the
* excess; if we used the same number, then we're done.
*/
- if (!len) {
- if (subreq == to)
+ if (!wreq->retry_buffered) {
+ if (!subreq_superfluous)
continue;
list_for_each_entry_safe_from(subreq, tmp,
&stream->subrequests, rreq_link) {
@@ -161,6 +181,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
subreq = netfs_alloc_subrequest(wreq);
subreq->source = to->source;
subreq->start = start;
+ subreq->len = wreq->retry_buffered;
subreq->stream_nr = to->stream_nr;
subreq->retry_count = 1;
@@ -169,49 +190,40 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
netfs_sreq_trace_new);
trace_netfs_sreq(subreq, netfs_sreq_trace_split);
+ __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
spin_lock(&wreq->lock);
+ /* Write IN_PROGRESS before pointer to new subreq */
+ smp_wmb();
list_add(&subreq->rreq_link, &to->rreq_link);
spin_unlock(&wreq->lock);
to = subreq;
- trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- stream->sreq_max_len = len;
- stream->sreq_max_segs = INT_MAX;
switch (stream->source) {
case NETFS_UPLOAD_TO_SERVER:
netfs_stat(&netfs_n_wh_upload);
- stream->sreq_max_len = umin(len, wreq->wsize);
break;
default:
WARN_ON_ONCE(1);
}
- stream->prepare_write(subreq);
-
- bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
- part = bvecq_slice(&dispatch_cursor,
- umin(len, stream->sreq_max_len),
- stream->sreq_max_segs,
- &subreq->nr_segs);
- subreq->len = subreq->transferred + part;
-
- len -= part;
- start += part;
- if (!len && boundary) {
- __set_bit(NETFS_SREQ_BOUNDARY, &to->flags);
- boundary = false;
- }
-
- netfs_reissue_write(stream, subreq);
- if (!len)
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us.
+ * Expand the subreq to consume the entire
+ * remaining amount to be retried and fail it.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ wreq->retry_buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
break;
-
- } while (len);
+ }
+ } while (wreq->retry_buffered > 0);
} while (!list_is_head(next, &stream->subrequests));
out:
- bvecq_pos_unset(&dispatch_cursor);
+ bvecq_pos_unset(&wreq->retry_cursor);
}
/*
diff --git a/fs/nfs/fscache.c b/fs/nfs/fscache.c
index 9b7fdad4a920..fb1441f88661 100644
--- a/fs/nfs/fscache.c
+++ b/fs/nfs/fscache.c
@@ -23,6 +23,7 @@
#include "iostat.h"
#include "fscache.h"
#include "nfstrace.h"
+#include <trace/events/netfs.h>
#define NFS_MAX_KEY_LEN 1000
@@ -273,8 +274,6 @@ static int nfs_netfs_init_request(struct netfs_io_request *rreq, struct file *fi
rreq->debug_id = atomic_inc_return(&nfs_netfs_debug_id);
/* [DEPRECATED] Use PG_private_2 to mark folio being written to the cache. */
__set_bit(NETFS_RREQ_USE_PGPRIV2, &rreq->flags);
- rreq->io_streams[0].sreq_max_len = NFS_SB(rreq->inode->i_sb)->rsize;
-
return 0;
}
@@ -296,8 +295,9 @@ static struct nfs_netfs_io_data *nfs_netfs_alloc(struct netfs_io_subrequest *sre
return netfs;
}
-static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
+static int nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
{
+ struct netfs_io_request *rreq = sreq->rreq;
struct nfs_netfs_io_data *netfs;
struct nfs_pageio_descriptor pgio;
struct inode *inode = sreq->rreq->inode;
@@ -307,6 +307,16 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
pgoff_t start, last;
int err;
+ if (sreq->len > NFS_SB(rreq->inode->i_sb)->rsize)
+ sreq->len = NFS_SB(rreq->inode->i_sb)->rsize;
+
+ err = netfs_prepare_read_buffer(sreq, INT_MAX);
+ if (err < 0) {
+ sreq->error = err;
+ return err;
+ }
+ /* After this point, must fail by termination. */
+
start = (sreq->start + sreq->transferred) >> PAGE_SHIFT;
last = ((sreq->start + sreq->len - sreq->transferred - 1) >> PAGE_SHIFT);
@@ -316,12 +326,15 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
netfs = nfs_netfs_alloc(sreq);
if (!netfs) {
sreq->error = -ENOMEM;
- return netfs_read_subreq_terminated(sreq);
+ netfs_read_subreq_terminated(sreq);
+ return 0;
}
+ trace_netfs_sreq(sreq, netfs_sreq_trace_submit);
+
pgio.pg_netfs = netfs; /* used in completion */
- xa_for_each_range(&sreq->rreq->mapping->i_pages, idx, page, start, last) {
+ xa_for_each_range(&rreq->mapping->i_pages, idx, page, start, last) {
/* nfs_read_add_folio() may schedule() due to pNFS layout and other RPCs */
err = nfs_read_add_folio(&pgio, ctx, page_folio(page));
if (err < 0) {
@@ -332,6 +345,7 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
out:
nfs_pageio_complete_read(&pgio);
nfs_netfs_put(netfs);
+ return 0;
}
void nfs_netfs_initiate_read(struct nfs_pgio_header *hdr)
diff --git a/fs/smb/client/cifssmb.c b/fs/smb/client/cifssmb.c
index 1f77512252e7..37d576d38837 100644
--- a/fs/smb/client/cifssmb.c
+++ b/fs/smb/client/cifssmb.c
@@ -1467,8 +1467,7 @@ cifs_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)
struct cifs_tcon *tcon = tlink_tcon(rdata->req->cfile->tlink);
struct inode *inode = &ictx->inode;
struct smb_rqst rqst = { .rq_iov = rdata->iov,
- .rq_nvec = 1,
- .rq_iter = rdata->subreq.io_iter };
+ .rq_nvec = 1};
struct cifs_credits credits = {
.value = 1,
.instance = 0,
@@ -1482,6 +1481,11 @@ cifs_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)
__func__, mid->mid, mid->mid_state, rdata->result,
rdata->subreq.len);
+ if (rdata->got_bytes)
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
+
switch (mid->mid_state) {
case MID_RESPONSE_RECEIVED:
/* result already set, check signature */
@@ -2007,7 +2011,10 @@ cifs_async_writev(struct cifs_io_subrequest *wdata)
rqst.rq_iov = iov;
rqst.rq_nvec = 1;
- rqst.rq_iter = wdata->subreq.io_iter;
+
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_SOURCE,
+ wdata->subreq.content.bvecq, wdata->subreq.content.slot,
+ wdata->subreq.content.offset, wdata->subreq.len);
cifs_dbg(FYI, "async write at %llu %zu bytes\n",
wdata->subreq.start, wdata->subreq.len);
diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c
index 7ad464782439..54db4e3a83fd 100644
--- a/fs/smb/client/file.c
+++ b/fs/smb/client/file.c
@@ -53,21 +53,23 @@ static int cifs_estimate_write(struct netfs_io_request *wreq,
}
/*
- * Prepare a subrequest to upload to the server. We need to allocate credits
- * so that we know the maximum amount of data that we can include in it.
+ * Issue a subrequest to upload to the server.
*/
-static void cifs_prepare_write(struct netfs_io_subrequest *subreq)
+static int cifs_issue_write(struct netfs_io_subrequest *subreq)
{
struct cifs_io_subrequest *wdata =
container_of(subreq, struct cifs_io_subrequest, subreq);
struct cifs_io_request *req = wdata->req;
- struct netfs_io_stream *stream = &req->rreq.io_streams[subreq->stream_nr];
struct TCP_Server_Info *server;
struct cifsFileInfo *open_file = req->cfile;
- struct cifs_sb_info *cifs_sb = CIFS_SB(wdata->rreq->inode->i_sb);
- size_t wsize = req->rreq.wsize;
+ struct cifs_sb_info *cifs_sb = CIFS_SB(subreq->rreq->inode->i_sb);
+ unsigned int max_segs = INT_MAX;
+ size_t len;
int rc;
+ if (cifs_forced_shutdown(cifs_sb))
+ return smb_EIO(smb_eio_trace_forced_shutdown);
+
if (!wdata->have_xid) {
wdata->xid = get_xid();
wdata->have_xid = true;
@@ -86,18 +88,16 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)
if (rc < 0) {
if (rc == -EAGAIN)
goto retry;
- subreq->error = rc;
- return netfs_prepare_write_failed(subreq);
+ return rc;
}
}
- rc = server->ops->wait_mtu_credits(server, wsize, &stream->sreq_max_len,
- &wdata->credits);
- if (rc < 0) {
- subreq->error = rc;
- return netfs_prepare_write_failed(subreq);
- }
+ len = umin(subreq->len, cifs_sb->ctx->wsize);
+ rc = server->ops->wait_mtu_credits(server, len, &len, &wdata->credits);
+ if (rc < 0)
+ return rc;
+ subreq->len = len;
wdata->credits.rreq_debug_id = subreq->rreq->debug_id;
wdata->credits.rreq_debug_index = subreq->debug_index;
wdata->credits.in_flight_check = 1;
@@ -113,46 +113,34 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)
const struct smbdirect_socket_parameters *sp =
smbd_get_parameters(server->smbd_conn);
- stream->sreq_max_segs = sp->max_frmr_depth;
+ max_segs = sp->max_frmr_depth;
}
#endif
-}
-/*
- * Issue a subrequest to upload to the server.
- */
-static void cifs_issue_write(struct netfs_io_subrequest *subreq)
-{
- struct cifs_io_subrequest *wdata =
- container_of(subreq, struct cifs_io_subrequest, subreq);
- struct cifs_sb_info *sbi = CIFS_SB(subreq->rreq->inode->i_sb);
- int rc;
-
- if (cifs_forced_shutdown(sbi)) {
- rc = smb_EIO(smb_eio_trace_forced_shutdown);
- goto fail;
+ rc = netfs_prepare_write_buffer(subreq, max_segs);
+ if (rc < 0) {
+ add_credits_and_wake_if(wdata->server, &wdata->credits, 0);
+ return rc;
}
+ /* After this point, must fail by termination. */
- rc = adjust_credits(wdata->server, wdata, cifs_trace_rw_credits_issue_write_adjust);
+ rc = adjust_credits(server, wdata, cifs_trace_rw_credits_issue_write_adjust);
if (rc)
- goto fail;
+ goto fail_with_credits;
rc = -EAGAIN;
if (wdata->req->cfile->invalidHandle)
- goto fail;
+ goto fail_with_credits;
wdata->server->ops->async_writev(wdata);
-out:
- return;
+ return 0;
-fail:
+fail_with_credits:
if (rc == -EAGAIN)
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- else
- trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
add_credits_and_wake_if(wdata->server, &wdata->credits, 0);
cifs_write_subrequest_terminated(wdata, rc);
- goto out;
+ return 0;
}
static void cifs_netfs_invalidate_cache(struct netfs_io_request *wreq)
@@ -161,17 +149,25 @@ static void cifs_netfs_invalidate_cache(struct netfs_io_request *wreq)
}
/*
- * Negotiate the size of a read operation on behalf of the netfs library.
+ * Issue a read operation on behalf of the netfs helper functions. We're asked
+ * to make a read of a certain size at a point in the file. We are permitted
+ * to only read a portion of that, but as long as we read something, the netfs
+ * helper will call us again so that we can issue another read.
*/
-static int cifs_prepare_read(struct netfs_io_subrequest *subreq)
+static int cifs_issue_read(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq);
struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq);
- struct TCP_Server_Info *server;
+ struct TCP_Server_Info *server = rdata->server;
struct cifs_sb_info *cifs_sb = CIFS_SB(rreq->inode->i_sb);
- size_t size;
- int rc = 0;
+ unsigned int max_segs = INT_MAX;
+ size_t len;
+ int rc;
+
+ cifs_dbg(FYI, "%s: op=%08x[%x] mapping=%p len=%zu/%zu\n",
+ __func__, rreq->debug_id, subreq->debug_index, rreq->mapping,
+ subreq->transferred, subreq->len);
if (!rdata->have_xid) {
rdata->xid = get_xid();
@@ -185,17 +181,15 @@ static int cifs_prepare_read(struct netfs_io_subrequest *subreq)
cifs_negotiate_rsize(server, cifs_sb->ctx,
tlink_tcon(req->cfile->tlink));
- rc = server->ops->wait_mtu_credits(server, cifs_sb->ctx->rsize,
- &size, &rdata->credits);
+ len = umin(subreq->len, cifs_sb->ctx->rsize);
+ rc = server->ops->wait_mtu_credits(server, len, &len, &rdata->credits);
if (rc)
return rc;
- rreq->io_streams[0].sreq_max_len = size;
-
- rdata->credits.in_flight_check = 1;
+ subreq->len = len;
rdata->credits.rreq_debug_id = rreq->debug_id;
rdata->credits.rreq_debug_index = subreq->debug_index;
-
+ rdata->credits.in_flight_check = 1;
trace_smb3_rw_credits(rdata->rreq->debug_id,
rdata->subreq.debug_index,
rdata->credits.value,
@@ -207,40 +201,27 @@ static int cifs_prepare_read(struct netfs_io_subrequest *subreq)
const struct smbdirect_socket_parameters *sp =
smbd_get_parameters(server->smbd_conn);
- rreq->io_streams[0].sreq_max_segs = sp->max_frmr_depth;
+ max_segs = sp->max_frmr_depth;
}
#endif
- return 0;
-}
-/*
- * Issue a read operation on behalf of the netfs helper functions. We're asked
- * to make a read of a certain size at a point in the file. We are permitted
- * to only read a portion of that, but as long as we read something, the netfs
- * helper will call us again so that we can issue another read.
- */
-static void cifs_issue_read(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_request *rreq = subreq->rreq;
- struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq);
- struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq);
- struct TCP_Server_Info *server = rdata->server;
- int rc = 0;
-
- cifs_dbg(FYI, "%s: op=%08x[%x] mapping=%p len=%zu/%zu\n",
- __func__, rreq->debug_id, subreq->debug_index, rreq->mapping,
- subreq->transferred, subreq->len);
+ rc = netfs_prepare_read_buffer(subreq, max_segs);
+ if (rc < 0) {
+ add_credits_and_wake_if(rdata->server, &rdata->credits, 0);
+ return rc;
+ }
+ /* After this point, must fail by termination. */
rc = adjust_credits(server, rdata, cifs_trace_rw_credits_issue_read_adjust);
if (rc)
- goto failed;
+ goto fail_with_credits;
if (req->cfile->invalidHandle) {
do {
rc = cifs_reopen_file(req->cfile, true);
} while (rc == -EAGAIN);
if (rc)
- goto failed;
+ goto fail_with_credits;
}
if (subreq->rreq->origin != NETFS_UNBUFFERED_READ &&
@@ -248,15 +229,22 @@ static void cifs_issue_read(struct netfs_io_subrequest *subreq)
__set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags);
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+
rc = rdata->server->ops->async_readv(rdata);
if (rc)
goto failed;
- return;
+ return 0;
+fail_with_credits:
+ if (rc == -EAGAIN)
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+ else
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
failed:
add_credits_and_wake_if(rdata->server, &rdata->credits, 0);
subreq->error = rc;
netfs_read_subreq_terminated(subreq);
+ return 0;
}
/*
@@ -371,12 +359,10 @@ const struct netfs_request_ops cifs_req_ops = {
.init_request = cifs_init_request,
.free_request = cifs_free_request,
.free_subrequest = cifs_free_subrequest,
- .prepare_read = cifs_prepare_read,
.issue_read = cifs_issue_read,
.done = cifs_rreq_done,
.begin_writeback = cifs_begin_writeback,
.estimate_write = cifs_estimate_write,
- .prepare_write = cifs_prepare_write,
.issue_write = cifs_issue_write,
.invalidate_cache = cifs_netfs_invalidate_cache,
};
diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c
index 97823bcf198c..1f3266f89cce 100644
--- a/fs/smb/client/smb2ops.c
+++ b/fs/smb/client/smb2ops.c
@@ -4884,6 +4884,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
unsigned int cur_page_idx;
unsigned int pad_len;
struct cifs_io_subrequest *rdata = mid->callback_data;
+ struct iov_iter iter;
struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
size_t copied;
bool use_rdma_mr = false;
@@ -4956,6 +4957,10 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
pad_len = data_offset - server->vals->read_rsp_size;
+ iov_iter_bvec_queue(&iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
+
if (buf_len <= data_offset) {
/* read response payload is in pages */
cur_page_idx = pad_len / PAGE_SIZE;
@@ -4985,7 +4990,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
/* Copy the data to the output I/O iterator. */
rdata->result = cifs_copy_bvecq_to_iter(buffer, data_len,
- cur_off, &rdata->subreq.io_iter);
+ cur_off, &iter);
if (rdata->result != 0) {
if (is_offloaded)
mid->mid_state = MID_RESPONSE_MALFORMED;
@@ -4999,7 +5004,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
buf_len >= end_off) {
/* read response payload is in buf */
WARN_ONCE(buffer, "read data can be either in buf or in buffer");
- copied = copy_to_iter(buf + data_offset, data_len, &rdata->subreq.io_iter);
+ copied = copy_to_iter(buf + data_offset, data_len, &iter);
if (copied == 0)
return smb_EIO2(smb_eio_trace_rx_copy_to_iter, copied, data_len);
rdata->got_bytes = copied;
diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c
index 4ce165e40657..085fdc7d8bec 100644
--- a/fs/smb/client/smb2pdu.c
+++ b/fs/smb/client/smb2pdu.c
@@ -4592,9 +4592,13 @@ smb2_new_read_req(void **buf, unsigned int *total_len,
*/
if (rdata && smb3_use_rdma_offload(io_parms)) {
struct smbdirect_buffer_descriptor_v1 *v1;
+ struct iov_iter iter;
bool need_invalidate = server->dialect == SMB30_PROT_ID;
- rdata->mr = smbd_register_mr(server->smbd_conn, &rdata->subreq.io_iter,
+ iov_iter_bvec_queue(&iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
+ rdata->mr = smbd_register_mr(server->smbd_conn, &iter,
true, need_invalidate);
if (!rdata->mr)
return -EAGAIN;
@@ -4658,9 +4662,10 @@ smb2_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)
unsigned int rreq_debug_id = rdata->rreq->debug_id;
unsigned int subreq_debug_index = rdata->subreq.debug_index;
- if (rdata->got_bytes) {
- rqst.rq_iter = rdata->subreq.io_iter;
- }
+ if (rdata->got_bytes)
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
WARN_ONCE(rdata->server != server,
"rdata server %p != mid server %p",
@@ -5148,7 +5153,9 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
goto out;
rqst.rq_iov = iov;
- rqst.rq_iter = wdata->subreq.io_iter;
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_SOURCE,
+ wdata->subreq.content.bvecq, wdata->subreq.content.slot,
+ wdata->subreq.content.offset, wdata->subreq.len);
rqst.rq_iov[0].iov_len = total_len - 1;
rqst.rq_iov[0].iov_base = (char *)req;
@@ -5187,9 +5194,14 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
*/
if (smb3_use_rdma_offload(io_parms)) {
struct smbdirect_buffer_descriptor_v1 *v1;
+ struct iov_iter iter;
bool need_invalidate = server->dialect == SMB30_PROT_ID;
- wdata->mr = smbd_register_mr(server->smbd_conn, &wdata->subreq.io_iter,
+ iov_iter_bvec_queue(&iter, ITER_SOURCE,
+ wdata->subreq.content.bvecq, wdata->subreq.content.slot,
+ wdata->subreq.content.offset, wdata->subreq.len);
+
+ wdata->mr = smbd_register_mr(server->smbd_conn, &iter,
false, need_invalidate);
if (!wdata->mr) {
rc = -EAGAIN;
@@ -5226,8 +5238,8 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
smb2_set_replay(server, &rqst);
}
- cifs_dbg(FYI, "async write at %llu %u bytes iter=%zx\n",
- io_parms->offset, io_parms->length, iov_iter_count(&wdata->subreq.io_iter));
+ cifs_dbg(FYI, "async write at %llu %u bytes len=%zx\n",
+ io_parms->offset, io_parms->length, wdata->subreq.len);
if (wdata->credits.value > 0) {
shdr->CreditCharge = cpu_to_le16(DIV_ROUND_UP(wdata->subreq.len,
diff --git a/fs/smb/client/transport.c b/fs/smb/client/transport.c
index fdf4e50c27ce..be2f6b909c34 100644
--- a/fs/smb/client/transport.c
+++ b/fs/smb/client/transport.c
@@ -1267,12 +1267,19 @@ cifs_readv_receive(struct TCP_Server_Info *server, struct mid_q_entry *mid)
}
#ifdef CONFIG_CIFS_SMB_DIRECT
- if (rdata->mr)
+ if (rdata->mr) {
length = data_len; /* An RDMA read is already done. */
- else
+ } else {
+#endif
+ struct iov_iter iter;
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, rdata->subreq.content.bvecq,
+ rdata->subreq.content.slot, rdata->subreq.content.offset,
+ data_len);
+ length = cifs_read_iter_from_socket(server, &iter, data_len);
+#ifdef CONFIG_CIFS_SMB_DIRECT
+ }
#endif
- length = cifs_read_iter_from_socket(server, &rdata->subreq.io_iter,
- data_len);
if (length > 0)
rdata->got_bytes += length;
server->total_read += length;
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 2c5b7f3a48b7..5eb836086329 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -162,23 +162,17 @@ struct netfs_write_estimate {
struct netfs_io_stream {
/* Submission tracking (main dispatch only; not retry) */
struct bvecq_pos dispatch_cursor; /* Point from which buffers are dispatched */
- struct netfs_io_subrequest *construct; /* Op being constructed */
uoff_t issue_from; /* Current issue point */
uoff_t last_end; /* End file pos of last folio added */
size_t buffered; /* Amount in buffer */
size_t post_gap; /* Length of partial folio tail */
- size_t sreq_max_len; /* Maximum size of a subrequest */
- unsigned int sreq_max_segs; /* 0 or max number of segments in an iterator */
- unsigned int submit_off; /* Folio offset we're submitting from */
- unsigned int submit_len; /* Amount of data left to submit */
unsigned int alignment; /* Required alignment */
u8 applicable; /* What sources are applicable (NOTE_* mask) */
bool buffering; /* T if buffering on this stream */
int (*estimate_write)(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
struct netfs_write_estimate *estimate);
- void (*prepare_write)(struct netfs_io_subrequest *subreq);
- void (*issue_write)(struct netfs_io_subrequest *subreq);
+ int (*issue_write)(struct netfs_io_subrequest *subreq);
atomic64_t issued_to; /* Point to which can be considered issued */
/* Collection tracking */
@@ -223,15 +217,14 @@ struct netfs_io_subrequest {
struct list_head rreq_link; /* Link in rreq->subrequests */
struct bvecq_pos dispatch_pos; /* Bookmark in the combined queue of the start */
struct bvecq_pos content; /* The (copied) content of the subrequest */
- struct iov_iter io_iter; /* Iterator for this subrequest */
uoff_t start; /* Where to start the I/O */
size_t len; /* Size of the I/O */
size_t post_gap; /* Length of partial folio tail */
size_t transferred; /* Amount of data transferred */
+ unsigned int nr_segs; /* Number of segments in content */
refcount_t ref;
short error; /* 0 or error that occurred */
unsigned short debug_index; /* Index in list (for debugging output) */
- unsigned int nr_segs; /* Number of segments in content */
u8 retry_count; /* The number of retries (0 on initial pass) */
enum netfs_io_source source; /* Where to read from/write to */
unsigned char stream_nr; /* I/O stream this belongs to */
@@ -239,7 +232,6 @@ struct netfs_io_subrequest {
#define NETFS_SREQ_COPY_TO_CACHE 0 /* Set if should copy the data to the cache */
#define NETFS_SREQ_CLEAR_TAIL 1 /* Set if the rest of the read should be cleared */
#define NETFS_SREQ_MADE_PROGRESS 4 /* Set if we transferred at least some data */
-#define NETFS_SREQ_BOUNDARY 6 /* Set if ends on hard boundary (eg. ceph object) */
#define NETFS_SREQ_HIT_EOF 7 /* Set if short due to EOF */
#define NETFS_SREQ_IN_PROGRESS 8 /* Unlocked when the subrequest completes */
#define NETFS_SREQ_NEED_RETRY 9 /* Set if the filesystem requests a retry */
@@ -288,12 +280,11 @@ struct netfs_io_request {
struct netfs_group *group; /* Writeback group being written back */
struct bvecq *spare; /* Advance allocation of bvecq */
struct bvecq_pos load_cursor; /* Point at which new folios are loaded in */
- struct bvecq_pos dispatch_cursor; /* Point from which buffers are dispatched */
struct bvecq_pos collect_cursor; /* Clear-up point of I/O buffer */
+ struct bvecq_pos retry_cursor; /* Point from which retries are dispatched */
wait_queue_head_t waitq; /* Processor waiter */
void *netfs_priv; /* Private data for the netfs */
void *netfs_priv2; /* Private data for the netfs */
- uoff_t last_end; /* End pos of last folio submitted */
uoff_t submitted; /* Amount submitted for I/O so far */
uoff_t len; /* Length of the request */
size_t transferred; /* Amount to be indicated as transferred */
@@ -305,6 +296,8 @@ struct netfs_io_request {
uoff_t cache_coll_to; /* Point the cache has collected to */
uoff_t cleaned_to; /* Position we've cleaned folios to */
uoff_t abandon_to; /* Position to abandon folios to */
+ uoff_t retry_start; /* Position to retry from */
+ size_t retry_buffered; /* Amount of data to retry */
const struct folio *no_unlock_folio; /* Don't unlock this folio after read */
gfp_t gfp; /* GFP flags to use */
unsigned int debug_id;
@@ -351,8 +344,7 @@ struct netfs_request_ops {
/* Read request handling */
void (*expand_readahead)(struct netfs_io_request *rreq);
- int (*prepare_read)(struct netfs_io_subrequest *subreq);
- void (*issue_read)(struct netfs_io_subrequest *subreq);
+ int (*issue_read)(struct netfs_io_subrequest *subreq);
bool (*is_still_valid)(struct netfs_io_request *rreq);
int (*check_write_begin)(struct file *file, uoff_t pos, unsigned len,
struct folio **foliop, void **_fsdata);
@@ -367,8 +359,7 @@ struct netfs_request_ops {
int (*estimate_write)(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
struct netfs_write_estimate *estimate);
- void (*prepare_write)(struct netfs_io_subrequest *subreq);
- void (*issue_write)(struct netfs_io_subrequest *subreq);
+ int (*issue_write)(struct netfs_io_subrequest *subreq);
void (*retry_request)(struct netfs_io_request *wreq, struct netfs_io_stream *stream);
void (*invalidate_cache)(struct netfs_io_request *wreq);
};
@@ -408,8 +399,11 @@ struct netfs_cache_ops {
struct netfs_io_stream *stream,
struct netfs_write_estimate *estimate);
+ /* Read data from the cache for a netfs subrequest. */
+ int (*issue_read)(struct netfs_io_subrequest *subreq);
+
/* Write data to the cache from a netfs subrequest. */
- void (*issue_write)(struct netfs_io_subrequest *subreq);
+ int (*issue_write)(struct netfs_io_subrequest *subreq);
/* Expand readahead request */
void (*expand_readahead)(struct netfs_cache_resources *cres,
@@ -417,30 +411,18 @@ struct netfs_cache_ops {
uoff_t *_len,
uoff_t i_size);
- /* Prepare a read operation, shortening it to a cached/uncached
- * boundary as appropriate.
- */
- int (*prepare_read)(struct netfs_io_subrequest *subreq);
-
- /* Prepare a write subrequest, working out if we're allowed to do it
- * and finding out the maximum amount of data to gather before
- * attempting to submit. If we're not permitted to do it, the
- * subrequest should be marked failed.
- */
- void (*prepare_write_subreq)(struct netfs_io_subrequest *subreq);
-
/* Prepare a write operation, working out what part of the write we can
* actually do.
*/
- int (*prepare_write)(struct netfs_cache_resources *cres,
- uoff_t *_start, size_t *_len, size_t upper_len,
- uoff_t i_size, bool no_space_allocated_yet);
+ int (*prepare_write_old)(struct netfs_cache_resources *cres,
+ uoff_t *_start, size_t *_len, size_t upper_len,
+ uoff_t i_size, bool no_space_allocated_yet);
/* Query the occupancy of the cache in a region, returning where the
* next chunk of data starts and how long it is.
*/
- int (*query_occupancy)(struct netfs_cache_resources *cres,
- struct fscache_occupancy *occ);
+ void (*query_occupancy)(struct netfs_cache_resources *cres,
+ struct fscache_occupancy *occ);
/* Collect the result of buffered writeback to the cache. This
* includes copying a read to the cache. block_type is one of:
@@ -477,7 +459,6 @@ int netfs_writeback_single(struct address_space *mapping,
struct iov_iter *iter, size_t len);
/* Address operations API */
-struct readahead_control;
void netfs_readahead(struct readahead_control *);
int netfs_read_folio(struct file *, struct folio *);
int netfs_write_begin(struct netfs_inode *, struct file *,
@@ -504,7 +485,8 @@ void netfs_put_subrequest(struct netfs_io_subrequest *subreq,
ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages,
struct bvecq **_bvecq_head,
iov_iter_extraction_t extraction_flags, gfp_t gfp);
-void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq);
+int netfs_prepare_read_buffer(struct netfs_io_subrequest *subreq, unsigned int max_segs);
+int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq, unsigned int max_segs);
void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error);
int netfs_start_io_read(struct inode *inode);
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 6ccf9de42858..dfe08492d1ad 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -49,6 +49,7 @@
#define netfs_rreq_traces \
EM(netfs_rreq_trace_all_queued, "ALL-Q ") \
EM(netfs_rreq_trace_assess, "ASSESS ") \
+ EM(netfs_rreq_trace_cache_align_error, "CA-ALN!") \
EM(netfs_rreq_trace_cache_cancelled, "CA-CNCL") \
EM(netfs_rreq_trace_cache_failed, "CA-FAIL") \
EM(netfs_rreq_trace_cache_fail_collect, "CA-F-CO") \
@@ -85,7 +86,8 @@
EM(netfs_rreq_trace_waited_quiesce, "DONE-QUIESCE") \
EM(netfs_rreq_trace_wake_ip, "WAKE-IP") \
EM(netfs_rreq_trace_wake_queue, "WAKE-Q ") \
- E_(netfs_rreq_trace_write_done, "WR-DONE")
+ EM(netfs_rreq_trace_write_done, "WR-DONE") \
+ E_(netfs_rreq_trace_zero_unread, "ZERO-UR")
#define netfs_sreq_sources \
EM(NETFS_SOURCE_UNKNOWN, "----") \
@@ -134,6 +136,7 @@
EM(netfs_sreq_trace_superfluous, "SPRFL") \
EM(netfs_sreq_trace_terminated, "TERM ") \
EM(netfs_sreq_trace_too_much, "!TOOM") \
+ EM(netfs_sreq_trace_too_many_retries, "!RETR") \
EM(netfs_sreq_trace_wait_for, "_WAIT") \
EM(netfs_sreq_trace_write, "WRITE") \
EM(netfs_sreq_trace_write_skip, "SKIP ") \
@@ -528,6 +531,31 @@ TRACE_EVENT(netfs_folio,
__print_symbolic(__entry->why, netfs_folio_traces))
);
+TRACE_EVENT(netfs_wback,
+ TP_PROTO(struct netfs_io_request *wreq, struct folio *folio, unsigned int notes),
+
+ TP_ARGS(wreq, folio, notes),
+
+ TP_STRUCT__entry(
+ __field(pgoff_t, index)
+ __field(unsigned int, wreq)
+ __field(unsigned int, nr)
+ __field(unsigned int, notes)
+ ),
+
+ TP_fast_assign(
+ __entry->wreq = wreq->debug_id;
+ __entry->notes = notes;
+ __entry->index = folio->index;
+ __entry->nr = folio_nr_pages(folio);
+ ),
+
+ TP_printk("R=%08x ix=%05lx-%05lx n=%02x",
+ __entry->wreq,
+ __entry->index, __entry->index + __entry->nr - 1,
+ __entry->notes)
+ );
+
TRACE_EVENT(netfs_write_iter,
TP_PROTO(const struct kiocb *iocb, const struct iov_iter *from),
diff --git a/net/9p/client.c b/net/9p/client.c
index ef64546c6d52..68d82f677a7a 100644
--- a/net/9p/client.c
+++ b/net/9p/client.c
@@ -1564,6 +1564,7 @@ void
p9_client_write_subreq(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *wreq = subreq->rreq;
+ struct iov_iter iter;
struct p9_fid *fid = wreq->netfs_priv;
struct p9_client *clnt = fid->clnt;
struct p9_req_t *req;
@@ -1574,14 +1575,17 @@ p9_client_write_subreq(struct netfs_io_subrequest *subreq)
p9_debug(P9_DEBUG_9P, ">>> TWRITE fid %d offset %llu len %d\n",
fid->fid, start, len);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
/* Don't bother zerocopy for small IO (< 1024) */
if (clnt->trans_mod->zc_request && len > 1024) {
- req = p9_client_zc_rpc(clnt, P9_TWRITE, NULL, &subreq->io_iter,
+ req = p9_client_zc_rpc(clnt, P9_TWRITE, NULL, &iter,
0, wreq->len, P9_ZC_HDR_SZ, "dqd",
fid->fid, start, len);
} else {
req = p9_client_rpc(clnt, P9_TWRITE, "dqV", fid->fid,
- start, len, &subreq->io_iter);
+ start, len, &iter);
}
if (IS_ERR(req)) {
netfs_write_subrequest_terminated(subreq, PTR_ERR(req));
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 35/35] netfs: Clean up now-unused code
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (33 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 34/35] netfs: Combine prepare and issue ops and grab the buffers on request David Howells
@ 2026-08-24 14:41 ` David Howells
2026-08-24 15:01 ` [PATCH v10 36/35] cachefiles: Preset the state xattr when creating a new file David Howells
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 14:41 UTC (permalink / raw)
To: Paulo Alcantara
Cc: David Howells, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel
Delete now-unused code from netfslib, given the changes to the writeback
dispatch code and the combination of ->prepare_xxx() and ->issue_xxx()
methods.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/netfs/write_issue.c | 216 -----------------------------------------
1 file changed, 216 deletions(-)
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index e0d14d77b751..961949689303 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -223,69 +223,6 @@ struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wr
return subreq;
}
-#if 0 // TODO: Remove old stuff
-/*
- * Prepare a write subrequest. We need to allocate a new subrequest
- * if we don't have one.
- */
-void netfs_prepare_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- uoff_t start)
-{
- struct netfs_io_subrequest *subreq;
-
- subreq = netfs_alloc_subrequest(wreq);
- subreq->source = stream->source;
- subreq->start = start;
- subreq->stream_nr = stream->stream_nr;
-
- bvecq_pos_set(&subreq->dispatch_pos, &wreq->dispatch_cursor);
-
- _enter("R=%x[%x]", wreq->debug_id, subreq->debug_index);
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
-
- stream->sreq_max_len = UINT_MAX;
- stream->sreq_max_segs = INT_MAX;
- switch (stream->source) {
- case NETFS_UPLOAD_TO_SERVER:
- netfs_stat(&netfs_n_wh_upload);
- stream->sreq_max_len = wreq->wsize;
- break;
- case NETFS_WRITE_TO_CACHE:
- netfs_stat(&netfs_n_wh_write);
- break;
- default:
- WARN_ON_ONCE(1);
- break;
- }
-
- if (stream->prepare_write)
- stream->prepare_write(subreq);
-
- __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
-
- /* We add to the end of the list whilst the collector may be walking
- * the list. The collector only goes nextwards and uses the lock to
- * remove entries off of the front.
- */
- spin_lock(&wreq->lock);
- /* Write IN_PROGRESS before pointer to new subreq */
- list_add_tail_release(&subreq->rreq_link, &stream->subrequests);
- if (list_is_first(&subreq->rreq_link, &stream->subrequests)) {
- if (!stream->active) {
- stream->collected_to = subreq->start;
- /* Write list pointers before active flag */
- smp_store_release(&stream->active, true);
- }
- }
-
- spin_unlock(&wreq->lock);
-
- stream->construct = subreq;
-}
-#endif
-
/*
* Advance the state of the amount of data buffered on a stream.
*/
@@ -368,159 +305,6 @@ int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq,
}
EXPORT_SYMBOL(netfs_prepare_write_buffer);
-#if 0 // TODO: Remove old stuff
-/*
- * Set the I/O iterator for the filesystem/cache to use and dispatch the I/O
- * operation. The operation may be asynchronous and should call
- * netfs_write_subrequest_terminated() when complete.
- */
-static void netfs_do_issue_write(struct netfs_io_stream *stream,
- struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_request *wreq = subreq->rreq;
-
- _enter("R=%x[%x],%zx", wreq->debug_id, subreq->debug_index, subreq->len);
-
- if (stream->source == NETFS_WRITE_TO_CACHE &&
- unlikely(test_bit(NETFS_RREQ_CACHE_STOP, &wreq->flags))) {
- size_t dio_size = wreq->cache_resources.dio_size;
- size_t len, disp;
-
- disp = subreq->start & (dio_size - 1);
- len = round_up(subreq->len + disp, dio_size);
-
- subreq->start -= disp;
- subreq->len = len;
-
- __set_bit(NETFS_SREQ_CANCELLED, &subreq->flags);
- return netfs_write_subrequest_terminated(subreq, subreq->len);
- }
-
- if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
- return netfs_write_subrequest_terminated(subreq, subreq->error);
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- stream->issue_write(subreq);
-}
-
-void netfs_reissue_write(struct netfs_io_stream *stream,
- struct netfs_io_subrequest *subreq)
-{
- // TODO: Use encrypted buffer
- bvecq_pos_unset(&subreq->content);
- bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
- iov_iter_bvec_queue(&subreq->io_iter, ITER_SOURCE,
- subreq->content.bvecq, subreq->content.slot,
- subreq->content.offset,
- subreq->len);
- iov_iter_advance(&subreq->io_iter, subreq->transferred);
-
- subreq->retry_count++;
- subreq->error = 0;
- __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
- __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
- netfs_stat(&netfs_n_wh_retry_write_subreq);
- netfs_do_issue_write(stream, subreq);
-}
-
-static void netfs_issue_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream)
-{
- struct netfs_io_subrequest *subreq = stream->construct;
-
- if (!subreq)
- return;
-
- bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
- iov_iter_bvec_queue(&subreq->io_iter, ITER_SOURCE,
- subreq->content.bvecq, subreq->content.slot,
- subreq->content.offset,
- subreq->len);
-
- stream->construct = NULL;
- netfs_do_issue_write(stream, subreq);
-}
-
-/*
- * Add data to the write subrequest, dispatching each as we fill it up or if it
- * is discontiguous with the previous. We only fill one part at a time so that
- * we can avoid overrunning the credits obtained (cifs) and try to parallelise
- * content-crypto preparation with network writes.
- */
-size_t netfs_advance_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- uoff_t start, size_t len, bool to_eof)
-{
- struct netfs_io_subrequest *subreq = stream->construct;
- size_t part;
-
- if (!stream->avail) {
- _leave("no write");
- return len;
- }
-
- _enter("R=%x[%x]", wreq->debug_id, subreq ? subreq->debug_index : 0);
-
- if (subreq && start != subreq->start + subreq->len) {
- netfs_issue_write(wreq, stream);
- subreq = NULL;
- }
-
- if (!stream->construct)
- netfs_prepare_write(wreq, stream, start);
- subreq = stream->construct;
-
- part = umin(stream->sreq_max_len - subreq->len, len);
- _debug("part %zx/%zx %zx/%zx", subreq->len, stream->sreq_max_len, part, len);
- subreq->len += part;
- subreq->nr_segs++;
-
- if (subreq->len >= stream->sreq_max_len ||
- subreq->nr_segs >= stream->sreq_max_segs ||
- to_eof) {
- netfs_issue_write(wreq, stream);
- subreq = NULL;
- }
-
- return part;
-}
-
-/*
- * Prepare and issue a subrequest.
- * TODO: Replace with combined ->prepare/->issue call().
- */
-static int netfs_prep_and_issue_subreq(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- struct netfs_io_subrequest *subreq)
-{
- stream->sreq_max_len = UINT_MAX;
- stream->sreq_max_segs = INT_MAX;
- switch (stream->source) {
- case NETFS_UPLOAD_TO_SERVER:
- netfs_stat(&netfs_n_wh_upload);
- stream->sreq_max_len = wreq->wsize;
- break;
- case NETFS_WRITE_TO_CACHE:
- netfs_stat(&netfs_n_wh_write);
- break;
- default:
- WARN_ON_ONCE(1);
- break;
- }
-
- if (stream->prepare_write)
- stream->prepare_write(subreq);
- netfs_prepare_buffered_write_buffer(subreq, stream->sreq_max_segs);
- iov_iter_bvec_queue(&subreq->io_iter, ITER_SOURCE,
- subreq->content.bvecq, subreq->content.slot,
- subreq->content.offset,
- subreq->len);
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- stream->issue_write(subreq);
- return 0;
-}
-#endif
-
/*
* Issue writes for a stream.
*/
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v10 36/35] cachefiles: Preset the state xattr when creating a new file
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
` (34 preceding siblings ...)
2026-08-24 14:41 ` [PATCH v10 35/35] netfs: Clean up now-unused code David Howells
@ 2026-08-24 15:01 ` David Howells
35 siblings, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 15:01 UTC (permalink / raw)
To: Paulo Alcantara
Cc: dhowells, Christian Brauner, Matthew Wilcox, Christoph Hellwig,
Jens Axboe, Leon Romanovsky, Namjae Jeon, ChenXiaoSong,
Marc Dionne, Stefan Metzmacher, Eric Van Hensbergen,
Dominique Martinet, Ilya Dryomov, netfs, linux-afs, linux-cifs,
linux-nfs, ceph-devel, v9fs, linux-erofs, linux-fsdevel,
linux-kernel
With a really small cache, cachefiles is likely to see a lot of writes
hitting ENOSPC - and this can include setxattr that sets the state xattr on
a cachefile - but we don't really want to successfully fill a cache file
only to have to scrap it because we can't set the xattr.
Instead, preset the xattr when we create the tmpfile we're going to use,
and scrap the file at that point if we get ENOSPC. Only if setxattr
succeeds do we allow data to be written to the file.
Note that there is a potential performance loss in that writes to the cache
have to be delayed until this is completed - but we do the tmpfile/setxattr
in parallel, starting when the file is opened and only have to wait once
writeback occurs.
Further, a new tracepoint, cachefiles_no_space, has been added to log ENOSPC
events, along with a couple of subreq traces to log failure detection in
netfslib.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Marc Dionne <marc.dionne@auristor.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/cachefiles/internal.h | 1
fs/cachefiles/io.c | 23 ++++++++++----
fs/cachefiles/namei.c | 15 ++++++++-
fs/cachefiles/xattr.c | 59 +++++++++++++++++++++++++++++++++++++-
include/trace/events/cachefiles.h | 30 ++++++++++++++++++-
include/trace/events/netfs.h | 2 +
6 files changed, 120 insertions(+), 10 deletions(-)
diff --git a/fs/cachefiles/internal.h b/fs/cachefiles/internal.h
index b2605111fd56..664be64ab538 100644
--- a/fs/cachefiles/internal.h
+++ b/fs/cachefiles/internal.h
@@ -283,6 +283,7 @@ void cachefiles_withdraw_volume(struct cachefiles_volume *volume);
/*
* xattr.c
*/
+int cachefiles_preset_object_xattr(struct cachefiles_object *object, struct file *file);
extern int cachefiles_set_object_xattr(struct cachefiles_object *object);
extern int cachefiles_check_auxdata(struct cachefiles_object *object,
struct file *file);
diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c
index a1580385d5fb..73c8ba01a9e2 100644
--- a/fs/cachefiles/io.c
+++ b/fs/cachefiles/io.c
@@ -672,10 +672,14 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
* space, we need to see if it's fully allocated. If it's not, we may
* want to cull it.
*/
- if (cachefiles_has_space(cache, 0, *_len / cache->bsize,
- cachefiles_has_space_check) == 0)
+ ret = cachefiles_has_space(cache, 0, *_len / cache->bsize,
+ cachefiles_has_space_check);
+ if (ret == 0)
return 0; /* Enough space to simply overwrite the whole block */
+ if (ret == -ENOBUFS)
+ trace_cachefiles_no_space(object, cachefiles_trace_write_nospace_2);
+
pos = cachefiles_inject_read_error();
if (pos == 0)
pos = vfs_llseek(file, start, SEEK_HOLE);
@@ -704,8 +708,11 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
return ret;
check_space:
- return cachefiles_has_space(cache, 0, *_len / cache->bsize,
- cachefiles_has_space_for_write);
+ ret = cachefiles_has_space(cache, 0, *_len / cache->bsize,
+ cachefiles_has_space_for_write);
+ if (ret == -ENOBUFS)
+ trace_cachefiles_no_space(object, cachefiles_trace_write_nospace);
+ return ret;
}
static int cachefiles_prepare_write_old(struct netfs_cache_resources *cres,
@@ -757,10 +764,14 @@ static int cachefiles_issue_write(struct netfs_io_subrequest *subreq)
wreq->debug_id, subreq->debug_index, start, start + len - 1);
if (!cachefiles_cres_file(cres)) {
- if (!fscache_wait_for_operation(cres, FSCACHE_WANT_WRITE))
+ if (!fscache_wait_for_operation(cres, FSCACHE_WANT_WRITE)) {
+ trace_netfs_sreq(subreq, netfs_sreq_trace_cache_waitfail);
return -ENOBUFS;
- if (!cachefiles_cres_file(cres))
+ }
+ if (!cachefiles_cres_file(cres)) {
+ trace_netfs_sreq(subreq, netfs_sreq_trace_cache_nofile);
return -ENOBUFS;
+ }
}
ret = netfs_prepare_write_buffer(subreq, BIO_MAX_VECS);
diff --git a/fs/cachefiles/namei.c b/fs/cachefiles/namei.c
index a30df0f91e5e..ef656a319ede 100644
--- a/fs/cachefiles/namei.c
+++ b/fs/cachefiles/namei.c
@@ -117,8 +117,11 @@ struct dentry *cachefiles_get_directory(struct cachefiles_cache *cache,
if (d_is_negative(subdir)) {
ret = cachefiles_has_space(cache, 1, 0,
cachefiles_has_space_for_create);
- if (ret < 0)
+ if (ret < 0) {
+ if (ret == -ENOBUFS)
+ trace_cachefiles_no_space(NULL, cachefiles_trace_mkdir_nospace);
goto mkdir_error;
+ }
_debug("attempt mkdir");
@@ -447,6 +450,11 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)
pr_notice("Cache does not support read_iter and write_iter\n");
goto err_unuse;
}
+
+ /* Preallocate space for the xattr. */
+ ret = cachefiles_preset_object_xattr(object, file);
+ if (ret < 0)
+ goto err_unuse;
out:
cachefiles_end_secure(cache, saved_cred);
object->content_info = CACHEFILES_CONTENT_ALL;
@@ -470,8 +478,11 @@ static bool cachefiles_create_file(struct cachefiles_object *object)
ret = cachefiles_has_space(object->volume->cache, 1, 0,
cachefiles_has_space_for_create);
- if (ret < 0)
+ if (ret < 0) {
+ if (ret == -ENOBUFS)
+ trace_cachefiles_no_space(object, cachefiles_trace_create_nospace);
return false;
+ }
file = cachefiles_create_tmpfile(object);
if (IS_ERR(file))
diff --git a/fs/cachefiles/xattr.c b/fs/cachefiles/xattr.c
index 5c1794beca93..9f78710bfe4d 100644
--- a/fs/cachefiles/xattr.c
+++ b/fs/cachefiles/xattr.c
@@ -34,6 +34,57 @@ struct cachefiles_vol_xattr {
__u8 data[]; /* netfs volume coherency data */
} __packed;
+/*
+ * Preset the state xattr on a cache file to allocate space for it.
+ */
+int cachefiles_preset_object_xattr(struct cachefiles_object *object, struct file *file)
+{
+ struct cachefiles_xattr *buf;
+ struct dentry *dentry = file->f_path.dentry;
+ unsigned int len = object->cookie->aux_len;
+ int ret;
+
+ buf = kzalloc(sizeof(struct cachefiles_xattr) + min(len, sizeof(__be64)), GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ buf->type = CACHEFILES_COOKIE_TYPE_DATA;
+ buf->content = CACHEFILES_CONTENT_DIRTY;
+
+ ret = cachefiles_inject_write_error();
+ if (ret == 0) {
+ ret = mnt_want_write_file(file);
+ if (ret == 0) {
+ ret = vfs_setxattr(&nop_mnt_idmap, dentry,
+ cachefiles_xattr_cache, buf,
+ sizeof(struct cachefiles_xattr) + len, 0);
+ mnt_drop_write_file(file);
+ }
+ }
+ if (ret < 0) {
+ trace_cachefiles_vfs_error(object, file_inode(file), ret,
+ cachefiles_trace_setxattr_error);
+ trace_cachefiles_coherency(object, file_inode(file)->i_ino,
+ object->object_size,
+ buf->data, buf->content,
+ cachefiles_coherency_set_fail);
+ switch (ret) {
+ case -ENOMEM:
+ case -ENOSPC:
+ break;
+ default:
+ cachefiles_io_error_obj(
+ object,
+ "Failed to set xattr with error %d", ret);
+ break;
+ }
+ }
+
+ kfree(buf);
+ _leave(" = %d", ret);
+ return ret;
+}
+
/*
* set the state xattr on a cache file
*/
@@ -82,10 +133,16 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
trace_cachefiles_coherency(object, file_inode(file)->i_ino,
object_size, buf->data, buf->content,
cachefiles_coherency_set_fail);
- if (ret != -ENOMEM)
+ switch (ret) {
+ case -ENOMEM:
+ break;
+ case -ENOSPC:
+ default:
cachefiles_io_error_obj(
object,
"Failed to set xattr with error %d", ret);
+ break;
+ }
} else {
trace_cachefiles_coherency(object, file_inode(file)->i_ino,
object_size, buf->data, buf->content,
diff --git a/include/trace/events/cachefiles.h b/include/trace/events/cachefiles.h
index 6146483300c2..3dca69126586 100644
--- a/include/trace/events/cachefiles.h
+++ b/include/trace/events/cachefiles.h
@@ -85,11 +85,13 @@ enum cachefiles_prepare_read_trace {
enum cachefiles_error_trace {
cachefiles_trace_alignment_error,
+ cachefiles_trace_create_nospace,
cachefiles_trace_fallocate_error,
cachefiles_trace_getxattr_error,
cachefiles_trace_link_error,
cachefiles_trace_lookup_error,
cachefiles_trace_mkdir_error,
+ cachefiles_trace_mkdir_nospace,
cachefiles_trace_notify_change_error,
cachefiles_trace_open_error,
cachefiles_trace_read_error,
@@ -102,6 +104,8 @@ enum cachefiles_error_trace {
cachefiles_trace_trunc_error,
cachefiles_trace_unlink_error,
cachefiles_trace_write_error,
+ cachefiles_trace_write_nospace,
+ cachefiles_trace_write_nospace_2,
};
#endif
@@ -171,11 +175,13 @@ enum cachefiles_error_trace {
#define cachefiles_error_traces \
EM(cachefiles_trace_alignment_error, "align") \
+ EM(cachefiles_trace_create_nospace, "create-nospace") \
EM(cachefiles_trace_fallocate_error, "fallocate") \
EM(cachefiles_trace_getxattr_error, "getxattr") \
EM(cachefiles_trace_link_error, "link") \
EM(cachefiles_trace_lookup_error, "lookup") \
EM(cachefiles_trace_mkdir_error, "mkdir") \
+ EM(cachefiles_trace_mkdir_nospace, "mkdir-nospace") \
EM(cachefiles_trace_notify_change_error, "notify_change") \
EM(cachefiles_trace_open_error, "open") \
EM(cachefiles_trace_read_error, "read") \
@@ -187,7 +193,9 @@ enum cachefiles_error_trace {
EM(cachefiles_trace_tmpfile_error, "tmpfile") \
EM(cachefiles_trace_trunc_error, "trunc") \
EM(cachefiles_trace_unlink_error, "unlink") \
- E_(cachefiles_trace_write_error, "write")
+ EM(cachefiles_trace_write_error, "write") \
+ EM(cachefiles_trace_write_nospace, "write-nospace") \
+ E_(cachefiles_trace_write_nospace_2, "write-nospace-2")
/*
@@ -707,6 +715,26 @@ TRACE_EVENT(cachefiles_io_error,
__entry->error)
);
+TRACE_EVENT(cachefiles_no_space,
+ TP_PROTO(struct cachefiles_object *obj, enum cachefiles_error_trace trace),
+
+ TP_ARGS(obj, trace),
+
+ TP_STRUCT__entry(
+ __field(unsigned int, obj)
+ __field(enum cachefiles_error_trace, trace)
+ ),
+
+ TP_fast_assign(
+ __entry->obj = obj ? obj->debug_id : 0;
+ __entry->trace = trace;
+ ),
+
+ TP_printk("o=%08x %s",
+ __entry->obj,
+ __print_symbolic(__entry->trace, cachefiles_error_traces))
+ );
+
#endif /* _TRACE_CACHEFILES_H */
/* This part must be outside protection */
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index dfe08492d1ad..54e90694eda7 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -102,8 +102,10 @@
EM(netfs_sreq_trace_abandoned, "ABNDN") \
EM(netfs_sreq_trace_add_donations, "+DON ") \
EM(netfs_sreq_trace_added, "ADD ") \
+ EM(netfs_sreq_trace_cache_nofile, "CA-!F") \
EM(netfs_sreq_trace_cache_nowrite, "CA-NW") \
EM(netfs_sreq_trace_cache_prepare, "CA-PR") \
+ EM(netfs_sreq_trace_cache_waitfail, "CA-!W") \
EM(netfs_sreq_trace_cache_write, "CA-WR") \
EM(netfs_sreq_trace_cancel, "CANCL") \
EM(netfs_sreq_trace_clear, "CLEAR") \
^ permalink raw reply [flat|nested] 39+ messages in thread
* READ_PLUS in NetFS? Re: [PATCH v10 30/35] netfs: Simplify read abandonment
2026-08-24 14:41 ` [PATCH v10 30/35] netfs: Simplify read abandonment David Howells
@ 2026-08-24 15:39 ` Aurélien Couderc
2026-08-24 19:47 ` David Howells
1 sibling, 0 replies; 39+ messages in thread
From: Aurélien Couderc @ 2026-08-24 15:39 UTC (permalink / raw)
To: David Howells
Cc: Paulo Alcantara, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, Linux NFS Mailing List, ceph-devel, v9fs,
linux-erofs, linux-fsdevel, linux-kernel
On Mon, Aug 24, 2026 at 5:04 PM David Howells <dhowells@redhat.com> wrote:
>
> Currently, if one or more of the subrequests in a read request fails, the
> read collection algorithm will attempt to salvage folios that are fully
> downloaded but that span multiple subreqs, provided all of the contributory
> subreqs succeeded, but this makes the algorithm quite complicated as a
> subreq can contribute to multiple folios and a folio can be contributed to
> by multiple subreqs.
>
> Simplify this by just abandoning the rest of a read request once a
> permanently failed subreq is hit. This is what read_folio, DIO/unbuffered
> read, gap filling, RMW and write preload all need to do; readahead is the
> odd one out as it doesn't have any return other than unlocking folios.
>
> With this change, even for readahead, the rest of the read is simply
> abandoned; read() and suchlike will call ->read_folio() on each
> non-uptodate folio to retry and retrieve the error.
>
> Note that retryable failures still get retried by netfslib as part of the
> request as before.
What about READ_PLUS, i.e. sparse file reads? If you do this kind of
work of handling partial abandonment, can such a work also benefit
READ_PLUS?
Aurélien
--
Aurélien Couderc <aurelien.couderc2002@gmail.com>
Big Data/Data mining expert, chess enthusiast
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: READ_PLUS in NetFS? Re: [PATCH v10 30/35] netfs: Simplify read abandonment
2026-08-24 14:41 ` [PATCH v10 30/35] netfs: Simplify read abandonment David Howells
2026-08-24 15:39 ` READ_PLUS in NetFS? " Aurélien Couderc
@ 2026-08-24 19:47 ` David Howells
1 sibling, 0 replies; 39+ messages in thread
From: David Howells @ 2026-08-24 19:47 UTC (permalink / raw)
To: =?UTF-8?Q?Aur=C3=A9lien_Couderc?=
Cc: dhowells, Paulo Alcantara, Christian Brauner, Matthew Wilcox,
Christoph Hellwig, Jens Axboe, Leon Romanovsky, Namjae Jeon,
ChenXiaoSong, Marc Dionne, Stefan Metzmacher,
Eric Van Hensbergen, Dominique Martinet, Ilya Dryomov, netfs,
linux-afs, linux-cifs, Linux NFS Mailing List, ceph-devel, v9fs,
linux-erofs, linux-fsdevel, linux-kernel
Aurélien Couderc <aurelien.couderc2002@gmail.com> wrote:
> What about READ_PLUS, i.e. sparse file reads? If you do this kind of
> work of handling partial abandonment, can such a work also benefit
> READ_PLUS?
READ_PLUS is NFS, right?
I'm not sure it would particularly help or hinder.
The VFS/VM only asks to read contiguous regions and netfslib breaks that up
into subrequests, but currently netfslib doesn't know anything about sparse
reads. What I'm looking at with Ceph, for example, is that Ceph knows where
the gaps are and will clear them before terminating the subrequest.
Partial abandonment is for when some sort of fatal error occurs and we cannot
finish the request.
David
^ permalink raw reply [flat|nested] 39+ messages in thread
end of thread, other threads:[~2026-08-24 19:47 UTC | newest]
Thread overview: 39+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
2026-08-24 14:40 ` [PATCH v10 01/35] netfs: Fix uninitialized return value in netfs_unbuffered_write() David Howells
2026-08-24 14:40 ` [PATCH v10 02/35] netfs: Fix read progress reporting David Howells
2026-08-24 14:40 ` [PATCH v10 03/35] cachefiles,netfs: sunset ondemand mode David Howells
2026-08-24 14:40 ` [PATCH v10 04/35] cachefiles: Fix potential UAF/KASAN warning David Howells
2026-08-24 14:40 ` [PATCH v10 05/35] netfs: Use uoff_t instead of unsigned long long and loff_t David Howells
2026-08-24 14:40 ` [PATCH v10 06/35] mm: Make readahead store folio count in readahead_control David Howells
2026-08-24 14:41 ` [PATCH v10 07/35] netfs: Bulk load the readahead-provided folios up front David Howells
2026-08-24 14:41 ` [PATCH v10 08/35] Add a function to kmap one page of a multipage bio_vec David Howells
2026-08-24 14:41 ` [PATCH v10 09/35] iov_iter: Make iov_iter_get_pages*() wrap iov_iter_extract_pages() David Howells
2026-08-24 14:41 ` [PATCH v10 10/35] iov_iter: Add a segmented queue of bio_vec[] David Howells
2026-08-24 14:41 ` [PATCH v10 11/35] netfs: Add some tools for managing bvecq chains David Howells
2026-08-24 14:41 ` [PATCH v10 12/35] netfs: Make mempool available for bvecq David Howells
2026-08-24 14:41 ` [PATCH v10 13/35] netfs: Add a function to extract from an iter into a bvecq David Howells
2026-08-24 14:41 ` [PATCH v10 14/35] afs: Use a bvecq to hold dir content rather than folioq David Howells
2026-08-24 14:41 ` [PATCH v10 15/35] cifs: Use a bvecq for buffering instead of a folioq David Howells
2026-08-24 14:41 ` [PATCH v10 16/35] smbdirect: Support ITER_BVECQ in smbdirect_map_sges_from_iter() David Howells
2026-08-24 14:41 ` [PATCH v10 17/35] netfs: Remove the writethrough code David Howells
2026-08-24 14:41 ` [PATCH v10 18/35] netfs: trace: Change the "clear" folio traces to "endwb" David Howells
2026-08-24 14:41 ` [PATCH v10 19/35] netfs: trace: Rejig a couple of the tracepoints David Howells
2026-08-24 14:41 ` [PATCH v10 20/35] netfs: Add some functions to wrap the all-queued handling David Howells
2026-08-24 14:41 ` [PATCH v10 21/35] netfs: Make deprecated PG_private_2 support optional David Howells
2026-08-24 14:41 ` [PATCH v10 22/35] cachefiles: Don't rely on backing fs storage map for most use cases David Howells
2026-08-24 14:41 ` [PATCH v10 23/35] netfs: Add the cache object ID to netfs_read/write tracepoints David Howells
2026-08-24 14:41 ` [PATCH v10 24/35] netfs: Switch to using bvecq rather than folio_queue and rolling_buffer David Howells
2026-08-24 14:41 ` [PATCH v10 25/35] smbdirect: Remove support for ITER_FOLIOQ from smbdirect_map_sges_from_iter() David Howells
2026-08-24 14:41 ` [PATCH v10 26/35] netfs: Remove netfs_alloc/free_folioq_buffer() David Howells
2026-08-24 14:41 ` [PATCH v10 27/35] netfs: Remove netfs_extract_user_iter() David Howells
2026-08-24 14:41 ` [PATCH v10 28/35] iov_iter: Remove ITER_FOLIOQ David Howells
2026-08-24 14:41 ` [PATCH v10 29/35] netfs: Remove folio_queue and rolling_buffer David Howells
2026-08-24 14:41 ` [PATCH v10 30/35] netfs: Simplify read abandonment David Howells
2026-08-24 15:39 ` READ_PLUS in NetFS? " Aurélien Couderc
2026-08-24 19:47 ` David Howells
2026-08-24 14:41 ` [PATCH v10 31/35] netfs: Check for too much data being read David Howells
2026-08-24 14:41 ` [PATCH v10 32/35] netfs: Add a method to get an estimate of the amount that can be written David Howells
2026-08-24 14:41 ` [PATCH v10 33/35] netfs: Rework writeback to use a separate list of regions to be unlocked David Howells
2026-08-24 14:41 ` [PATCH v10 34/35] netfs: Combine prepare and issue ops and grab the buffers on request David Howells
2026-08-24 14:41 ` [PATCH v10 35/35] netfs: Clean up now-unused code David Howells
2026-08-24 15:01 ` [PATCH v10 36/35] cachefiles: Preset the state xattr when creating a new file David Howells
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®