From: David Howells <dhowells@redhat.com>
To: Paulo Alcantara <pc@manguebit.org>
Cc: David Howells <dhowells@redhat.com>,
Christian Brauner <christian@brauner.io>,
Matthew Wilcox <willy@infradead.org>,
Christoph Hellwig <hch@infradead.org>,
Jens Axboe <axboe@kernel.dk>, Leon Romanovsky <leon@kernel.org>,
Namjae Jeon <linkinjeon@kernel.org>,
ChenXiaoSong <chenxiaosong@chenxiaosong.com>,
Marc Dionne <marc.dionne@auristor.com>,
Stefan Metzmacher <metze@samba.org>,
Eric Van Hensbergen <ericvh@kernel.org>,
Dominique Martinet <asmadeus@codewreck.org>,
Ilya Dryomov <idryomov@gmail.com>,
netfs@lists.linux.dev, linux-afs@lists.infradead.org,
linux-cifs@vger.kernel.org, linux-nfs@vger.kernel.org,
ceph-devel@vger.kernel.org, v9fs@lists.linux.dev,
linux-erofs@lists.ozlabs.org, linux-fsdevel@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: [PATCH v10 34/35] netfs: Combine prepare and issue ops and grab the buffers on request
Date: Mon, 24 Aug 2026 15:41:27 +0100 [thread overview]
Message-ID: <20260824144130.759997-35-dhowells@redhat.com> (raw)
In-Reply-To: <20260824144130.759997-1-dhowells@redhat.com>
Modify the way subrequests are generated in netfslib to try and simplify
the code. The issue, primarily, is in writeback: the code has to create
multiple streams of write requests to disparate targets with different
properties (e.g. server and fscache), where not every folio needs to go to
every target (e.g. data just read from the server may only need writing to
the cache).
The current model in writeback, at least, is to go carefully through every
folio, preparing a subrequest for each stream when it was detected that
part of the current folio needed to go to that stream, and repeating this
within and across contiguous folios; then to issue subrequests as they
become full or hit boundaries after first setting up the buffer. However,
this is quite difficult to follow - and makes it tricky to handle
discontiguous folios in a request.
This is changed such that netfs now accumulates buffers and attaches them
to each stream when they become valid for that stream, then flushes the
stream when a limit or a boundary is hit. The issuing code in netfs then
loops around creating and issuing subrequests without calling a separate
prepare stage (though a function is provided to get an estimate of when
flushing should occur). The filesystem (or cache) then gets to take a
slice of the master bvec chain as its I/O buffer for each subrequest,
including discontiguities if it can support a sparse/vectored RPC (as Ceph
can).
Similar-ish changes also apply to buffered read and unbuffered read and
write, though in each of those cases there is only a single contiguous
stream. Though for buffered read this consists of interwoven requests from
multiple sources (server or cache).
To this end, netfslib is changed in the following ways:
(1) ->prepare_xxx(), buffer selection and ->issue_xxx() are now collapsed
together such that one ->issue_xxx() call is made with the subrequest
defined to the maximum extent; the filesystem/cache then reduces the
length of the subrequest and calls back to netfslib to grab a slice of
the buffer, which may reduce the subrequest further if a maximum
segment limit is set. The filesystem/cache then dispatches the
operation.
(2) Retry buffer tracking is added to the netfs_io_request struct. This
is then selected by the subrequest retry counter being non-zero.
(3) The use of iov_iter is pushed down to the filesystem. Netfslib now
provides the filesystem with a bvecq holding the buffer rather than an
iov_iter. The bvecq can be duplicated and headers/trailers attached
to hold protocol and several duplicated bvecqs can be linked together
to create a compound operation.
(4) If the ->issue_xxx() functions terminate with -ENOMEM, a flag is set
on the request to abort further subrequest generation/retrying. If
it fails early with any other error, generally the subreq will be
extended as much as possible and failed with the error returned.
(5) During writeback, netfslib now builds up an accumulation of buffered
data before issuing writes on each stream (one server, one cache). It
asks each stream for an estimate of how much data to accumulate before
it next generates subrequests on the stream. The filesystem or cache
is not required to use up all the data accumulated on a stream at that
time unless the end of the pagecache is hit.
(6) During read-gaps, in which there are two gaps on either end of a dirty
streaming write page that need to be filled, a buffer is constructed
consisting of the two ends plus a sink page repeated to cover the
middle portion. This is passed to the server as a single write. For
something like Ceph, this should probably be done either as a
vectored/sparse read or as two separate reads (if different Ceph
objects are involved).
(7) During unbuffered/DIO read/write, there is a single contiguous file
region to be read or written as a single stream. The dispatching
function just creates subrequests and calls ->issue_xxx() repeatedly
to eat through the bufferage.
(8) At the start of buffered read, the entire set of folios allocated by
VM readahead is loaded into a bvecq chain, rather than trying to do it
piecemeal as-needed. As the pages were already added and locked by
the VM, this is slightly more efficient than loading piecemeal as only
a single iteration of the xarray is required.
(9) During buffered read, there is a single contiguous file region, to
read as a single stream - however, this stream may be stitched
together from subrequests to multiple sources. Which sources are used
where is now determined by querying the cache to find the next couple
of extents in which it has data; netfslib uses this to direct the
subrequests towards the appropriate sources.
Each subrequest is given the maximum length in the current extent and
then ->issue_read() is called. The filesystem then limits the size
and slices off a piece of the buffer for that extent.
(10) Cachefiles now provides an estimation function that indicates the
standard maxima for doing DIO (MAX_RW_COUNT and BIO_MAX_VECS).
Note that sparse cachefiles still rely on the backing filesystem for
content mapping. That will need to be addressed in a future patch and is
not trivial to fix.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Paulo Alcantara <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
fs/9p/vfs_addr.c | 43 ++++--
fs/afs/file.c | 27 ++--
fs/afs/fsclient.c | 8 +-
fs/afs/internal.h | 5 +-
fs/afs/write.c | 33 ++---
fs/afs/yfsclient.c | 6 +-
fs/cachefiles/io.c | 230 ++++++++++++++++++++++---------
fs/ceph/addr.c | 123 ++++++++++-------
fs/netfs/buffered_read.c | 257 +++++++++++++++++++----------------
fs/netfs/direct_read.c | 84 +++++++-----
fs/netfs/direct_write.c | 141 ++++++++++---------
fs/netfs/fscache_io.c | 2 +-
fs/netfs/internal.h | 43 ++++--
fs/netfs/main.c | 1 +
fs/netfs/misc.c | 33 ++++-
fs/netfs/objects.c | 4 +-
fs/netfs/read_collect.c | 38 ++----
fs/netfs/read_pgpriv2.c | 125 +++++++++++------
fs/netfs/read_retry.c | 208 +++++++++++++---------------
fs/netfs/read_single.c | 155 +++++++++++++--------
fs/netfs/write_issue.c | 157 +++++++++++++--------
fs/netfs/write_retry.c | 168 ++++++++++++-----------
fs/nfs/fscache.c | 24 +++-
fs/smb/client/cifssmb.c | 13 +-
fs/smb/client/file.c | 134 ++++++++----------
fs/smb/client/smb2ops.c | 9 +-
fs/smb/client/smb2pdu.c | 28 ++--
fs/smb/client/transport.c | 15 +-
include/linux/netfs.h | 54 +++-----
include/trace/events/netfs.h | 30 +++-
net/9p/client.c | 8 +-
31 files changed, 1301 insertions(+), 905 deletions(-)
diff --git a/fs/9p/vfs_addr.c b/fs/9p/vfs_addr.c
index 9b163a13ec3f..fc5b9c677986 100644
--- a/fs/9p/vfs_addr.c
+++ b/fs/9p/vfs_addr.c
@@ -67,28 +67,51 @@ static int v9fs_estimate_write(struct netfs_io_request *wreq,
/*
* Issue a subrequest to write to the server.
*/
-static void v9fs_issue_write(struct netfs_io_subrequest *subreq)
+static int v9fs_issue_write(struct netfs_io_subrequest *subreq)
{
+ struct iov_iter iter;
struct p9_fid *fid = subreq->rreq->netfs_priv;
- int err, len;
+ int err, len = 0;
- len = p9_client_write(fid, subreq->start, &subreq->io_iter, &err);
+ subreq->len = umin(subreq->len, fid->clnt->msize - P9_IOHDRSZ);
+
+ err = netfs_prepare_write_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
+ len = p9_client_write(fid, subreq->start, &iter, &err);
if (len > 0)
__set_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+
netfs_write_subrequest_terminated(subreq, len ?: err);
+ return 0;
}
/**
* v9fs_issue_read - Issue a read from 9P
* @subreq: The read to make
*/
-static void v9fs_issue_read(struct netfs_io_subrequest *subreq)
+static int v9fs_issue_read(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
+ struct iov_iter iter;
struct p9_fid *fid = rreq->netfs_priv;
char *target;
unsigned long long pos = subreq->start + subreq->transferred;
- int total = 0, err, len, n;
+ size_t len;
+ int total = 0, err;
+
+ err = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
if (S_ISLNK(rreq->inode->i_mode)) {
/* p9_client_readlink() must not be called for legacy protocols
@@ -105,12 +128,13 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)
err = p9_client_readlink(fid, &target);
if (err != 0)
goto fill_subreq;
- len = strlen(target);
- n = copy_to_iter(target, len, &subreq->io_iter);
+ len = min(strlen(target), subreq->len);
+ total = copy_to_iter(target, len, &iter);
kfree(target);
- total = n;
} else {
- total = p9_client_read(fid, pos, &subreq->io_iter, &err);
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+
+ total = p9_client_read(fid, pos, &iter, &err);
}
fill_subreq:
@@ -128,6 +152,7 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)
subreq->error = err;
netfs_read_subreq_terminated(subreq);
+ return 0;
}
/**
diff --git a/fs/afs/file.c b/fs/afs/file.c
index 4a412e0472ec..a6b61d609f02 100644
--- a/fs/afs/file.c
+++ b/fs/afs/file.c
@@ -332,11 +332,12 @@ void afs_fetch_data_immediate_cancel(struct afs_call *call)
/*
* Fetch file data from the volume.
*/
-static void afs_issue_read(struct netfs_io_subrequest *subreq)
+static int afs_issue_read(struct netfs_io_subrequest *subreq)
{
struct afs_operation *op;
struct afs_vnode *vnode = AFS_FS_I(subreq->rreq->inode);
struct key *key = subreq->rreq->netfs_priv;
+ int ret;
_enter("%s{%llx:%llu.%u},%x,,,",
vnode->volume->name,
@@ -345,11 +346,15 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq)
vnode->fid.unique,
key_serial(key));
+ ret = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
+
op = afs_alloc_operation(key, vnode->volume);
if (IS_ERR(op)) {
- subreq->error = PTR_ERR(op);
- netfs_read_subreq_terminated(subreq);
- return;
+ ret = PTR_ERR(op);
+ goto failed;
}
afs_op_set_vnode(op, 0, vnode);
@@ -364,20 +369,24 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq)
op->flags |= AFS_OPERATION_ASYNC;
if (!afs_begin_vnode_operation(op)) {
- subreq->error = afs_put_operation(op);
- netfs_read_subreq_terminated(subreq);
- return;
+ ret = afs_put_operation(op);
+ goto failed;
}
if (!afs_select_fileserver(op)) {
- afs_end_read(op);
- return;
+ afs_end_read(op); /* Error recorded here. */
+ return 0;
}
afs_issue_read_call(op);
} else {
afs_do_sync_operation(op);
}
+ return 0;
+failed:
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ return 0;
}
static int afs_init_request(struct netfs_io_request *rreq, struct file *file)
diff --git a/fs/afs/fsclient.c b/fs/afs/fsclient.c
index 1a3f186a6a11..9713551dd842 100644
--- a/fs/afs/fsclient.c
+++ b/fs/afs/fsclient.c
@@ -339,7 +339,9 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call)
if (call->remaining == 0)
goto no_more_data;
- call->iter = &subreq->io_iter;
+ iov_iter_bvec_queue(&call->def_iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
call->iov_len = umin(call->remaining, subreq->len - subreq->transferred);
call->unmarshall++;
fallthrough;
@@ -1088,7 +1090,7 @@ static void afs_fs_store_data64(struct afs_operation *op)
if (!call)
return afs_op_nomem(op);
- call->write_iter = op->store.write_iter;
+ call->write_iter = &op->store.write_iter;
/* marshall the parameters */
bp = call->request;
@@ -1142,7 +1144,7 @@ void afs_fs_store_data(struct afs_operation *op)
if (!call)
return afs_op_nomem(op);
- call->write_iter = op->store.write_iter;
+ call->write_iter = &op->store.write_iter;
/* marshall the parameters */
bp = call->request;
diff --git a/fs/afs/internal.h b/fs/afs/internal.h
index ae2bc699bfde..5c536cc7b53c 100644
--- a/fs/afs/internal.h
+++ b/fs/afs/internal.h
@@ -914,7 +914,7 @@ struct afs_operation {
afs_lock_type_t type;
} lock;
struct {
- struct iov_iter *write_iter;
+ struct iov_iter write_iter;
loff_t pos;
loff_t size;
loff_t i_size;
@@ -1701,8 +1701,7 @@ extern int afs_check_volume_status(struct afs_volume *, struct afs_operation *);
int afs_estimate_write(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
struct netfs_write_estimate *estimate);
-void afs_prepare_write(struct netfs_io_subrequest *subreq);
-void afs_issue_write(struct netfs_io_subrequest *subreq);
+int afs_issue_write(struct netfs_io_subrequest *subreq);
void afs_begin_writeback(struct netfs_io_request *wreq);
void afs_retry_request(struct netfs_io_request *wreq, struct netfs_io_stream *stream);
extern int afs_writepages(struct address_space *, struct writeback_control *);
diff --git a/fs/afs/write.c b/fs/afs/write.c
index a0c9916f594b..46f81d9e7d7f 100644
--- a/fs/afs/write.c
+++ b/fs/afs/write.c
@@ -99,20 +99,6 @@ int afs_estimate_write(struct netfs_io_request *wreq,
return 0;
}
-/*
- * Prepare a subrequest to write to the server. This sets the max_len
- * parameter.
- */
-void afs_prepare_write(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_stream *stream = &subreq->rreq->io_streams[subreq->stream_nr];
-
- //if (test_bit(NETFS_SREQ_RETRYING, &subreq->flags))
- // subreq->max_len = 512 * 1024;
- //else
- stream->sreq_max_len = 256 * 1024 * 1024;
-}
-
/*
* Issue a subrequest to write to the server.
*/
@@ -156,12 +142,15 @@ static void afs_issue_write_worker(struct work_struct *work)
op->flags |= AFS_OPERATION_UNINTR;
op->ops = &afs_store_data_operation;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
afs_begin_vnode_operation(op);
- op->store.write_iter = &subreq->io_iter;
op->store.i_size = umax(pos + len, netfs_read_remote_i_size(&vnode->netfs.inode));
op->mtime = inode_get_mtime(&vnode->netfs.inode);
+ iov_iter_bvec_queue(&op->store.write_iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
afs_wait_for_operation(op);
ret = afs_put_operation(op);
switch (ret) {
@@ -185,11 +174,21 @@ static void afs_issue_write_worker(struct work_struct *work)
netfs_write_subrequest_terminated(subreq, ret < 0 ? ret : subreq->len);
}
-void afs_issue_write(struct netfs_io_subrequest *subreq)
+int afs_issue_write(struct netfs_io_subrequest *subreq)
{
+ int ret;
+
+ if (subreq->len > 256 * 1024 * 1024)
+ subreq->len = 256 * 1024 * 1024;
+ ret = netfs_prepare_write_buffer(subreq, INT_MAX);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
+
subreq->work.func = afs_issue_write_worker;
if (!queue_work(system_dfl_wq, &subreq->work))
WARN_ON_ONCE(1);
+ return 0;
}
/*
@@ -200,6 +199,8 @@ void afs_begin_writeback(struct netfs_io_request *wreq)
{
if (S_ISREG(wreq->inode->i_mode))
afs_get_writeback_key(wreq);
+
+ wreq->io_streams[0].avail = true;
}
/*
diff --git a/fs/afs/yfsclient.c b/fs/afs/yfsclient.c
index d941179730a9..52c588092050 100644
--- a/fs/afs/yfsclient.c
+++ b/fs/afs/yfsclient.c
@@ -385,7 +385,9 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call)
if (call->remaining == 0)
goto no_more_data;
- call->iter = &subreq->io_iter;
+ iov_iter_bvec_queue(&call->def_iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
call->iov_len = min(call->remaining, subreq->len - subreq->transferred);
call->unmarshall++;
fallthrough;
@@ -1357,7 +1359,7 @@ void yfs_fs_store_data(struct afs_operation *op)
if (!call)
return afs_op_nomem(op);
- call->write_iter = op->store.write_iter;
+ call->write_iter = &op->store.write_iter;
/* marshall the parameters */
bp = call->request;
diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c
index 91b69be78c6b..a1580385d5fb 100644
--- a/fs/cachefiles/io.c
+++ b/fs/cachefiles/io.c
@@ -197,12 +197,131 @@ static int cachefiles_read(struct netfs_cache_resources *cres,
return ret;
}
+/*
+ * Handle completion of a read from the cache issued by netfslib.
+ */
+static void cachefiles_issue_read_complete(struct kiocb *iocb, long ret)
+{
+ struct cachefiles_kiocb *ki = container_of(iocb, struct cachefiles_kiocb, iocb);
+ struct netfs_io_subrequest *subreq = ki->subreq;
+ struct inode *inode = file_inode(ki->iocb.ki_filp);
+
+ _enter("%ld", ret);
+
+ if (ret < 0) {
+ subreq->error = -ESTALE;
+ trace_cachefiles_io_error(ki->object, inode, ret,
+ cachefiles_trace_read_error);
+ }
+
+ if (ret >= 0) {
+ if (ki->object->cookie->inval_counter == ki->inval_counter) {
+ subreq->error = 0;
+ if (ret > 0) {
+ subreq->transferred += ret;
+ __set_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ }
+ } else {
+ subreq->error = -ESTALE;
+ }
+ }
+
+ netfs_read_subreq_terminated(subreq);
+ cachefiles_put_kiocb(ki);
+}
+
+/*
+ * Issue a read operation to the cache. We assume that the cache is readable
+ * as the caller should have called ->query_occupancy() first.
+ */
+static int cachefiles_issue_read(struct netfs_io_subrequest *subreq)
+{
+ struct netfs_cache_resources *cres = &subreq->rreq->cache_resources;
+ struct cachefiles_object *object = cachefiles_cres_object(cres);
+ struct cachefiles_kiocb *ki;
+ struct iov_iter iter;
+ struct file *file = cachefiles_cres_file(cres);
+ unsigned int old_nofs;
+ ssize_t ret;
+
+ fscache_count_read();
+
+ _enter("%pD,%lli,%llx,%zx/%llx",
+ file, file_inode(file)->i_ino, subreq->start, subreq->len,
+ i_size_read(file_inode(file)));
+
+ if (subreq->len > MAX_RW_COUNT)
+ subreq->len = MAX_RW_COUNT;
+
+ ret = netfs_prepare_read_buffer(subreq, BIO_MAX_VECS);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
+ ret = -ENOMEM;
+ ki = kzalloc_obj(struct cachefiles_kiocb);
+ if (!ki)
+ goto failed;
+
+ refcount_set(&ki->ki_refcnt, 2);
+ ki->iocb.ki_filp = file;
+ ki->iocb.ki_pos = subreq->start;
+ ki->iocb.ki_flags = IOCB_DIRECT;
+ ki->iocb.ki_ioprio = get_current_ioprio();
+ ki->iocb.ki_complete = cachefiles_issue_read_complete;
+ ki->object = object;
+ ki->inval_counter = cres->inval_counter;
+ ki->subreq = subreq;
+ ki->was_async = true;
+
+ get_file(ki->iocb.ki_filp);
+ cachefiles_grab_object(object, cachefiles_obj_get_ioreq);
+
+ trace_cachefiles_read(object, file_inode(file), ki->iocb.ki_pos, subreq->len);
+ old_nofs = memalloc_nofs_save();
+ ret = cachefiles_inject_read_error();
+ if (ret == 0)
+ ret = vfs_iocb_iter_read(file, &ki->iocb, &iter);
+ memalloc_nofs_restore(old_nofs);
+
+ switch (ret) {
+ case -EIOCBQUEUED:
+ break;
+
+ case -ERESTARTSYS:
+ case -ERESTARTNOINTR:
+ case -ERESTARTNOHAND:
+ case -ERESTART_RESTARTBLOCK:
+ /* There's no easy way to restart the syscall since other AIO's
+ * may be already running. Just fail this IO with EINTR.
+ */
+ ret = -EINTR;
+ fallthrough;
+ default:
+ ki->was_async = false;
+ cachefiles_issue_read_complete(&ki->iocb, ret);
+ break;
+ }
+
+ cachefiles_put_kiocb(ki);
+ _leave(" = %zd", ret);
+ return 0;
+failed:
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ return 0;
+}
+
/*
* Query the occupancy of the cache in a region, returning the extent of the
- * next two chunks of cached data and the next hole.
+ * next two chunks of cached data and the next hole. The occupancy map is
+ * preloaded to show just one giant hole.
*/
-static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
- struct fscache_occupancy *occ)
+static void cachefiles_query_occupancy(struct netfs_cache_resources *cres,
+ struct fscache_occupancy *occ)
{
struct cachefiles_object *object;
struct inode *inode;
@@ -212,7 +331,7 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
int i;
if (!fscache_wait_for_operation(cres, FSCACHE_WANT_READ))
- return -ENOBUFS;
+ return;
object = cachefiles_cres_object(cres);
file = cachefiles_cres_file(cres);
@@ -247,7 +366,7 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
ret = vfs_llseek(file, occ->query_from, SEEK_DATA);
if (IS_ERR_VALUE_LL(ret)) {
if (ret != -ENXIO)
- return ret;
+ goto done;
occ->query_from = ULLONG_MAX;
goto done;
}
@@ -260,7 +379,7 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
ret = vfs_llseek(file, occ->query_from, SEEK_HOLE);
if (IS_ERR_VALUE_LL(ret)) {
if (ret != -ENXIO)
- return ret;
+ goto done;
occ->query_from = ULLONG_MAX;
goto done;
}
@@ -273,7 +392,6 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
done:
_debug("query[0] %llx-%llx", occ->cached_from[0], occ->cached_to[0]);
_debug("query[1] %llx-%llx", occ->cached_from[1], occ->cached_to[1]);
- return 0;
}
/*
@@ -590,9 +708,9 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
cachefiles_has_space_for_write);
}
-static int cachefiles_prepare_write(struct netfs_cache_resources *cres,
- uoff_t *_start, size_t *_len, size_t upper_len,
- uoff_t i_size, bool no_space_allocated_yet)
+static int cachefiles_prepare_write_old(struct netfs_cache_resources *cres,
+ uoff_t *_start, size_t *_len, size_t upper_len,
+ uoff_t i_size, bool no_space_allocated_yet)
{
struct cachefiles_object *object = cachefiles_cres_object(cres);
struct cachefiles_cache *cache = object->volume->cache;
@@ -623,80 +741,63 @@ static int cachefiles_estimate_write(struct netfs_io_request *wreq,
return 0;
}
-static void cachefiles_prepare_write_subreq(struct netfs_io_subrequest *subreq)
+static int cachefiles_issue_write(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *wreq = subreq->rreq;
struct netfs_cache_resources *cres = &wreq->cache_resources;
- struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
-
- _enter("W=%x[%x] %llx", wreq->debug_id, subreq->debug_index, subreq->start);
+ struct cachefiles_object *object = cachefiles_cres_object(cres);
+ struct cachefiles_cache *cache = object->volume->cache;
+ struct iov_iter iter;
+ const struct cred *saved_cred;
+ uoff_t start = subreq->start;
+ size_t len = subreq->len;
+ int ret = -EINVAL;
- stream->sreq_max_len = MAX_RW_COUNT;
- stream->sreq_max_segs = BIO_MAX_VECS;
+ _enter("W=%x[%x] %llx-%llx",
+ wreq->debug_id, subreq->debug_index, start, start + len - 1);
if (!cachefiles_cres_file(cres)) {
if (!fscache_wait_for_operation(cres, FSCACHE_WANT_WRITE))
- return netfs_prepare_write_failed(subreq);
+ return -ENOBUFS;
if (!cachefiles_cres_file(cres))
- return netfs_prepare_write_failed(subreq);
+ return -ENOBUFS;
}
-}
-static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_request *wreq = subreq->rreq;
- struct netfs_cache_resources *cres = &wreq->cache_resources;
- struct cachefiles_object *object = cachefiles_cres_object(cres);
- struct cachefiles_cache *cache = object->volume->cache;
- const struct cred *saved_cred;
- size_t off, pre, post, len = subreq->len;
- uoff_t start = subreq->start;
- int ret;
+ ret = netfs_prepare_write_buffer(subreq, BIO_MAX_VECS);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
- _enter("W=%x[%x] %llx-%llx",
- wreq->debug_id, subreq->debug_index, start, start + len - 1);
+ /* The buffer extraction func may round out start and end. */
+ start = subreq->start;
+ len = subreq->len;
- /* We need to start on the cache granularity boundary */
- off = start & (cache->bsize - 1);
- if (off) {
- pre = cache->bsize - off;
- if (pre >= len) {
- fscache_count_dio_misfit();
- netfs_write_subrequest_terminated(subreq, len);
- return;
- }
- subreq->transferred += pre;
- start += pre;
- len -= pre;
- iov_iter_advance(&subreq->io_iter, pre);
- }
-
- /* We also need to end on the cache granularity boundary */
- post = len & (cache->bsize - 1);
- if (post) {
- len -= post;
- if (len == 0) {
- fscache_count_dio_misfit();
- netfs_write_subrequest_terminated(subreq, post);
- return;
- }
- iov_iter_truncate(&subreq->io_iter, len);
+ /* We need to start and end on cache granularity boundaries. */
+ if (WARN_ON_ONCE(start & (cache->bsize - 1)) ||
+ WARN_ON_ONCE(len & (cache->bsize - 1))) {
+ fscache_count_dio_misfit();
+ ret = -EIO;
+ goto failed;
}
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, len);
+
trace_netfs_sreq(subreq, netfs_sreq_trace_cache_prepare);
cachefiles_begin_secure(cache, &saved_cred);
ret = __cachefiles_prepare_write(object, cachefiles_cres_file(cres),
&start, &len, len, true);
cachefiles_end_secure(cache, saved_cred);
- if (ret < 0) {
- netfs_write_subrequest_terminated(subreq, ret);
- return;
- }
+ if (ret < 0)
+ goto failed;
trace_netfs_sreq(subreq, netfs_sreq_trace_cache_write);
- cachefiles_write(&subreq->rreq->cache_resources,
- subreq->start, &subreq->io_iter,
+ cachefiles_write(&subreq->rreq->cache_resources, subreq->start, &iter,
netfs_write_subrequest_terminated, subreq);
+ return 0;
+failed:
+ netfs_write_subrequest_terminated(subreq, ret);
+ return 0;
}
/*
@@ -753,6 +854,7 @@ static void cachefiles_collect_write(struct netfs_io_request *wreq,
WARN_ON(new_size & (cache->bsize - 1)) ||
WARN_ON(start & (cache->bsize - 1)) ||
WARN_ON(len & (cache->bsize - 1))) {
+ trace_netfs_rreq(wreq, netfs_rreq_trace_cache_align_error);
trace_cachefiles_io_error(object, inode, -EIO,
cachefiles_trace_alignment_error);
cachefiles_remove_object_xattr(cache, object, file->f_path.dentry);
@@ -907,10 +1009,10 @@ static const struct netfs_cache_ops cachefiles_netfs_cache_ops = {
.end_operation = cachefiles_end_operation,
.read = cachefiles_read,
.write = cachefiles_write,
+ .issue_read = cachefiles_issue_read,
.issue_write = cachefiles_issue_write,
- .prepare_write = cachefiles_prepare_write,
- .prepare_write_subreq = cachefiles_prepare_write_subreq,
.estimate_write = cachefiles_estimate_write,
+ .prepare_write_old = cachefiles_prepare_write_old,
.query_occupancy = cachefiles_query_occupancy,
.collect_write = cachefiles_collect_write,
};
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index 6586f6c1dc73..b13c071bd574 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -274,7 +274,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
ceph_dec_osd_stopping_blocker(fsc->mdsc);
}
-static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
+static int ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
struct inode *inode = rreq->inode;
@@ -283,7 +283,8 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
struct ceph_mds_request *req;
struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(inode->i_sb);
struct ceph_inode_info *ci = ceph_inode(inode);
- ssize_t err = 0;
+ struct iov_iter iter;
+ ssize_t err;
size_t len;
int mode;
@@ -292,8 +293,20 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
__set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags);
__clear_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags);
- if (subreq->start >= inode->i_size)
+ err = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ if (subreq->start >= inode->i_size) {
+ __set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags);
+ err = 0;
goto out;
+ }
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset,
+ subreq->len);
/* We need to fetch the inline data. */
mode = ceph_try_to_choose_auth_mds(inode, CEPH_STAT_CAP_INLINE_DATA);
@@ -302,11 +315,13 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
err = PTR_ERR(req);
goto out;
}
+
req->r_ino1 = ci->i_vino;
req->r_args.getattr.mask = cpu_to_le32(CEPH_STAT_CAP_INLINE_DATA);
req->r_num_caps = 2;
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+
err = ceph_mdsc_do_request(mdsc, NULL, req);
if (err < 0)
goto out;
@@ -316,11 +331,13 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
if (iinfo->inline_version == CEPH_INLINE_NONE) {
/* The data got uninlined */
ceph_mdsc_put_request(req);
- return false;
+ __set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ err = -EAGAIN;
+ goto out;
}
len = min_t(size_t, iinfo->inline_len - subreq->start, subreq->len);
- err = copy_to_iter(iinfo->inline_data + subreq->start, len, &subreq->io_iter);
+ err = copy_to_iter(iinfo->inline_data + subreq->start, len, &iter);
if (err == 0) {
err = -EFAULT;
} else {
@@ -333,26 +350,10 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
subreq->error = err;
trace_netfs_sreq(subreq, netfs_sreq_trace_io_progress);
netfs_read_subreq_terminated(subreq);
- return true;
-}
-
-static int ceph_netfs_prepare_read(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_request *rreq = subreq->rreq;
- struct inode *inode = rreq->inode;
- struct ceph_inode_info *ci = ceph_inode(inode);
- struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
- u64 objno, objoff;
- u32 xlen;
-
- /* Truncate the extent at the end of the current block */
- ceph_calc_file_object_mapping(&ci->i_layout, subreq->start, subreq->len,
- &objno, &objoff, &xlen);
- rreq->io_streams[0].sreq_max_len = umin(xlen, fsc->mount_options->rsize);
return 0;
}
-static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
+static int ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
struct inode *inode = rreq->inode;
@@ -361,19 +362,18 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
struct ceph_client *cl = fsc->client;
struct ceph_osd_request *req = NULL;
struct ceph_vino vino = ceph_vino(inode);
- int err;
- u64 len;
+ struct iov_iter iter;
+ u64 objno, objoff, len, off = subreq->start;
+ u32 maxlen;
+ int err = -EIO;
bool sparse = IS_ENCRYPTED(inode) || ceph_test_mount_opt(fsc, SPARSEREAD);
- u64 off = subreq->start;
int extent_cnt;
- if (ceph_inode_is_shutdown(inode)) {
- err = -EIO;
- goto out;
- }
+ if (ceph_inode_is_shutdown(inode))
+ return -EIO;
- if (ceph_has_inline_data(ci) && ceph_netfs_issue_op_inline(subreq))
- return;
+ if (ceph_has_inline_data(ci))
+ return ceph_netfs_issue_op_inline(subreq);
// TODO: This rounding here is slightly dodgy. It *should* work, for
// now, as the cache only deals in blocks that are a multiple of
@@ -383,26 +383,48 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
len = subreq->len;
ceph_fscrypt_adjust_off_and_len(inode, &off, &len);
+ /* Truncate the extent at the end of the current block */
+ ceph_calc_file_object_mapping(&ci->i_layout, subreq->start, len,
+ &objno, &objoff, &maxlen);
+ maxlen = min(maxlen, fsc->mount_options->rsize);
+ len = min(len, maxlen);
+ subreq->len = len;
+
+ /* Grab a slice of read buffer. This may shrink the subreq. */
+ err = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ /* Create a request. In theory, this may shrink the request again, but
+ * it shouldn't since we calculated the object size above and already
+ * shrank to that, but if it does, we'll just end up doing a short read
+ * and retrying to get the rest.
+ */
+ len = subreq->len;
req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout, vino,
off, &len, 0, 1, sparse ? CEPH_OSD_OP_SPARSE_READ : CEPH_OSD_OP_READ,
CEPH_OSD_FLAG_READ, NULL, ci->i_truncate_seq,
ci->i_truncate_size, false);
if (IS_ERR(req)) {
err = PTR_ERR(req);
- req = NULL;
- goto out;
+ goto failed_noput;
}
if (sparse) {
extent_cnt = __ceph_sparse_read_ext_count(inode, len);
err = ceph_alloc_sparse_ext_map(&req->r_ops[0], extent_cnt);
if (err)
- goto out;
+ goto failed;
}
doutc(cl, "%llx.%llx pos=%llu orig_len=%zu len=%llu\n",
ceph_vinop(inode), subreq->start, subreq->len, len);
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset,
+ subreq->len);
+
/*
* FIXME: For now, use CEPH_OSD_DATA_TYPE_PAGES instead of _ITER for
* encrypted inodes. We'd need infrastructure that handles an iov_iter
@@ -421,13 +443,11 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
* ceph_msg_data_cursor_init() triggers BUG_ON() in the case
* if msg->sparse_read_total > msg->data_length.
*/
- subreq->io_iter.count = len;
-
- err = iov_iter_get_pages_alloc2(&subreq->io_iter, &pages, len, &page_off);
+ err = iov_iter_get_pages_alloc2(&iter, &pages, len, &page_off);
if (err < 0) {
doutc(cl, "%llx.%llx failed to allocate pages, %d\n",
ceph_vinop(inode), err);
- goto out;
+ goto eio;
}
/* should always give us a page-aligned read */
@@ -438,12 +458,10 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
osd_req_op_extent_osd_data_pages(req, 0, pages, len, 0, false,
false);
} else {
- osd_req_op_extent_osd_iter(req, 0, &subreq->io_iter);
- }
- if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) {
- err = -EIO;
- goto out;
+ osd_req_op_extent_osd_iter(req, 0, &iter);
}
+ if (!ceph_inc_osd_stopping_blocker(fsc->mdsc))
+ goto eio;
req->r_callback = finish_netfs_read;
req->r_priv = subreq;
req->r_inode = inode;
@@ -451,19 +469,22 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
ceph_osdc_start_request(req->r_osdc, req);
-out:
ceph_osdc_put_request(req);
- if (err) {
- subreq->error = err;
- netfs_read_subreq_terminated(subreq);
- }
- doutc(cl, "%llx.%llx result %d\n", ceph_vinop(inode), err);
+ doutc(cl, "%llx.%llx result -EIOCBQUEUED\n", ceph_vinop(inode));
+ return 0;
+eio:
+ err = -EIO;
+failed:
+ ceph_osdc_put_request(req);
+failed_noput:
+ subreq->error = err;
+ netfs_read_subreq_terminated(subreq);
+ return 0;
}
static int ceph_init_request(struct netfs_io_request *rreq, struct file *file)
{
struct inode *inode = rreq->inode;
- struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
struct ceph_client *cl = ceph_inode_to_client(inode);
int got = 0, want = CEPH_CAP_FILE_CACHE;
struct ceph_netfs_request_data *priv;
@@ -515,7 +536,6 @@ static int ceph_init_request(struct netfs_io_request *rreq, struct file *file)
priv->caps = got;
rreq->netfs_priv = priv;
- rreq->io_streams[0].sreq_max_len = fsc->mount_options->rsize;
out:
if (ret < 0) {
@@ -543,7 +563,6 @@ static void ceph_netfs_free_request(struct netfs_io_request *rreq)
const struct netfs_request_ops ceph_netfs_ops = {
.init_request = ceph_init_request,
.free_request = ceph_netfs_free_request,
- .prepare_read = ceph_netfs_prepare_read,
.issue_read = ceph_netfs_issue_read,
.expand_readahead = ceph_netfs_expand_readahead,
.check_write_begin = ceph_netfs_check_write_begin,
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 88e5404080e8..fbb0c312b723 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -100,70 +100,98 @@ static int netfs_begin_cache_read(struct netfs_io_request *rreq, struct netfs_in
}
/*
- * netfs_prepare_read_iterator - Prepare the subreq iterator for I/O
- * @subreq: The subrequest to be set up
- *
- * Prepare the I/O iterator representing the read buffer on a subrequest for
- * the filesystem to use for I/O (it can be passed directly to a socket). This
- * is intended to be called from the ->issue_read() method once the filesystem
- * has trimmed the request to the size it wants.
- *
- * Returns the limited size if successful and -ENOMEM if insufficient memory
- * available.
+ * Prepare the I/O buffer on a buffered read subrequest for the filesystem to
+ * use as a bvec queue.
*/
-static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq)
+static int netfs_prepare_buffered_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
struct netfs_io_request *rreq = subreq->rreq;
struct netfs_io_stream *stream = &rreq->io_streams[0];
ssize_t extracted;
- size_t rsize = subreq->len;
- if (subreq->source == NETFS_DOWNLOAD_FROM_SERVER)
- rsize = umin(rsize, stream->sreq_max_len);
+ _enter("R=%08x[%x] l=%zx s=%u",
+ rreq->debug_id, subreq->debug_index, subreq->len, max_segs);
- bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor);
- extracted = bvecq_slice(&rreq->dispatch_cursor, rsize,
- stream->sreq_max_segs, &subreq->nr_segs);
- if (extracted < rsize) {
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+ extracted = bvecq_slice(&stream->dispatch_cursor, subreq->len,
+ max_segs, &subreq->nr_segs);
+
+ if (extracted < subreq->len) {
subreq->len = extracted;
trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
}
+ stream->buffered -= extracted;
+ stream->issue_from = subreq->start + subreq->len;
+ rreq->submitted = stream->issue_from;
- return subreq->len;
+ if (!stream->buffered)
+ netfs_all_subreqs_queued(rreq);
+ return 0;
}
-/*
- * Issue a read against the cache.
- * - Eats the caller's ref on subreq.
+/**
+ * netfs_prepare_read_buffer - Get the buffer for a subrequest
+ * @subreq: The subrequest to get the buffer for
+ * @max_segs: Maximum number of segments in buffer (or INT_MAX)
+ *
+ * Extract a slice of buffer from the stream and attach it to the subrequest as
+ * a bio_vec queue. The maximum amount of data attached is set by
+ * @subreq->len, but this may be shortened if @max_segs would be exceeded.
+ *
+ * [!] NOTE: This must be run in the same thread as ->issue_read() was called
+ * in as we access the readahead_control struct if there is one.
*/
-static void netfs_read_cache_to_pagecache(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
+int netfs_prepare_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
- struct netfs_cache_resources *cres = &rreq->cache_resources;
-
- netfs_stat(&netfs_n_rh_read);
- cres->ops->read(cres, subreq->start, &subreq->io_iter, NETFS_READ_HOLE_IGNORE,
- netfs_cache_read_terminated, subreq);
+ switch (subreq->rreq->origin) {
+ case NETFS_READAHEAD:
+ case NETFS_READPAGE:
+ case NETFS_READ_FOR_WRITE:
+ if (subreq->retry_count)
+ return netfs_prepare_buffered_read_retry_buffer(subreq, max_segs);
+ return netfs_prepare_buffered_read_buffer(subreq, max_segs);
+
+ case NETFS_UNBUFFERED_READ:
+ case NETFS_DIO_READ:
+ case NETFS_READ_GAPS:
+ return netfs_prepare_unbuffered_read_buffer(subreq, max_segs);
+ case NETFS_READ_SINGLE:
+ return netfs_prepare_read_single_buffer(subreq, max_segs);
+ default:
+ WARN_ON_ONCE(1);
+ return -EIO;
+ }
}
+EXPORT_SYMBOL(netfs_prepare_read_buffer);
-int netfs_read_query_cache(struct netfs_io_request *rreq, struct fscache_occupancy *occ)
+void netfs_read_query_cache(struct netfs_io_request *rreq, struct fscache_occupancy *occ)
{
struct netfs_cache_resources *cres = &rreq->cache_resources;
occ->granularity = PAGE_SIZE;
if (occ->query_from >= occ->query_to)
- return 0;
+ return;
if (!cres->ops)
- return 0;
+ return;
occ->query_from = round_up(occ->query_from, occ->granularity);
- return cres->ops->query_occupancy(cres, occ);
+ cres->ops->query_occupancy(cres, occ);
}
-void netfs_queue_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
+/*
+ * Allocate and prepare a read subrequest.
+ */
+struct netfs_io_subrequest *netfs_alloc_read_subrequest(struct netfs_io_request *rreq)
{
+ struct netfs_io_subrequest *subreq;
struct netfs_io_stream *stream = &rreq->io_streams[0];
+ subreq = netfs_alloc_subrequest(rreq);
+ if (!subreq)
+ return subreq;
+
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
/* We add to the end of the list whilst the collector may be walking
@@ -173,40 +201,56 @@ void netfs_queue_read(struct netfs_io_request *rreq,
spin_lock(&rreq->lock);
/* Write IN_PROGRESS before pointer to new subreq */
list_add_tail_release(&subreq->rreq_link, &stream->subrequests);
- if (list_is_first(&subreq->rreq_link, &stream->subrequests)) {
- if (!stream->active) {
- stream->collected_to = subreq->start;
- /* Store list pointers before active flag */
- smp_store_release(&stream->active, true);
- }
- }
+ if (!stream->active)
+ /* Store list pointers before active flag */
+ smp_store_release(&stream->active, true);
spin_unlock(&rreq->lock);
+ return subreq;
}
static void netfs_issue_read(struct netfs_io_request *rreq,
struct netfs_io_subrequest *subreq)
{
- bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
- iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
- subreq->content.slot, subreq->content.offset, subreq->len);
+ struct netfs_cache_resources *cres = &rreq->cache_resources;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+ int ret;
+
+ _enter("R=%08x[%x]", rreq->debug_id, subreq->debug_index);
switch (subreq->source) {
case NETFS_DOWNLOAD_FROM_SERVER:
- rreq->netfs_ops->issue_read(subreq);
- break;
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0)
+ goto fail;
+ return;
case NETFS_READ_FROM_CACHE:
- netfs_read_cache_to_pagecache(rreq, subreq);
- break;
+ netfs_stat(&netfs_n_rh_read);
+ ret = cres->ops->issue_read(subreq);
+ if (ret < 0)
+ goto fail;
+ return;
default:
- bvecq_zero(&subreq->content, subreq->len);
+ WARN_ON_ONCE(1);
+ fallthrough;
+ case NETFS_FILL_WITH_ZEROES:
+ stream->issue_from = subreq->start + subreq->len;
+ stream->buffered -= subreq->len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+ if (!stream->buffered)
+ netfs_all_subreqs_queued(rreq);
+ bvecq_zero(&stream->dispatch_cursor, subreq->len);
subreq->transferred = subreq->len;
subreq->error = 0;
- iov_iter_zero(subreq->len, &subreq->io_iter);
- subreq->transferred = subreq->len;
- netfs_read_subreq_terminated(subreq);
- break;
+ return netfs_read_subreq_terminated(subreq);
}
+
+fail:
+ /* Ownership of subreq was returned to us. */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ stream->buffered -= subreq->len;
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
}
/*
@@ -225,19 +269,16 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
.cached_to[1] = ULLONG_MAX,
};
struct fscache_occupancy *occ = &_occ;
- ssize_t size = rreq->len;
- uoff_t start = rreq->start;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
int ret = 0;
_enter("R=%08x", rreq->debug_id);
- bvecq_pos_set(&rreq->dispatch_cursor, &rreq->load_cursor);
+ bvecq_pos_set(&stream->dispatch_cursor, &rreq->load_cursor);
do {
- int (*prepare_read)(struct netfs_io_subrequest *subreq) = NULL;
struct netfs_io_subrequest *subreq;
- ssize_t slice;
- uoff_t hole_to, cache_to;
+ uoff_t hole_to, cache_to, stop;
/* If we don't have any, find out the next couple of data
* extents from the cache, containing of following the
@@ -246,7 +287,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
*/
hole_to = occ->cached_from[0];
cache_to = occ->cached_to[0];
- if (start >= cache_to) {
+ if (stream->issue_from >= cache_to) {
/* Extent exhausted; shuffle down. */
int i;
@@ -261,52 +302,50 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
if (occ->cached_from[0] != ULLONG_MAX)
continue;
- /* Get new extents */
- ret = netfs_read_query_cache(rreq, occ);
- if (ret < 0)
- break;
+ /* Wait for the cache to enter the readable state and
+ * get more extents. If the cache doesn't become
+ * readable, this leaves the default negative extent in
+ * place.
+ */
+ netfs_read_query_cache(rreq, occ);
continue;
}
- subreq = netfs_alloc_subrequest(rreq);
+ subreq = netfs_alloc_read_subrequest(rreq);
if (!subreq) {
ret = -ENOMEM;
break;
}
- subreq->start = start;
- subreq->len = size;
-
- netfs_queue_read(rreq, subreq);
+ subreq->start = stream->issue_from;
+ stop = stream->issue_from + stream->buffered;
uoff_t zero_point = netfs_read_zero_point(rreq->inode);
uoff_t zlimit = umin(zero_point, rreq->i_size);
_debug("rsub %llx %llx-%llx", subreq->start, hole_to, cache_to);
- if (start >= hole_to && start < cache_to) {
+ if (stream->issue_from >= hole_to && stream->issue_from < cache_to) {
/* Overlap with a cached region, where the cache may
* record a block of zeroes.
*/
- _debug("cached s=%llx c=%llx l=%zx", start, cache_to, size);
- subreq->len = umin(cache_to - start, size);
- subreq->len = round_up(subreq->len, occ->granularity);
+ _debug("cached s=%llx c=%llx l=%zx",
+ stream->issue_from, cache_to, stream->buffered);
+ subreq->len = umin(cache_to - stream->issue_from, stream->buffered);
if (occ->cached_type[0] == FSCACHE_EXTENT_ZERO) {
subreq->source = NETFS_FILL_WITH_ZEROES;
netfs_stat(&netfs_n_rh_zero);
} else {
+ subreq->len = round_up(subreq->len, occ->granularity);
subreq->source = NETFS_READ_FROM_CACHE;
- prepare_read = rreq->cache_resources.ops->prepare_read;
}
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
-
- } else if (subreq->start >= zlimit && size > 0) {
+ } else if (subreq->start >= zlimit &&
+ subreq->start < stop) {
/* If this range lies beyond the zero-point, that part
* can just be cleared locally.
*/
- _debug("zero %llx-%llx", start, start + size);
- subreq->len = size;
+ _debug("zero %llx-%llx", subreq->start, stop);
+ subreq->len = stream->buffered;
subreq->source = NETFS_FILL_WITH_ZEROES;
if (rreq->cache_resources.ops)
__set_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags);
@@ -316,10 +355,10 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
* this range lies beyond the zero-point or the EOF,
* that part can just be cleared locally.
*/
- uoff_t limit = min3(zlimit, start + size, hole_to);
+ uoff_t limit = min3(zlimit, stop, hole_to);
_debug("limit %llx %llx", rreq->i_size, zero_point);
- _debug("download %llx-%llx", start, start + size);
+ _debug("download %llx-%llx", subreq->start, stop);
subreq->len = umin(limit - subreq->start, ULONG_MAX);
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
if (rreq->cache_resources.ops)
@@ -327,39 +366,15 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
netfs_stat(&netfs_n_rh_download);
}
- if (size == 0) {
+ if (subreq->len == 0) {
pr_err("ZERO-LEN READ: R=%08x[%x] l=%zx/%zx s=%llx z=%llx i=%llx",
rreq->debug_id, subreq->debug_index,
- subreq->len, size,
+ subreq->len, stream->buffered,
subreq->start, zero_point, rreq->i_size);
netfs_cancel_read(subreq, ret);
break;
}
- rreq->io_streams[0].sreq_max_len = MAX_RW_COUNT;
- rreq->io_streams[0].sreq_max_segs = INT_MAX;
-
- if (prepare_read) {
- ret = prepare_read(subreq);
- if (ret < 0) {
- netfs_cancel_read(subreq, ret);
- break;
- }
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
- }
-
- slice = netfs_prepare_read_iterator(subreq);
- if (slice < 0) {
- ret = slice;
- netfs_cancel_read(subreq, ret);
- break;
- }
- start += slice;
- size -= slice;
- if (size <= 0)
- netfs_all_subreqs_queued(rreq);
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
netfs_issue_read(rreq, subreq);
netfs_maybe_bulk_drop_ra_refs(rreq);
@@ -368,18 +383,19 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
if (test_bit(NETFS_RREQ_FAILED, &rreq->flags))
break;
cond_resched();
- } while (size > 0);
+ } while (stream->buffered > 0);
- if (unlikely(size > 0)) {
+ if (unlikely(!netfs_are_all_subreqs_queued(rreq))) {
netfs_all_subreqs_queued(rreq);
netfs_wake_collector(rreq);
}
/* Defer error return as we may need to wait for outstanding I/O. */
- cmpxchg(&rreq->error, 0, ret);
+ if (ret < 0)
+ cmpxchg(&rreq->error, 0, ret);
bvecq_pos_unset(&rreq->load_cursor);
- bvecq_pos_unset(&rreq->dispatch_cursor);
+ bvecq_pos_unset(&stream->dispatch_cursor);
}
/**
@@ -400,17 +416,22 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
void netfs_readahead(struct readahead_control *ractl)
{
struct netfs_io_request *rreq;
+ struct netfs_io_stream *stream;
struct netfs_inode *ictx = netfs_inode(ractl->mapping->host);
ssize_t added;
uoff_t start = readahead_pos(ractl);
size_t size = readahead_length(ractl);
int ret;
+ _enter("");
+
rreq = netfs_alloc_request(ractl->mapping, ractl->file, start, size,
NETFS_READAHEAD);
if (IS_ERR(rreq))
return;
+ stream = &rreq->io_streams[0];
+
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &rreq->flags);
ret = netfs_begin_cache_read(rreq, ictx);
@@ -435,9 +456,9 @@ void netfs_readahead(struct readahead_control *ractl)
__set_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags);
rreq->submitted = rreq->start + added;
- rreq->cleaned_to = rreq->start;
bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
netfs_read_set_unlock_at(rreq);
+ stream->buffered = added;
netfs_read_to_pagecache(rreq);
netfs_maybe_bulk_drop_ra_refs(rreq);
@@ -453,18 +474,20 @@ EXPORT_SYMBOL(netfs_readahead);
*/
static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio)
{
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
struct bvecq *bq;
size_t fsize = folio_size(folio);
if (bvecq_buffer_init(&rreq->load_cursor, rreq->gfp, false) < 0)
return -ENOMEM;
+ bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
bq = rreq->load_cursor.bvecq;
bvec_set_folio(&bq->bv[0], folio, fsize, 0);
bvecq_filled_to(bq, 1);
rreq->submitted = rreq->start + fsize;
rreq->progress_at = fsize;
- bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
+ stream->buffered = fsize;
return 0;
}
@@ -474,6 +497,7 @@ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct fo
static int netfs_read_gaps(struct file *file, struct folio *folio)
{
struct netfs_io_request *rreq;
+ struct netfs_io_stream *stream;
struct address_space *mapping = folio->mapping;
struct netfs_group *group = netfs_folio_group(folio);
struct netfs_folio *finfo = netfs_folio_info(folio);
@@ -495,6 +519,7 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
ret = PTR_ERR(rreq);
goto alloc_error;
}
+ stream = &rreq->io_streams[0];
ret = netfs_begin_cache_read(rreq, ctx);
if (ret == -ENOMEM || ret == -EINTR || ret == -ERESTARTSYS)
@@ -549,6 +574,7 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
bvecq_filled_to(bq, slot);
rreq->submitted = rreq->start + flen;
+ stream->buffered = flen;
netfs_read_to_pagecache(rreq);
@@ -627,6 +653,7 @@ int netfs_read_folio(struct file *file, struct folio *folio)
goto discard;
netfs_read_to_pagecache(rreq);
+
ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret < 0 ? ret : 0;
diff --git a/fs/netfs/direct_read.c b/fs/netfs/direct_read.c
index 058cc6bb7124..8ded0197af91 100644
--- a/fs/netfs/direct_read.c
+++ b/fs/netfs/direct_read.c
@@ -16,6 +16,32 @@
#include <linux/netfs.h>
#include "internal.h"
+int netfs_prepare_unbuffered_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *rreq = subreq->rreq;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ stream->buffered -= subreq->len;
+ stream->issue_from += subreq->len;
+ rreq->submitted = stream->issue_from;
+
+ if (stream->buffered == 0)
+ netfs_all_subreqs_queued(rreq);
+ return 0;
+}
+
/*
* Perform a read to a buffer from the server, slicing up the region to be read
* according to the network rsize.
@@ -23,17 +49,15 @@
static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
{
struct netfs_io_stream *stream = &rreq->io_streams[0];
- ssize_t size = rreq->len;
- uoff_t start = rreq->start;
- int ret;
- bvecq_pos_set(&rreq->dispatch_cursor, &rreq->load_cursor);
- bvecq_pos_transfer(&rreq->collect_cursor, &rreq->load_cursor);
+ bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
+ bvecq_pos_transfer(&stream->dispatch_cursor, &rreq->load_cursor);
do {
struct netfs_io_subrequest *subreq;
+ int ret;
- subreq = netfs_alloc_subrequest(rreq);
+ subreq = netfs_alloc_read_subrequest(rreq);
if (!subreq) {
/* Stash the error in the request if there's not
* already an error set.
@@ -43,51 +67,32 @@ static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
}
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- subreq->start = start;
- subreq->len = size;
-
- netfs_queue_read(rreq, subreq);
+ subreq->start = stream->issue_from;
+ subreq->len = stream->buffered;
netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read) {
- ret = rreq->netfs_ops->prepare_read(subreq);
- if (ret < 0) {
- netfs_cancel_read(subreq, ret);
- break;
- }
- }
-
- bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor);
- bvecq_pos_set(&subreq->content, &rreq->dispatch_cursor);
- subreq->len = bvecq_slice(&rreq->dispatch_cursor,
- umin(size, stream->sreq_max_len),
- stream->sreq_max_segs,
- &subreq->nr_segs);
-
- size -= subreq->len;
- start += subreq->len;
- rreq->submitted += subreq->len;
- if (size <= 0)
- netfs_all_subreqs_queued(rreq);
- iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
- subreq->content.slot, subreq->content.offset, subreq->len);
-
- rreq->netfs_ops->issue_read(subreq);
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ stream->buffered -= subreq->len;
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ break;
+ }
if (test_bit(NETFS_RREQ_PAUSE, &rreq->flags))
netfs_wait_for_paused_read(rreq);
if (test_bit(NETFS_RREQ_FAILED, &rreq->flags))
break;
cond_resched();
- } while (size > 0);
+ } while (stream->buffered > 0);
- if (unlikely(size > 0)) {
+ if (unlikely(stream->buffered > 0)) {
netfs_all_subreqs_queued(rreq);
netfs_wake_collector(rreq);
}
- bvecq_pos_unset(&rreq->dispatch_cursor);
+ bvecq_pos_unset(&stream->dispatch_cursor);
}
/*
@@ -138,6 +143,7 @@ static ssize_t netfs_unbuffered_read(struct netfs_io_request *rreq, bool sync)
ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *iter)
{
struct netfs_io_request *rreq;
+ struct netfs_io_stream *stream;
ssize_t ret;
size_t orig_count = iov_iter_count(iter);
bool sync = is_sync_kiocb(iocb);
@@ -162,6 +168,8 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i
netfs_stat(&netfs_n_rh_dio_read);
trace_netfs_read(rreq, rreq->start, rreq->len, netfs_read_trace_dio_read);
+ stream = &rreq->io_streams[0];
+
/* If this is an async op, we have to keep track of the destination
* buffer for ourselves as the caller's iterator will be trashed when
* we return.
@@ -177,6 +185,8 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i
goto error_put;
rreq->len = ret;
+ stream->buffered = ret;
+ stream->issue_from = rreq->start;
// TODO: Set up bounce buffer if needed
diff --git a/fs/netfs/direct_write.c b/fs/netfs/direct_write.c
index 5bc3ec261f6c..40a2d34789ad 100644
--- a/fs/netfs/direct_write.c
+++ b/fs/netfs/direct_write.c
@@ -9,6 +9,34 @@
#include <linux/uio.h>
#include "internal.h"
+/*
+ * Prepare the buffer for an unbuffered/DIO write.
+ */
+int netfs_prepare_unbuffered_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_stream *stream = &subreq->rreq->io_streams[subreq->stream_nr];
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ // TODO: Wait here for completion of prev subreq
+
+ stream->issue_from += subreq->len;
+ stream->buffered -= subreq->len;
+ if (stream->buffered == 0)
+ netfs_all_subreqs_queued(subreq->rreq);
+ return 0;
+}
+
/*
* Perform the cleanup rituals after an unbuffered write is complete.
*/
@@ -74,9 +102,9 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq,
wreq->transferred += subreq->transferred;
if (subreq->transferred < subreq->len) {
- bvecq_pos_unset(&wreq->dispatch_cursor);
- bvecq_pos_transfer(&wreq->dispatch_cursor, &subreq->dispatch_pos);
- bvecq_pos_advance(&wreq->dispatch_cursor, subreq->transferred);
+ bvecq_pos_unset(&stream->dispatch_cursor);
+ bvecq_pos_transfer(&stream->dispatch_cursor, &subreq->dispatch_pos);
+ bvecq_pos_advance(&stream->dispatch_cursor, subreq->transferred);
}
stream->collected_to = subreq->start + subreq->transferred;
@@ -85,6 +113,7 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq,
trace_netfs_collect_stream(wreq, stream);
trace_netfs_collect_state(wreq, wreq->collected_to, 0);
+ /* TODO: Progressively clean up wreq->direct_bq */
}
/*
@@ -103,60 +132,50 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
_enter("%llx", wreq->len);
- bvecq_pos_set(&wreq->dispatch_cursor, &wreq->load_cursor);
- bvecq_pos_set(&wreq->collect_cursor, &wreq->dispatch_cursor);
+ stream->issue_from = wreq->start;
+ stream->buffered = wreq->len;
+ bvecq_pos_set(&stream->dispatch_cursor, &wreq->load_cursor);
if (wreq->origin == NETFS_DIO_WRITE)
inode_dio_begin(wreq->inode);
- stream->collected_to = wreq->start;
-
for (;;) {
bool retry = false;
if (!subreq) {
- netfs_prepare_write(wreq, stream, wreq->start + wreq->transferred);
- subreq = stream->construct;
- stream->construct = NULL;
- } else {
- bvecq_pos_set(&subreq->dispatch_pos, &wreq->dispatch_cursor);
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ if (!subreq) {
+ ret = -ENOMEM;
+ goto failed;
+ }
}
- /* Check if (re-)preparation failed. */
- if (unlikely(test_bit(NETFS_SREQ_FAILED, &subreq->flags))) {
- netfs_write_subrequest_terminated(subreq, subreq->error);
- wreq->error = subreq->error;
- break;
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. Note that
+ * ->dispatch_pos may or may not be initialised.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ netfs_write_subrequest_terminated(subreq, ret);
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed);
+ subreq = NULL;
+ goto failed;
}
- subreq->len = bvecq_slice(&wreq->dispatch_cursor, stream->sreq_max_len,
- stream->sreq_max_segs, &subreq->nr_segs);
- bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
-
- iov_iter_bvec_queue(&subreq->io_iter, ITER_SOURCE,
- subreq->content.bvecq, subreq->content.slot,
- subreq->content.offset,
- subreq->len);
-
- if (!iov_iter_count(&subreq->io_iter))
- break;
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- stream->issue_write(subreq);
-
- /* Async, need to wait. */
- netfs_wait_for_in_progress_stream(wreq, stream);
-
- if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
+ ret = netfs_wait_for_in_progress_subreq(wreq, subreq);
+ if (ret < 0) {
+ if (ret != -EAGAIN) {
+ /* Don't need to lock here as the collection is
+ * done in this thread.
+ */
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed);
+ subreq = NULL;
+ goto failed;
+ }
retry = true;
- } else if (test_bit(NETFS_SREQ_FAILED, &subreq->flags)) {
- ret = subreq->error;
- wreq->error = ret;
- netfs_see_subrequest(subreq, netfs_sreq_trace_see_failed);
- subreq = NULL;
- break;
}
- ret = 0;
if (!retry) {
netfs_unbuffered_write_collect(wreq, stream, subreq);
@@ -171,20 +190,21 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
continue;
}
- /* We need to retry the last subrequest, so first reset the
- * iterator, taking into account what, if anything, we managed
- * to transfer.
+ /* We need to retry the last subrequest, so first wind back the
+ * buffer position.
*/
subreq->error = -EAGAIN;
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
bvecq_pos_unset(&subreq->content);
- bvecq_pos_unset(&wreq->dispatch_cursor);
- bvecq_pos_transfer(&wreq->dispatch_cursor, &subreq->dispatch_pos);
+ bvecq_pos_unset(&stream->dispatch_cursor);
+ bvecq_pos_transfer(&stream->dispatch_cursor, &subreq->dispatch_pos);
+ stream->issue_from -= subreq->len - subreq->transferred;
+ stream->buffered += subreq->len - subreq->transferred;
if (subreq->transferred > 0) {
- wreq->transferred += subreq->transferred;
- bvecq_pos_advance(&wreq->dispatch_cursor, subreq->transferred);
+ wreq->transferred += subreq->transferred;
+ bvecq_pos_advance(&stream->dispatch_cursor, subreq->transferred);
}
if (stream->source == NETFS_UPLOAD_TO_SERVER &&
@@ -193,25 +213,21 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
__clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
__clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
- __clear_bit(NETFS_SREQ_BOUNDARY, &subreq->flags);
__clear_bit(NETFS_SREQ_FAILED, &subreq->flags);
- subreq->start = wreq->start + wreq->transferred;
- subreq->len = wreq->len - wreq->transferred;
+ __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ subreq->start = stream->issue_from;
+ subreq->len = stream->buffered;
subreq->transferred = 0;
subreq->retry_count += 1;
- stream->sreq_max_len = UINT_MAX;
- stream->sreq_max_segs = INT_MAX;
netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- if (stream->prepare_write)
- stream->prepare_write(subreq);
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
netfs_stat(&netfs_n_wh_retry_write_subreq);
}
- bvecq_pos_unset(&wreq->dispatch_cursor);
- bvecq_pos_unset(&wreq->load_cursor);
+failed:
+ bvecq_pos_unset(&stream->dispatch_cursor);
netfs_unbuffered_write_done(wreq);
_leave(" = %d", ret);
return ret;
@@ -255,6 +271,7 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
if (IS_ERR(wreq))
return PTR_ERR(wreq);
+ wreq->len = iov_iter_count(iter);
wreq->io_streams[0].avail = true;
trace_netfs_write(wreq, (iocb->ki_flags & IOCB_DIRECT ?
netfs_write_trace_dio_write :
@@ -265,9 +282,7 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
* we have to save the source buffer as the iterator is only
* good until we return. In such a case, extract an iterator
* to represent as much of the the output buffer as we can
- * manage. Note that the extraction might not be able to
- * allocate a sufficiently large bvec array and may shorten the
- * request.
+ * manage. Note that the extraction may shorten the request.
*/
ssize_t n = netfs_extract_iter(iter, len, INT_MAX,
&wreq->load_cursor.bvecq, 0, wreq->gfp);
@@ -282,8 +297,6 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
wreq->load_cursor.bvecq->max_slots);
}
- __set_bit(NETFS_RREQ_USE_IO_ITER, &wreq->flags);
-
/* Copy the data into the bounce buffer and encrypt it. */
// TODO
diff --git a/fs/netfs/fscache_io.c b/fs/netfs/fscache_io.c
index 056a2bae5d99..0b85bd33dfe5 100644
--- a/fs/netfs/fscache_io.c
+++ b/fs/netfs/fscache_io.c
@@ -239,7 +239,7 @@ void __fscache_write_to_cache(struct fscache_cookie *cookie,
fscache_access_io_write) < 0)
goto abandon_free;
- ret = cres->ops->prepare_write(cres, &start, &len, len, i_size, false);
+ ret = cres->ops->prepare_write_old(cres, &start, &len, len, i_size, false);
if (ret < 0)
goto abandon_end;
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 3aa9c7ef85de..3d345c9d6d5a 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -22,10 +22,9 @@
/*
* buffered_read.c
*/
-int netfs_read_query_cache(struct netfs_io_request *rreq,
- struct fscache_occupancy *occ);
-void netfs_queue_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq);
+void netfs_read_query_cache(struct netfs_io_request *rreq,
+ struct fscache_occupancy *occ);
+struct netfs_io_subrequest *netfs_alloc_read_subrequest(struct netfs_io_request *rreq);
void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error);
int netfs_prefetch_for_write(struct file *file, struct folio *folio,
size_t offset, size_t len);
@@ -36,6 +35,18 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,
void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode,
uoff_t pos, size_t copied);
+/*
+ * direct_read.c
+ */
+int netfs_prepare_unbuffered_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
+/*
+ * direct_write.c
+ */
+int netfs_prepare_unbuffered_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
/*
* main.c
*/
@@ -74,6 +85,8 @@ struct bvecq *netfs_buffer_make_space(struct netfs_io_request *rreq,
enum netfs_bvecq_trace trace);
void netfs_wake_collector(struct netfs_io_request *rreq);
void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq);
+int netfs_wait_for_in_progress_subreq(struct netfs_io_request *rreq,
+ struct netfs_io_subrequest *subreq);
void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,
struct netfs_io_stream *stream);
ssize_t netfs_wait_for_read(struct netfs_io_request *rreq);
@@ -155,9 +168,18 @@ static inline bool netfs_using_pgpriv2(const struct netfs_io_request *rreq)
/*
* read_retry.c
*/
+int netfs_prepare_buffered_read_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+int netfs_reset_for_read_retry(struct netfs_io_subrequest *subreq);
void netfs_retry_reads(struct netfs_io_request *rreq);
void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq);
+/*
+ * read_single.c
+ */
+int netfs_prepare_read_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
/*
* stats.c
*/
@@ -228,26 +250,19 @@ void netfs_write_collection_worker(struct work_struct *work);
/*
* write_issue.c
*/
+struct netfs_writethrough;
struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
struct file *file,
uoff_t start,
enum netfs_io_origin origin);
struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wreq,
struct netfs_io_stream *stream);
-void netfs_prepare_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- uoff_t start);
-void netfs_reissue_write(struct netfs_io_stream *stream,
- struct netfs_io_subrequest *subreq);
-void netfs_issue_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream);
-size_t netfs_advance_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- uoff_t start, size_t len, bool to_eof);
/*
* write_retry.c
*/
+int netfs_prepare_write_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
void netfs_retry_writes(struct netfs_io_request *wreq);
/*
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index b8da5e85cc67..3736a2a7662d 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -18,6 +18,7 @@ MODULE_DESCRIPTION("Network fs support");
MODULE_AUTHOR("Red Hat, Inc.");
MODULE_LICENSE("GPL");
+EXPORT_TRACEPOINT_SYMBOL(netfs_rreq);
EXPORT_TRACEPOINT_SYMBOL(netfs_sreq);
unsigned netfs_debug;
diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c
index 4fa09560fb4b..a729e51a8ab2 100644
--- a/fs/netfs/misc.c
+++ b/fs/netfs/misc.c
@@ -250,6 +250,37 @@ void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq)
netfs_wake_collector(rreq);
}
+/*
+ * Wait for a subrequest to come to completion.
+ */
+int netfs_wait_for_in_progress_subreq(struct netfs_io_request *rreq,
+ struct netfs_io_subrequest *subreq)
+{
+ if (netfs_check_subreq_in_progress(subreq)) {
+ DEFINE_WAIT(myself);
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_wait_quiesce);
+ for (;;) {
+ prepare_to_wait(&rreq->waitq, &myself, TASK_UNINTERRUPTIBLE);
+
+ if (!netfs_check_subreq_in_progress(subreq))
+ break;
+
+ trace_netfs_sreq(subreq, netfs_sreq_trace_wait_for);
+ schedule();
+ }
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_waited_quiesce);
+ finish_wait(&rreq->waitq, &myself);
+ }
+
+ if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
+ return -EAGAIN;
+ if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
+ return subreq->error;
+ return 0;
+}
+
/*
* Wait for all outstanding I/O in a stream to quiesce.
*/
@@ -380,7 +411,7 @@ static ssize_t netfs_wait_for_in_progress(struct netfs_io_request *rreq,
case NETFS_UNBUFFERED_WRITE:
break;
default:
- if (rreq->submitted < rreq->len) {
+ if (rreq->transferred < rreq->len) {
trace_netfs_failure(rreq, NULL, ret, netfs_fail_short_read);
ret = -EIO;
}
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index a1d0de65484e..3b22748474b9 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -55,8 +55,6 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
rreq->i_size = i_size_read(inode);
rreq->debug_id = atomic_inc_return(&debug_ids);
rreq->wsize = INT_MAX;
- rreq->io_streams[0].sreq_max_len = ULONG_MAX;
- rreq->io_streams[0].sreq_max_segs = 0;
spin_lock_init(&rreq->lock);
init_waitqueue_head(&rreq->waitq);
refcount_set(&rreq->ref, 2);
@@ -150,8 +148,8 @@ static void netfs_deinit_request(struct netfs_io_request *rreq)
if (rreq->cache_resources.ops)
rreq->cache_resources.ops->end_operation(&rreq->cache_resources);
bvecq_pos_unset(&rreq->load_cursor);
- bvecq_pos_unset(&rreq->dispatch_cursor);
bvecq_pos_unset(&rreq->collect_cursor);
+ bvecq_pos_unset(&rreq->retry_cursor);
bvecq_put(rreq->spare);
while (rreq->writebacks) {
struct netfs_writeback *wback = rreq->writebacks;
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index e91f7962df04..9e50759591a9 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -36,6 +36,7 @@ static void netfs_clear_unread(struct netfs_io_subrequest *subreq)
if (subreq->start + subreq->transferred >= subreq->rreq->i_size)
__set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags);
+ trace_netfs_rreq(subreq->rreq, netfs_rreq_trace_zero_unread);
}
/*
@@ -407,8 +408,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
if (test_bit(NETFS_SREQ_COPY_TO_CACHE, &front_flags))
notes |= COPY_TO_CACHE;
- if (front->start + transferred >= unlock_at ||
- test_bit(NETFS_SREQ_HIT_EOF, &front_flags))
+ if (front->start + transferred >= unlock_at)
netfs_read_unlock_folios(rreq, ¬es);
} else {
stream->collected_to = front->start + transferred;
@@ -500,31 +500,6 @@ static void netfs_rreq_assess_dio(struct netfs_io_request *rreq)
inode_dio_end(rreq->inode);
}
-/*
- * Do processing after reading a monolithic single object.
- */
-static void netfs_rreq_assess_single(struct netfs_io_request *rreq)
-{
- struct netfs_io_stream *stream = &rreq->io_streams[0];
-
- if (!rreq->error && stream->source == NETFS_DOWNLOAD_FROM_SERVER &&
- fscache_resources_valid(&rreq->cache_resources)) {
- trace_netfs_rreq(rreq, netfs_rreq_trace_dirty);
- netfs_single_mark_inode_dirty(rreq->inode);
- }
-
- if (rreq->iocb) {
- rreq->iocb->ki_pos += rreq->transferred;
- if (rreq->iocb->ki_complete) {
- trace_netfs_rreq(rreq, netfs_rreq_trace_ki_complete);
- rreq->iocb->ki_complete(
- rreq->iocb, rreq->error ? rreq->error : rreq->transferred);
- }
- }
- if (rreq->netfs_ops->done)
- rreq->netfs_ops->done(rreq);
-}
-
/*
* Perform the collection of subrequests and folios.
*
@@ -558,7 +533,7 @@ bool netfs_read_collection(struct netfs_io_request *rreq)
netfs_rreq_assess_dio(rreq);
break;
case NETFS_READ_SINGLE:
- netfs_rreq_assess_single(rreq);
+ WARN_ON_ONCE(1);
break;
default:
break;
@@ -692,6 +667,11 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)
} else if (test_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags)) {
__set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
trace_netfs_sreq(subreq, netfs_sreq_trace_partial_read);
+ } else if (subreq->source == NETFS_READ_FROM_CACHE) {
+ netfs_stat(&netfs_n_rh_read_failed);
+ __set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ subreq->error = -ENODATA;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_short);
} else {
__set_bit(NETFS_SREQ_FAILED, &subreq->flags);
subreq->error = -ENODATA;
@@ -710,6 +690,8 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)
if (unlikely(subreq->error < 0)) {
trace_netfs_failure(rreq, subreq, subreq->error, netfs_fail_read);
+ if (subreq->error == -ENOMEM)
+ set_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags);
if (subreq->source == NETFS_READ_FROM_CACHE) {
netfs_stat(&netfs_n_rh_read_failed);
__set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c
index 14fe90751b5b..3c239682bbaf 100644
--- a/fs/netfs/read_pgpriv2.c
+++ b/fs/netfs/read_pgpriv2.c
@@ -13,8 +13,37 @@
#include <linux/task_io_accounting_ops.h>
#include "internal.h"
+int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *creq = subreq->rreq;
+ struct netfs_io_stream *stream = &creq->io_streams[1];
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ // TODO: Wait here for completion of prev subreq
+
+ stream->issue_from += subreq->len;
+ stream->buffered -= subreq->len;
+ if (stream->buffered == 0)
+ netfs_all_subreqs_queued(creq);
+ return 0;
+}
+
/*
- * [DEPRECATED] Copy a folio to the cache with PG_private_2 set.
+ * [DEPRECATED] Copy a folio to the cache with PG_private_2 set. Note that the
+ * folio won't necessarily be contiguous with the previous one as there might
+ * be a mixture of folios read from the cache and downloaded from the server
+ * (or just zeroed).
*/
static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio *folio)
{
@@ -24,7 +53,6 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
size_t dio_size = PAGE_SIZE;
size_t fsize = folio_size(folio), flen = fsize;
uoff_t fpos = folio_pos(folio), i_size;
- bool to_eof = false;
_enter("");
@@ -44,12 +72,8 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
if (fpos + fsize > creq->i_size)
creq->i_size = i_size;
- if (flen > i_size - fpos) {
+ if (flen > i_size - fpos)
flen = i_size - fpos;
- to_eof = true;
- } else if (flen == i_size - fpos) {
- to_eof = true;
- }
flen = round_up(flen, dio_size);
@@ -63,7 +87,7 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
*/
queue = creq->load_cursor.bvecq;
if (bvecq_is_full(queue) ||
- (fpos != creq->last_end && creq->last_end > 0 && queue->nr_slots > 0)) {
+ (fpos != cache->last_end && cache->last_end > 0 && queue->nr_slots > 0)) {
bvecq_buffer_append(&creq->load_cursor, creq->spare);
creq->spare = NULL;
@@ -78,38 +102,10 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
bvecq_filled_to(queue, slot);
creq->load_cursor.slot = slot;
creq->load_cursor.offset = 0;
- creq->last_end = fpos + flen;
-
- bvecq_pos_nudge(&creq->dispatch_cursor);
-
- cache->submit_off = 0;
- cache->submit_len = flen;
+ cache->last_end = fpos + flen;
+ trace_netfs_wback(creq, folio, 0);
- /* Attach the folio to one or more subrequests. For a big folio, we
- * could end up with thousands of subrequests if the wsize is small -
- * but we might need to wait during the creation of subrequests for
- * network resources (eg. SMB credits).
- */
- do {
- ssize_t part;
-
- creq->dispatch_cursor.offset = cache->submit_off;
-
- atomic64_set(&cache->issued_to, fpos + cache->submit_off);
- part = netfs_advance_write(creq, cache, fpos + cache->submit_off,
- cache->submit_len, to_eof);
- cache->submit_off += part;
- if (part > cache->submit_len)
- cache->submit_len = 0;
- else
- cache->submit_len -= part;
- } while (cache->submit_len > 0);
-
- bvecq_pos_step(&creq->dispatch_cursor);
- atomic64_set(&cache->issued_to, fpos + fsize);
-
- if (flen < fsize)
- netfs_issue_write(creq, cache);
+ cache->buffered += flen;
}
/*
@@ -119,6 +115,7 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache(
struct netfs_io_request *rreq, struct folio *folio)
{
struct netfs_io_request *creq;
+ struct netfs_io_stream *cache;
if (!fscache_resources_valid(&rreq->cache_resources))
goto cancel;
@@ -128,13 +125,14 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache(
if (IS_ERR(creq))
goto cancel;
- if (!creq->io_streams[1].avail)
+ cache = &creq->io_streams[1];
+ if (!cache->avail)
goto cancel_put;
if (bvecq_buffer_init(&creq->load_cursor, creq->gfp, false) < 0)
goto cancel_put;
- bvecq_pos_set(&creq->dispatch_cursor, &creq->load_cursor);
- bvecq_pos_set(&creq->collect_cursor, &creq->dispatch_cursor);
+ bvecq_pos_set(&cache->dispatch_cursor, &creq->load_cursor);
+ bvecq_pos_set(&creq->collect_cursor, &creq->load_cursor);
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &creq->flags);
trace_netfs_copy2cache(rreq, creq);
@@ -177,24 +175,61 @@ void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *fo
netfs_pgpriv2_copy_folio(creq, folio);
}
+/*
+ * Issue all pending writes on the cache stream.
+ */
+static void netfs_pgpriv2_issue_stream(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream)
+{
+ atomic64_set_release(&stream->issued_to, wreq->start);
+
+ do {
+ struct netfs_io_subrequest *subreq;
+ int ret;
+
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ if (!subreq)
+ break;
+
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. Punt the
+ * error to the collector and stop the issuance of new
+ * subreqs.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ stream->buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
+ break;
+ }
+ /* We no longer own subreq. */
+
+ if (test_bit(NETFS_RREQ_SAW_ENOMEM, &wreq->flags))
+ break;
+
+ } while (stream->buffered > 0);
+
+ netfs_all_subreqs_queued(wreq);
+}
+
/*
* [DEPRECATED] End writing to the cache, flushing out any outstanding writes.
*/
void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)
{
struct netfs_io_request *creq = rreq->copy_to_cache;
+ struct netfs_io_stream *stream = &creq->io_streams[1];
if (IS_ERR_OR_NULL(creq))
return;
- netfs_issue_write(creq, &creq->io_streams[1]);
- netfs_all_subreqs_queued(creq);
+ netfs_pgpriv2_issue_stream(creq, stream);
trace_netfs_rreq(rreq, netfs_rreq_trace_end_copy_to_cache);
if (list_empty_careful(&creq->io_streams[1].subrequests))
netfs_wake_collector(creq);
netfs_put_request(creq, netfs_rreq_trace_put_return);
- creq->copy_to_cache = NULL;
+ rreq->copy_to_cache = NULL;
}
/*
diff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c
index 733d9ff452c7..51ee780b11ed 100644
--- a/fs/netfs/read_retry.c
+++ b/fs/netfs/read_retry.c
@@ -9,20 +9,55 @@
#include <linux/slab.h>
#include "internal.h"
-static void netfs_reissue_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
+/*
+ * Prepare the I/O buffer on a buffered read subrequest for the filesystem to
+ * use as a bvec queue.
+ */
+int netfs_prepare_buffered_read_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
- bvecq_pos_unset(&subreq->content);
+ struct netfs_io_request *rreq = subreq->rreq;
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &rreq->retry_cursor);
bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
- iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
- subreq->content.slot, subreq->content.offset, subreq->len);
- iov_iter_advance(&subreq->io_iter, subreq->transferred);
+ len = bvecq_slice(&rreq->retry_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+ rreq->retry_buffered -= subreq->len;
+ rreq->retry_start += subreq->len;
+ return 0;
+}
- subreq->error = 0;
+/*
+ * Reset the state of the subrequest and discard any buffering so that we can
+ * retry (where this may include sending it to the server instead of the
+ * cache).
+ */
+int netfs_reset_for_read_retry(struct netfs_io_subrequest *subreq)
+{
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+
+ if (subreq->retry_count > 3) {
+ trace_netfs_sreq(subreq, netfs_sreq_trace_too_many_retries);
+ return subreq->error;
+ }
+
+ subreq->retry_count++;
__clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ __clear_bit(NETFS_SREQ_FAILED, &subreq->flags);
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
- netfs_stat(&netfs_n_rh_retry_read_subreq);
- subreq->rreq->netfs_ops->issue_read(subreq);
+ bvecq_pos_unset(&subreq->content);
+ bvecq_pos_unset(&subreq->dispatch_pos);
+ subreq->error = 0;
+ subreq->transferred = 0;
+ netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
+ netfs_stat(&netfs_n_wh_retry_write_subreq);
+ return 0;
}
/*
@@ -33,8 +68,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
{
struct netfs_io_subrequest *subreq;
struct netfs_io_stream *stream = &rreq->io_streams[0];
- struct bvecq_pos dispatch_cursor = {};
struct list_head *next;
+ int ret;
_enter("R=%x", rreq->debug_id);
@@ -44,46 +79,19 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
if (rreq->netfs_ops->retry_request)
rreq->netfs_ops->retry_request(rreq, NULL);
- /* If there's no renegotiation to do, just resend each retryable subreq
- * up to the first permanently failed one.
- */
- if (!rreq->netfs_ops->prepare_read &&
- !rreq->cache_resources.ops) {
- list_for_each_entry(subreq, &stream->subrequests, rreq_link) {
- if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
- break;
- if (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
- subreq->retry_count++;
- netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_read(rreq, subreq);
- }
- }
- return;
- }
+ /* Read pointer to subreq before reading subreq state. */
+ next = smp_load_acquire(&stream->subrequests.next);
- /* Okay, we need to renegotiate all the download requests and flip any
- * failed cache reads over to being download requests and negotiate
- * those also. All fully successful subreqs have been removed from the
- * list and any spare data from those has been donated.
- *
- * What we do is decant the list and rebuild it one subreq at a time so
- * that we don't end up with donations jumping over a gap we're busy
- * populating with smaller subrequests. In the event that the subreq
- * we just launched finishes before we insert the next subreq, it'll
- * fill in rreq->prev_donated instead.
- *
- * Note: Alternatively, we could split the tail subrequest right before
- * we reissue it and fix up the donations under lock.
+ /* Renegotiate all the download requests and flip any failed cache
+ * reads over to being download requests and negotiate those also.
*/
- next = stream->subrequests.next;
-
do {
struct netfs_io_subrequest *from, *to, *tmp;
- uoff_t start, len;
- size_t part;
- bool boundary = false, subreq_superfluous = false;
+ uoff_t start;
+ size_t len;
+ bool subreq_superfluous = false;
- bvecq_pos_unset(&dispatch_cursor);
+ bvecq_pos_unset(&rreq->retry_cursor);
/* Go through the subreqs and find the next span of contiguous
* buffer that we then rejig (cifs, for example, needs the
@@ -98,8 +106,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
rreq->debug_id, from->debug_index,
from->start, from->transferred, from->len);
- if (test_bit(NETFS_SREQ_FAILED, &from->flags) ||
- !test_bit(NETFS_SREQ_NEED_RETRY, &from->flags)) {
+ if (!test_bit(NETFS_SREQ_NEED_RETRY, &from->flags)) {
subreq = from;
goto abandon;
}
@@ -113,20 +120,21 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
subreq = list_entry(next, struct netfs_io_subrequest, rreq_link);
if (subreq->start != start + len ||
subreq->transferred > 0 ||
- test_bit(NETFS_SREQ_BOUNDARY, &subreq->flags) ||
!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
break;
to = subreq;
len += to->len;
}
- _debug(" - range: %llx-%llx %llx", start, start + len - 1, len);
+ _debug(" - range: %llx-%llx %zx", start, start + len - 1, len);
/* Determine the set of buffers we're going to use. Each
- * subreq gets a subset of a single overall contiguous buffer.
+ * subreq takes a subset of a single overall contiguous buffer.
*/
- bvecq_pos_transfer(&dispatch_cursor, &from->dispatch_pos);
- bvecq_pos_advance(&dispatch_cursor, from->transferred);
+ bvecq_pos_transfer(&rreq->retry_cursor, &from->dispatch_pos);
+ bvecq_pos_advance(&rreq->retry_cursor, from->transferred);
+ rreq->retry_start = start;
+ rreq->retry_buffered = len;
from->transferred = 0;
/* Work through the sublist. The chain of buffers we're going
@@ -135,51 +143,30 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
*/
subreq = from;
list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) {
- if (!len) {
+ if (rreq->retry_buffered == 0) {
subreq_superfluous = true;
break;
}
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- subreq->start = start;
- subreq->len = len;
- __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
- __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
- subreq->retry_count++;
- subreq->transferred = 0;
+ subreq->start = rreq->retry_start;
+ subreq->len = rreq->retry_buffered;
- bvecq_pos_unset(&subreq->content);
- bvecq_pos_unset(&subreq->dispatch_pos);
- bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
-
- /* Renegotiate max_len (rsize) */
- stream->sreq_max_len = len;
- stream->sreq_max_segs = INT_MAX;
- if (rreq->netfs_ops->prepare_read &&
- rreq->netfs_ops->prepare_read(subreq) < 0) {
- trace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed);
+ ret = netfs_reset_for_read_retry(subreq);
+ if (ret < 0) {
__set_bit(NETFS_SREQ_FAILED, &subreq->flags);
+ rreq->error = ret;
goto abandon;
}
- part = bvecq_slice(&dispatch_cursor,
- umin(len, stream->sreq_max_len),
- stream->sreq_max_segs,
- &subreq->nr_segs);
- subreq->len = part;
-
- len -= part;
- start += part;
- if (!len) {
- if (boundary)
- __set_bit(NETFS_SREQ_BOUNDARY, &subreq->flags);
- } else {
- __clear_bit(NETFS_SREQ_BOUNDARY, &subreq->flags);
+ netfs_stat(&netfs_n_rh_download);
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ goto abandon_after;
}
-
- netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_read(rreq, subreq);
+ if (test_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags))
+ goto abandon_after;
if (subreq == to) {
subreq_superfluous = false;
break;
@@ -189,7 +176,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
/* If we managed to use fewer subreqs, we can discard the
* excess; if we used the same number, then we're done.
*/
- if (!len) {
+ if (rreq->retry_buffered == 0) {
if (!subreq_superfluous)
continue;
list_for_each_entry_safe_from(subreq, tmp,
@@ -207,7 +194,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
}
/* We ran out of subrequests, so we need to allocate some more
- * and insert them after.
+ * and insert them after. They must start with being marked
+ * for retry to switch to the retry cursor.
*/
do {
subreq = netfs_alloc_subrequest(rreq);
@@ -216,8 +204,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
goto abandon_after;
}
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- subreq->start = start;
- subreq->len = len;
+ subreq->start = rreq->retry_start;
+ subreq->len = rreq->retry_buffered;
subreq->stream_nr = stream->stream_nr;
subreq->retry_count = 1;
@@ -225,43 +213,32 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
refcount_read(&subreq->ref),
netfs_sreq_trace_new);
+ __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
+
spin_lock(&rreq->lock);
+ /* Write IN_PROGRESS before pointer to new subreq */
+ smp_wmb();
list_add(&subreq->rreq_link, &to->rreq_link);
spin_unlock(&rreq->lock);
to = subreq;
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- stream->sreq_max_len = umin(len, rreq->rsize);
- stream->sreq_max_segs = INT_MAX;
-
netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read(subreq) < 0) {
- trace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed);
- __set_bit(NETFS_SREQ_FAILED, &subreq->flags);
- goto abandon;
- }
-
- bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
- part = bvecq_slice(&dispatch_cursor,
- umin(len, stream->sreq_max_len),
- stream->sreq_max_segs,
- &subreq->nr_segs);
- subreq->len = part;
-
- len -= part;
- start += part;
- if (!len && boundary) {
- __set_bit(NETFS_SREQ_BOUNDARY, &to->flags);
- boundary = false;
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ goto abandon_after;
}
+ if (test_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags))
+ goto abandon_after;
- netfs_reissue_read(rreq, subreq);
- } while (len);
+ } while (rreq->retry_buffered > 0);
} while (!list_is_head(next, &stream->subrequests));
out:
- bvecq_pos_unset(&dispatch_cursor);
+ bvecq_pos_unset(&rreq->retry_cursor);
return;
/* If we hit an error, fail all remaining incomplete subrequests */
@@ -334,6 +311,7 @@ void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq)
}
trace_netfs_folio(folio, netfs_folio_trace_abandon);
folio_unlock(folio);
+ p->bv[slot].bv_page = NULL;
}
}
}
diff --git a/fs/netfs/read_single.c b/fs/netfs/read_single.c
index 9fd07dbb08b5..4d387db98668 100644
--- a/fs/netfs/read_single.c
+++ b/fs/netfs/read_single.c
@@ -16,6 +16,22 @@
#include <linux/netfs.h>
#include "internal.h"
+int netfs_prepare_read_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *rreq = subreq->rreq;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+
+ bvecq_pos_set(&subreq->dispatch_pos, &rreq->load_cursor);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+
+ stream->buffered = 0;
+ stream->issue_from += subreq->len;
+ rreq->submitted = stream->issue_from;
+ netfs_all_subreqs_queued(rreq);
+ return 0;
+}
+
/**
* netfs_single_mark_inode_dirty - Mark a single, monolithic object inode dirty
* @inode: The inode to mark
@@ -58,17 +74,6 @@ static int netfs_single_begin_cache_read(struct netfs_io_request *rreq, struct n
return fscache_begin_read_operation(&rreq->cache_resources, netfs_i_cookie(ctx));
}
-static void netfs_single_read_cache(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
-{
- struct netfs_cache_resources *cres = &rreq->cache_resources;
-
- _enter("R=%08x[%x]", rreq->debug_id, subreq->debug_index);
- netfs_stat(&netfs_n_rh_read);
- cres->ops->read(cres, subreq->start, &subreq->io_iter, NETFS_READ_HOLE_FAIL,
- netfs_cache_read_terminated, subreq);
-}
-
/*
* Perform a read to a buffer from the cache or the server. Only a single
* subreq is permitted as the object must be fetched in a single transaction.
@@ -84,70 +89,84 @@ static int netfs_single_dispatch_read(struct netfs_io_request *rreq)
.cached_to[1] = ULLONG_MAX,
};
struct netfs_io_subrequest *subreq;
- int ret = 0;
+ int ret;
+
+ netfs_read_query_cache(rreq, &occ);
- subreq = netfs_alloc_subrequest(rreq);
+ subreq = netfs_alloc_read_subrequest(rreq);
if (!subreq)
return -ENOMEM;
- subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
subreq->start = 0;
subreq->len = rreq->len;
- bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor);
- bvecq_pos_set(&subreq->content, &rreq->dispatch_cursor);
-
- iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq,
- subreq->content.slot, subreq->content.offset, subreq->len);
-
- netfs_queue_read(rreq, subreq);
+ trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
/* Try to use the cache if the cache content matches the size of the
* remote file.
*/
- netfs_read_query_cache(rreq, &occ);
if (occ.cached_from[0] == 0 &&
- occ.cached_to[0] >= rreq->len)
- subreq->source = NETFS_READ_FROM_CACHE;
+ occ.cached_to[0] >= rreq->len) {
+ struct netfs_cache_resources *cres = &rreq->cache_resources;
- switch (subreq->source) {
- case NETFS_DOWNLOAD_FROM_SERVER:
- netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read) {
- ret = rreq->netfs_ops->prepare_read(subreq);
- if (ret < 0)
- goto cancel;
+ subreq->source = NETFS_READ_FROM_CACHE;
+ netfs_stat(&netfs_n_rh_read);
+ ret = cres->ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
}
- netfs_all_subreqs_queued(rreq);
- rreq->netfs_ops->issue_read(subreq);
- rreq->submitted += subreq->len;
- break;
- case NETFS_READ_FROM_CACHE:
- if (rreq->cache_resources.ops->prepare_read) {
- ret = rreq->cache_resources.ops->prepare_read(subreq);
- if (ret < 0)
- goto cancel;
+ ret = netfs_wait_for_in_progress_subreq(rreq, subreq);
+ if (ret == 0)
+ goto success;
+ if (ret == -ENOMEM)
+ goto cancel;
+
+ /* Didn't manage to retrieve from the cache, so toss it to the
+ * server instead.
+ */
+ if (netfs_reset_for_read_retry(subreq) < 0)
+ goto cancel;
+ }
+
+ __set_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags);
+
+ /* Try to send it to the cache. */
+ for (;;) {
+ subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
+ netfs_stat(&netfs_n_rh_download);
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
}
- netfs_all_subreqs_queued(rreq);
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- netfs_single_read_cache(rreq, subreq);
- rreq->submitted += subreq->len;
- ret = 0;
- break;
- default:
- pr_warn("Unexpected single-read source %u\n", subreq->source);
- WARN_ON_ONCE(true);
- ret = -EIO;
- goto cancel;
+ ret = netfs_wait_for_in_progress_subreq(rreq, subreq);
+ if (ret == 0)
+ goto success;
+ if (ret == -ENOMEM)
+ goto cancel;
+ if (ret != -EAGAIN)
+ goto failed;
+ if (netfs_reset_for_read_retry(subreq) < 0)
+ goto cancel;
}
- return ret;
+success:
+ rreq->transferred = subreq->transferred;
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_consumed);
+ return 0;
cancel:
- netfs_cancel_read(subreq, ret);
- netfs_all_subreqs_queued(rreq);
- netfs_wake_collector(rreq);
+ rreq->error = ret;
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_cancel);
+ return ret;
+failed:
+ rreq->error = ret;
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed);
return ret;
}
@@ -179,7 +198,7 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite
if (IS_ERR(rreq))
return PTR_ERR(rreq);
- ret = netfs_extract_iter(iter, rreq->len, INT_MAX, &rreq->dispatch_cursor.bvecq,
+ ret = netfs_extract_iter(iter, rreq->len, INT_MAX, &rreq->load_cursor.bvecq,
0, rreq->gfp);
if (ret < 0)
goto cleanup_free;
@@ -197,9 +216,29 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite
netfs_stat(&netfs_n_rh_read_single);
trace_netfs_read(rreq, 0, rreq->len, netfs_read_trace_read_single);
- netfs_single_dispatch_read(rreq);
+ ret = netfs_single_dispatch_read(rreq);
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_complete);
+ if (ret == 0) {
+ task_io_account_read(rreq->transferred);
+
+ if (test_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags) &&
+ fscache_resources_valid(&rreq->cache_resources)) {
+ trace_netfs_rreq(rreq, netfs_rreq_trace_dirty);
+ netfs_single_mark_inode_dirty(rreq->inode);
+ }
+ ret = rreq->transferred;
+ }
+
+ if (rreq->netfs_ops->done)
+ rreq->netfs_ops->done(rreq);
+
+ netfs_wake_rreq_flag(rreq, NETFS_RREQ_IN_PROGRESS, netfs_rreq_trace_wake_ip);
+ /* As we cleared NETFS_RREQ_IN_PROGRESS, we acquired its ref. */
+ netfs_put_request(rreq, netfs_rreq_trace_put_work_ip);
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_done);
- ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret;
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index 0c71f8c151c2..e0d14d77b751 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -68,6 +68,9 @@ struct netfs_wb_params {
struct netfs_write_estimate estimates[NR_IO_STREAMS];
};
+static int netfs_prepare_write_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
/*
* Kill all dirty folios in the event of an unrecoverable error, starting with
* a locked folio we've already obtained from writeback_iter().
@@ -148,7 +151,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
wreq->io_streams[0].source = NETFS_UPLOAD_TO_SERVER;
wreq->io_streams[0].applicable = NOTE_UPLOAD;
wreq->io_streams[0].estimate_write = ictx->ops->estimate_write;
- wreq->io_streams[0].prepare_write = ictx->ops->prepare_write;
wreq->io_streams[0].issue_write = ictx->ops->issue_write;
wreq->io_streams[0].collected_to = start;
wreq->io_streams[0].transferred = 0;
@@ -162,7 +164,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
wreq->io_streams[1].avail = true;
wreq->io_streams[1].active = true;
wreq->io_streams[1].estimate_write = wreq->cache_resources.ops->estimate_write;
- wreq->io_streams[1].prepare_write = wreq->cache_resources.ops->prepare_write_subreq;
wreq->io_streams[1].issue_write = wreq->cache_resources.ops->issue_write;
wreq->io_streams[1].alignment = wreq->cache_resources.dio_size;
}
@@ -170,19 +171,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
return wreq;
}
-/**
- * netfs_prepare_write_failed - Note write preparation failed
- * @subreq: The subrequest to mark
- *
- * Mark a subrequest to note that preparation for write failed.
- */
-void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq)
-{
- __set_bit(NETFS_SREQ_FAILED, &subreq->flags);
- trace_netfs_sreq(subreq, netfs_sreq_trace_prep_failed);
-}
-EXPORT_SYMBOL(netfs_prepare_write_failed);
-
/*
* Allocate and prepare a write subrequest. Will only return NULL if not
* performing writeback; if performing writeback, mempools may be accessed and
@@ -235,6 +223,7 @@ struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wr
return subreq;
}
+#if 0 // TODO: Remove old stuff
/*
* Prepare a write subrequest. We need to allocate a new subrequest
* if we don't have one.
@@ -295,6 +284,7 @@ void netfs_prepare_write(struct netfs_io_request *wreq,
stream->construct = subreq;
}
+#endif
/*
* Advance the state of the amount of data buffered on a stream.
@@ -341,6 +331,44 @@ static int netfs_prepare_buffered_write_buffer(struct netfs_io_subrequest *subre
return 0;
}
+/**
+ * netfs_prepare_write_buffer - Get the buffer for a subrequest
+ * @subreq: The subrequest to get the buffer for
+ * @max_segs: Maximum number of segments in buffer (or INT_MAX)
+ *
+ * Extract a slice of buffer from the stream and attach it to the subrequest as
+ * a bio_vec queue. The maximum amount of data attached is set by
+ * @subreq->len, but this may be shortened if @max_segs would be exceeded.
+ */
+int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *rreq = subreq->rreq;
+
+ switch (rreq->origin) {
+ case NETFS_WRITEBACK:
+ if (test_bit(NETFS_RREQ_RETRYING, &rreq->flags))
+ return netfs_prepare_write_retry_buffer(subreq, max_segs);
+ return netfs_prepare_buffered_write_buffer(subreq, max_segs);
+
+ case NETFS_UNBUFFERED_WRITE:
+ case NETFS_DIO_WRITE:
+ return netfs_prepare_unbuffered_write_buffer(subreq, max_segs);
+
+ case NETFS_WRITEBACK_SINGLE:
+ return netfs_prepare_write_single_buffer(subreq, max_segs);
+
+ case NETFS_PGPRIV2_COPY_TO_CACHE:
+ return netfs_prepare_pgpriv2_write_buffer(subreq, max_segs);
+
+ default:
+ WARN_ON_ONCE(1);
+ return -EIO;
+ }
+}
+EXPORT_SYMBOL(netfs_prepare_write_buffer);
+
+#if 0 // TODO: Remove old stuff
/*
* Set the I/O iterator for the filesystem/cache to use and dispatch the I/O
* operation. The operation may be asynchronous and should call
@@ -395,8 +423,8 @@ void netfs_reissue_write(struct netfs_io_stream *stream,
netfs_do_issue_write(stream, subreq);
}
-void netfs_issue_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream)
+static void netfs_issue_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream)
{
struct netfs_io_subrequest *subreq = stream->construct;
@@ -491,6 +519,7 @@ static int netfs_prep_and_issue_subreq(struct netfs_io_request *wreq,
stream->issue_write(subreq);
return 0;
}
+#endif
/*
* Issue writes for a stream.
@@ -523,7 +552,7 @@ static void netfs_writeback_flush(struct netfs_io_request *wreq,
return;
}
- ret = netfs_prep_and_issue_subreq(wreq, stream, subreq);
+ ret = stream->issue_write(subreq);
if (ret < 0) {
/* Ownership of subreq was returned to us. */
trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
@@ -649,6 +678,7 @@ static void netfs_writeback_add_folio_to_stream(struct netfs_io_request *wreq,
wreq->load_cursor.slot--;
trace_netfs_bv_slot(wreq->load_cursor.bvecq, wreq->load_cursor.slot - 1);
+ trace_netfs_wback(wreq, folio, params->notes);
for (int s = 0; s < NR_IO_STREAMS; s++) {
struct netfs_io_stream *stream = &wreq->io_streams[s];
@@ -846,31 +876,6 @@ static void netfs_writeback_folio(struct netfs_io_request *wreq,
goto out;
}
-#if 0 // TODO: Remove
-/*
- * End the issuing of writes, letting the collector know we're done.
- */
-static void netfs_end_issue_write(struct netfs_io_request *wreq)
-{
- bool needs_poke = true;
-
- netfs_all_subreqs_queued(wreq);
-
- for (int s = 0; s < NR_IO_STREAMS; s++) {
- struct netfs_io_stream *stream = &wreq->io_streams[s];
-
- if (!stream->active)
- continue;
- if (!list_empty(&stream->subrequests))
- needs_poke = false;
- netfs_issue_write(wreq, stream);
- }
-
- if (needs_poke)
- netfs_wake_collector(wreq);
-}
-#endif
-
/*
* Write some of the pending data back to the server
*/
@@ -957,6 +962,25 @@ int netfs_writepages(struct address_space *mapping,
}
EXPORT_SYMBOL(netfs_writepages);
+/*
+ * Prepare a buffer for a single monolithic write.
+ */
+static int netfs_prepare_write_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *wreq = subreq->rreq;
+ struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+
+ stream->buffered = 0;
+ stream->issue_from = subreq->len;
+ wreq->submitted = subreq->len;
+ netfs_all_subreqs_queued(wreq);
+ return 0;
+}
+
/**
* netfs_writeback_single - Write back a monolithic payload
* @mapping: The mapping to write from
@@ -968,6 +992,11 @@ EXPORT_SYMBOL(netfs_writepages);
* cache. There's a maximum of one subrequest per stream. The buffer should
* be rounded out sufficiently that it can accommodate cache DIO rounding.
*
+ * This is normally only used to write to the cache (for AFS directories and
+ * symlinks); it doesn't normally write to the server as well. The filesystem
+ * can override that by setting NETFS_RREQ_UPLOAD_TO_SERVER when the request is
+ * initialised.
+ *
* Return: 0 if successful; 1 if skipped due to lock conflict and WB_SYNC_NONE;
* or a negative error code.
* the cache. There's a maximum of one subrequest per stream.
@@ -981,6 +1010,11 @@ int netfs_writeback_single(struct address_space *mapping,
size_t clen;
int ret;
+ _enter("%zx,%zx", iov_iter_count(iter), len);
+
+ if (!len)
+ return 0;
+
if (!netfs_wb_begin(ictx, wbc->sync_mode == WB_SYNC_NONE)) {
/* The VFS will have undirtied the inode. */
netfs_single_mark_inode_dirty(&ictx->inode);
@@ -992,9 +1026,9 @@ int netfs_writeback_single(struct address_space *mapping,
ret = PTR_ERR(wreq);
goto couldnt_start;
}
+
wreq->len = len;
clen = len;
-
if (wreq->cache_resources.dio_size > 1) {
clen = round_up(len, wreq->cache_resources.dio_size);
if (clen > iov_iter_count(iter)) {
@@ -1003,7 +1037,7 @@ int netfs_writeback_single(struct address_space *mapping,
}
}
- ret = netfs_extract_iter(iter, clen, INT_MAX, &wreq->dispatch_cursor.bvecq,
+ ret = netfs_extract_iter(iter, clen, INT_MAX, &wreq->load_cursor.bvecq,
0, wreq->gfp);
if (ret < 0)
goto cleanup_free;
@@ -1012,12 +1046,14 @@ int netfs_writeback_single(struct address_space *mapping,
goto cleanup_free;
}
- bvecq_pos_set(&wreq->collect_cursor, &wreq->dispatch_cursor);
-
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags);
trace_netfs_write(wreq, netfs_write_trace_writeback_single);
netfs_stat(&netfs_n_wh_writepages);
+ /* This normally just writes to the cache; if the filesystem wants to
+ * write to the server too, it must set UPLOAD_TO_SERVER in
+ * ->init_request().
+ */
if (test_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags))
wreq->netfs_ops->begin_writeback(wreq);
@@ -1028,14 +1064,29 @@ int netfs_writeback_single(struct address_space *mapping,
if (!stream->avail)
continue;
- netfs_prepare_write(wreq, stream, 0);
-
- subreq = stream->construct;
- subreq->len = wreq->len;
+ stream->issue_from = 0;
+ stream->buffered = len;
if (stream->source == NETFS_WRITE_TO_CACHE)
- subreq->len = clen;
+ stream->buffered = clen;
- netfs_issue_write(wreq, stream);
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ if (!subreq) {
+ ret = -ENOMEM;
+ break;
+ }
+
+ bvecq_pos_set(&stream->dispatch_cursor, &wreq->load_cursor);
+
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ stream->buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
+ /* Punt the error to the collector. */
+ }
+
+ bvecq_pos_unset(&stream->dispatch_cursor);
}
wreq->submitted = wreq->len;
diff --git a/fs/netfs/write_retry.c b/fs/netfs/write_retry.c
index c61bed687244..153c180f4fed 100644
--- a/fs/netfs/write_retry.c
+++ b/fs/netfs/write_retry.c
@@ -12,13 +12,44 @@
#include "internal.h"
/*
- * Perform retries on the streams that need it.
+ * Prepare the write buffer for a retry. We can't necessarily reuse the write
+ * buffer from the previous run of a subrequest because the filesystem is
+ * permitted to modify it (add headers/trailers, encrypt it). Further, the
+ * subrequest may now be a different size (e.g. cifs has to negotiate for
+ * maximum transfer size). Also, we can't look at *stream as that may still
+ * refer to the source material being broken up into original subrequests.
+ */
+int netfs_prepare_write_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *wreq = subreq->rreq;
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &wreq->retry_cursor);
+ bvecq_pos_set(&subreq->content, &wreq->retry_cursor);
+ len = bvecq_slice(&wreq->retry_cursor, subreq->len, max_segs, &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ wreq->retry_start += len;
+ wreq->retry_buffered -= len;
+ if (wreq->retry_buffered == 0)
+ bvecq_pos_unset(&wreq->retry_cursor);
+ return 0;
+}
+
+/*
+ * Perform retries on the streams that need it. This only has to deal with
+ * buffered writes; unbuffered write retry is handled in direct_write.c.
*/
static void netfs_retry_write_stream(struct netfs_io_request *wreq,
struct netfs_io_stream *stream)
{
- struct bvecq_pos dispatch_cursor = {};
struct list_head *next;
+ int ret;
_enter("R=%x[%x:]", wreq->debug_id, stream->stream_nr);
@@ -33,30 +64,15 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
if (unlikely(stream->failed))
return;
- /* If there's no renegotiation to do, just resend each failed subreq. */
- if (!stream->prepare_write) {
- struct netfs_io_subrequest *subreq;
-
- list_for_each_entry(subreq, &stream->subrequests, rreq_link) {
- if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
- break;
- if (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
- netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_write(stream, subreq);
- }
- }
- return;
- }
-
- next = stream->subrequests.next;
+ /* Read pointer to subreq before reading subreq state. */
+ next = smp_load_acquire(&stream->subrequests.next);
do {
struct netfs_io_subrequest *subreq = NULL, *from, *to, *tmp;
uoff_t start, len;
- size_t part;
- bool boundary = false;
+ bool subreq_superfluous = false;
- bvecq_pos_unset(&dispatch_cursor);
+ bvecq_pos_unset(&wreq->retry_cursor);
/* Go through the stream and find the next span of contiguous
* data that we then rejig (cifs, for example, needs the wsize
@@ -80,7 +96,6 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
subreq = list_entry(next, struct netfs_io_subrequest, rreq_link);
if (subreq->start != start + len ||
subreq->transferred > 0 ||
- test_bit(NETFS_SREQ_BOUNDARY, &subreq->flags) ||
!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
break;
to = subreq;
@@ -90,8 +105,10 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
/* Determine the set of buffers we're going to use. Each
* subreq gets a subset of a single overall contiguous buffer.
*/
- bvecq_pos_transfer(&dispatch_cursor, &from->dispatch_pos);
- bvecq_pos_advance(&dispatch_cursor, from->transferred);
+ bvecq_pos_transfer(&wreq->retry_cursor, &from->dispatch_pos);
+ bvecq_pos_advance(&wreq->retry_cursor, from->transferred);
+ wreq->retry_start = start;
+ wreq->retry_buffered = len;
/* Work through the sublist. The chain of buffers we're going
* to fill is attached to dispatch_cursor and we need to read
@@ -99,47 +116,50 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
*/
subreq = from;
list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) {
- if (!len)
+ if (!wreq->retry_buffered) {
+ subreq_superfluous = true;
break;
-
- subreq->start = start;
- subreq->len = len;
- __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
- trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- subreq->transferred = 0;
+ }
bvecq_pos_unset(&subreq->content);
bvecq_pos_unset(&subreq->dispatch_pos);
- /* Renegotiate max_len (wsize) */
- stream->sreq_max_len = len;
- stream->sreq_max_segs = INT_MAX;
- stream->prepare_write(subreq);
-
- bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
- part = bvecq_slice(&dispatch_cursor,
- umin(len, stream->sreq_max_len),
- stream->sreq_max_segs,
- &subreq->nr_segs);
- subreq->len = part;
-
- len -= part;
- start += part;
- if (len && subreq == to &&
- __test_and_clear_bit(NETFS_SREQ_BOUNDARY, &to->flags))
- boundary = true;
-
+ __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ __clear_bit(NETFS_SREQ_FAILED, &subreq->flags);
+ __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
+ subreq->start = wreq->retry_start;
+ subreq->len = wreq->retry_buffered;
+ subreq->transferred = 0;
+ subreq->retry_count += 1;
+ subreq->error = 0;
+
+ netfs_stat(&netfs_n_wh_retry_write_subreq);
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_write(stream, subreq);
- if (subreq == to)
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us.
+ * Expand the subreq to consume the entire
+ * remaining amount to be retried and fail it.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ wreq->retry_buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
+ break;
+ }
+
+ if (subreq == to) {
+ subreq_superfluous = false;
break;
+ }
}
/* If we managed to use fewer subreqs, we can discard the
* excess; if we used the same number, then we're done.
*/
- if (!len) {
- if (subreq == to)
+ if (!wreq->retry_buffered) {
+ if (!subreq_superfluous)
continue;
list_for_each_entry_safe_from(subreq, tmp,
&stream->subrequests, rreq_link) {
@@ -161,6 +181,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
subreq = netfs_alloc_subrequest(wreq);
subreq->source = to->source;
subreq->start = start;
+ subreq->len = wreq->retry_buffered;
subreq->stream_nr = to->stream_nr;
subreq->retry_count = 1;
@@ -169,49 +190,40 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
netfs_sreq_trace_new);
trace_netfs_sreq(subreq, netfs_sreq_trace_split);
+ __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
spin_lock(&wreq->lock);
+ /* Write IN_PROGRESS before pointer to new subreq */
+ smp_wmb();
list_add(&subreq->rreq_link, &to->rreq_link);
spin_unlock(&wreq->lock);
to = subreq;
- trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- stream->sreq_max_len = len;
- stream->sreq_max_segs = INT_MAX;
switch (stream->source) {
case NETFS_UPLOAD_TO_SERVER:
netfs_stat(&netfs_n_wh_upload);
- stream->sreq_max_len = umin(len, wreq->wsize);
break;
default:
WARN_ON_ONCE(1);
}
- stream->prepare_write(subreq);
-
- bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor);
- part = bvecq_slice(&dispatch_cursor,
- umin(len, stream->sreq_max_len),
- stream->sreq_max_segs,
- &subreq->nr_segs);
- subreq->len = subreq->transferred + part;
-
- len -= part;
- start += part;
- if (!len && boundary) {
- __set_bit(NETFS_SREQ_BOUNDARY, &to->flags);
- boundary = false;
- }
-
- netfs_reissue_write(stream, subreq);
- if (!len)
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us.
+ * Expand the subreq to consume the entire
+ * remaining amount to be retried and fail it.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ wreq->retry_buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
break;
-
- } while (len);
+ }
+ } while (wreq->retry_buffered > 0);
} while (!list_is_head(next, &stream->subrequests));
out:
- bvecq_pos_unset(&dispatch_cursor);
+ bvecq_pos_unset(&wreq->retry_cursor);
}
/*
diff --git a/fs/nfs/fscache.c b/fs/nfs/fscache.c
index 9b7fdad4a920..fb1441f88661 100644
--- a/fs/nfs/fscache.c
+++ b/fs/nfs/fscache.c
@@ -23,6 +23,7 @@
#include "iostat.h"
#include "fscache.h"
#include "nfstrace.h"
+#include <trace/events/netfs.h>
#define NFS_MAX_KEY_LEN 1000
@@ -273,8 +274,6 @@ static int nfs_netfs_init_request(struct netfs_io_request *rreq, struct file *fi
rreq->debug_id = atomic_inc_return(&nfs_netfs_debug_id);
/* [DEPRECATED] Use PG_private_2 to mark folio being written to the cache. */
__set_bit(NETFS_RREQ_USE_PGPRIV2, &rreq->flags);
- rreq->io_streams[0].sreq_max_len = NFS_SB(rreq->inode->i_sb)->rsize;
-
return 0;
}
@@ -296,8 +295,9 @@ static struct nfs_netfs_io_data *nfs_netfs_alloc(struct netfs_io_subrequest *sre
return netfs;
}
-static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
+static int nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
{
+ struct netfs_io_request *rreq = sreq->rreq;
struct nfs_netfs_io_data *netfs;
struct nfs_pageio_descriptor pgio;
struct inode *inode = sreq->rreq->inode;
@@ -307,6 +307,16 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
pgoff_t start, last;
int err;
+ if (sreq->len > NFS_SB(rreq->inode->i_sb)->rsize)
+ sreq->len = NFS_SB(rreq->inode->i_sb)->rsize;
+
+ err = netfs_prepare_read_buffer(sreq, INT_MAX);
+ if (err < 0) {
+ sreq->error = err;
+ return err;
+ }
+ /* After this point, must fail by termination. */
+
start = (sreq->start + sreq->transferred) >> PAGE_SHIFT;
last = ((sreq->start + sreq->len - sreq->transferred - 1) >> PAGE_SHIFT);
@@ -316,12 +326,15 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
netfs = nfs_netfs_alloc(sreq);
if (!netfs) {
sreq->error = -ENOMEM;
- return netfs_read_subreq_terminated(sreq);
+ netfs_read_subreq_terminated(sreq);
+ return 0;
}
+ trace_netfs_sreq(sreq, netfs_sreq_trace_submit);
+
pgio.pg_netfs = netfs; /* used in completion */
- xa_for_each_range(&sreq->rreq->mapping->i_pages, idx, page, start, last) {
+ xa_for_each_range(&rreq->mapping->i_pages, idx, page, start, last) {
/* nfs_read_add_folio() may schedule() due to pNFS layout and other RPCs */
err = nfs_read_add_folio(&pgio, ctx, page_folio(page));
if (err < 0) {
@@ -332,6 +345,7 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
out:
nfs_pageio_complete_read(&pgio);
nfs_netfs_put(netfs);
+ return 0;
}
void nfs_netfs_initiate_read(struct nfs_pgio_header *hdr)
diff --git a/fs/smb/client/cifssmb.c b/fs/smb/client/cifssmb.c
index 1f77512252e7..37d576d38837 100644
--- a/fs/smb/client/cifssmb.c
+++ b/fs/smb/client/cifssmb.c
@@ -1467,8 +1467,7 @@ cifs_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)
struct cifs_tcon *tcon = tlink_tcon(rdata->req->cfile->tlink);
struct inode *inode = &ictx->inode;
struct smb_rqst rqst = { .rq_iov = rdata->iov,
- .rq_nvec = 1,
- .rq_iter = rdata->subreq.io_iter };
+ .rq_nvec = 1};
struct cifs_credits credits = {
.value = 1,
.instance = 0,
@@ -1482,6 +1481,11 @@ cifs_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)
__func__, mid->mid, mid->mid_state, rdata->result,
rdata->subreq.len);
+ if (rdata->got_bytes)
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
+
switch (mid->mid_state) {
case MID_RESPONSE_RECEIVED:
/* result already set, check signature */
@@ -2007,7 +2011,10 @@ cifs_async_writev(struct cifs_io_subrequest *wdata)
rqst.rq_iov = iov;
rqst.rq_nvec = 1;
- rqst.rq_iter = wdata->subreq.io_iter;
+
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_SOURCE,
+ wdata->subreq.content.bvecq, wdata->subreq.content.slot,
+ wdata->subreq.content.offset, wdata->subreq.len);
cifs_dbg(FYI, "async write at %llu %zu bytes\n",
wdata->subreq.start, wdata->subreq.len);
diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c
index 7ad464782439..54db4e3a83fd 100644
--- a/fs/smb/client/file.c
+++ b/fs/smb/client/file.c
@@ -53,21 +53,23 @@ static int cifs_estimate_write(struct netfs_io_request *wreq,
}
/*
- * Prepare a subrequest to upload to the server. We need to allocate credits
- * so that we know the maximum amount of data that we can include in it.
+ * Issue a subrequest to upload to the server.
*/
-static void cifs_prepare_write(struct netfs_io_subrequest *subreq)
+static int cifs_issue_write(struct netfs_io_subrequest *subreq)
{
struct cifs_io_subrequest *wdata =
container_of(subreq, struct cifs_io_subrequest, subreq);
struct cifs_io_request *req = wdata->req;
- struct netfs_io_stream *stream = &req->rreq.io_streams[subreq->stream_nr];
struct TCP_Server_Info *server;
struct cifsFileInfo *open_file = req->cfile;
- struct cifs_sb_info *cifs_sb = CIFS_SB(wdata->rreq->inode->i_sb);
- size_t wsize = req->rreq.wsize;
+ struct cifs_sb_info *cifs_sb = CIFS_SB(subreq->rreq->inode->i_sb);
+ unsigned int max_segs = INT_MAX;
+ size_t len;
int rc;
+ if (cifs_forced_shutdown(cifs_sb))
+ return smb_EIO(smb_eio_trace_forced_shutdown);
+
if (!wdata->have_xid) {
wdata->xid = get_xid();
wdata->have_xid = true;
@@ -86,18 +88,16 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)
if (rc < 0) {
if (rc == -EAGAIN)
goto retry;
- subreq->error = rc;
- return netfs_prepare_write_failed(subreq);
+ return rc;
}
}
- rc = server->ops->wait_mtu_credits(server, wsize, &stream->sreq_max_len,
- &wdata->credits);
- if (rc < 0) {
- subreq->error = rc;
- return netfs_prepare_write_failed(subreq);
- }
+ len = umin(subreq->len, cifs_sb->ctx->wsize);
+ rc = server->ops->wait_mtu_credits(server, len, &len, &wdata->credits);
+ if (rc < 0)
+ return rc;
+ subreq->len = len;
wdata->credits.rreq_debug_id = subreq->rreq->debug_id;
wdata->credits.rreq_debug_index = subreq->debug_index;
wdata->credits.in_flight_check = 1;
@@ -113,46 +113,34 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)
const struct smbdirect_socket_parameters *sp =
smbd_get_parameters(server->smbd_conn);
- stream->sreq_max_segs = sp->max_frmr_depth;
+ max_segs = sp->max_frmr_depth;
}
#endif
-}
-/*
- * Issue a subrequest to upload to the server.
- */
-static void cifs_issue_write(struct netfs_io_subrequest *subreq)
-{
- struct cifs_io_subrequest *wdata =
- container_of(subreq, struct cifs_io_subrequest, subreq);
- struct cifs_sb_info *sbi = CIFS_SB(subreq->rreq->inode->i_sb);
- int rc;
-
- if (cifs_forced_shutdown(sbi)) {
- rc = smb_EIO(smb_eio_trace_forced_shutdown);
- goto fail;
+ rc = netfs_prepare_write_buffer(subreq, max_segs);
+ if (rc < 0) {
+ add_credits_and_wake_if(wdata->server, &wdata->credits, 0);
+ return rc;
}
+ /* After this point, must fail by termination. */
- rc = adjust_credits(wdata->server, wdata, cifs_trace_rw_credits_issue_write_adjust);
+ rc = adjust_credits(server, wdata, cifs_trace_rw_credits_issue_write_adjust);
if (rc)
- goto fail;
+ goto fail_with_credits;
rc = -EAGAIN;
if (wdata->req->cfile->invalidHandle)
- goto fail;
+ goto fail_with_credits;
wdata->server->ops->async_writev(wdata);
-out:
- return;
+ return 0;
-fail:
+fail_with_credits:
if (rc == -EAGAIN)
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- else
- trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
add_credits_and_wake_if(wdata->server, &wdata->credits, 0);
cifs_write_subrequest_terminated(wdata, rc);
- goto out;
+ return 0;
}
static void cifs_netfs_invalidate_cache(struct netfs_io_request *wreq)
@@ -161,17 +149,25 @@ static void cifs_netfs_invalidate_cache(struct netfs_io_request *wreq)
}
/*
- * Negotiate the size of a read operation on behalf of the netfs library.
+ * Issue a read operation on behalf of the netfs helper functions. We're asked
+ * to make a read of a certain size at a point in the file. We are permitted
+ * to only read a portion of that, but as long as we read something, the netfs
+ * helper will call us again so that we can issue another read.
*/
-static int cifs_prepare_read(struct netfs_io_subrequest *subreq)
+static int cifs_issue_read(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq);
struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq);
- struct TCP_Server_Info *server;
+ struct TCP_Server_Info *server = rdata->server;
struct cifs_sb_info *cifs_sb = CIFS_SB(rreq->inode->i_sb);
- size_t size;
- int rc = 0;
+ unsigned int max_segs = INT_MAX;
+ size_t len;
+ int rc;
+
+ cifs_dbg(FYI, "%s: op=%08x[%x] mapping=%p len=%zu/%zu\n",
+ __func__, rreq->debug_id, subreq->debug_index, rreq->mapping,
+ subreq->transferred, subreq->len);
if (!rdata->have_xid) {
rdata->xid = get_xid();
@@ -185,17 +181,15 @@ static int cifs_prepare_read(struct netfs_io_subrequest *subreq)
cifs_negotiate_rsize(server, cifs_sb->ctx,
tlink_tcon(req->cfile->tlink));
- rc = server->ops->wait_mtu_credits(server, cifs_sb->ctx->rsize,
- &size, &rdata->credits);
+ len = umin(subreq->len, cifs_sb->ctx->rsize);
+ rc = server->ops->wait_mtu_credits(server, len, &len, &rdata->credits);
if (rc)
return rc;
- rreq->io_streams[0].sreq_max_len = size;
-
- rdata->credits.in_flight_check = 1;
+ subreq->len = len;
rdata->credits.rreq_debug_id = rreq->debug_id;
rdata->credits.rreq_debug_index = subreq->debug_index;
-
+ rdata->credits.in_flight_check = 1;
trace_smb3_rw_credits(rdata->rreq->debug_id,
rdata->subreq.debug_index,
rdata->credits.value,
@@ -207,40 +201,27 @@ static int cifs_prepare_read(struct netfs_io_subrequest *subreq)
const struct smbdirect_socket_parameters *sp =
smbd_get_parameters(server->smbd_conn);
- rreq->io_streams[0].sreq_max_segs = sp->max_frmr_depth;
+ max_segs = sp->max_frmr_depth;
}
#endif
- return 0;
-}
-/*
- * Issue a read operation on behalf of the netfs helper functions. We're asked
- * to make a read of a certain size at a point in the file. We are permitted
- * to only read a portion of that, but as long as we read something, the netfs
- * helper will call us again so that we can issue another read.
- */
-static void cifs_issue_read(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_request *rreq = subreq->rreq;
- struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq);
- struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq);
- struct TCP_Server_Info *server = rdata->server;
- int rc = 0;
-
- cifs_dbg(FYI, "%s: op=%08x[%x] mapping=%p len=%zu/%zu\n",
- __func__, rreq->debug_id, subreq->debug_index, rreq->mapping,
- subreq->transferred, subreq->len);
+ rc = netfs_prepare_read_buffer(subreq, max_segs);
+ if (rc < 0) {
+ add_credits_and_wake_if(rdata->server, &rdata->credits, 0);
+ return rc;
+ }
+ /* After this point, must fail by termination. */
rc = adjust_credits(server, rdata, cifs_trace_rw_credits_issue_read_adjust);
if (rc)
- goto failed;
+ goto fail_with_credits;
if (req->cfile->invalidHandle) {
do {
rc = cifs_reopen_file(req->cfile, true);
} while (rc == -EAGAIN);
if (rc)
- goto failed;
+ goto fail_with_credits;
}
if (subreq->rreq->origin != NETFS_UNBUFFERED_READ &&
@@ -248,15 +229,22 @@ static void cifs_issue_read(struct netfs_io_subrequest *subreq)
__set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags);
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+
rc = rdata->server->ops->async_readv(rdata);
if (rc)
goto failed;
- return;
+ return 0;
+fail_with_credits:
+ if (rc == -EAGAIN)
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+ else
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
failed:
add_credits_and_wake_if(rdata->server, &rdata->credits, 0);
subreq->error = rc;
netfs_read_subreq_terminated(subreq);
+ return 0;
}
/*
@@ -371,12 +359,10 @@ const struct netfs_request_ops cifs_req_ops = {
.init_request = cifs_init_request,
.free_request = cifs_free_request,
.free_subrequest = cifs_free_subrequest,
- .prepare_read = cifs_prepare_read,
.issue_read = cifs_issue_read,
.done = cifs_rreq_done,
.begin_writeback = cifs_begin_writeback,
.estimate_write = cifs_estimate_write,
- .prepare_write = cifs_prepare_write,
.issue_write = cifs_issue_write,
.invalidate_cache = cifs_netfs_invalidate_cache,
};
diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c
index 97823bcf198c..1f3266f89cce 100644
--- a/fs/smb/client/smb2ops.c
+++ b/fs/smb/client/smb2ops.c
@@ -4884,6 +4884,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
unsigned int cur_page_idx;
unsigned int pad_len;
struct cifs_io_subrequest *rdata = mid->callback_data;
+ struct iov_iter iter;
struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
size_t copied;
bool use_rdma_mr = false;
@@ -4956,6 +4957,10 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
pad_len = data_offset - server->vals->read_rsp_size;
+ iov_iter_bvec_queue(&iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
+
if (buf_len <= data_offset) {
/* read response payload is in pages */
cur_page_idx = pad_len / PAGE_SIZE;
@@ -4985,7 +4990,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
/* Copy the data to the output I/O iterator. */
rdata->result = cifs_copy_bvecq_to_iter(buffer, data_len,
- cur_off, &rdata->subreq.io_iter);
+ cur_off, &iter);
if (rdata->result != 0) {
if (is_offloaded)
mid->mid_state = MID_RESPONSE_MALFORMED;
@@ -4999,7 +5004,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
buf_len >= end_off) {
/* read response payload is in buf */
WARN_ONCE(buffer, "read data can be either in buf or in buffer");
- copied = copy_to_iter(buf + data_offset, data_len, &rdata->subreq.io_iter);
+ copied = copy_to_iter(buf + data_offset, data_len, &iter);
if (copied == 0)
return smb_EIO2(smb_eio_trace_rx_copy_to_iter, copied, data_len);
rdata->got_bytes = copied;
diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c
index 4ce165e40657..085fdc7d8bec 100644
--- a/fs/smb/client/smb2pdu.c
+++ b/fs/smb/client/smb2pdu.c
@@ -4592,9 +4592,13 @@ smb2_new_read_req(void **buf, unsigned int *total_len,
*/
if (rdata && smb3_use_rdma_offload(io_parms)) {
struct smbdirect_buffer_descriptor_v1 *v1;
+ struct iov_iter iter;
bool need_invalidate = server->dialect == SMB30_PROT_ID;
- rdata->mr = smbd_register_mr(server->smbd_conn, &rdata->subreq.io_iter,
+ iov_iter_bvec_queue(&iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
+ rdata->mr = smbd_register_mr(server->smbd_conn, &iter,
true, need_invalidate);
if (!rdata->mr)
return -EAGAIN;
@@ -4658,9 +4662,10 @@ smb2_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)
unsigned int rreq_debug_id = rdata->rreq->debug_id;
unsigned int subreq_debug_index = rdata->subreq.debug_index;
- if (rdata->got_bytes) {
- rqst.rq_iter = rdata->subreq.io_iter;
- }
+ if (rdata->got_bytes)
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
WARN_ONCE(rdata->server != server,
"rdata server %p != mid server %p",
@@ -5148,7 +5153,9 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
goto out;
rqst.rq_iov = iov;
- rqst.rq_iter = wdata->subreq.io_iter;
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_SOURCE,
+ wdata->subreq.content.bvecq, wdata->subreq.content.slot,
+ wdata->subreq.content.offset, wdata->subreq.len);
rqst.rq_iov[0].iov_len = total_len - 1;
rqst.rq_iov[0].iov_base = (char *)req;
@@ -5187,9 +5194,14 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
*/
if (smb3_use_rdma_offload(io_parms)) {
struct smbdirect_buffer_descriptor_v1 *v1;
+ struct iov_iter iter;
bool need_invalidate = server->dialect == SMB30_PROT_ID;
- wdata->mr = smbd_register_mr(server->smbd_conn, &wdata->subreq.io_iter,
+ iov_iter_bvec_queue(&iter, ITER_SOURCE,
+ wdata->subreq.content.bvecq, wdata->subreq.content.slot,
+ wdata->subreq.content.offset, wdata->subreq.len);
+
+ wdata->mr = smbd_register_mr(server->smbd_conn, &iter,
false, need_invalidate);
if (!wdata->mr) {
rc = -EAGAIN;
@@ -5226,8 +5238,8 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
smb2_set_replay(server, &rqst);
}
- cifs_dbg(FYI, "async write at %llu %u bytes iter=%zx\n",
- io_parms->offset, io_parms->length, iov_iter_count(&wdata->subreq.io_iter));
+ cifs_dbg(FYI, "async write at %llu %u bytes len=%zx\n",
+ io_parms->offset, io_parms->length, wdata->subreq.len);
if (wdata->credits.value > 0) {
shdr->CreditCharge = cpu_to_le16(DIV_ROUND_UP(wdata->subreq.len,
diff --git a/fs/smb/client/transport.c b/fs/smb/client/transport.c
index fdf4e50c27ce..be2f6b909c34 100644
--- a/fs/smb/client/transport.c
+++ b/fs/smb/client/transport.c
@@ -1267,12 +1267,19 @@ cifs_readv_receive(struct TCP_Server_Info *server, struct mid_q_entry *mid)
}
#ifdef CONFIG_CIFS_SMB_DIRECT
- if (rdata->mr)
+ if (rdata->mr) {
length = data_len; /* An RDMA read is already done. */
- else
+ } else {
+#endif
+ struct iov_iter iter;
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, rdata->subreq.content.bvecq,
+ rdata->subreq.content.slot, rdata->subreq.content.offset,
+ data_len);
+ length = cifs_read_iter_from_socket(server, &iter, data_len);
+#ifdef CONFIG_CIFS_SMB_DIRECT
+ }
#endif
- length = cifs_read_iter_from_socket(server, &rdata->subreq.io_iter,
- data_len);
if (length > 0)
rdata->got_bytes += length;
server->total_read += length;
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 2c5b7f3a48b7..5eb836086329 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -162,23 +162,17 @@ struct netfs_write_estimate {
struct netfs_io_stream {
/* Submission tracking (main dispatch only; not retry) */
struct bvecq_pos dispatch_cursor; /* Point from which buffers are dispatched */
- struct netfs_io_subrequest *construct; /* Op being constructed */
uoff_t issue_from; /* Current issue point */
uoff_t last_end; /* End file pos of last folio added */
size_t buffered; /* Amount in buffer */
size_t post_gap; /* Length of partial folio tail */
- size_t sreq_max_len; /* Maximum size of a subrequest */
- unsigned int sreq_max_segs; /* 0 or max number of segments in an iterator */
- unsigned int submit_off; /* Folio offset we're submitting from */
- unsigned int submit_len; /* Amount of data left to submit */
unsigned int alignment; /* Required alignment */
u8 applicable; /* What sources are applicable (NOTE_* mask) */
bool buffering; /* T if buffering on this stream */
int (*estimate_write)(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
struct netfs_write_estimate *estimate);
- void (*prepare_write)(struct netfs_io_subrequest *subreq);
- void (*issue_write)(struct netfs_io_subrequest *subreq);
+ int (*issue_write)(struct netfs_io_subrequest *subreq);
atomic64_t issued_to; /* Point to which can be considered issued */
/* Collection tracking */
@@ -223,15 +217,14 @@ struct netfs_io_subrequest {
struct list_head rreq_link; /* Link in rreq->subrequests */
struct bvecq_pos dispatch_pos; /* Bookmark in the combined queue of the start */
struct bvecq_pos content; /* The (copied) content of the subrequest */
- struct iov_iter io_iter; /* Iterator for this subrequest */
uoff_t start; /* Where to start the I/O */
size_t len; /* Size of the I/O */
size_t post_gap; /* Length of partial folio tail */
size_t transferred; /* Amount of data transferred */
+ unsigned int nr_segs; /* Number of segments in content */
refcount_t ref;
short error; /* 0 or error that occurred */
unsigned short debug_index; /* Index in list (for debugging output) */
- unsigned int nr_segs; /* Number of segments in content */
u8 retry_count; /* The number of retries (0 on initial pass) */
enum netfs_io_source source; /* Where to read from/write to */
unsigned char stream_nr; /* I/O stream this belongs to */
@@ -239,7 +232,6 @@ struct netfs_io_subrequest {
#define NETFS_SREQ_COPY_TO_CACHE 0 /* Set if should copy the data to the cache */
#define NETFS_SREQ_CLEAR_TAIL 1 /* Set if the rest of the read should be cleared */
#define NETFS_SREQ_MADE_PROGRESS 4 /* Set if we transferred at least some data */
-#define NETFS_SREQ_BOUNDARY 6 /* Set if ends on hard boundary (eg. ceph object) */
#define NETFS_SREQ_HIT_EOF 7 /* Set if short due to EOF */
#define NETFS_SREQ_IN_PROGRESS 8 /* Unlocked when the subrequest completes */
#define NETFS_SREQ_NEED_RETRY 9 /* Set if the filesystem requests a retry */
@@ -288,12 +280,11 @@ struct netfs_io_request {
struct netfs_group *group; /* Writeback group being written back */
struct bvecq *spare; /* Advance allocation of bvecq */
struct bvecq_pos load_cursor; /* Point at which new folios are loaded in */
- struct bvecq_pos dispatch_cursor; /* Point from which buffers are dispatched */
struct bvecq_pos collect_cursor; /* Clear-up point of I/O buffer */
+ struct bvecq_pos retry_cursor; /* Point from which retries are dispatched */
wait_queue_head_t waitq; /* Processor waiter */
void *netfs_priv; /* Private data for the netfs */
void *netfs_priv2; /* Private data for the netfs */
- uoff_t last_end; /* End pos of last folio submitted */
uoff_t submitted; /* Amount submitted for I/O so far */
uoff_t len; /* Length of the request */
size_t transferred; /* Amount to be indicated as transferred */
@@ -305,6 +296,8 @@ struct netfs_io_request {
uoff_t cache_coll_to; /* Point the cache has collected to */
uoff_t cleaned_to; /* Position we've cleaned folios to */
uoff_t abandon_to; /* Position to abandon folios to */
+ uoff_t retry_start; /* Position to retry from */
+ size_t retry_buffered; /* Amount of data to retry */
const struct folio *no_unlock_folio; /* Don't unlock this folio after read */
gfp_t gfp; /* GFP flags to use */
unsigned int debug_id;
@@ -351,8 +344,7 @@ struct netfs_request_ops {
/* Read request handling */
void (*expand_readahead)(struct netfs_io_request *rreq);
- int (*prepare_read)(struct netfs_io_subrequest *subreq);
- void (*issue_read)(struct netfs_io_subrequest *subreq);
+ int (*issue_read)(struct netfs_io_subrequest *subreq);
bool (*is_still_valid)(struct netfs_io_request *rreq);
int (*check_write_begin)(struct file *file, uoff_t pos, unsigned len,
struct folio **foliop, void **_fsdata);
@@ -367,8 +359,7 @@ struct netfs_request_ops {
int (*estimate_write)(struct netfs_io_request *wreq,
struct netfs_io_stream *stream,
struct netfs_write_estimate *estimate);
- void (*prepare_write)(struct netfs_io_subrequest *subreq);
- void (*issue_write)(struct netfs_io_subrequest *subreq);
+ int (*issue_write)(struct netfs_io_subrequest *subreq);
void (*retry_request)(struct netfs_io_request *wreq, struct netfs_io_stream *stream);
void (*invalidate_cache)(struct netfs_io_request *wreq);
};
@@ -408,8 +399,11 @@ struct netfs_cache_ops {
struct netfs_io_stream *stream,
struct netfs_write_estimate *estimate);
+ /* Read data from the cache for a netfs subrequest. */
+ int (*issue_read)(struct netfs_io_subrequest *subreq);
+
/* Write data to the cache from a netfs subrequest. */
- void (*issue_write)(struct netfs_io_subrequest *subreq);
+ int (*issue_write)(struct netfs_io_subrequest *subreq);
/* Expand readahead request */
void (*expand_readahead)(struct netfs_cache_resources *cres,
@@ -417,30 +411,18 @@ struct netfs_cache_ops {
uoff_t *_len,
uoff_t i_size);
- /* Prepare a read operation, shortening it to a cached/uncached
- * boundary as appropriate.
- */
- int (*prepare_read)(struct netfs_io_subrequest *subreq);
-
- /* Prepare a write subrequest, working out if we're allowed to do it
- * and finding out the maximum amount of data to gather before
- * attempting to submit. If we're not permitted to do it, the
- * subrequest should be marked failed.
- */
- void (*prepare_write_subreq)(struct netfs_io_subrequest *subreq);
-
/* Prepare a write operation, working out what part of the write we can
* actually do.
*/
- int (*prepare_write)(struct netfs_cache_resources *cres,
- uoff_t *_start, size_t *_len, size_t upper_len,
- uoff_t i_size, bool no_space_allocated_yet);
+ int (*prepare_write_old)(struct netfs_cache_resources *cres,
+ uoff_t *_start, size_t *_len, size_t upper_len,
+ uoff_t i_size, bool no_space_allocated_yet);
/* Query the occupancy of the cache in a region, returning where the
* next chunk of data starts and how long it is.
*/
- int (*query_occupancy)(struct netfs_cache_resources *cres,
- struct fscache_occupancy *occ);
+ void (*query_occupancy)(struct netfs_cache_resources *cres,
+ struct fscache_occupancy *occ);
/* Collect the result of buffered writeback to the cache. This
* includes copying a read to the cache. block_type is one of:
@@ -477,7 +459,6 @@ int netfs_writeback_single(struct address_space *mapping,
struct iov_iter *iter, size_t len);
/* Address operations API */
-struct readahead_control;
void netfs_readahead(struct readahead_control *);
int netfs_read_folio(struct file *, struct folio *);
int netfs_write_begin(struct netfs_inode *, struct file *,
@@ -504,7 +485,8 @@ void netfs_put_subrequest(struct netfs_io_subrequest *subreq,
ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages,
struct bvecq **_bvecq_head,
iov_iter_extraction_t extraction_flags, gfp_t gfp);
-void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq);
+int netfs_prepare_read_buffer(struct netfs_io_subrequest *subreq, unsigned int max_segs);
+int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq, unsigned int max_segs);
void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error);
int netfs_start_io_read(struct inode *inode);
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 6ccf9de42858..dfe08492d1ad 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -49,6 +49,7 @@
#define netfs_rreq_traces \
EM(netfs_rreq_trace_all_queued, "ALL-Q ") \
EM(netfs_rreq_trace_assess, "ASSESS ") \
+ EM(netfs_rreq_trace_cache_align_error, "CA-ALN!") \
EM(netfs_rreq_trace_cache_cancelled, "CA-CNCL") \
EM(netfs_rreq_trace_cache_failed, "CA-FAIL") \
EM(netfs_rreq_trace_cache_fail_collect, "CA-F-CO") \
@@ -85,7 +86,8 @@
EM(netfs_rreq_trace_waited_quiesce, "DONE-QUIESCE") \
EM(netfs_rreq_trace_wake_ip, "WAKE-IP") \
EM(netfs_rreq_trace_wake_queue, "WAKE-Q ") \
- E_(netfs_rreq_trace_write_done, "WR-DONE")
+ EM(netfs_rreq_trace_write_done, "WR-DONE") \
+ E_(netfs_rreq_trace_zero_unread, "ZERO-UR")
#define netfs_sreq_sources \
EM(NETFS_SOURCE_UNKNOWN, "----") \
@@ -134,6 +136,7 @@
EM(netfs_sreq_trace_superfluous, "SPRFL") \
EM(netfs_sreq_trace_terminated, "TERM ") \
EM(netfs_sreq_trace_too_much, "!TOOM") \
+ EM(netfs_sreq_trace_too_many_retries, "!RETR") \
EM(netfs_sreq_trace_wait_for, "_WAIT") \
EM(netfs_sreq_trace_write, "WRITE") \
EM(netfs_sreq_trace_write_skip, "SKIP ") \
@@ -528,6 +531,31 @@ TRACE_EVENT(netfs_folio,
__print_symbolic(__entry->why, netfs_folio_traces))
);
+TRACE_EVENT(netfs_wback,
+ TP_PROTO(struct netfs_io_request *wreq, struct folio *folio, unsigned int notes),
+
+ TP_ARGS(wreq, folio, notes),
+
+ TP_STRUCT__entry(
+ __field(pgoff_t, index)
+ __field(unsigned int, wreq)
+ __field(unsigned int, nr)
+ __field(unsigned int, notes)
+ ),
+
+ TP_fast_assign(
+ __entry->wreq = wreq->debug_id;
+ __entry->notes = notes;
+ __entry->index = folio->index;
+ __entry->nr = folio_nr_pages(folio);
+ ),
+
+ TP_printk("R=%08x ix=%05lx-%05lx n=%02x",
+ __entry->wreq,
+ __entry->index, __entry->index + __entry->nr - 1,
+ __entry->notes)
+ );
+
TRACE_EVENT(netfs_write_iter,
TP_PROTO(const struct kiocb *iocb, const struct iov_iter *from),
diff --git a/net/9p/client.c b/net/9p/client.c
index ef64546c6d52..68d82f677a7a 100644
--- a/net/9p/client.c
+++ b/net/9p/client.c
@@ -1564,6 +1564,7 @@ void
p9_client_write_subreq(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *wreq = subreq->rreq;
+ struct iov_iter iter;
struct p9_fid *fid = wreq->netfs_priv;
struct p9_client *clnt = fid->clnt;
struct p9_req_t *req;
@@ -1574,14 +1575,17 @@ p9_client_write_subreq(struct netfs_io_subrequest *subreq)
p9_debug(P9_DEBUG_9P, ">>> TWRITE fid %d offset %llu len %d\n",
fid->fid, start, len);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
/* Don't bother zerocopy for small IO (< 1024) */
if (clnt->trans_mod->zc_request && len > 1024) {
- req = p9_client_zc_rpc(clnt, P9_TWRITE, NULL, &subreq->io_iter,
+ req = p9_client_zc_rpc(clnt, P9_TWRITE, NULL, &iter,
0, wreq->len, P9_ZC_HDR_SZ, "dqd",
fid->fid, start, len);
} else {
req = p9_client_rpc(clnt, P9_TWRITE, "dqV", fid->fid,
- start, len, &subreq->io_iter);
+ start, len, &iter);
}
if (IS_ERR(req)) {
netfs_write_subrequest_terminated(subreq, PTR_ERR(req));
next prev parent reply other threads:[~2026-08-24 14:46 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-24 14:40 [PATCH v10 00/35] netfs: Keep track of folios in a segmented bio_vec[] chain David Howells
2026-08-24 14:40 ` [PATCH v10 01/35] netfs: Fix uninitialized return value in netfs_unbuffered_write() David Howells
2026-08-24 14:40 ` [PATCH v10 02/35] netfs: Fix read progress reporting David Howells
2026-08-24 14:40 ` [PATCH v10 03/35] cachefiles,netfs: sunset ondemand mode David Howells
2026-08-24 14:40 ` [PATCH v10 04/35] cachefiles: Fix potential UAF/KASAN warning David Howells
2026-08-24 14:40 ` [PATCH v10 05/35] netfs: Use uoff_t instead of unsigned long long and loff_t David Howells
2026-08-24 14:40 ` [PATCH v10 06/35] mm: Make readahead store folio count in readahead_control David Howells
2026-08-24 14:41 ` [PATCH v10 07/35] netfs: Bulk load the readahead-provided folios up front David Howells
2026-08-24 14:41 ` [PATCH v10 08/35] Add a function to kmap one page of a multipage bio_vec David Howells
2026-08-24 14:41 ` [PATCH v10 09/35] iov_iter: Make iov_iter_get_pages*() wrap iov_iter_extract_pages() David Howells
2026-08-24 14:41 ` [PATCH v10 10/35] iov_iter: Add a segmented queue of bio_vec[] David Howells
2026-08-24 14:41 ` [PATCH v10 11/35] netfs: Add some tools for managing bvecq chains David Howells
2026-08-24 14:41 ` [PATCH v10 12/35] netfs: Make mempool available for bvecq David Howells
2026-08-24 14:41 ` [PATCH v10 13/35] netfs: Add a function to extract from an iter into a bvecq David Howells
2026-08-24 14:41 ` [PATCH v10 14/35] afs: Use a bvecq to hold dir content rather than folioq David Howells
2026-08-24 14:41 ` [PATCH v10 15/35] cifs: Use a bvecq for buffering instead of a folioq David Howells
2026-08-24 14:41 ` [PATCH v10 16/35] smbdirect: Support ITER_BVECQ in smbdirect_map_sges_from_iter() David Howells
2026-08-24 14:41 ` [PATCH v10 17/35] netfs: Remove the writethrough code David Howells
2026-08-24 14:41 ` [PATCH v10 18/35] netfs: trace: Change the "clear" folio traces to "endwb" David Howells
2026-08-24 14:41 ` [PATCH v10 19/35] netfs: trace: Rejig a couple of the tracepoints David Howells
2026-08-24 14:41 ` [PATCH v10 20/35] netfs: Add some functions to wrap the all-queued handling David Howells
2026-08-24 14:41 ` [PATCH v10 21/35] netfs: Make deprecated PG_private_2 support optional David Howells
2026-08-24 14:41 ` [PATCH v10 22/35] cachefiles: Don't rely on backing fs storage map for most use cases David Howells
2026-08-24 14:41 ` [PATCH v10 23/35] netfs: Add the cache object ID to netfs_read/write tracepoints David Howells
2026-08-24 14:41 ` [PATCH v10 24/35] netfs: Switch to using bvecq rather than folio_queue and rolling_buffer David Howells
2026-08-24 14:41 ` [PATCH v10 25/35] smbdirect: Remove support for ITER_FOLIOQ from smbdirect_map_sges_from_iter() David Howells
2026-08-24 14:41 ` [PATCH v10 26/35] netfs: Remove netfs_alloc/free_folioq_buffer() David Howells
2026-08-24 14:41 ` [PATCH v10 27/35] netfs: Remove netfs_extract_user_iter() David Howells
2026-08-24 14:41 ` [PATCH v10 28/35] iov_iter: Remove ITER_FOLIOQ David Howells
2026-08-24 14:41 ` [PATCH v10 29/35] netfs: Remove folio_queue and rolling_buffer David Howells
2026-08-24 14:41 ` [PATCH v10 30/35] netfs: Simplify read abandonment David Howells
2026-08-24 15:39 ` READ_PLUS in NetFS? " Aurélien Couderc
2026-08-24 19:47 ` David Howells
2026-08-24 14:41 ` [PATCH v10 31/35] netfs: Check for too much data being read David Howells
2026-08-24 14:41 ` [PATCH v10 32/35] netfs: Add a method to get an estimate of the amount that can be written David Howells
2026-08-24 14:41 ` [PATCH v10 33/35] netfs: Rework writeback to use a separate list of regions to be unlocked David Howells
2026-08-24 14:41 ` David Howells [this message]
2026-08-24 14:41 ` [PATCH v10 35/35] netfs: Clean up now-unused code David Howells
2026-08-24 15:01 ` [PATCH v10 36/35] cachefiles: Preset the state xattr when creating a new file David Howells
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260824144130.759997-35-dhowells@redhat.com \
--to=dhowells@redhat.com \
--cc=asmadeus@codewreck.org \
--cc=axboe@kernel.dk \
--cc=ceph-devel@vger.kernel.org \
--cc=chenxiaosong@chenxiaosong.com \
--cc=christian@brauner.io \
--cc=ericvh@kernel.org \
--cc=hch@infradead.org \
--cc=idryomov@gmail.com \
--cc=leon@kernel.org \
--cc=linkinjeon@kernel.org \
--cc=linux-afs@lists.infradead.org \
--cc=linux-cifs@vger.kernel.org \
--cc=linux-erofs@lists.ozlabs.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=marc.dionne@auristor.com \
--cc=metze@samba.org \
--cc=netfs@lists.linux.dev \
--cc=pc@manguebit.org \
--cc=v9fs@lists.linux.dev \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®