[PATCH 0/4] splice: remove pipe buffer stealing

[PATCH 0/4] splice: remove pipe buffer stealing

[Kiryl Shutsemau (Meta)]

->try_steal() lets the consumer of a pipe buffer take its page instead
of copying the data.  Two consumers use it:

  - fuse, when a server splices a read reply into /dev/fuse with
    SPLICE_F_MOVE: the page goes into the fuse file's page cache;
  - virtio_console, which passes the page to the host.

The page can come from the page cache of another file, from a write()
into the pipe, or from vmsplice(SPLICE_F_GIFT).  Stealing it takes page
cache folios out of their files, and turns pages into page cache folios
and back.  That gets in the way of reading the page cache without folio
references [1] and of telling folios apart with a PG_folio flag.  It
works only on pages that nothing else holds, for fuse only on clean,
unmapped, order-0 ones, and it has needed fixes again and again:

  32f98877c57b ("fuse: don't check refcount after stealing page")
  b89ecd60d38e ("fuse: ignore PG_workingset after stealing")
  712a951025c0 ("fuse: fix page stealing")
  59ba47b6be9c ("fuse: Check for large folio with SPLICE_F_MOVE")
  15c20483d915 ("fuse: validate folio state before pipe_buf_try_steal() in fuse_try_move_folio()")

Linus and David suggested removing it [2].  Both consumers already copy
when a steal fails; with this series they always do.  vmsplice() and
splice() still accept SPLICE_F_GIFT and SPLICE_F_MOVE, and fuse still
advertises FUSE_SPLICE_MOVE, so that servers which ask for it mount as
before.

Miklos, Amit: do you know of users of the fuse move, or of virtio_console
splice, that depend on the copy being avoided?

Without ->try_steal(), nosteal_pipe_buf_ops is the same as
default_pipe_buf_ops, and the last patch keeps only the latter.

Tested on x86_64 with a FUSE server that answers reads by splicing the
page cache pages of a backing file into /dev/fuse with SPLICE_F_MOVE.  On
v7.4-rc1 how much moves depends on folio sizes: with order-0 folios on
both sides, one read of a 16M fuse file took 3968 of the backing file's
4096 pages out of its page cache.  With the series nothing moves, and the
data read through fuse is the same, also when the server reads its
requests with splice().  A virtio-serial port fed by splice() from a pipe
and from a file, and vmsplice(SPLICE_F_GIFT), SPLICE_F_MOVE, tee() and
AF_UNIX socket splice checks also give the same data before and after.

[1] https://lore.kernel.org/all/20261104130118.2507513-1-kas@kernel.org/
[2] https://lore.kernel.org/all/CAHk-=whhp=Q8qR+wPoM9+mxYPQ_OXMOEoKmT7J0JZikFZusuqw@mail.gmail.com/

Kiryl Shutsemau (Meta) (4):
  virtio_console: copy pipe buffers in splice_write()
  fuse: don't move pipe buffers into the page cache
  splice: remove pipe buffer stealing
  splice: replace nosteal_pipe_buf_ops with default_pipe_buf_ops

 drivers/char/virtio_console.c |  39 ++++-----
 fs/fuse/args.h                |   1 -
 fs/fuse/dev.c                 | 146 +---------------------------------
 fs/fuse/file.c                |   1 -
 fs/fuse/fuse_dev_i.h          |   1 -
 fs/pipe.c                     |  43 ----------
 fs/splice.c                   | 104 ++----------------------
 include/linux/pipe_fs_i.h     |  38 ---------
 kernel/watch_queue.c          |   4 -
 mm/shmem.c                    |   7 --
 net/core/skbuff.c             |   2 +-
 11 files changed, 26 insertions(+), 360 deletions(-)


base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.54.0

[PATCH 1/4] virtio_console: copy pipe buffers in splice_write()

[Kiryl Shutsemau (Meta)]

pipe_to_sg() tries to steal the page of each pipe buffer and pass it to
the host as it is.  When the steal fails, it copies the data into a page
of its own.

Pages written into the pipe with write(), gifted user pages and page
cache pages can be stolen.  The ftrace ring buffer pages that
virtio-trace splices cannot, so that in-tree user always gets the copy.

Always copy.  The data that reaches the host does not change.

Preparation for removing ->try_steal() from pipe buffers.

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 drivers/char/virtio_console.c | 39 +++++++++++++----------------------
 1 file changed, 14 insertions(+), 25 deletions(-)

diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index 019bcae81af5..b1b1aaf43e07 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -891,45 +891,34 @@ static int pipe_to_sg(struct pipe_inode_info *pipe, struct pipe_buffer *buf,
 {
 	struct sg_list *sgl = sd->u.data;
 	unsigned int offset, len;
+	struct page *page;
+	char *src;
 
 	if (sgl->n == sgl->size)
 		return 0;
 
-	/* Try lock this page */
-	if (pipe_buf_try_steal(pipe, buf)) {
-		/* Get reference and unlock page for moving */
-		get_page(buf->page);
-		unlock_page(buf->page);
+	page = alloc_page(GFP_KERNEL);
+	if (!page)
+		return -ENOMEM;
 
-		len = min(buf->len, sd->len);
-		sg_set_page(&(sgl->sg[sgl->n]), buf->page, len, buf->offset);
-	} else {
-		/* Failback to copying a page */
-		struct page *page = alloc_page(GFP_KERNEL);
-		char *src;
+	offset = sd->pos & ~PAGE_MASK;
 
-		if (!page)
-			return -ENOMEM;
+	len = sd->len;
+	if (len + offset > PAGE_SIZE)
+		len = PAGE_SIZE - offset;
 
-		offset = sd->pos & ~PAGE_MASK;
+	src = kmap_local_page(buf->page);
+	memcpy(page_address(page) + offset, src + buf->offset, len);
+	kunmap_local(src);
 
-		len = sd->len;
-		if (len + offset > PAGE_SIZE)
-			len = PAGE_SIZE - offset;
-
-		src = kmap_local_page(buf->page);
-		memcpy(page_address(page) + offset, src + buf->offset, len);
-		kunmap_local(src);
-
-		sg_set_page(&(sgl->sg[sgl->n]), page, len, offset);
-	}
+	sg_set_page(&sgl->sg[sgl->n], page, len, offset);
 	sgl->n++;
 	sgl->len += len;
 
 	return len;
 }
 
-/* Faster zero-copy write by splicing */
+/* Write by splicing, copying the data from the pipe */
 static ssize_t port_fops_splice_write(struct pipe_inode_info *pipe,
 				      struct file *filp, loff_t *ppos,
 				      size_t len, unsigned int flags)
-- 
2.54.0

[PATCH 2/4] fuse: don't move pipe buffers into the page cache

[Kiryl Shutsemau (Meta)]

A splice() into /dev/fuse with SPLICE_F_MOVE lets the reply to a read
request steal full pages from the pipe.  Each stolen page replaces, in
the fuse file's page cache, the folio that the request allocated.  The
page can be a page cache folio of another file, a page written into the
pipe, or a gifted user page.

Always copy.  The server sees the same data, and the move was a hint
that splice() never promised to honour.

Keep advertising FUSE_SPLICE_MOVE.  libfuse fails the mount of a
filesystem that asks for a capability the kernel does not offer.

Preparation for removing ->try_steal() from pipe buffers.

Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 fs/fuse/args.h       |   1 -
 fs/fuse/dev.c        | 144 +------------------------------------------
 fs/fuse/file.c       |   1 -
 fs/fuse/fuse_dev_i.h |   1 -
 4 files changed, 3 insertions(+), 144 deletions(-)

diff --git a/fs/fuse/args.h b/fs/fuse/args.h
index 5173264a1261..36ef0c6d5999 100644
--- a/fs/fuse/args.h
+++ b/fs/fuse/args.h
@@ -36,7 +36,6 @@ struct fuse_args {
 	bool user_pages:1;
 	bool out_argvar:1;
 	bool page_zeroing:1;
-	bool page_replace:1;
 	bool may_block:1;
 	bool is_ext:1;
 	bool is_pinned:1;
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 3641abb90d6b..e7238eed21fb 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -1099,128 +1099,6 @@ static int fuse_copy_do(struct fuse_copy_state *cs, void **val, unsigned *size)
 	return ncpy;
 }
 
-static int fuse_check_folio(struct folio *folio)
-{
-	if (folio_mapped(folio) ||
-	    folio->mapping != NULL ||
-	    (folio->flags.f & PAGE_FLAGS_CHECK_AT_PREP &
-	     ~(1 << PG_locked |
-	       1 << PG_referenced |
-	       1 << PG_lru |
-	       1 << PG_active |
-	       1 << PG_workingset |
-	       1 << PG_reclaim |
-	       1 << PG_waiters |
-	       LRU_GEN_MASK | LRU_REFS_MASK))) {
-		dump_page(&folio->page, "fuse: trying to steal weird page");
-		return 1;
-	}
-	return 0;
-}
-
-/*
- * Attempt to steal a page from the splice() pipe and move it into the
- * pagecache. If successful, the pointer in @pagep will be updated. The
- * folio that was originally in @pagep will lose a reference and the new
- * folio returned in @pagep will carry a reference.
- */
-static int fuse_try_move_folio(struct fuse_copy_state *cs, struct folio **foliop)
-{
-	int err;
-	struct folio *oldfolio = *foliop;
-	struct folio *newfolio;
-	struct pipe_buffer *buf = cs->pipebufs;
-
-	folio_get(oldfolio);
-	err = unlock_request(cs->req);
-	if (err)
-		goto out_put_old;
-
-	fuse_copy_finish(cs);
-
-	err = pipe_buf_confirm(cs->pipe, buf);
-	if (err)
-		goto out_put_old;
-
-	BUG_ON(!cs->nr_segs);
-	cs->currbuf = buf;
-	cs->len = buf->len;
-	cs->pipebufs++;
-	cs->nr_segs--;
-
-	if (cs->len != folio_size(oldfolio))
-		goto out_fallback;
-
-	newfolio = page_folio(buf->page);
-
-	if (folio_test_large(newfolio))
-		goto out_fallback;
-
-	/*
-	 * This is a new and locked page, it shouldn't be mapped or
-	 * have any special flags on it
-	 */
-	if (WARN_ON(folio_mapped(oldfolio)))
-		goto out_fallback;
-	if (WARN_ON(folio_has_private(oldfolio)))
-		goto out_fallback;
-	if (WARN_ON(folio_test_dirty(oldfolio) ||
-				folio_test_writeback(oldfolio)))
-		goto out_fallback;
-	if (WARN_ON(folio_test_mlocked(oldfolio)))
-		goto out_fallback;
-
-	if (!pipe_buf_try_steal(cs->pipe, buf))
-		goto out_fallback;
-
-	folio_clear_uptodate(newfolio);
-	folio_clear_mappedtodisk(newfolio);
-
-	if (fuse_check_folio(newfolio) != 0)
-		goto out_fallback_unlock;
-
-	err = lock_request(cs->req);
-	if (err)
-		goto out_fallback_unlock;
-
-	replace_page_cache_folio(oldfolio, newfolio);
-
-	folio_get(newfolio);
-
-	if (!(buf->flags & PIPE_BUF_FLAG_LRU))
-		folio_add_lru(newfolio);
-
-	/*
-	 * Release while we have extra ref on stolen page.  Otherwise
-	 * anon_pipe_buf_release() might think the page can be reused.
-	 */
-	pipe_buf_release(cs->pipe, buf);
-
-	*foliop = newfolio;
-	folio_unlock(oldfolio);
-	/* Drop ref for ap->pages[] array */
-	folio_put(oldfolio);
-	cs->len = 0;
-
-	err = 0;
-out_put_old:
-	/* Drop ref obtained in this function */
-	folio_put(oldfolio);
-	return err;
-
-out_fallback_unlock:
-	folio_unlock(newfolio);
-out_fallback:
-	cs->pg = buf->page;
-	cs->offset = buf->offset;
-
-	err = lock_request(cs->req);
-	if (!err)
-		err = 1;
-
-	goto out_put_old;
-}
-
 static int fuse_ref_folio(struct fuse_copy_state *cs, struct folio *folio,
 			  unsigned offset, unsigned count)
 {
@@ -1275,16 +1153,9 @@ int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
 				return fuse_ref_folio(cs, folio, offset, count);
 			}
 		} else if (!cs->len) {
-			if (cs->move_folios && folio &&
-			    offset == 0 && count == folio_size(folio)) {
-				err = fuse_try_move_folio(cs, foliop);
-				if (err <= 0)
-					return err;
-			} else {
-				err = fuse_copy_fill(cs);
-				if (err)
-					return err;
-			}
+			err = fuse_copy_fill(cs);
+			if (err)
+				return err;
 		}
 		if (folio) {
 			void *mapaddr = kmap_local_folio(folio, offset);
@@ -1920,9 +1791,6 @@ static ssize_t fuse_dev_do_write(struct fuse_dev *fud,
 		if (!smp_load_acquire(&fch->initialized) || !fch->connected)
 			goto copy_finish;
 
-		/* Don't try to move folios (yet) */
-		cs->move_folios = false;
-
 		err = fuse_notify(fch->conn, oh.error, nbytes - sizeof(oh), cs);
 		goto copy_finish;
 	}
@@ -1966,8 +1834,6 @@ static ssize_t fuse_dev_do_write(struct fuse_dev *fud,
 	set_bit(FR_LOCKED, &req->flags);
 	spin_unlock(&fpq->lock);
 	cs->req = req;
-	if (!req->args->page_replace)
-		cs->move_folios = false;
 
 	if (oh.error)
 		err = nbytes != sizeof(oh) ? -EINVAL : 0;
@@ -2068,7 +1934,6 @@ static ssize_t fuse_dev_splice_write(struct pipe_inode_info *pipe,
 				goto out_free;
 
 			*obuf = *ibuf;
-			obuf->flags &= ~PIPE_BUF_FLAG_GIFT;
 			obuf->len = rem;
 			ibuf->offset += obuf->len;
 			ibuf->len -= obuf->len;
@@ -2083,9 +1948,6 @@ static ssize_t fuse_dev_splice_write(struct pipe_inode_info *pipe,
 	cs.nr_segs = nbuf;
 	cs.pipe = pipe;
 
-	if (flags & SPLICE_F_MOVE)
-		cs.move_folios = true;
-
 	ret = fuse_dev_do_write(fud, &cs, len);
 
 	pipe_lock(pipe);
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index d5e1cb53d86a..ab955e4c3e5d 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -1096,7 +1096,6 @@ static void fuse_send_readpages(struct fuse_io_args *ia, struct file *file,
 
 	ap->args.out_pages = true;
 	ap->args.page_zeroing = true;
-	ap->args.page_replace = true;
 
 	/*
 	 * Don't overflow end offset.
diff --git a/fs/fuse/fuse_dev_i.h b/fs/fuse/fuse_dev_i.h
index 4b412a76225f..a90ce26ded31 100644
--- a/fs/fuse/fuse_dev_i.h
+++ b/fs/fuse/fuse_dev_i.h
@@ -326,7 +326,6 @@ struct fuse_copy_state {
 	unsigned int len;
 	unsigned int offset;
 	bool write:1;
-	bool move_folios:1;
 	bool is_uring:1;
 	/* set when the payload is zero-copied. folios are filled in place */
 	bool skip_folio_copy:1;
-- 
2.54.0

[PATCH 3/4] splice: remove pipe buffer stealing

[Kiryl Shutsemau (Meta)]

Nothing calls pipe_buf_try_steal() any more.  Remove ->try_steal() from
struct pipe_buf_operations, its implementations and the exported
generic_pipe_buf_try_steal().

PIPE_BUF_FLAG_LRU and PIPE_BUF_FLAG_GIFT were set only for stealing, and
go too.  page_cache_pipe_buf_release() did nothing but drop the page and
the LRU flag, so page cache and user page buffers use
generic_pipe_buf_release().

vmsplice() still accepts SPLICE_F_GIFT, and splice() SPLICE_F_MOVE.
Both were hints, and nothing acts on them now.

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 fs/pipe.c                 | 43 ------------------
 fs/splice.c               | 96 +++------------------------------------
 include/linux/pipe_fs_i.h | 36 ---------------
 kernel/watch_queue.c      |  4 --
 mm/shmem.c                |  7 ---
 5 files changed, 6 insertions(+), 180 deletions(-)

diff --git a/fs/pipe.c b/fs/pipe.c
index 4fa69192e6fb..4c1122597a3c 100644
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -222,48 +222,6 @@ static void anon_pipe_buf_release(struct pipe_inode_info *pipe,
 	anon_pipe_put_page(pipe, page);
 }
 
-static bool anon_pipe_buf_try_steal(struct pipe_inode_info *pipe,
-		struct pipe_buffer *buf)
-{
-	struct page *page = buf->page;
-
-	if (page_count(page) != 1)
-		return false;
-	memcg_kmem_uncharge_page(page, 0);
-	__SetPageLocked(page);
-	return true;
-}
-
-/**
- * generic_pipe_buf_try_steal - attempt to take ownership of a &pipe_buffer
- * @pipe:	the pipe that the buffer belongs to
- * @buf:	the buffer to attempt to steal
- *
- * Description:
- *	This function attempts to steal the &struct page attached to
- *	@buf. If successful, this function returns 0 and returns with
- *	the page locked. The caller may then reuse the page for whatever
- *	he wishes; the typical use is insertion into a different file
- *	page cache.
- */
-bool generic_pipe_buf_try_steal(struct pipe_inode_info *pipe,
-		struct pipe_buffer *buf)
-{
-	struct page *page = buf->page;
-
-	/*
-	 * A reference of one is golden, that means that the owner of this
-	 * page is the only one holding a reference to it. lock the page
-	 * and return OK.
-	 */
-	if (page_count(page) == 1) {
-		lock_page(page);
-		return true;
-	}
-	return false;
-}
-EXPORT_SYMBOL(generic_pipe_buf_try_steal);
-
 /**
  * generic_pipe_buf_get - get a reference to a &struct pipe_buffer
  * @pipe:	the pipe that the buffer belongs to
@@ -297,7 +255,6 @@ EXPORT_SYMBOL(generic_pipe_buf_release);
 
 static const struct pipe_buf_operations anon_pipe_buf_ops = {
 	.release	= anon_pipe_buf_release,
-	.try_steal	= anon_pipe_buf_try_steal,
 	.get		= generic_pipe_buf_get,
 };
 
diff --git a/fs/splice.c b/fs/splice.c
index e6d43329ed3f..ba8f6f9aa3f0 100644
--- a/fs/splice.c
+++ b/fs/splice.c
@@ -55,63 +55,6 @@ static noinline void pipe_clear_nowait(struct file *file)
 	} while (!try_cmpxchg(&file->f_mode, &fmode, fmode & ~FMODE_NOWAIT));
 }
 
-/*
- * Attempt to steal a page from a pipe buffer. This should perhaps go into
- * a vm helper function, it's already simplified quite a bit by the
- * addition of remove_mapping(). If success is returned, the caller may
- * attempt to reuse this page for another destination.
- */
-static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info *pipe,
-		struct pipe_buffer *buf)
-{
-	struct folio *folio = page_folio(buf->page);
-	struct address_space *mapping;
-
-	folio_lock(folio);
-
-	mapping = folio_mapping(folio);
-	if (mapping) {
-		WARN_ON(!folio_test_uptodate(folio));
-
-		/*
-		 * At least for ext2 with nobh option, we need to wait on
-		 * writeback completing on this folio, since we'll remove it
-		 * from the pagecache.  Otherwise truncate wont wait on the
-		 * folio, allowing the disk blocks to be reused by someone else
-		 * before we actually wrote our data to them. fs corruption
-		 * ensues.
-		 */
-		folio_wait_writeback(folio);
-
-		if (!filemap_release_folio(folio, GFP_KERNEL))
-			goto out_unlock;
-
-		/*
-		 * If we succeeded in removing the mapping, set LRU flag
-		 * and return good.
-		 */
-		if (remove_mapping(mapping, folio)) {
-			buf->flags |= PIPE_BUF_FLAG_LRU;
-			return true;
-		}
-	}
-
-	/*
-	 * Raced with truncate or failed to remove folio from current
-	 * address space, unlock and return failure.
-	 */
-out_unlock:
-	folio_unlock(folio);
-	return false;
-}
-
-static void page_cache_pipe_buf_release(struct pipe_inode_info *pipe,
-					struct pipe_buffer *buf)
-{
-	put_page(buf->page);
-	buf->flags &= ~PIPE_BUF_FLAG_LRU;
-}
-
 /*
  * Check whether the contents of buf is OK to access. Since the content
  * is a page cache page, IO may be in flight.
@@ -154,24 +97,12 @@ static int page_cache_pipe_buf_confirm(struct pipe_inode_info *pipe,
 
 const struct pipe_buf_operations page_cache_pipe_buf_ops = {
 	.confirm	= page_cache_pipe_buf_confirm,
-	.release	= page_cache_pipe_buf_release,
-	.try_steal	= page_cache_pipe_buf_try_steal,
+	.release	= generic_pipe_buf_release,
 	.get		= generic_pipe_buf_get,
 };
 
-static bool user_page_pipe_buf_try_steal(struct pipe_inode_info *pipe,
-		struct pipe_buffer *buf)
-{
-	if (!(buf->flags & PIPE_BUF_FLAG_GIFT))
-		return false;
-
-	buf->flags |= PIPE_BUF_FLAG_LRU;
-	return generic_pipe_buf_try_steal(pipe, buf);
-}
-
 static const struct pipe_buf_operations user_page_pipe_buf_ops = {
-	.release	= page_cache_pipe_buf_release,
-	.try_steal	= user_page_pipe_buf_try_steal,
+	.release	= generic_pipe_buf_release,
 	.get		= generic_pipe_buf_get,
 };
 
@@ -400,7 +331,6 @@ EXPORT_SYMBOL(copy_splice_read);
 
 const struct pipe_buf_operations default_pipe_buf_ops = {
 	.release	= generic_pipe_buf_release,
-	.try_steal	= generic_pipe_buf_try_steal,
 	.get		= generic_pipe_buf_get,
 };
 
@@ -1469,12 +1399,10 @@ static ssize_t __do_splice(struct file *in, loff_t __user *off_in,
 }
 
 static ssize_t iter_to_pipe(struct iov_iter *from,
-			    struct pipe_inode_info *pipe,
-			    unsigned int flags)
+			    struct pipe_inode_info *pipe)
 {
 	struct pipe_buffer buf = {
 		.ops = &user_page_pipe_buf_ops,
-		.flags = flags
 	};
 	size_t total = 0;
 	ssize_t ret = 0;
@@ -1564,10 +1492,6 @@ static ssize_t vmsplice_to_pipe(struct file *file, struct iov_iter *iter,
 {
 	struct pipe_inode_info *pipe;
 	ssize_t ret = 0;
-	unsigned buf_flag = 0;
-
-	if (flags & SPLICE_F_GIFT)
-		buf_flag = PIPE_BUF_FLAG_GIFT;
 
 	pipe = get_pipe_info(file, true);
 	if (!pipe)
@@ -1578,7 +1502,7 @@ static ssize_t vmsplice_to_pipe(struct file *file, struct iov_iter *iter,
 	pipe_lock(pipe);
 	ret = wait_for_space(pipe, flags);
 	if (!ret)
-		ret = iter_to_pipe(iter, pipe, buf_flag);
+		ret = iter_to_pipe(iter, pipe);
 	pipe_unlock(pipe);
 	if (ret > 0) {
 		wakeup_pipe_readers(pipe);
@@ -1839,11 +1763,7 @@ static int splice_pipe_to_pipe(struct pipe_inode_info *ipipe,
 			}
 			*obuf = *ibuf;
 
-			/*
-			 * Don't inherit the gift and merge flags, we need to
-			 * prevent multiple steals of this page.
-			 */
-			obuf->flags &= ~PIPE_BUF_FLAG_GIFT;
+			/* The page is shared with ipipe: don't append to it */
 			obuf->flags &= ~PIPE_BUF_FLAG_CAN_MERGE;
 
 			obuf->len = len;
@@ -1956,11 +1876,7 @@ static ssize_t link_pipe(struct pipe_inode_info *ipipe,
 
 		*obuf = *ibuf;
 
-		/*
-		 * Don't inherit the gift and merge flag, we need to prevent
-		 * multiple steals of this page.
-		 */
-		obuf->flags &= ~PIPE_BUF_FLAG_GIFT;
+		/* The page is shared with ipipe: don't append to it */
 		obuf->flags &= ~PIPE_BUF_FLAG_CAN_MERGE;
 
 		if (obuf->len > len)
diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h
index 6402930282e5..fb6570e80d01 100644
--- a/include/linux/pipe_fs_i.h
+++ b/include/linux/pipe_fs_i.h
@@ -4,9 +4,7 @@
 
 #define PIPE_DEF_BUFFERS	16
 
-#define PIPE_BUF_FLAG_LRU	0x01	/* page is on the LRU */
 #define PIPE_BUF_FLAG_ATOMIC	0x02	/* was atomically mapped */
-#define PIPE_BUF_FLAG_GIFT	0x04	/* page is a gift */
 #define PIPE_BUF_FLAG_PACKET	0x08	/* read() as a packet */
 #define PIPE_BUF_FLAG_CAN_MERGE	0x10	/* can merge buffers */
 #define PIPE_BUF_FLAG_WHOLE	0x20	/* read() must return entire buffer or error */
@@ -127,16 +125,6 @@ struct pipe_inode_info {
 #endif
 };
 
-/*
- * Note on the nesting of these functions:
- *
- * ->confirm()
- *	->try_steal()
- *
- * That is, ->try_steal() must be called on a confirmed buffer.  See below for
- * the meaning of each operation.  Also see the kerneldoc in fs/pipe.c for the
- * pipe and generic variants of these hooks.
- */
 struct pipe_buf_operations {
 	/*
 	 * ->confirm() verifies that the data in the pipe buffer is there
@@ -153,16 +141,6 @@ struct pipe_buf_operations {
 	 */
 	void (*release)(struct pipe_inode_info *, struct pipe_buffer *);
 
-	/*
-	 * Attempt to take ownership of the pipe buffer and its contents.
-	 * ->try_steal() returns %true for success, in which case the contents
-	 * of the pipe (the buf->page) is locked and now completely owned by the
-	 * caller. The page may then be transferred to a different mapping, the
-	 * most often used case is insertion into different file address space
-	 * cache.
-	 */
-	bool (*try_steal)(struct pipe_inode_info *, struct pipe_buffer *);
-
 	/*
 	 * Get a reference to the pipe buffer.
 	 */
@@ -304,19 +282,6 @@ static inline int pipe_buf_confirm(struct pipe_inode_info *pipe,
 	return buf->ops->confirm(pipe, buf);
 }
 
-/**
- * pipe_buf_try_steal - attempt to take ownership of a pipe_buffer
- * @pipe:	the pipe that the buffer belongs to
- * @buf:	the buffer to attempt to steal
- */
-static inline bool pipe_buf_try_steal(struct pipe_inode_info *pipe,
-		struct pipe_buffer *buf)
-{
-	if (!buf->ops->try_steal)
-		return false;
-	return buf->ops->try_steal(pipe, buf);
-}
-
 /* Differs from PIPE_BUF in that PIPE_SIZE is the length of the actual
    memory allocation, whereas PIPE_BUF makes atomicity guarantees.  */
 #define PIPE_SIZE		PAGE_SIZE
@@ -335,7 +300,6 @@ void free_pipe_info(struct pipe_inode_info *);
 
 /* Generic pipe buffer ops functions */
 bool generic_pipe_buf_get(struct pipe_inode_info *, struct pipe_buffer *);
-bool generic_pipe_buf_try_steal(struct pipe_inode_info *, struct pipe_buffer *);
 void generic_pipe_buf_release(struct pipe_inode_info *, struct pipe_buffer *);
 
 extern const struct pipe_buf_operations nosteal_pipe_buf_ops;
diff --git a/kernel/watch_queue.c b/kernel/watch_queue.c
index 453dcf613205..e2bdd5fb0db1 100644
--- a/kernel/watch_queue.c
+++ b/kernel/watch_queue.c
@@ -77,13 +77,9 @@ static void watch_queue_pipe_buf_release(struct pipe_inode_info *pipe,
 	generic_pipe_buf_release(pipe, buf);
 }
 
-// No try_steal function => no stealing
-#define watch_queue_pipe_buf_try_steal NULL
-
 /* New data written to a pipe may be appended to a buffer with this type. */
 static const struct pipe_buf_operations watch_queue_pipe_buf_ops = {
 	.release	= watch_queue_pipe_buf_release,
-	.try_steal	= watch_queue_pipe_buf_try_steal,
 	.get		= generic_pipe_buf_get,
 };
 
diff --git a/mm/shmem.c b/mm/shmem.c
index af21a89abb0f..4a9440a1e1f8 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -3655,15 +3655,8 @@ static void zero_pipe_buf_release(struct pipe_inode_info *pipe,
 {
 }
 
-static bool zero_pipe_buf_try_steal(struct pipe_inode_info *pipe,
-				    struct pipe_buffer *buf)
-{
-	return false;
-}
-
 static const struct pipe_buf_operations zero_pipe_buf_ops = {
 	.release	= zero_pipe_buf_release,
-	.try_steal	= zero_pipe_buf_try_steal,
 	.get		= zero_pipe_buf_get,
 };
 
-- 
2.54.0

[PATCH 4/4] splice: replace nosteal_pipe_buf_ops with default_pipe_buf_ops

[Kiryl Shutsemau (Meta)]

Without ->try_steal(), nosteal_pipe_buf_ops and default_pipe_buf_ops are
the same: generic_pipe_buf_release() and generic_pipe_buf_get().

Keep default_pipe_buf_ops, and use it in skb_splice_bits() and
fuse_dev_splice_read().  Export it in place of nosteal_pipe_buf_ops,
since fuse can be built as a module.

Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 fs/fuse/dev.c             | 2 +-
 fs/splice.c               | 8 +-------
 include/linux/pipe_fs_i.h | 2 --
 net/core/skbuff.c         | 2 +-
 4 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index e7238eed21fb..a5c08a00cf83 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -1627,7 +1627,7 @@ static ssize_t fuse_dev_splice_read(struct file *in, loff_t *ppos,
 		 * Need to be careful about this.  Having buf->ops in module
 		 * code can Oops if the buffer persists after module unload.
 		 */
-		bufs[page_nr].ops = &nosteal_pipe_buf_ops;
+		bufs[page_nr].ops = &default_pipe_buf_ops;
 		bufs[page_nr].flags = 0;
 		ret = add_to_pipe(pipe, &bufs[page_nr++]);
 		if (unlikely(ret < 0))
diff --git a/fs/splice.c b/fs/splice.c
index ba8f6f9aa3f0..e9e0c6a7b23c 100644
--- a/fs/splice.c
+++ b/fs/splice.c
@@ -333,13 +333,7 @@ const struct pipe_buf_operations default_pipe_buf_ops = {
 	.release	= generic_pipe_buf_release,
 	.get		= generic_pipe_buf_get,
 };
-
-/* Pipe buffer operations for a socket and similar. */
-const struct pipe_buf_operations nosteal_pipe_buf_ops = {
-	.release	= generic_pipe_buf_release,
-	.get		= generic_pipe_buf_get,
-};
-EXPORT_SYMBOL(nosteal_pipe_buf_ops);
+EXPORT_SYMBOL(default_pipe_buf_ops);
 
 static void wakeup_pipe_writers(struct pipe_inode_info *pipe)
 {
diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h
index fb6570e80d01..95f6236c5c5b 100644
--- a/include/linux/pipe_fs_i.h
+++ b/include/linux/pipe_fs_i.h
@@ -302,8 +302,6 @@ void free_pipe_info(struct pipe_inode_info *);
 bool generic_pipe_buf_get(struct pipe_inode_info *, struct pipe_buffer *);
 void generic_pipe_buf_release(struct pipe_inode_info *, struct pipe_buffer *);
 
-extern const struct pipe_buf_operations nosteal_pipe_buf_ops;
-
 unsigned long account_pipe_buffers(struct user_struct *user,
 				   unsigned long old, unsigned long new);
 bool too_many_pipe_buffers_soft(unsigned long user_bufs);
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 48f676411724..e063241b2005 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -3281,7 +3281,7 @@ int skb_splice_bits(struct sk_buff *skb, struct sock *sk, unsigned int offset,
 		.pages = pages,
 		.partial = partial,
 		.nr_pages_max = MAX_SKB_FRAGS,
-		.ops = &nosteal_pipe_buf_ops,
+		.ops = &default_pipe_buf_ops,
 		.spd_release = sock_spd_release,
 	};
 	int ret = 0;
-- 
2.54.0

Re: [PATCH 0/4] splice: remove pipe buffer stealing

[Linus Torvalds]

On Tue, 10 Nov 2026 at 06:52, Kiryl Shutsemau (Meta) <kas@kernel.org> wrote:
>
>  11 files changed, 26 insertions(+), 360 deletions(-)

Ack. Lovely. I'm considering just taking this as-is now, rather than
delaying it. Comments?

               Linus

Re: [PATCH 1/4] virtio_console: copy pipe buffers in splice_write()

[Zi Yan]

On 10 Nov 2026, at 9:52, Kiryl Shutsemau (Meta) wrote:

> pipe_to_sg() tries to steal the page of each pipe buffer and pass it to
> the host as it is.  When the steal fails, it copies the data into a page
> of its own.
>
> Pages written into the pipe with write(), gifted user pages and page
> cache pages can be stolen.  The ftrace ring buffer pages that
> virtio-trace splices cannot, so that in-tree user always gets the copy.
>
> Always copy.  The data that reaches the host does not change.
>
> Preparation for removing ->try_steal() from pipe buffers.
>
> Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>  drivers/char/virtio_console.c | 39 +++++++++++++----------------------
>  1 file changed, 14 insertions(+), 25 deletions(-)
>
LGTM.

Acked-by: Zi Yan <ziy@nvidia.com>

Best Regards,
Yan, Zi

Re: [PATCH 2/4] fuse: don't move pipe buffers into the page cache

[Zi Yan]

On 10 Nov 2026, at 9:52, Kiryl Shutsemau (Meta) wrote:

> A splice() into /dev/fuse with SPLICE_F_MOVE lets the reply to a read
> request steal full pages from the pipe.  Each stolen page replaces, in
> the fuse file's page cache, the folio that the request allocated.  The
> page can be a page cache folio of another file, a page written into the
> pipe, or a gifted user page.
>
> Always copy.  The server sees the same data, and the move was a hint
> that splice() never promised to honour.
>
> Keep advertising FUSE_SPLICE_MOVE.  libfuse fails the mount of a
> filesystem that asks for a capability the kernel does not offer.
>
> Preparation for removing ->try_steal() from pipe buffers.
>
> Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
> Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>  fs/fuse/args.h       |   1 -
>  fs/fuse/dev.c        | 144 +------------------------------------------
>  fs/fuse/file.c       |   1 -
>  fs/fuse/fuse_dev_i.h |   1 -
>  4 files changed, 3 insertions(+), 144 deletions(-)
>
LGTM.

Acked-by: Zi Yan <ziy@nvidia.com>

Best Regards,
Yan, Zi

Re: [PATCH 1/4] virtio_console: copy pipe buffers in splice_write()

[Matthew Wilcox]

On Tue, Nov 10, 2026 at 02:52:36PM +0000, Kiryl Shutsemau (Meta) wrote:
> +	page = alloc_page(GFP_KERNEL);
> +	if (!page)
> +		return -ENOMEM;
>  
> +	offset = sd->pos & ~PAGE_MASK;
>  
> +	len = sd->len;
> +	if (len + offset > PAGE_SIZE)
> +		len = PAGE_SIZE - offset;
>  
> +	src = kmap_local_page(buf->page);
> +	memcpy(page_address(page) + offset, src + buf->offset, len);
> +	kunmap_local(src);

	memcpy_from_page(page_address(page) + offset, buf->page, buf->offset,
			len);

> +	sg_set_page(&sgl->sg[sgl->n], page, len, offset);
>  	sgl->n++;
>  	sgl->len += len;
>  
>  	return len;
>  }