[PATCH v2 0/5] btrfs: handle -ENOMEM errors in some synchronous dirops without aborting

[Jeff Layton <jlayton@kernel.org>]

We've had a (relatively small) number of ENOMEM btrfs aborts occur in
synchronous directory morphing codepaths. It's not terribly common, but
there are a few places where an memory allocation failure results in an
abort.

This patchset reworks the code to do the allocations up front, before the
point where we'd have to abort the fs if it fails.

This does not cover all potential cases where this can currently occur:

In particular, a rename that overwrites the target can still abort the
fs if a memory allocation fails. Fixing that is substantially more work,
unfortunately.

This also doesn't cover orphaning a new inode on failure (which can
trigger new memory allocations), so this series is designed to work in
conjunction with with Boris' GFP_NOFAIL series [1].

AFAICT, these are ancient problems, dating back at least to ~2011. I
didn't bother adding Fixes: tags.

AI disclosure: I made heavy use of an LLM in this patchset, from
drafting the initial series to helping test it.

[1] https://lore.kernel.org/linux-btrfs/cover.1784673567.git.boris@bur.io/

Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
Changes in v2:
- Use an on-stack btrfs_path in btrfs_insert_orphan_item() so the ENOMEM
  recovery does not itself fail on a path allocation.
- Simplify the recovery in btrfs_create_new_inode() to rely on
  btrfs_orphan_add()'s internal abort instead of aborting twice.
- Add ALLOW_ERROR_INJECTION() on btrfs_prealloc_delayed_dir_index() and a
  new fstest (btrfs/341) to exercise the ENOMEM path.
- Link to v1: https://lore.kernel.org/r/20260717-btrfs-enomem-v1-0-cdc9c0e265d0@kernel.org

---
Jeff Layton (5):
      btrfs: use an on-stack path in btrfs_insert_orphan_item()
      btrfs: split btrfs_insert_delayed_dir_index() into prealloc and commit phases
      btrfs: pre-allocate delayed dir index before btree modification
      btrfs: handle ENOMEM from btrfs_insert_dir_item() without aborting
      btrfs: pre-allocate delayed dir index for non-overwrite rename

 fs/btrfs/btrfs_inode.h   |  4 ++-
 fs/btrfs/delayed-inode.c | 90 +++++++++++++++++++++++++++++++++++-------------
 fs/btrfs/delayed-inode.h | 22 +++++++++---
 fs/btrfs/dir-item.c      | 30 ++++++++++++----
 fs/btrfs/dir-item.h      |  5 +--
 fs/btrfs/inode.c         | 57 ++++++++++++++++++++++++------
 fs/btrfs/orphan.c        | 11 +++---
 fs/btrfs/transaction.c   |  2 +-
 fs/btrfs/tree-log.c      |  4 +--
 9 files changed, 168 insertions(+), 57 deletions(-)
---
base-commit: 76a2100a0e8d81636586449024e7558656300481
change-id: 20260715-btrfs-enomem-988f2cc36ffd

Best regards,
-- 
Jeff Layton <jlayton@kernel.org>