[PATCH] ceph: fix leaked inode reference on writeback abort at umount

[matthew@bargrove.com]

From: Matthew Brown <matthew@bargrove.com>

When ceph_submit_write() fails to take an osd_stopping_blocker -- which
only happens once the filesystem has begun unmounting
(mdsc->stopping >= CEPH_MDSC_STOPPING_FLUSHING) -- it redirties and
unlocks the folios it had already pulled out of
ceph_process_folio_batch() via redirty_page_for_writepage(), instead of
submitting them for writeback.

Those folios have already been through folio_clear_dirty_for_io(), so
PG_dirty is clear, but they are still "claimed" by the original
ceph_dirty_folio() call: ci->i_wrbuffer_ref is still counting them,
folio->private still pins a ceph_snap_context, and on the 0 -> 1
i_wrbuffer_ref transition ceph_dirty_folio() took an ihold() on the
inode. None of that has been released, because the folio was never
submitted, so writepages_finish() -- the only place that would release
it -- never runs.

redirty_page_for_writepage() calls folio_mark_dirty(), which invokes
->dirty_folio() again as soon as it observes PG_dirty clear. That
re-enters ceph_dirty_folio() on a folio that is still claimed.
ceph_dirty_folio() cannot distinguish "still claimed, never submitted"
from "genuinely clean again": it increments i_wrbuffer_ref again and
re-attaches a second snap_context reference, but does not take a second
ihold(), because ihold() only fires on the 0 -> 1 transition and
i_wrbuffer_ref is already non-zero.

The result is two i_wrbuffer_ref increments backed by only one ihold().
When the folio is eventually written back successfully on a later,
unrelated writeback pass, writepages_finish() drops exactly one wrbuffer
ref for it, so i_wrbuffer_ref never returns to 0 and the iput() that
should balance the original ihold() never runs. The inode's i_count
stays elevated, evict_inodes() cannot free it, and
generic_shutdown_super() trips:

  VFS: Busy inodes after unmount of ceph (ceph)
  kernel BUG at fs/super.c:650!

(a WARNING rather than a BUG without CONFIG_BUG_ON_DATA_CORRUPTION; on
an affected production node with that config set it is a fatal panic and
a full node reboot).

Fix it by releasing the existing wrbuffer/snap_context claim before
redirtying, mirroring what ceph_invalidate_folio() and
write_folio_nounlock() already do on their own abort paths. This
restores the invariant that ceph_dirty_folio() only ever runs against a
folio that is not already claimed, so it takes exactly one ihold() per
0 -> 1 transition and one matching wrbuffer ref per claimed folio.

Only the locked_pages loop is changed here. The preceding fbatch loop is
deliberately left alone: those entries can include folios that are
legitimately still under a different, in-flight write, where releasing
the claim would be a premature release. The two structurally similar
redirty-after-clear-dirty-for-io sites in ceph_process_folio_batch()
appear to share the same defect class and are left for a follow-up to
keep this fix minimal.

Fixes: fd7449d937e7 ("ceph: fix generic/421 test failure")
Link: https://tracker.ceph.com/issues/79149
Signed-off-by: Matthew Brown <matthew@bargrove.com>
---
Reproduced and verified on a 6.18.40 kernel: forcing the
ceph_submit_write() osd_stopping_blocker-failure branch and unmounting a
CephFS kernel-client mount with dirty pages reliably produces the
"VFS: Busy inodes after unmount of ceph" / fs/super.c:650 splat on the
unpatched kernel, and produces no such splat with this patch applied
(identical kernel, identical trigger). The original failure was captured
in production off-box via the serial console (pstore could not persist
the panic).

 fs/ceph/addr.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index ecf33b6..97d9ef1 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -1426,6 +1426,16 @@ void ceph_shift_unused_folios_left(struct folio_batch *fbatch)
 	fbatch->nr = n;
 }
 
+static void ceph_undo_wrbuffer_claim(struct inode *inode, struct page *page)
+{
+	struct ceph_snap_context *snapc = detach_page_private(page);
+
+	if (!snapc)
+		return;
+	ceph_put_wrbuffer_cap_refs(ceph_inode(inode), 1, snapc);
+	ceph_put_snap_context(snapc);
+}
+
 static
 int ceph_submit_write(struct address_space *mapping,
 			struct writeback_control *wbc,
@@ -1489,6 +1499,7 @@ int ceph_submit_write(struct address_space *mapping,
 			if (!page)
 				continue;
 
+			ceph_undo_wrbuffer_claim(inode, page);
 			redirty_page_for_writepage(wbc, page);
 			unlock_page(page);
 		}
-- 
2.44.0