Debugging stuck mount

Debugging stuck mount

[Morten Hein Tiljeset]

Hi folks,

I'm trying to debug an issue that occurs sporadically in production where the
ext4 filesystem on a device, say dm-1, is never fully closed. This is visible
from userspace only via the existence of /sys/fs/ext4/dm-1 which cryptsetup
uses to determine that the device is still mounted.

My initial thought was that it was mounted in some mount namespace, but this is
not the case. I've used a debugger (drgn) on /proc/kcore to find the
superblock. I can see that this is kept alive by a single mount which looks
like this (leaving out all fields that are NULL/empty lists):

*(struct mount *)0xffff888af92c5cc0 = {
	.mnt_parent = (struct mount *)0xffff888af92c5cc0,
	.mnt_mountpoint = (struct dentry *)0xffff888850331980,  // an application defined path
	.mnt = (struct vfsmount){
		.mnt_root = (struct dentry *)0xffff888850331980,    // note: same path as path as mnt_mountpoint
		.mnt_sb = (struct super_block *)0xffff88a89f7bc800, // points to the superblock I want cleaned up
		.mnt_flags = (int)134217760,                        // 0x8000020 = MNT_UMOUNT | MNT_RELATIME
		.mnt_userns = (struct user_namespace *)init_user_ns+0x0 = 0xffffffffb384b400,
	},
	.mnt_pcp = (struct mnt_pcp *)0x37dfbfa2c338,
	.mnt_instance = (struct list_head){
		.next = (struct list_head *)0xffff88a89f7bc8d0,
		.prev = (struct list_head *)0xffff88a89f7bc8d0,
	},
	.mnt_devname = (const char *)0xffff88a7d0fe7cc0 = "/dev/mapper/<my device>_crypt", // maps to /dev/dm-1
	.mnt_id = (int)3605,
}

In particular I notice that the mount namespace is NULL. As far as I understand
the only way to get this state is through a lazy unmount (MNT_DETACH). I can at
least manage to create a similar state by lazily unmounting but keeping the
mount alive with a shell with CWD inside the mountpoint.

I've tried to search for the superblock pointer on cwd/root of all tasks, which
works in my synthetic example but not for the real case. I've had similar
results searching for the superblock pointer using drgn's fsrefs.py script[1]
which has support for searching additional kernel data structures.

Is there any good way of detecting why this mount is kept alive? Where am I not
looking? Any pointers would be greatly appreciated!

[1] https://drgn.readthedocs.io/en/latest/release_highlights/0.0.26.html

Re: Debugging stuck mount

[Christian Brauner]

On Thu, Aug 08, 2024 at 11:40:07AM GMT, Morten Hein Tiljeset wrote:
> Hi folks,
> 
> I'm trying to debug an issue that occurs sporadically in production where the
> ext4 filesystem on a device, say dm-1, is never fully closed. This is visible
> from userspace only via the existence of /sys/fs/ext4/dm-1 which cryptsetup
> uses to determine that the device is still mounted.
> 
> My initial thought was that it was mounted in some mount namespace, but this is
> not the case. I've used a debugger (drgn) on /proc/kcore to find the
> superblock. I can see that this is kept alive by a single mount which looks
> like this (leaving out all fields that are NULL/empty lists):
> 
> *(struct mount *)0xffff888af92c5cc0 = {
> 	.mnt_parent = (struct mount *)0xffff888af92c5cc0,
> 	.mnt_mountpoint = (struct dentry *)0xffff888850331980,  // an application defined path
> 	.mnt = (struct vfsmount){
> 		.mnt_root = (struct dentry *)0xffff888850331980,    // note: same path as path as mnt_mountpoint
> 		.mnt_sb = (struct super_block *)0xffff88a89f7bc800, // points to the superblock I want cleaned up
> 		.mnt_flags = (int)134217760,                        // 0x8000020 = MNT_UMOUNT | MNT_RELATIME
> 		.mnt_userns = (struct user_namespace *)init_user_ns+0x0 = 0xffffffffb384b400,
> 	},
> 	.mnt_pcp = (struct mnt_pcp *)0x37dfbfa2c338,
> 	.mnt_instance = (struct list_head){
> 		.next = (struct list_head *)0xffff88a89f7bc8d0,
> 		.prev = (struct list_head *)0xffff88a89f7bc8d0,
> 	},
> 	.mnt_devname = (const char *)0xffff88a7d0fe7cc0 = "/dev/mapper/<my device>_crypt", // maps to /dev/dm-1
> 	.mnt_id = (int)3605,
> }

That's the root mount of the filesystem here.

> 
> In particular I notice that the mount namespace is NULL. As far as I understand
> the only way to get this state is through a lazy unmount (MNT_DETACH). I can at
> least manage to create a similar state by lazily unmounting but keeping the
> mount alive with a shell with CWD inside the mountpoint.
> 
> I've tried to search for the superblock pointer on cwd/root of all tasks, which
> works in my synthetic example but not for the real case. I've had similar
> results searching for the superblock pointer using drgn's fsrefs.py script[1]
> which has support for searching additional kernel data structures.

It's likely held alive by some random file descriptor someone has open.
IOW, try and walk all /proc/<pid>/fd/<nr> in that case and see whether
anything keeps it alive.

Re: Debugging stuck mount

[Morten Hein Tiljeset]

> It's likely held alive by some random file descriptor someone has open.
> IOW, try and walk all /proc/<pid>/fd/<nr> in that case and see whether
> anything keeps it alive.

Thanks for the suggestion, but I've already tried the equivalent of that by
using a debugger to find the superblock in question and then walking all open
fds and comparing the superblock pointer. I've validated that this approach
works in a synthetic example where I create a new namespace, mount the
filesystem under /mnt, run a program to open /mnt/foo and lazy unmount /mnt.

Walking procfs seems less precise. I've tried iterating through /proc/*/fd/*
and comparing the Device entry of stat -L, also without luck.

Re: Debugging stuck mount

[Christian Brauner]

On Thu, Aug 08, 2024 at 04:20:43PM GMT, Morten Hein Tiljeset wrote:
> > It's likely held alive by some random file descriptor someone has open.
> > IOW, try and walk all /proc/<pid>/fd/<nr> in that case and see whether
> > anything keeps it alive.
> 
> Thanks for the suggestion, but I've already tried the equivalent of that by
> using a debugger to find the superblock in question and then walking all open
> fds and comparing the superblock pointer. I've validated that this approach
> works in a synthetic example where I create a new namespace, mount the
> filesystem under /mnt, run a program to open /mnt/foo and lazy unmount /mnt.
> 
> Walking procfs seems less precise. I've tried iterating through /proc/*/fd/*
> and comparing the Device entry of stat -L, also without luck.

The file descriptor could already be closed but the task could be stuck
exiting and so queued task work including destroying files wouldn't be
done yet. You could also try and see if you can figure out what tasks
require your workload to do a lazy umount in the first place. That might
bring you closer to the root cause.

What kernel version are you running anyway?

Re: Debugging stuck mount

[Morten Hein Tiljeset]

> The file descriptor could already be closed but the task could be stuck
> exiting and so queued task work including destroying files wouldn't be
> done yet. You could also try and see if you can figure out what tasks
> require your workload to do a lazy umount in the first place. That might
> bring you closer to the root cause.
>
> What kernel version are you running anyway?
That's an interesting idea about the stuck task, I'll try looking into that. We're not
doing lazy unmounts on purpose -- I'm only concluding that it must be happening based on
the mount namespace being NULL. I wonder if the lazy unmount could happen implicitly
as a consequence of some other operation?

I'm on kernel 6.1.53 and I've also experienced the problem on the 5.15.x series.

Re: Debugging stuck mount

[Christian Brauner]

On Fri, Aug 09, 2024 at 09:39:41AM GMT, Morten Hein Tiljeset wrote:
> > The file descriptor could already be closed but the task could be stuck
> > exiting and so queued task work including destroying files wouldn't be
> > done yet. You could also try and see if you can figure out what tasks
> > require your workload to do a lazy umount in the first place. That might
> > bring you closer to the root cause.
> >
> > What kernel version are you running anyway?
> That's an interesting idea about the stuck task, I'll try looking into that. We're not
> doing lazy unmounts on purpose -- I'm only concluding that it must be happening based on
> the mount namespace being NULL. I wonder if the lazy unmount could happen implicitly
> as a consequence of some other operation?

You don't need a lazy umount for this. You could also just have a
container or service exiting that uses a mount namespace. In that case
the last exiting task will close all of its open files and then put the
mount namespace. The mount namespace will unmount all mounts that were
still around NULLing the mntns.

Say P1 has somehow managed to get hold of a file descriptor in some
other mount namespace M1 and holds a file descriptor referring to some
mount in there. Now the last process pinning M1 exists and puts all
mounts associated with M1. Now P1 holds a file descriptor pointing to a
mount that has a NULL mount namespace.

A task could hang running task work because of some device file or
whatever that it's closing. Or it could hang exiting namespaces. I
vaguely remember that years ago on earlier kernels we had some issues
there.