Re: PROBLEM: Duplicated entries in /proc/<pid>/mountinfo

[Christian Brauner <brauner@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 4812 bytes --]

On Wed, Jan 28, 2026 at 08:49:12AM -0600, Zachary M. Raines wrote:
> Greetings,
> 
> When mounting and unmounting many filesystems, /proc/<pid>/mountinfo sometimes
> contains entries which are duplicated many times.
> 
> Summary
> =======
> 
> Sometimes on a system that is mounting and unmounting filesystems frequently,
> for example running lots of docker containers, the size of /proc/1/mountinfo,
> can become very large -- 100s, to 1000s of entries or more -- with the vast
> majority being a single entry duplicated many times.
> 
> This causes other problems on the system, due to systemd parsing the mount table
> whenever it changes, and eating up a lot of memory, for example [1]. Waiting
> long enough there are rare events where the length of mountinfo can go into the
> millions of lines and lead to OOM and kernel panics.
> 
> Running the reproducers below, I pretty reliably see an Ubuntu virtual machine
> kernel panic due to lack of memory within about 24hrs.
> 
> Versions
> ========
> 
> Bisecting the kernel git history, I was able to track the issue back to
> '2eea9ce4310d8 mounts: keep list of mounts in an rbtree' [2].
> 
> I've tested on 6.19-rc7 in a virtual machine and the issue is still present
> there. /proc/version:
> 
> Linux version 6.19.0-rc7+ (ubuntu@kernel-builder) (gcc (Ubuntu 15.2.0-4ubuntu4)
> 15.2.0, GNU ld (GNU Binutils for Ubuntu) 2.45) #8 SMP PREEMPT_DYNAMIC Tue Jan 27
> 22:33:35 UTC 2026
> 
> running on Ubuntu 25.10
> 
> Reproducer
> ==========
> 
> The problem can be reproduced by mounting and then unmounting tmpfs in a loop
> and in a seperate process reading /proc/1/mountinfo and checking for duplicates.
> 
> I used the following scripts:
> 
> 1. Mounts and unmounts tmpfs
> 
> #!/bin/bash
> counter=0
> while true; do
>     unique_name="tmpfs_$$_$counter"
>    	mkdir -p "/tmp/$unique_name"
>    	sudo mount -t tmpfs "$unique_name" "/tmp/$unique_name"
>    	sudo umount "/tmp/$unique_name"
>    	rmdir "/tmp/$unique_name"
>    	((counter++))
>    	sleep 0.1
> done
> 
> 2. Reads `/prod/1/mountinfo` and checks for duplicates
> 
> #!/bin/bash
> THRESHOLD=75
> echo "Starting monitoring at $(date)"
> while true; do
>     # Get mountinfo entries and count total
>     mountinfo="$(cat /proc/1/mountinfo)"
>     mountinfo_count=$(echo "$mountinfo" | wc -l)
> 
>     if ((mountinfo_count > THRESHOLD)); then
>         echo "$(date): Mount count ($mountinfo_count) exceeds threshold ($THRESHOLD)"
> 
>         # Find and log duplicate mount points with their counts
>         duplicates=$(echo "$mountinfo" | sort | uniq -cd)
> 
>         if [[ -n "$duplicates" ]]; then
>             echo "Duplicate mounts :"
>             echo "$duplicates"
>         fi
>         echo "====="
>         echo "$mountinfo"
>         echo "---"
>     fi
> 
>     sleep 0.1
> done
> 
> Typically, within 5-10 minutes duplicates can be observed, often including
> hundreds or thousands of copies of the same mount point -- although the number
> can rarely spike to much higher values. Given a long enough uptime, I've
> observed up to 1.4 million duplicates at a time.
> 
> The duplication in mountinfo is very intermittent. `cat /proc/1/mountinfo` 100ms
> later shows no duplication.
> 
> Additional diagnostics
> ======================
> 
> While running the script (2.) above, I also ran the following bpftrace script
> 
> 3. Trace vfs_mounts as by `cat /proc/1/mountinfo`
> 
> #!/usr/bin/env bpftrace
> 
> fentry:show_mountinfo / comm == "cat"/ {
>     @mnts[args->mnt] = count();
> }
> 
> tracepoint:sched:sched_process_exit / comm == "cat"/ {
>     for ($mnt : @mnts) {
>         if ($mnt.1 > 1) {
>             printf("Duplicate mount %p\n", $mnt.0);
>             @dups[$mnt.0] = $mnt.1;
>         }
>     }
>     clear(@mnts);
> }
> 
> and observed that a single mount struct was reached multiple times -- perhaps
> unsurprisingly exactly the same number as there were duplicates detected by
> the above script.
> 
> Typical outputs of script (2.) and the bpftrace script above are
> 
> Starting monitoring at Tue Jan 27 20:48:13 UTC 2026
> Tue Jan 27 20:50:43 UTC 2026: Mount count (696) exceeds threshold (75)
> Duplicate mounts :
>   /proc/sys/fs/binfmt_misc: 2 occurrences
>   /tmp/tmpfs_856614_41491: 666 occurrences
> 
> and
> 
> @dups[0xffff88e5fb9f10a0]: 666

Thanks for the report. So it's a bit unfortunate that you're showing
duplication by source path. That's not as useful as that can
legitimately happen. So the better test would be to see whether you get
any duplicated unique mount ids, i.e., whether the same
mnt->mnt_id_unique appears multiple times. Because that's a bug for
sure.

I suspect the issue is real though. I'm appending a patch as a proposed
fix. Can you test that and report back, please? I'm traveling tomorrow
so might take a little.

[-- Attachment #2: 0001-namespace-fix-proc-mount-iteration.patch --]
[-- Type: text/x-diff, Size: 2099 bytes --]

From 26907be24e631329ba963d66196fe4f5d0863388 Mon Sep 17 00:00:00 2001
From: Christian Brauner <brauner@kernel.org>
Date: Thu, 29 Jan 2026 14:52:22 +0100
Subject: [PATCH] namespace: fix proc mount iteration

The m->index isn't updated when m->show() overflows and retains its
value before the current mount causing a restart to start at the same
value. If that happens in short order to due a quickly expanding mount
table this would cause the same mount to be shown again and again.

Ensure that *pos always equals the mount id of the mount that was
returned by start/next. On restart after overflow mnt_find_id_at(*pos)
finds the exact mount. This should avoid duplicates, avoid skips and
should handle concurrent modification just fine.

Cc: <stable@vger.kernel.org>
Fixed: 2eea9ce4310d8 ("mounts: keep list of mounts in an rbtree")
Signed-off-by: Christian Brauner <brauner@kernel.org>
---
 fs/namespace.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/fs/namespace.c b/fs/namespace.c
index c58674a20cad..ad35f8c961ef 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -1537,23 +1537,33 @@ static struct mount *mnt_find_id_at_reverse(struct mnt_namespace *ns, u64 mnt_id
 static void *m_start(struct seq_file *m, loff_t *pos)
 {
 	struct proc_mounts *p = m->private;
+	struct mount *mnt;
 
 	down_read(&namespace_sem);
 
-	return mnt_find_id_at(p->ns, *pos);
+	mnt = mnt_find_id_at(p->ns, *pos);
+	if (mnt)
+		*pos = mnt->mnt_id_unique;
+	return mnt;
 }
 
 static void *m_next(struct seq_file *m, void *v, loff_t *pos)
 {
-	struct mount *next = NULL, *mnt = v;
+	struct mount *mnt = v;
 	struct rb_node *node = rb_next(&mnt->mnt_node);
 
-	++*pos;
 	if (node) {
-		next = node_to_mount(node);
+		struct mount *next = node_to_mount(node);
 		*pos = next->mnt_id_unique;
+		return next;
 	}
-	return next;
+
+	/*
+	 * No more mounts. Set pos past current mount's ID so that if
+	 * iteration restarts, mnt_find_id_at() returns NULL.
+	 */
+	*pos = mnt->mnt_id_unique + 1;
+	return NULL;
 }
 
 static void m_stop(struct seq_file *m, void *v)
-- 
2.47.3