mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Tejun Heo <tj@kernel.org>
To: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Christian Brauner <christian@brauner.io>,
	Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
	Meta kernel team <kernel-team@meta.com>,
	linux-fsdevel@vger.kernel.org, driver-core@lists.linux.dev,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH] kernfs: don't hold kernfs_rwsem across notification delivery
Date: Thu, 10 Sep 2026 11:12:22 -1000	[thread overview]
Message-ID: <aqMdNoXA0LIYFpQW@slm.duckdns.org> (raw)
In-Reply-To: <20260910045406.485295-1-shakeel.butt@linux.dev>

On Wed, Sep 09, 2026 at 09:54:06PM -0700, Shakeel Butt wrote:
> Recently at Meta we noticed that on multi-tenant systems, a workload
> under extreme memory pressure negatively impacts unrelated workloads and
> system daemons like below [1] and fb-oomd [2]. Mainly we observed that
> system daemons were stuck on kernfs_rwsem in cgroup-related interfaces
> for long periods of time.
> 
> On further inspection, we found that the workload under pressure had
> registered inotify watches on its memory.events and was continuously
> receiving limit-hitting notifications. The cgroup notification is done in
> kernfs_notify_workfn(), which holds kernfs_supers_rwsem and kernfs_rwsem
> and calls fsnotify(), which allocates the event with
> kmalloc(GFP_KERNEL_ACCOUNT|__GFP_RETRY_MAYFAIL) charged to the watching
> cgroup, which is already under memory pressure. So the notify worker
> ended up in memory reclaim of a cgroup already under pressure, and itself
> kept triggering the limit notifications.
> 
> This notification path takes the locks in read mode, but a waiting writer
> blocks all future readers. That is exactly what is happening in the Meta
> fleet, and it leaves daemons that are critical to the reliability of the
> system stuck for long periods of time.
> 
> Commit 400188ae361a ("kernfs: Acquire kernfs_rwsem in
> kernfs_notify_workfn().") added kernfs_rwsem just to safely read
> kernfs_node::name.  Let's sample the name once before the loop and drop
> the lock.  kernfs_supers_rwsem still covers the list, and the removal
> paths take that one for reading too, so a stalled worker can still hold
> a removal up once a mount or unmount is queued behind it.
> 
> Two things change for a watcher.  A name longer than NAME_MAX now gives
> an event with no name, where before it gave one with the full name;
> fsnotify() takes the name as optional, so the event still arrives.  And
> the name and the parent are no longer sampled under one lock, so a rename
> between the two would name a file against the directory it moved to.
> That cannot happen today: kernfs_notify() rejects anything that is not a
> file, and no caller of kernfs_rename_ns() renames one.
> 
> [1] https://github.com/facebookincubator/below
> [2] https://github.com/facebookincubator/oomd
> 
> Fixes: 400188ae361a ("kernfs: Acquire kernfs_rwsem in kernfs_notify_workfn().")
> Assisted-by: LLM
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

-- 
tejun

      reply	other threads:[~2026-09-10 21:12 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-09-10  4:54 Shakeel Butt
2026-09-10 21:12 ` Tejun Heo [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=aqMdNoXA0LIYFpQW@slm.duckdns.org \
    --to=tj@kernel.org \
    --cc=bigeasy@linutronix.de \
    --cc=christian@brauner.io \
    --cc=driver-core@lists.linux.dev \
    --cc=gregkh@linuxfoundation.org \
    --cc=kernel-team@meta.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=shakeel.butt@linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®