From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fhigh-a5-smtp.messagingengine.com (fhigh-a5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1BFCA29D26B for ; Thu, 2 Jul 2026 02:28:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782959335; cv=none; b=mx5PqNQriYOLM6WiZ4tAgRsotdbvOHIL16ZkNwq5t2iR5ECu5azCYReeFiGpJQhomjGh2vib8zEIAxnELupOx9SP/s+DQLkZF1i6R0E7YqP/+fjqOtPphxIv/KnD7TKh3FKFaXKKwooeQ6PdTwx4f85F/+2cicngC5Q3SeXX2xM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782959335; c=relaxed/simple; bh=T0GUB+JCtJp6VLh1+L75irLPn0zYSiyXyzIF1spo98M=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=g+f61QzDxhKYd+gkvbJ2UFUUvrSYQ60fyFsM9RqGXUaP/f8DxJOaBiixRu+2yDym46VV6Y/eWNUGLDWTIzYddLBpJJXKyyNvHt73N7rYefqXTFJMBb5ODuGpvj7mHVaJPvmBEKxOioeEzWqXBpSSiMug/h8PNUeaegfOHiQbsO4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=themaw.net; spf=pass smtp.mailfrom=themaw.net; dkim=pass (2048-bit key) header.d=themaw.net header.i=@themaw.net header.b=af9GWIdN; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Df/NmW9J; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=themaw.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=themaw.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=themaw.net header.i=@themaw.net header.b="af9GWIdN"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Df/NmW9J" Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfhigh.phl.internal (Postfix) with ESMTP id 25E6B1400152; Wed, 1 Jul 2026 22:28:53 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Wed, 01 Jul 2026 22:28:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=themaw.net; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1782959333; x=1783045733; bh=berWtSeX2X6M44bxhQtveNCtJ0Y2JGkYnUJjeY/rkvw=; b= af9GWIdNRLnjp9+S7BU/9d843z7qP9uNwqZKTGnscffBBNanwHR0hz2kWkUfL2Y6 itFcRUXMlVX5FRUUgTa7f74cHuqsz8Ccu1W30hVg5VqOnOoinzmo6a/ujO54ZDmm DgEYrpuYvjaPQagNvMjUInoQgXmZE83KJGYIKhPBxYn3tY8HuGeK11HXv3KoWOmy r34Ks72H9phcpfdqHb1T+ZBa11adlRiNnrkzy7l+7UzTEcgGq8HGO96CuuzrFwkx 6BzA2V4LV4+/ceH3iqrOV2NzGxuk2ykofcHhlakZL2O6m73GdfhUNfUIcX2oJrfB UdfbU8haQ5FuEsksCidGRw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1782959333; x= 1783045733; bh=berWtSeX2X6M44bxhQtveNCtJ0Y2JGkYnUJjeY/rkvw=; b=D f/NmW9J6SRTcb2X8UDgQrPS32IqzKmnc9oqDJ9N3n65nVoAB4Uyn/0Nk6nBAEdbA ZE/svZbqeQRpMujpbzvWS1i7rpyHKmmtXinj2OjCwfHuSXtbTgOdUwadwfaqvlTm luQpJzAdGb28KDAfQCtTPmqxMb68qiwScgCxO9ZnHnsIMWmzl5HuBDbjUMw9xT36 sVtwk0HGPOl0eH3mfAj+UFIZYm4kXWDSCZDV9mxy5XBH7+FGBVzviXnC/KyBT5P5 /rOZjQxTPK4pzd8Ag7aloQvtBGynPB2wn1DcwDbqEduVZq+FOG+SHi8r3jPoIltJ V/smMVyYjVtrrk8QxYHQg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFCRRykEd4ByJIkX6+aoLV6MUw2QhGklemr34PRjfhpjZ9oIQ5N/HueEl/uslzKyL /1aGG08K4xHgCrKSe7gNlZ7LWSGCmwCtFYXrbllzq2fMa+gsIhajIqbhkcZtiRV7EDcNxE 69Ja+u15+Yf4c84hYmyEvwJC2cEVfSX3vXGWwhwvZ4gvhq/ACTtSWXCiNKxW+OqlX5Hhl5 s3jImPu21o/mD5kyvXHeCSusFoxtO+FaftO+LvioJ/bwhf+zO5hWnFQX33xdULUgF3ytaY MIkW9Q9Mxd6j6Mu0YDY33+Cf3r4jLNr3Z1ii+lTioxRGLL2IhRezfUU/+UmsHVhpgnOpLi nN9seeU5lPkkmlfsh9ihChalGvDL6SqrABgKcMYPeTSZZI9p2RyissJZN7vgYFJ0YOlgYC IJ+NdmdUriBZQcr/MGQaySzov4duOPeiqm4nofBH0VOkmmtE9+2saikkLs4/MXE9SjqHGy XRjZMDbVihy1Q9Skrt0IPub95P3nrbwbDJaBdn5kY9bNOZN9XCpYVG9nZQM3K275+HTWcs rVxtxfYs7Liw1arKJy+CGwQWceple3l0kPKihoSDdTl4LynY4pfkBd5vUT1+j7Uq6TUSPr bgANqsbeqi6SYMch+j4lceMAwf+bmiuHB65rxSR/D1H3mJBAUC37IENCN6fQ X-ME-Proxy: Feedback-ID: i31e841b0:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Jul 2026 22:28:50 -0400 (EDT) Message-ID: Date: Thu, 2 Jul 2026 10:28:46 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation To: Wenwu Hou , Greg Kroah-Hartman Cc: Minchan Kim , Tejun Heo , LKML References: <20211116194317.1430399-1-minchan@kernel.org> Content-Language: en-AU From: Ian Kent Autocrypt: addr=raven@themaw.net; keydata= xsFNBE6c/ycBEADdYbAI5BKjE+yw+dOE+xucCEYiGyRhOI9JiZLUBh+PDz8cDnNxcCspH44o E7oTH0XPn9f7Zh0TkXWA8G6BZVCNifG7mM9K8Ecp3NheQYCk488ucSV/dz6DJ8BqX4psd4TI gpcs2iDQlg5CmuXDhc5z1ztNubv8hElSlFX/4l/U18OfrdTbbcjF/fivBkzkVobtltiL+msN bDq5S0K2KOxRxuXGaDShvfbz6DnajoVLEkNgEnGpSLxQNlJXdQBTE509MA30Q2aGk6oqHBQv zxjVyOu+WLGPSj7hF8SdYOjizVKIARGJzDy8qT4v/TLdVqPa2d0rx7DFvBRzOqYQL13/Zvie kuGbj3XvFibVt2ecS87WCJ/nlQxCa0KjGy0eb3i4XObtcU23fnd0ieZsQs4uDhZgzYB8LNud WXx9/Q0qsWfvZw7hEdPdPRBmwRmt2O1fbfk5CQN1EtNgS372PbOjQHaIV6n+QQP2ELIa3X5Z RnyaXyzwaCt6ETUHTslEaR9nOG6N3sIohIwlIywGK6WQmRBPyz5X1oF2Ld9E0crlaZYFPMRH hQtFxdycIBpTlc59g7uIXzwRx65HJcyBflj72YoTzwchN6Wf2rKq9xmtkV2Eihwo8WH3XkL9 cjVKjg8rKRmqIMSRCpqFBWJpT1FzecQ8EMV0fk18Q5MLj441yQARAQABzRtJYW4gS2VudCA8 cmF2ZW5AdGhlbWF3Lm5ldD7CwXsEEwECACUCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheA BQJOnjOcAhkBAAoJEOdnc4D1T9iphrYQALHK3J5rjzy4qPiLJ0EE9eJkyV1rqtzct5Ah9pu6 LSkqxgQCfN3NmKOoj+TpbXGagg28qTGjkFvJSlpNY7zAj+fA11UVCxERgQBOJcPrbgaeYZua E4ST+w/inOdatNZRnNWGugqvez80QGuxFRQl1ttMaky7VxgwNTXcFNjClW3ifdD75gHlrU0V ZUULa1a0UVip0rNc7mFUKxhEUk+8NhowRZUk0nt1JUwezlyIYPysaN7ToVeYE4W0VgpWczmA tHtkRGIAgwL7DCNNJ6a+H50FEsyixmyr/pMuNswWbr3+d2MiJ1IYreZLhkGfNq9nG/+YK/0L Q2/OkIsz8bOrkYLTw8WwzfTz2RXV1N2NtsMKB/APMcuuodkSI5bzzgyu1cDrGLz43faFFmB9 xAmKjibRLk6ChbmrZhuCYL0nn+RkL036jMLw5F1xiu2ltEgK2/gNJhm29iBhvScUKOqUnbPw DSMZ2NipMqj7Xy3hjw1CStEy3pCXp8/muaB8KRnf92VvjO79VEls29KuX6rz32bcBM4qxsVn cOqyghSE69H3q4SY7EbhdIfacUSEUV+m/pZK5gnJIl6n1Rh6u0MFXWttvu0j9JEl92Ayj8u8 J/tYvFMpag3nTeC3I+arPSKpeWDX08oisrEp0Yw15r+6jbPjZNz7LvrYZ2fa3Am6KRn0zsFN BE6c/ycBEADZzcb88XlSiooYoEt3vuGkYoSkz7potX864MSNGekek1cwUrXeUdHUlw5zwPoC 4H5JF7D8q7lYoelBYJ+Mf0vdLzJLbbEtN5+v+s2UEbkDlnUQS1yRo1LxyNhJiXsQVr7WVA/c 8qcDWUYX7q/4Ckg77UO4l/eHCWNnHu7GkvKLVEgRjKPKroIEnjI0HMK3f6ABDReoc741RF5X X3qwmCgKZx0AkLjObXE3W769dtbNbWmW0lgFKe6dxlYrlZbq25Aubhcu2qTdQ/okx6uQ41+v QDxgYtocsT/CG1u0PpbtMeIm3mVQRXmjDFKjKAx9WOX/BHpk7VEtsNQUEp1lZo6hH7jeo5me CYFzgIbXdsMA9TjpzPpiWK9GetbD5KhnDId4ANMrWPNuGC/uPHDjtEJyf0cwknsRFLhL4/NJ KvqAuiXQ57x6qxrkuuinBQ3S9RR3JY7R7c3rqpWyaTuNNGPkIrRNyePky/ZTgTMA5of8Wioy z06XNhr6mG5xT+MHztKAQddV3xFy9f3Jrvtd6UvFbQPwG7Lv+/UztY5vPAzp7aJGz2pDbb0Q BC9u1mrHICB4awPlja/ljn+uuIb8Ow3jSy+Sx58VFEK7ctIOULdmnHXMFEihnOZO3NlNa6q+ XZOK7J00Ne6y0IBAaNTM+xMF+JRc7Gx6bChES9vxMyMbXwARAQABwsFfBBgBAgAJBQJOnP8n AhsMAAoJEOdnc4D1T9iphf4QAJuR1jVyLLSkBDOPCa3ejvEqp4H5QUogl1ASkEboMiWcQJQd LaH6zHNySMnsN6g/UVhuviANBxtW2DFfANPiydox85CdH71gLkcOE1J7J6Fnxgjpc1Dq5kxh imBSqa2hlsKUt3MLXbjEYL5OTSV2RtNP04KwlGS/xMfNwQf2O2aJoC4mSs4OeZwsHJFVF8rK XDvL/NzMCnysWCwjVIDhHBBIOC3mecYtXrasv9nl77LgffyyaAAQZz7yZcvn8puj9jH9h+mr L02W+gd+Sh6Grvo5Kk4ngzfT/FtscVGv9zFWxfyoQHRyuhk0SOsoTNYN8XIWhosp9GViyDtE FXmrhiazz7XHc32u+o9+WugpTBZktYpORxLVwf9h1PY7CPDNX4EaIO64oyy9O3/huhOTOGha nVvqlYHyEYCFY7pIfaSNhgZs2aV0oP13XV6PGb5xir5ah+NW9gQk/obnvY5TAVtgTjAte5tZ +coCSBkOU1xMiW5Td7QwkNmtXKHyEF6dxCAMK1KHIqxrBaZO27PEDSHaIPHePi7y4KKq9C9U 8k5V5dFA0mqH/st9Sw6tFbqPkqjvvMLETDPVxOzinpU2VBGhce4wufSIoVLOjQnbIo1FIqWg Dx24eHv235mnNuGHrG+EapIh7g/67K0uAzwp17eyUYlE5BMcwRlaHMuKTil6 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 30/6/26 11:26, Wenwu Hou wrote: > On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote: >> On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote: >>> The kernfs implementation has big lock granularity(kernfs_rwsem) so >>> every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete >>> the lock. Thus, if one of userspace goes the sleep under holding >>> the lock for a long time, rest of them should wait it. A example is >>> the holder goes direct reclaim with the lock since it needs memory >>> allocation. Let's fix it at common technique that release the lock >>> and then allocate the memory. Fortunately, kernfs looks like have >>> an refcount so I hope it's fine. >>> >>> Signed-off-by: Minchan Kim >>> --- >>> fs/kernfs/dir.c | 14 +++++++++++--- >>> fs/kernfs/inode.c | 2 +- >>> fs/kernfs/kernfs-internal.h | 1 + >>> 3 files changed, 13 insertions(+), 4 deletions(-) >> What workload hits this lock to cause it to be noticable? >> >> There was a bunch of recent work in this area to make this much more >> fine-grained, and the theoritical benchmarks that people created (adding >> 10s of thousands of scsi disks at boot time) have gotten better. > Hi, > > By 2026, the kernfs_rwsem has been split into per-fs. But the problem still > exists. In spite of my confusion due to your reference to a really old post I do have a recent similar report (although I'm still struggling to work out the cause of my report) I do agree that moving the inode allocation out from under the rwsem is a good idea regardless. And I do still think the old patch you are referring to isn't quite good enough. Once I have worked through the back trace I'm looking at I'll have a go at doing it, unless Greg or Tejun has any objections? Ian > > In a k8s cluster environment, there are a lot of processes reading /sys, for example: > > Container runtime: runc > Prometheus Exporter: node_exporter > > If a tenant's container processes read /sys and sleep on the OOM (or direct > reclaim) path, both runc and node_exporter will be blocked, which will cause > the cluster worker node to become unschedulable. > > For example, the nodejs runtime reads > /sys/devices/system/cpu/cpu%u/cpufreq/scaling_cur_freq [1] > > Jun 22 21:43:17 kernel: task:node state:D stack:0 pid:2621530 ppid:1062663 flags:0x00004002 > Jun 22 21:43:17 kernel: Call Trace: > Jun 22 21:43:17 kernel: > Jun 22 21:43:17 kernel: __schedule+0x278/0x740 > Jun 22 21:43:17 kernel: schedule+0x5a/0xd0 > Jun 22 21:43:17 kernel: schedule_preempt_disabled+0x11/0x20 > Jun 22 21:43:17 kernel: __mutex_lock.constprop.0+0x376/0x680 > Jun 22 21:43:17 kernel: ? kvm_sched_clock_read+0xd/0x20 > Jun 22 21:43:17 kernel: mem_cgroup_out_of_memory+0x53/0x150 > Jun 22 21:43:17 kernel: try_charge_memcg+0x682/0x7d0 > Jun 22 21:43:17 kernel: ? memcg_list_lru_alloc+0xa7/0x330 > Jun 22 21:43:17 kernel: obj_cgroup_charge+0x70/0x170 > Jun 22 21:43:17 kernel: slab_pre_alloc_hook.constprop.0+0xb6/0x1d0 > Jun 22 21:43:17 kernel: ? alloc_inode+0x59/0xc0 > Jun 22 21:43:17 kernel: kmem_cache_alloc_lru+0x54/0x2c0 > Jun 22 21:43:17 kernel: alloc_inode+0x59/0xc0 > Jun 22 21:43:17 kernel: iget_locked+0xe3/0x220 > Jun 22 21:43:17 kernel: kernfs_get_inode+0x18/0x110 > Jun 22 21:43:17 kernel: kernfs_iop_lookup+0x74/0xd0 > Jun 22 21:43:17 kernel: __lookup_slow+0x82/0x130 > Jun 22 21:43:17 kernel: walk_component+0xdb/0x150 > Jun 22 21:43:17 kernel: link_path_walk.part.0.constprop.0+0x240/0x380 > Jun 22 21:43:17 kernel: ? path_init+0x293/0x3d0 > Jun 22 21:43:17 kernel: path_openat+0x85/0x2a0 > Jun 22 21:43:17 kernel: ? seq_printf+0x8e/0xb0 > Jun 22 21:43:17 kernel: do_filp_open+0xb4/0x160 > Jun 22 21:43:17 kernel: ? __check_object_size.part.0+0x5e/0x130 > Jun 22 21:43:17 kernel: do_sys_openat2+0x91/0xc0 > Jun 22 21:43:17 kernel: __x64_sys_openat+0x53/0xa0 > Jun 22 21:43:17 kernel: do_syscall_64+0x35/0x80 > Jun 22 21:43:17 kernel: entry_SYSCALL_64_after_hwframe+0x78/0xe2 > > A rwsem writer is involved: > > Jun 22 21:46:17 kernel: task:kworker/u232:0 state:D stack:0 pid:2886179 ppid:2 flags:0x00004000 > Jun 22 21:46:17 kernel: Workqueue: netns cleanup_net > Jun 22 21:46:17 kernel: Call Trace: > Jun 22 21:46:17 kernel: > Jun 22 21:46:17 kernel: __schedule+0x278/0x740 > Jun 22 21:46:17 kernel: ? psi_group_change+0x226/0x3d0 > Jun 22 21:46:17 kernel: schedule+0x5a/0xd0 > Jun 22 21:46:17 kernel: schedule_preempt_disabled+0x11/0x20 > Jun 22 21:46:17 kernel: rwsem_down_write_slowpath+0x1e2/0x4f0 > Jun 22 21:46:17 kernel: down_write+0x57/0x60 > Jun 22 21:46:17 kernel: kernfs_remove_by_name_ns+0x38/0xc0 > Jun 22 21:46:17 kernel: remove_files+0x2b/0x70 > Jun 22 21:46:17 kernel: sysfs_remove_group+0x38/0x80 > Jun 22 21:46:17 kernel: sysfs_remove_groups+0x29/0x50 > Jun 22 21:46:17 kernel: ib_free_port_attrs+0x92/0x170 [ib_core] > Jun 22 21:46:17 kernel: rdma_dev_exit_net+0x117/0x1d0 [ib_core] > Jun 22 21:46:17 kernel: ops_exit_list+0x30/0x70 > Jun 22 21:46:17 kernel: cleanup_net+0x273/0x430 > Jun 22 21:46:17 kernel: process_one_work+0x18a/0x3a0 > Jun 22 21:46:17 kernel: worker_thread+0x277/0x3a0 > Jun 22 21:46:17 kernel: ? __pfx_worker_thread+0x10/0x10 > Jun 22 21:46:17 kernel: kthread+0xe1/0x110 > Jun 22 21:46:17 kernel: ? __pfx_kthread+0x10/0x10 > Jun 22 21:46:17 kernel: ret_from_fork+0x2d/0x50 > Jun 22 21:46:17 kernel: ? __pfx_kthread+0x10/0x10 > Jun 22 21:46:17 kernel: ret_from_fork_asm+0x1b/0x30 > Jun 22 21:46:17 kernel: > > > In our real-world case, thousands of nodejs processes hit the cgroup memory > limit and sleep on the OOM path, runc gets blocked, and consequently the node > remains in an unschedulable state for many hours. > > > Link: > - [1] https://github.com/nodejs/node/blob/v22.19.0/deps/uv/src/unix/linux.c#L1889 > >> But in that work, no one could find a real benchmark or use case that >> anyone could even notice this type of thing. What do you have that >> shows this? >> thanks, >> >> greg k-h