From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6489D43C04D for ; Tue, 30 Jun 2026 14:30:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782829853; cv=none; b=oD/tHpy5SuFYN1JviM21RH1uISjA1W81UJsqD+8yxvtgcvxi69fa9RjZwOhvdwgRKUQKzxoIJlPJ9pkp7oHksjguuMNx+zTfZ+NxxUdZUaeC5JmvYCJAkmPA1BLBRBF7rzxppRUEmTzUGG+SaYa79N2ML+zqAHW8Fvg0zDuztqE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782829853; c=relaxed/simple; bh=M5ulQE6BKX8pAaLEt4yJ0Yvr5NEuNMRwVZ9jO3GqiW8=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=Ugf41sy6DXdGsJ7uxDyCbVa042zb10V4oTHan5BR+AaDkgXzm2oAYs9plUDXhKqf2kqfCt58/wZnmcsOvyzGVoTNWJIQ5ICead+YwjZJdAard38Bk6gx23Ebbpk9/gFYgr+JQZU2/KLvGzMwkWG68zyyiFxbnDNDPqsORcIB5uk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=themaw.net; spf=pass smtp.mailfrom=themaw.net; dkim=pass (2048-bit key) header.d=themaw.net header.i=@themaw.net header.b=dZBq8T4Q; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=F3FIkZSt; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=themaw.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=themaw.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=themaw.net header.i=@themaw.net header.b="dZBq8T4Q"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="F3FIkZSt" Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfhigh.stl.internal (Postfix) with ESMTP id 83B027A0123; Tue, 30 Jun 2026 10:30:50 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Tue, 30 Jun 2026 10:30:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=themaw.net; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1782829850; x=1782916250; bh=L0W990WqMoLiii8LNE4neasyqkx6FVeJxe7R8Hwijzo=; b= dZBq8T4QjtZKnLEutZB+WCWtWCs0I7XAaoWZNa8mPfXeJ/QX2U3dmSH0siDqrw4E vb92pXlL89tTsWYBnXa0jgC5R4oDYGZdnWm9bdsODYl+R66OhEgCCUHBMzk+laiv kIkc2RcVr+HO+/R2zUdXqpC6VIyeNG1rIyPSw7q3abImJQr/V7W5K3pFeGVP4YJp rbC0J1ph32VkkSZgZPIvsnRsM0HrZTv54p/eqfciQGSZepaKBV2ywxJMSkOCwVpr kYxOhO4MGOO05iGLTx8mIBI/Sxdb5R9vM95+CPiAlYK5r7/mQSfNZwpdVueJQ2Xd p4Yu5+VkFtt/siL4cb9VIg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1782829850; x= 1782916250; bh=L0W990WqMoLiii8LNE4neasyqkx6FVeJxe7R8Hwijzo=; b=F 3FIkZStVwVZuSrzPdmUZli8WxGDvxNrVBtaaurJF0Zl1iMe3mqN21LFVLr5+e83v EjqiCqgtTTZWSICvrJ2S8q2FHb0St/ftS9aHDAOIxYR1gMSD0ZfVcgRhKzzfGVHF EoPhU06tbfj0K5xWvvxl1f3RlM37UrtSjktjB8rXtd4gJDh6DyX+k15iWB90WLRt JjZtYuECtIm8E0SKzLGeMoEFf/wWSiqUaXRmNYSBE9F03rC+gEZ8SNcajjF2jLoy viaUhAuUwUmiJ1qyPKgK3PA4A86qNi6M8Fq4glxHccDYhJJqQ3mH/uVJ+SdYRlwZ VvFgbDEiyX0y/5SbJAriA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTGxhxP0lei+oeUikhXELKjooOwg+QNmImP+HwlRHnX6eHeG/dA2BTlgJNRtIqEPpZ M7lEoJ/8zD8289zoD1H7JBUrgx3Oz6EAUiEVkFjmSmpKInlOykDggyBeRv1op9TIbTR0Ob 9uMZeCNJFu+iezfGrQ4Z/B8k6If12zMZUhzx1qLbpmHWm/WFXqDz8miU3dd02i4YZEGvMR 7v0a7JQkm4y4EGq60yctFji4DeejJYz+J3RwymbitnMhZwgfopFLec+VyOtX5NPRDbY9Zk 8IlsLnzvWcr7cH6frDmmM8bRUFXWLvQi7J+mkZIAvZYNG2LXIk3CFa2vWJ+V0sA6eSKQq+ WI3an7J5HhFV7BiXXf+ucemirrqMEUfXsjAxd+N5XcsD1bHIlzwzlA9aQrzm16xR7Z0SI1 PEF4dcGzZp4Eu4bOgCNqulCbujSvUZwH4SQTLs2hOY8i1j/ipPSmeu8BQn8jyX6zKTbvDA Q5v1uX2j4iq4yannx4l4RHTAITXvwgV57LjxD1pRQOeRdzznYjjX9l3O7dZk988sjUL4y8 AvMn0B864Dg/zehakZc5hL0o4vqBt3arZfSLByL7e/+KWnrrwOJCDQRRGxhvzm2j389itq DS2fxQRiZSRZ56hphGmyCl5BtU7szR3V4VwXwuTNwkxoUimxvjK1RGgs09VA X-ME-Proxy: Feedback-ID: i31e841b0:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 30 Jun 2026 10:30:47 -0400 (EDT) Message-ID: Date: Tue, 30 Jun 2026 22:30:43 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation To: Wenwu Hou , Greg Kroah-Hartman Cc: Minchan Kim , Tejun Heo , LKML References: <20211116194317.1430399-1-minchan@kernel.org> Content-Language: en-AU From: Ian Kent Autocrypt: addr=raven@themaw.net; keydata= xsFNBE6c/ycBEADdYbAI5BKjE+yw+dOE+xucCEYiGyRhOI9JiZLUBh+PDz8cDnNxcCspH44o E7oTH0XPn9f7Zh0TkXWA8G6BZVCNifG7mM9K8Ecp3NheQYCk488ucSV/dz6DJ8BqX4psd4TI gpcs2iDQlg5CmuXDhc5z1ztNubv8hElSlFX/4l/U18OfrdTbbcjF/fivBkzkVobtltiL+msN bDq5S0K2KOxRxuXGaDShvfbz6DnajoVLEkNgEnGpSLxQNlJXdQBTE509MA30Q2aGk6oqHBQv zxjVyOu+WLGPSj7hF8SdYOjizVKIARGJzDy8qT4v/TLdVqPa2d0rx7DFvBRzOqYQL13/Zvie kuGbj3XvFibVt2ecS87WCJ/nlQxCa0KjGy0eb3i4XObtcU23fnd0ieZsQs4uDhZgzYB8LNud WXx9/Q0qsWfvZw7hEdPdPRBmwRmt2O1fbfk5CQN1EtNgS372PbOjQHaIV6n+QQP2ELIa3X5Z RnyaXyzwaCt6ETUHTslEaR9nOG6N3sIohIwlIywGK6WQmRBPyz5X1oF2Ld9E0crlaZYFPMRH hQtFxdycIBpTlc59g7uIXzwRx65HJcyBflj72YoTzwchN6Wf2rKq9xmtkV2Eihwo8WH3XkL9 cjVKjg8rKRmqIMSRCpqFBWJpT1FzecQ8EMV0fk18Q5MLj441yQARAQABzRtJYW4gS2VudCA8 cmF2ZW5AdGhlbWF3Lm5ldD7CwXsEEwECACUCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheA BQJOnjOcAhkBAAoJEOdnc4D1T9iphrYQALHK3J5rjzy4qPiLJ0EE9eJkyV1rqtzct5Ah9pu6 LSkqxgQCfN3NmKOoj+TpbXGagg28qTGjkFvJSlpNY7zAj+fA11UVCxERgQBOJcPrbgaeYZua E4ST+w/inOdatNZRnNWGugqvez80QGuxFRQl1ttMaky7VxgwNTXcFNjClW3ifdD75gHlrU0V ZUULa1a0UVip0rNc7mFUKxhEUk+8NhowRZUk0nt1JUwezlyIYPysaN7ToVeYE4W0VgpWczmA tHtkRGIAgwL7DCNNJ6a+H50FEsyixmyr/pMuNswWbr3+d2MiJ1IYreZLhkGfNq9nG/+YK/0L Q2/OkIsz8bOrkYLTw8WwzfTz2RXV1N2NtsMKB/APMcuuodkSI5bzzgyu1cDrGLz43faFFmB9 xAmKjibRLk6ChbmrZhuCYL0nn+RkL036jMLw5F1xiu2ltEgK2/gNJhm29iBhvScUKOqUnbPw DSMZ2NipMqj7Xy3hjw1CStEy3pCXp8/muaB8KRnf92VvjO79VEls29KuX6rz32bcBM4qxsVn cOqyghSE69H3q4SY7EbhdIfacUSEUV+m/pZK5gnJIl6n1Rh6u0MFXWttvu0j9JEl92Ayj8u8 J/tYvFMpag3nTeC3I+arPSKpeWDX08oisrEp0Yw15r+6jbPjZNz7LvrYZ2fa3Am6KRn0zsFN BE6c/ycBEADZzcb88XlSiooYoEt3vuGkYoSkz7potX864MSNGekek1cwUrXeUdHUlw5zwPoC 4H5JF7D8q7lYoelBYJ+Mf0vdLzJLbbEtN5+v+s2UEbkDlnUQS1yRo1LxyNhJiXsQVr7WVA/c 8qcDWUYX7q/4Ckg77UO4l/eHCWNnHu7GkvKLVEgRjKPKroIEnjI0HMK3f6ABDReoc741RF5X X3qwmCgKZx0AkLjObXE3W769dtbNbWmW0lgFKe6dxlYrlZbq25Aubhcu2qTdQ/okx6uQ41+v QDxgYtocsT/CG1u0PpbtMeIm3mVQRXmjDFKjKAx9WOX/BHpk7VEtsNQUEp1lZo6hH7jeo5me CYFzgIbXdsMA9TjpzPpiWK9GetbD5KhnDId4ANMrWPNuGC/uPHDjtEJyf0cwknsRFLhL4/NJ KvqAuiXQ57x6qxrkuuinBQ3S9RR3JY7R7c3rqpWyaTuNNGPkIrRNyePky/ZTgTMA5of8Wioy z06XNhr6mG5xT+MHztKAQddV3xFy9f3Jrvtd6UvFbQPwG7Lv+/UztY5vPAzp7aJGz2pDbb0Q BC9u1mrHICB4awPlja/ljn+uuIb8Ow3jSy+Sx58VFEK7ctIOULdmnHXMFEihnOZO3NlNa6q+ XZOK7J00Ne6y0IBAaNTM+xMF+JRc7Gx6bChES9vxMyMbXwARAQABwsFfBBgBAgAJBQJOnP8n AhsMAAoJEOdnc4D1T9iphf4QAJuR1jVyLLSkBDOPCa3ejvEqp4H5QUogl1ASkEboMiWcQJQd LaH6zHNySMnsN6g/UVhuviANBxtW2DFfANPiydox85CdH71gLkcOE1J7J6Fnxgjpc1Dq5kxh imBSqa2hlsKUt3MLXbjEYL5OTSV2RtNP04KwlGS/xMfNwQf2O2aJoC4mSs4OeZwsHJFVF8rK XDvL/NzMCnysWCwjVIDhHBBIOC3mecYtXrasv9nl77LgffyyaAAQZz7yZcvn8puj9jH9h+mr L02W+gd+Sh6Grvo5Kk4ngzfT/FtscVGv9zFWxfyoQHRyuhk0SOsoTNYN8XIWhosp9GViyDtE FXmrhiazz7XHc32u+o9+WugpTBZktYpORxLVwf9h1PY7CPDNX4EaIO64oyy9O3/huhOTOGha nVvqlYHyEYCFY7pIfaSNhgZs2aV0oP13XV6PGb5xir5ah+NW9gQk/obnvY5TAVtgTjAte5tZ +coCSBkOU1xMiW5Td7QwkNmtXKHyEF6dxCAMK1KHIqxrBaZO27PEDSHaIPHePi7y4KKq9C9U 8k5V5dFA0mqH/st9Sw6tFbqPkqjvvMLETDPVxOzinpU2VBGhce4wufSIoVLOjQnbIo1FIqWg Dx24eHv235mnNuGHrG+EapIh7g/67K0uAzwp17eyUYlE5BMcwRlaHMuKTil6 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 30/6/26 11:26, Wenwu Hou wrote: > On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote: >> On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote: >>> The kernfs implementation has big lock granularity(kernfs_rwsem) so >>> every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete >>> the lock. Thus, if one of userspace goes the sleep under holding >>> the lock for a long time, rest of them should wait it. A example is >>> the holder goes direct reclaim with the lock since it needs memory >>> allocation. Let's fix it at common technique that release the lock >>> and then allocate the memory. Fortunately, kernfs looks like have >>> an refcount so I hope it's fine. >>> >>> Signed-off-by: Minchan Kim >>> --- >>> fs/kernfs/dir.c | 14 +++++++++++--- >>> fs/kernfs/inode.c | 2 +- >>> fs/kernfs/kernfs-internal.h | 1 + >>> 3 files changed, 13 insertions(+), 4 deletions(-) >> What workload hits this lock to cause it to be noticable? Hi Greg, ;) Sadly, perhaps by coincidence, I have a case that looks almost identical to this so I'm pretty interested in the discussion. This scenario didn't immediately occur to me looking at the vmcores but, yes, I think this is a serious (potential) problem. But I don't have enough information to put together a causal scenario either which is a problem. The type of systems that show these sort of problem are extremely heavy kernfs users (obviously) so sleeping, even under a read lock, can be a problem, again obviously. I'm not sure about dropping and re-taking the lock, although at first sight it looks like it would be ok, I just don't like this sort of execution continuity break but a refactor that avoids memory allocations under a lock seems like a very sensible thing to do. I don't yet have a feel for how far to go with it either. But given this code path is a very hot in the systems for which I get these problem reports the inode lookup op might be enough. >> >> There was a bunch of recent work in this area to make this much more >> fine-grained, and the theoritical benchmarks that people created (adding >> 10s of thousands of scsi disks at boot time) have gotten better. > Hi, > > By 2026, the kernfs_rwsem has been split into per-fs. But the problem still > exists. Yes, but for my case I'm not sue how much of that has made it into kernels people with these systems are using. I have done a fair bit to add these improvements but change is slow in larger environments. > > In a k8s cluster environment, there are a lot of processes reading /sys, for example: > > Container runtime: runc > Prometheus Exporter: node_exporter Indeed, these and a few other friends too, thend to show up again and again. > > If a tenant's container processes read /sys and sleep on the OOM (or direct > reclaim) path, both runc and node_exporter will be blocked, which will cause > the cluster worker node to become unschedulable. In the case I saw it looked like there was a directory remove going on and there was about 7M negative dentries, roughly double the positives, so I've been chasing info. to work out if the number of directory children was large enough to cause this. Perhaps, as demand grows over time, we will see other scenarios beside the allocation delays you have seen. > > For example, the nodejs runtime reads > /sys/devices/system/cpu/cpu%u/cpufreq/scaling_cur_freq [1] > > Jun 22 21:43:17 kernel: task:node state:D stack:0 pid:2621530 ppid:1062663 flags:0x00004002 > Jun 22 21:43:17 kernel: Call Trace: > Jun 22 21:43:17 kernel: > Jun 22 21:43:17 kernel: __schedule+0x278/0x740 > Jun 22 21:43:17 kernel: schedule+0x5a/0xd0 > Jun 22 21:43:17 kernel: schedule_preempt_disabled+0x11/0x20 > Jun 22 21:43:17 kernel: __mutex_lock.constprop.0+0x376/0x680 > Jun 22 21:43:17 kernel: ? kvm_sched_clock_read+0xd/0x20 > Jun 22 21:43:17 kernel: mem_cgroup_out_of_memory+0x53/0x150 > Jun 22 21:43:17 kernel: try_charge_memcg+0x682/0x7d0 > Jun 22 21:43:17 kernel: ? memcg_list_lru_alloc+0xa7/0x330 > Jun 22 21:43:17 kernel: obj_cgroup_charge+0x70/0x170 > Jun 22 21:43:17 kernel: slab_pre_alloc_hook.constprop.0+0xb6/0x1d0 > Jun 22 21:43:17 kernel: ? alloc_inode+0x59/0xc0 > Jun 22 21:43:17 kernel: kmem_cache_alloc_lru+0x54/0x2c0 > Jun 22 21:43:17 kernel: alloc_inode+0x59/0xc0 > Jun 22 21:43:17 kernel: iget_locked+0xe3/0x220 > Jun 22 21:43:17 kernel: kernfs_get_inode+0x18/0x110 > Jun 22 21:43:17 kernel: kernfs_iop_lookup+0x74/0xd0 > Jun 22 21:43:17 kernel: __lookup_slow+0x82/0x130 > Jun 22 21:43:17 kernel: walk_component+0xdb/0x150 > Jun 22 21:43:17 kernel: link_path_walk.part.0.constprop.0+0x240/0x380 > Jun 22 21:43:17 kernel: ? path_init+0x293/0x3d0 > Jun 22 21:43:17 kernel: path_openat+0x85/0x2a0 > Jun 22 21:43:17 kernel: ? seq_printf+0x8e/0xb0 > Jun 22 21:43:17 kernel: do_filp_open+0xb4/0x160 > Jun 22 21:43:17 kernel: ? __check_object_size.part.0+0x5e/0x130 > Jun 22 21:43:17 kernel: do_sys_openat2+0x91/0xc0 > Jun 22 21:43:17 kernel: __x64_sys_openat+0x53/0xa0 > Jun 22 21:43:17 kernel: do_syscall_64+0x35/0x80 > Jun 22 21:43:17 kernel: entry_SYSCALL_64_after_hwframe+0x78/0xe2 > > A rwsem writer is involved: > > Jun 22 21:46:17 kernel: task:kworker/u232:0 state:D stack:0 pid:2886179 ppid:2 flags:0x00004000 > Jun 22 21:46:17 kernel: Workqueue: netns cleanup_net > Jun 22 21:46:17 kernel: Call Trace: > Jun 22 21:46:17 kernel: > Jun 22 21:46:17 kernel: __schedule+0x278/0x740 > Jun 22 21:46:17 kernel: ? psi_group_change+0x226/0x3d0 > Jun 22 21:46:17 kernel: schedule+0x5a/0xd0 > Jun 22 21:46:17 kernel: schedule_preempt_disabled+0x11/0x20 > Jun 22 21:46:17 kernel: rwsem_down_write_slowpath+0x1e2/0x4f0 > Jun 22 21:46:17 kernel: down_write+0x57/0x60 > Jun 22 21:46:17 kernel: kernfs_remove_by_name_ns+0x38/0xc0 > Jun 22 21:46:17 kernel: remove_files+0x2b/0x70 > Jun 22 21:46:17 kernel: sysfs_remove_group+0x38/0x80 > Jun 22 21:46:17 kernel: sysfs_remove_groups+0x29/0x50 > Jun 22 21:46:17 kernel: ib_free_port_attrs+0x92/0x170 [ib_core] > Jun 22 21:46:17 kernel: rdma_dev_exit_net+0x117/0x1d0 [ib_core] > Jun 22 21:46:17 kernel: ops_exit_list+0x30/0x70 > Jun 22 21:46:17 kernel: cleanup_net+0x273/0x430 > Jun 22 21:46:17 kernel: process_one_work+0x18a/0x3a0 > Jun 22 21:46:17 kernel: worker_thread+0x277/0x3a0 > Jun 22 21:46:17 kernel: ? __pfx_worker_thread+0x10/0x10 > Jun 22 21:46:17 kernel: kthread+0xe1/0x110 > Jun 22 21:46:17 kernel: ? __pfx_kthread+0x10/0x10 > Jun 22 21:46:17 kernel: ret_from_fork+0x2d/0x50 > Jun 22 21:46:17 kernel: ? __pfx_kthread+0x10/0x10 > Jun 22 21:46:17 kernel: ret_from_fork_asm+0x1b/0x30 > Jun 22 21:46:17 kernel: This looks very much like the senario I've seen recently, that remove looked like a sub-tree removal to me at the time so the question I have is what's your dentry-state like? If I could get our customer to do so I'd be asking for a locking status report but I wouldn't want to use a debug kernel build, just a build the lock reporting enabled ... that;s just not going to happen. > > > In our real-world case, thousands of nodejs processes hit the cgroup memory > limit and sleep on the OOM path, runc gets blocked, and consequently the node > remains in an unschedulable state for many hours. Right, that does sound like a different workload to my case so maybe your not alone, ;) > > > Link: > - [1] https://github.com/nodejs/node/blob/v22.19.0/deps/uv/src/unix/linux.c#L1889 > >> But in that work, no one could find a real benchmark or use case that >> anyone could even notice this type of thing. What do you have that >> shows this? I wish I could be more helpful but I too am guilty of not being able to get enough info. and have no reproducer either. Ian