From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out30-111.freemail.mail.aliyun.com (out30-111.freemail.mail.aliyun.com [115.124.30.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 036D829D26B for ; Thu, 2 Jul 2026 01:53:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.111 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782957233; cv=none; b=CCkCYV17Du66Sq3suVBMIuOWu/DMYlJuWr1V1IBPdrN3n+AEElZeIrXGSEHq4SCJ9WTU2QGTcd1rThqu6ca5Rh+k+mW14SPkAhueX8iOF5SOcNOI9yobEZ3+OkDHxOcqCW34cepWIOtZSS11t652SxUW/yZjneM75FvYr92Snqk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782957233; c=relaxed/simple; bh=7j2DGLWgEIzf1jdSur1120dsO5h7elCpn8csBiUkz+o=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=Z/ah7f2DKoTAfauaLqaMHjupHxPt6P2rfsDC2a0czHk0wFIdRqWrFxHr6F7cHPVmBFoKrUYHRYvFVYFPq+UQOR2jwkhTh3fqmlQHj4pmaZZ9CXqSxw1bwmc0R2lvCUMXWxM7NeQXHYQeAgQEt5dJUz1O13HfxlLBd3KbGMUeWA0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=dTOSjmpr; arc=none smtp.client-ip=115.124.30.111 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="dTOSjmpr" DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1782957227; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; bh=aCdQZWdDSpFmbhuq+yfAxjlCHtgqeTO1sTjh6FuuDtA=; b=dTOSjmprnN/dDXbQ/Hb0TR8ieG2EKVVPs6LnzEogT2dJb+BeWs6D75YooTaurz98g/RgYGdZrUASJFcDV7ysp//OhMkx/yfIZV+apndv7EPHn8gEfv8pVDNiC4pyzvyLtaOsAFe7CPWe7t4pQnAQ2oNkZ1hy4vQhgK4KfNsdPKw= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R581e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037033178;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=9;SR=0;TI=SMTPD_---0X6CCsq7_1782957226; Received: from 30.74.144.121(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0X6CCsq7_1782957226 cluster:ay36) by smtp.aliyun-inc.com; Thu, 02 Jul 2026 09:53:46 +0800 Message-ID: <20ca5e4a-fa83-4471-8f8c-17bca1d31be7@linux.alibaba.com> Date: Thu, 2 Jul 2026 09:53:45 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [BUG] shmem: shmem_get_folio_gfp livelock To: Kairui Song Cc: =?UTF-8?B?6ams6LaF?= , Andrew Morton , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , =?UTF-8?B?55Sw5a2d5paM?= , =?UTF-8?B?5L+e5Lic5paM?= , =?UTF-8?B?5p2O6bmP56iL?= , "hughd@google.com" References: <126cb4ced14f4a3fa40c3189bf8a5920@xiaomi.com> <49858bc642844e3bbf6449c0f241af04@xiaomi.com> <700a2cbf90a2484f979aac858f08f5d4@xiaomi.com> <623db71c-daab-451c-909f-a8efa56b998b@linux.alibaba.com> From: Baolin Wang In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 7/2/26 1:25 AM, Kairui Song wrote: > On Wed, Jul 1, 2026 at 6:09 PM Baolin Wang > wrote: >> >> CC Hugh and Kairui. > > Hello! > >> >> On 6/30/26 9:15 PM, 马超 wrote: >>> Hello, >>> I encountered a bug in the shmem subsystem. Details below. >>> >>> [Summary] >>> shmem_get_folio_gfp() can livelock when multiple threads fault on the >>> same shmem page concurrently. The -EEXIST retry loop (goto repeat) has >>> no cond_resched(), causing busy-looping threads to starve the thread >>> that holds the swapcache slot, resulting in an indefinite RCU stall and >>> system hang. >>> [Environment] >>> 1.Kernel: 6.18.21 (ARM64, PREEMPT, CONFIG_LRU_GEN=y) >>> 2.Triggered by: multi-threaded app with threads constrained to 2 CPUs >>> via cpuset >>> [Root Cause] >>> When multiple threads in the same process fault on the same shmem >>> swap entry: >>> 1.Thread A enters shmem_swap_alloc_folio(), succeeds at >>> swapcache_prepare() (sets SWAP_HAS_CACHE), then enters >>> workingset_refault() → lru_gen_refault() → rcu_read_lock(). >>> While inside the RCU read-side critical section, it is >>> preempted via preempt_schedule_irq (IRQ exit path detects >>> TIF_NEED_RESCHED). >>> 2.Threads B & C enter shmem_swap_alloc_folio(), fail at >>> swapcache_prepare() (slot already taken by A), return -EEXIST. >>> 3.In shmem_get_folio_gfp(): >>> error = shmem_swapin_folio(...); >>> if (error == -EEXIST) >>> goto repeat;// no cond_resched(), tight loop >>> 4.Threads B & C spin at 100% CPU on the retry loop. All three >>> threads share the same cpuset (CPU0-1,cpus_allowed=0x3).Thread A >>> is perpetually preempted and starved — it cannot complete the >>> few instructions needed to call rcu_read_unlock(). >>> 5.The held RCU read lock blocks the grace period indefinitely, >>> causing all synchronize_rcu() callers (cgroup operations, >>> fd allocation, etc.) to hang, eventually blocking init. >>> [Scheduling Details] >>> Key observations: >>> 1.Thread A was RCU-boosted to prio 98 but accumulated only >>> 99ms of execution over the entire stall period (~1200s). >>> It was effectively starved despite the priority boost. >>> 2.Threads B & C have vruntime=0 and prio 91, indicating >>> they run in an RT-equivalent scheduling class (SCHED_FIFO/RT >>> policy). Each accumulated ~1134 seconds of execution with >>> only ~1600 context switches, meaning they ran uninterrupted >>> for ~700ms per scheduling quantum on average. >>> 3.Thread A cannot preempt Threads B & C: Although RCU boost >>> raised Thread A to prio 98, Threads B & C at prio 91 (lower >>> numeric value = higher priority in RT class) have equal or >>> higher effective priority. The busy-looping threads never >>> voluntarily yield (no cond_resched(), no blocking calls in >>> the loop), so Thread A never gets scheduled. >>> 4.CPU contention: CPU0 had nr_running=28 and CPU1 had >>> nr_running=24, with 3-4 RT tasks per CPU. Thread A competed >>> with Thread B on CPU0 but could not win scheduling. >>> [Observed Impact] >>> 1.RCU stall lasting 910+ seconds (19 consecutive stall >>> warnings, grace period g=4398761 never advanced) >>> 2.synchronize_rcu_expedited() callers blocked 742+ seconds >>> 3.init process hung > 720 seconds → system unresponsive >>> [Call Traces] >>> Thread A (RCU stall source, sampled 19 times identically): >>> __switch_to+0x1a4/0x360 (T) >>> __schedule+0x96c/0xf3c >>> preempt_schedule_irq+0xec/0x198 >>> raw_irqentry_exit_cond_resched+0x2c/0x44 >>> irqentry_exit+0x38/0x64 >>> exit_to_kernel_mode+0x28/0x38 >>> el1_interrupt+0x5c/0xa8 >>> el1h_64_irq_handler+0x18/0x24 >>> el1h_64_irq+0x84/0x88 >>> workingset_refault+0x16c/0x79c (P) >>> shmem_swapin_folio+0x8e4/0xd44 >>> shmem_get_folio_gfp+0xb8/0x710 >>> shmem_fault+0xa0/0x174 >>> __do_fault >>> do_pte_missing >>> handle_mm_fault >>> do_page_fault >>> el0_ia >>> >>> Thread B (busy-loop on CPU0, sum_exec_runtime=1134s): >>> xas_load+0x78/0xe4 (P) >>> shmem_swapin_folio+0x950/0xd44 >>> shmem_get_folio_gfp+0xb8/0x710 >>> shmem_fault → ... → el0_ia >>> >>> Thread C (busy-loop on CPU1, sum_exec_runtime=1134s): >>> xas_load+0x50/0xe4 (P) >>> shmem_swapin_folio+0xd8/0xd44 >>> shmem_get_folio_gfp+0xb8/0x710 >>> shmem_fault → ... → el0_ia >>> [Question] >>> What is the recommended approach to fix this livelock? >>> We are considering adding a cond_resched() before the >>> goto repeat in shmem_get_folio_gfp() to break the tight >>> loop and allow the swapcache-holding thread to make >>> progress. Would this be an acceptable fix, or is there >>> a better strategy (e.g., bounded retry with fallback, >>> or yielding to the specific waiter)? > > So this is a 6.18 issue, the SWAP_HAS_CACHE design was a long time > issue and we had a workaround for anon in commit 13ddaf26be32, the > `schedule_timeout_uninterruptible(1)`. That's fine as a workaroud and > we remove the workaround with proper redesign later in commit > f1879e8a0c60. Shmem never used a workaround like > schedule_timeout_uninterruptible and we jump to the right design > directly. > > For 6.18 LTS, do we need a fix though? I think we can have a similar > timeout for shmem as well, just do a > schedule_timeout_uninterruptible(1), yes it's really ugly, and we have > another long time workaround like that in mm/swap_state.c as well in > 6.18. Without the later proper redesign there seems no better way. > Barry provided an extra improvement on top of that workaround in > commit 01626a182302, maybe can also be carried into shmem. So maybe > having both for shmem is a good choice for a 6.18 fix. Thanks for the explanation. I think we need a fix for 6.18 LTS, and it sounds reasonable to me to follow what anon folios did. Would you mind sending a proper fix for this issue? If you don't have time, I can fix it based on your suggestion.