From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1762812AbYEORqP (ORCPT ); Thu, 15 May 2008 13:46:15 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752863AbYEORp7 (ORCPT ); Thu, 15 May 2008 13:45:59 -0400 Received: from wx-out-0506.google.com ([66.249.82.225]:43327 "EHLO wx-out-0506.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751823AbYEORp5 (ORCPT ); Thu, 15 May 2008 13:45:57 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=PpMNs5u1D/j1Ex6X59XQjJfhGCwrNoJqwz65kU8lu38ctohCAmnCtsiG1qeUClL8GoplCX4BLf7RRAn6IpKu7vsq6xIns96OQAml0T5UsPh1FVUWROua5xCqTUSfHHBX86WGqdMKO1wmtyA6dksTz1aR7zuiJ8k6wMVijnDffIY= Message-ID: Date: Thu, 15 May 2008 21:45:55 +0400 From: "Alexander Beregalov" To: "David Chinner" Subject: Re: 2.6.26-rc1: possible circular locking dependency with xfs filesystem Cc: "Kamalesh Babulal" , pvp-lsts@fs.ru.acad.bg, kernel-testers@vger.kernel.org, "kernel list" , "Ingo Molnar" , peterz@infradead.org, xfs@oss.sgi.com In-Reply-To: <20080511231002.GN103491721@sgi.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4825DF71.1030209@linux.vnet.ibm.com> <48266C77.3040102@linux.vnet.ibm.com> <20080511231002.GN103491721@sgi.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 2008/5/12 David Chinner : > On Sun, May 11, 2008 at 09:18:07AM +0530, Kamalesh Babulal wrote: >> Kamalesh Babulal wrote: >> > Adding the cc to kernel-list, Ingo Molnar and Peter Zijlstra >> > >> > Alexander Beregalov wrote: >> >> [ INFO: possible circular locking dependency detected ] >> >> 2.6.26-rc1-00279-g28a4acb #13 >> >> ------------------------------------------------------- >> >> nfsd/3087 is trying to acquire lock: >> >> (iprune_mutex){--..}, at: [] shrink_icache_memory+0x38/0x19b >> >> >> >> but task is already holding lock: >> >> (&(&ip->i_iolock)->mr_lock){----}, at: [] xfs_ilock+0xa2/0xd6 >> >> >> >> which lock already depends on the new lock. >> >> >> >> >> >> the existing dependency chain (in reverse order) is: >> >> >> >> -> #1 (&(&ip->i_iolock)->mr_lock){----}: >> >> [] __lock_acquire+0xa0c/0xbc6 >> >> [] lock_acquire+0x6a/0x86 >> >> [] down_write_nested+0x33/0x6a >> >> [] xfs_ilock+0x7b/0xd6 >> >> [] xfs_ireclaim+0x1d/0x59 >> >> [] xfs_finish_reclaim+0x173/0x195 >> >> [] xfs_reclaim+0xb3/0x138 >> >> [] xfs_fs_clear_inode+0x55/0x8e >> >> [] clear_inode+0x83/0xd2 >> >> [] dispose_list+0x3c/0xc1 >> >> [] shrink_icache_memory+0x173/0x19b >> >> [] shrink_slab+0xda/0x14e >> >> [] try_to_free_pages+0x1e4/0x2a2 >> >> [] __alloc_pages_internal+0x23a/0x39d >> >> [] __alloc_pages+0xa/0xc >> >> [] __do_page_cache_readahead+0xaa/0x16a >> >> [] force_page_cache_readahead+0x4a/0x74 >> >> [] sys_madvise+0x308/0x400 >> >> [] sysenter_past_esp+0x6a/0xb1 >> >> [] 0xffffffff >> >> >> >> -> #0 (iprune_mutex){--..}: >> >> [] __lock_acquire+0x929/0xbc6 >> >> [] lock_acquire+0x6a/0x86 >> >> [] mutex_lock_nested+0xb4/0x226 >> >> [] shrink_icache_memory+0x38/0x19b >> >> [] shrink_slab+0xda/0x14e >> >> [] try_to_free_pages+0x1e4/0x2a2 >> >> [] __alloc_pages_internal+0x23a/0x39d >> >> [] __alloc_pages+0xa/0xc >> >> [] __do_page_cache_readahead+0xaa/0x16a >> >> [] ondemand_readahead+0x119/0x127 >> >> [] page_cache_async_readahead+0x52/0x5d >> >> [] generic_file_splice_read+0x290/0x4a8 >> >> [] xfs_splice_read+0x4b/0x78 >> >> [] xfs_file_splice_read+0x24/0x29 >> >> [] do_splice_to+0x45/0x63 >> >> [] splice_direct_to_actor+0xab/0x150 >> >> [] nfsd_vfs_read+0x1ed/0x2d0 >> >> [] nfsd_read+0x82/0x99 >> >> [] nfsd3_proc_read+0xdf/0x12a >> >> [] nfsd_dispatch+0xcf/0x19e >> >> [] svc_process+0x3b3/0x68b >> >> [] nfsd+0x168/0x26b >> >> [] kernel_thread_helper+0x7/0x10 >> >> [] 0xffffffff > > Oh, yeah, that. Direct inode reclaim through memory pressure. > > Effectively memory reclaim inverts locking order w.r.t. iprune_mutex > when it recurses into the filesystem. False positive - can never > cause a deadlock on XFS. Can't be solved from the XFS side of things > without effectively turning off lockdep checking for xfs inode > locking. Yes, it is not a deadlock, but machine hangs for few seconds. It still happens about once a day for me. Every kernel report looks similar to the above. I cannot reproduce it quickly, so bisect is not possible. > > The fix is needed to lockdep via iprune_mutex annotations here.... > >> May 9 02:16:46 nomad64 kernel: [42951853.992965] the existing dependency chain (in reverse order) is: >> May 9 02:16:46 nomad64 kernel: [42951853.992967] >> May 9 02:16:46 nomad64 kernel: [42951853.992968] -> #1 (&(&ip->i_iolock)->mr_lock){----}: >> May 9 02:16:46 nomad64 kernel: [42951853.992974] [] __lock_acquire+0xf92/0x1080 >> May 9 02:16:46 nomad64 kernel: [42951853.992989] [] lock_acquire+0xa2/0xd0 >> May 9 02:16:46 nomad64 kernel: [42951853.993002] [] down_write_nested+0x46/0x80 >> May 9 02:16:46 nomad64 kernel: [42951853.993018] [] xfs_ilock+0x99/0xa0 >> May 9 02:16:46 nomad64 kernel: [42951853.993034] [] xfs_free_eofblocks+0x1c7/0x250 >> May 9 02:16:46 nomad64 kernel: [42951853.993049] [] xfs_release+0x186/0x1d0 >> May 9 02:16:46 nomad64 kernel: [42951853.993062] [] xfs_file_release+0x10/0x20 >> May 9 02:16:46 nomad64 kernel: [42951853.993076] [] __fput+0xcc/0x1c0 >> May 9 02:16:46 nomad64 kernel: [42951853.993091] [] fput+0x16/0x20 >> May 9 02:16:46 nomad64 kernel: [42951853.993105] [] remove_vma+0x4a/0x80 >> May 9 02:16:46 nomad64 kernel: [42951853.993120] [] do_munmap+0x281/0x2e0 >> May 9 02:16:46 nomad64 kernel: [42951853.993134] [] sys_munmap+0x4b/0x70 >> May 9 02:16:46 nomad64 kernel: [42951853.993148] [] system_call_after_swapgs+0x7b/0x80 >> May 9 02:16:46 nomad64 kernel: [42951853.993161] [] 0xffffffffffffffff > > hmmmm. Sounds like: > > fd = open() > addr = mmap(fd) > close(fd) > ..... > munmap(addr); > > But yes, XFS takes locks in ->release which means..... > >> May 9 02:16:46 nomad64 kernel: [42951853.993293] Call Trace: >> May 9 02:16:46 nomad64 kernel: [42951853.993297] [] print_circular_bug_tail+0x83/0x90 >> May 9 02:16:46 nomad64 kernel: [42951853.993302] [] __lock_acquire+0xdb0/0x1080 >> May 9 02:16:46 nomad64 kernel: [42951853.993306] [] ? do_page_fault+0xdd/0x890 >> May 9 02:16:46 nomad64 kernel: [42951853.993310] [] lock_acquire+0xa2/0xd0 >> May 9 02:16:46 nomad64 kernel: [42951853.993313] [] ? do_page_fault+0xdd/0x890 >> May 9 02:16:46 nomad64 kernel: [42951853.993317] [] down_read+0x3b/0x70 >> May 9 02:16:46 nomad64 kernel: [42951853.993320] [] do_page_fault+0xdd/0x890 >> May 9 02:16:46 nomad64 kernel: [42951853.993324] [] error_exit+0x0/0xa9 >> May 9 02:16:46 nomad64 kernel: [42951853.993328] [] ? file_read_actor+0x46/0x1b0 >> May 9 02:16:46 nomad64 kernel: [42951853.993331] [] ? _read_unlock_irq+0x36/0x60 >> May 9 02:16:46 nomad64 kernel: [42951853.993335] [] ? generic_file_aio_read+0x2cc/0x5d0 >> May 9 02:16:46 nomad64 kernel: [42951853.993339] [] ? get_lock_stats+0x19/0x70 >> May 9 02:16:46 nomad64 kernel: [42951853.993343] [] ? xfs_read+0x139/0x220 >> May 9 02:16:46 nomad64 kernel: [42951853.993347] [] ? xfs_file_aio_read+0x4d/0x60 >> May 9 02:16:46 nomad64 kernel: [42951853.993350] [] ? do_sync_read+0xf1/0x130 >> May 9 02:16:46 nomad64 kernel: [42951853.993354] [] ? autoremove_wake_function+0x0/0x40 >> May 9 02:16:46 nomad64 kernel: [42951853.993358] [] ? trace_hardirqs_on+0xda/0x170 >> May 9 02:16:46 nomad64 kernel: [42951853.993361] [] ? __rcu_read_unlock+0xb5/0xc0 >> May 9 02:16:46 nomad64 kernel: [42951853.993365] [] ? trace_hardirqs_on+0xda/0x170 >> May 9 02:16:46 nomad64 kernel: [42951853.993369] [] ? security_file_permission+0x11/0x20 >> May 9 02:16:46 nomad64 kernel: [42951853.993374] [] ? vfs_read+0xc4/0x160 >> May 9 02:16:46 nomad64 kernel: [42951853.993377] [] ? sys_read+0x50/0x90 >> May 9 02:16:46 nomad64 kernel: [42951853.993380] [] ? system_call_after_swapgs+0x7b/0x80 > > Oh, joy - a page fault during a read() call triggers lock order > inversions on the mmap->sem. I don't think this can deadlock > (can't be page faulting in a vma that is being torn down), but > it's clear from the last trace that the VM has a mmap->sem > inversion problem with ->release vs ->read and page faults... > > Basically what we are seeing here in both cases is that the VM is > calling inode ->release or ->clear_inode methods with different high > level locks held. If the filesystem has to take the same locks in > these methods as it does in, say, ->read (like XFS does), then we > are guaranteed to get reports like this. AFAICT there's nothing we > can do from the filesystem perspective to prevent false positives like > this from being reported.... > > Cheers, > > Dave. > -- > Dave Chinner > Principal Engineer > SGI Australian Software Group > -- > To unsubscribe from this list: send the line "unsubscribe kernel-testers" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >