From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757775AbYCVWOn (ORCPT ); Sat, 22 Mar 2008 18:14:43 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752570AbYCVWOg (ORCPT ); Sat, 22 Mar 2008 18:14:36 -0400 Received: from smtp1.linux-foundation.org ([140.211.169.13]:56463 "EHLO smtp1.linux-foundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752637AbYCVWOf (ORCPT ); Sat, 22 Mar 2008 18:14:35 -0400 Date: Sat, 22 Mar 2008 15:13:39 -0700 From: Andrew Morton To: Christian Kujau Cc: "Rafael J. Wysocki" , LKML , Greg KH , Tejun Heo , Kay Sievers , xfs-masters@oss.sgi.com Subject: Re: 2.6.25-rc6: kernel BUG at fs/sysfs/file.c:89 Message-Id: <20080322151339.adb146bd.akpm@linux-foundation.org> In-Reply-To: References: <200803220226.35552.rjw@sisk.pl> <20080321222606.16407d74.akpm@linux-foundation.org> X-Mailer: Sylpheed 2.4.8 (GTK+ 2.12.5; x86_64-redhat-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, 22 Mar 2008 22:54:51 +0100 (CET) Christian Kujau wrote: > On Sat, 22 Mar 2008, Christian Kujau wrote: > >> If so, the below (already merged) patch should fix this crash. > >> If this patch does not fix it then please apply this debug patch: > >> http://userweb.kernel.org/~akpm/mmotm/broken-out/gregkh-driver-driver-core-debug-for-bad-dev_attr_show-return-value.patch > >> then rerun the test. > > > > Ah, sorry, I misread your post: I applied the aforementioned dm-crypt > > patch[0] and Greg's debug patch and got quite nasty SCSI errors, leading to a > > complete lockup. I'll try again and only apply Neil's patch...stay tuned... > > Hm, this was strange: I applied Neil's patch (and the dm-crypt patch) on > 2.6.25-rc6 and I kept getting SCSI errors (and lockups) when doing "tar > -cf - | dd of=/dev/null" - which I did to generate disk I/O. > > Doing the same as a normal user, nothing bad happened (except a > "possible circular locking dependency" warning, copied below) and the box > is still up & running for a few hours now, with a constant read of ~30MB/s > across (md-)disks (running tar, rsync). So all in all I'm very happy, > because with these two patches applied, 2.6.25-rc seems to be usable > again. > > Of course, it'd be interesting to know where the scsi errors come from, > but that's another story I guess... > > Thanks to all involved, > Christian. > > [ 4657.715881] ======================================================= > [ 4657.716512] [ INFO: possible circular locking dependency detected ] > [ 4657.716895] 2.6.25-rc6 #5 > [ 4657.717170] ------------------------------------------------------- > [ 4657.717552] rsync/14184 is trying to acquire lock: > [ 4657.717892] (iprune_mutex){--..}, at: [] shrink_icache_memory+0x72/0x220 > [ 4657.718239] > [ 4657.718239] but task is already holding lock: > [ 4657.718463] (&(&ip->i_iolock)->mr_lock){----}, at: [] xfs_ilock+0x96/0xb0 > [ 4657.718714] > [ 4657.718715] which lock already depends on the new lock. > [ 4657.718716] > [ 4657.719047] > [ 4657.719047] the existing dependency chain (in reverse order) is: > [ 4657.719291] > [ 4657.719292] -> #1 (&(&ip->i_iolock)->mr_lock){----}: > [ 4657.719526] [] add_lock_to_list+0x44/0xc0 > [ 4657.719779] [] __lock_acquire+0xc26/0x10b0 > [ 4657.720142] [] xfs_ilock+0x96/0xb0 > [ 4657.720483] [] mark_held_locks+0x3d/0x70 > [ 4657.720483] [] lock_acquire+0x5e/0x80 > [ 4657.720483] [] xfs_ilock+0x96/0xb0 > [ 4657.720483] [] down_write_nested+0x41/0x60 > [ 4657.720483] [] xfs_ilock+0x96/0xb0 > [ 4657.720483] [] xfs_ilock+0x96/0xb0 > [ 4657.720483] [] xfs_ireclaim+0x1a/0x60 > [ 4657.720483] [] xfs_finish_reclaim+0x53/0x1a0 > [ 4657.720483] [] xfs_fs_clear_inode+0x5e/0x90 > [ 4657.720483] [] clear_inode+0xa9/0x130 > [ 4657.720483] [] destroy_inode+0x20/0x40 > [ 4657.720483] [] dispose_list+0x1a/0xc0 > [ 4657.720483] [] shrink_icache_memory+0x1e2/0x220 > [ 4657.720483] [] shrink_slab+0x101/0x160 > [ 4657.720483] [] kswapd+0x298/0x3f0 > [ 4657.720483] [] isolate_pages_global+0x0/0x60 > [ 4657.720483] [] autoremove_wake_function+0x0/0x40 > [ 4657.720483] [] trace_hardirqs_on+0x9c/0x110 > [ 4657.720483] [] kswapd+0x0/0x3f0 > [ 4657.720483] [] kthread+0x42/0x70 > [ 4657.720483] [] kthread+0x0/0x70 > [ 4657.720483] [] kernel_thread_helper+0x7/0x18 > [ 4657.720483] [] 0xffffffff > [ 4657.720483] > [ 4657.720483] -> #0 (iprune_mutex){--..}: > [ 4657.720483] [] print_circular_bug_entry+0x40/0x50 > [ 4657.720483] [] __lock_acquire+0xa27/0x10b0 > [ 4657.720483] [] __lock_acquire+0x18f/0x10b0 > [ 4657.720483] [] lock_acquire+0x5e/0x80 > [ 4657.720483] [] shrink_icache_memory+0x72/0x220 > [ 4657.720483] [] mutex_lock_nested+0x89/0x240 > [ 4657.720483] [] shrink_icache_memory+0x72/0x220 > [ 4657.720483] [] shrink_icache_memory+0x72/0x220 > [ 4657.720483] [] shrink_icache_memory+0x72/0x220 > [ 4657.720483] [] shrink_slab+0x21/0x160 > [ 4657.720483] [] shrink_slab+0x101/0x160 > [ 4657.720483] [] try_to_free_pages+0x152/0x230 > [ 4657.720483] [] isolate_pages_global+0x0/0x60 > [ 4657.720483] [] __alloc_pages+0x14b/0x370 > [ 4657.720483] [] _read_unlock_irq+0x20/0x30 > [ 4657.720483] [] __grab_cache_page+0x81/0xc0 > [ 4657.720483] [] block_write_begin+0x76/0xe0 > [ 4657.720483] [] xfs_vm_write_begin+0x46/0x50 > [ 4657.720483] [] xfs_get_blocks+0x0/0x30 > [ 4657.720483] [] generic_file_buffered_write+0x117/0x650 > [ 4657.720483] [] __mark_inode_dirty+0x53/0x180 > [ 4657.720483] [] _spin_lock+0x29/0x40 > [ 4657.720483] [] __mark_inode_dirty+0x53/0x180 > [ 4657.720483] [] xfs_write+0x7ac/0x8a0 > [ 4657.720483] [] core_sys_select+0x21/0x350 > [ 4657.720483] [] xfs_file_aio_write+0x5c/0x70 > [ 4657.720483] [] do_sync_write+0xd5/0x120 > [ 4657.720483] [] autoremove_wake_function+0x0/0x40 > [ 4657.720483] [] dnotify_parent+0x35/0x90 > [ 4657.720483] [] do_sync_write+0x0/0x120 > [ 4657.720483] [] vfs_write+0x9f/0x140 > [ 4657.720483] [] sys_write+0x41/0x70 > [ 4657.720483] [] sysenter_past_esp+0x5f/0xa5 > [ 4657.720483] [] 0xffffffff > [ 4657.720483] > [ 4657.720483] other info that might help us debug this: > [ 4657.720483] > [ 4657.720483] 3 locks held by rsync/14184: > [ 4657.720483] #0: (&sb->s_type->i_mutex_key#12){--..}, at: [] xfs_write+0x3f9/0x8a0 > [ 4657.720483] #1: (&(&ip->i_iolock)->mr_lock){----}, at: [] xfs_ilock+0x96/0xb0 > [ 4657.720483] #2: (shrinker_rwsem){----}, at: [] shrink_slab+0x21/0x160 > [ 4657.720483] > [ 4657.720483] stack backtrace: > [ 4657.720483] Pid: 14184, comm: rsync Not tainted 2.6.25-rc6 #5 > [ 4657.720483] [] print_circular_bug_tail+0x72/0x80 > [ 4657.720483] [] __lock_acquire+0xa27/0x10b0 > [ 4657.720483] [] __lock_acquire+0x18f/0x10b0 > [ 4657.720483] [] lock_acquire+0x5e/0x80 > [ 4657.720483] [] shrink_icache_memory+0x72/0x220 > [ 4657.720483] [] mutex_lock_nested+0x89/0x240 > [ 4657.720483] [] shrink_icache_memory+0x72/0x220 > [ 4657.720483] [] shrink_icache_memory+0x72/0x220 > [ 4657.720483] [] shrink_icache_memory+0x72/0x220 > [ 4657.720483] [] shrink_slab+0x21/0x160 > [ 4657.720483] [] shrink_slab+0x101/0x160 > [ 4657.720483] [] try_to_free_pages+0x152/0x230 > [ 4657.720483] [] isolate_pages_global+0x0/0x60 > [ 4657.720483] [] __alloc_pages+0x14b/0x370 > [ 4657.720483] [] _read_unlock_irq+0x20/0x30 > [ 4657.720483] [] __grab_cache_page+0x81/0xc0 > [ 4657.720483] [] block_write_begin+0x76/0xe0 > [ 4657.720483] [] xfs_vm_write_begin+0x46/0x50 > [ 4657.720483] [] xfs_get_blocks+0x0/0x30 > [ 4657.720483] [] generic_file_buffered_write+0x117/0x650 > [ 4657.720483] [] __mark_inode_dirty+0x53/0x180 > [ 4657.720483] [] _spin_lock+0x29/0x40 > [ 4657.720483] [] __mark_inode_dirty+0x53/0x180 > [ 4657.720483] [] xfs_write+0x7ac/0x8a0 > [ 4657.720483] [] core_sys_select+0x21/0x350 > [ 4657.720483] [] xfs_file_aio_write+0x5c/0x70 > [ 4657.720483] [] do_sync_write+0xd5/0x120 > [ 4657.720483] [] autoremove_wake_function+0x0/0x40 > [ 4657.720483] [] dnotify_parent+0x35/0x90 > [ 4657.720483] [] do_sync_write+0x0/0x120 > [ 4657.720483] [] vfs_write+0x9f/0x140 > [ 4657.720483] [] sys_write+0x41/0x70 > [ 4657.720483] [] sysenter_past_esp+0x5f/0xa5 > [ 4657.720483] ======================= > That's an XFS bug. To clarify, I believe you are running 2.6.25-rc6 plus http://lkml.org/lkml/2008/3/22/8 plus some dm-crypt patch?