From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965636AbXCVDtN (ORCPT ); Wed, 21 Mar 2007 23:49:13 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S965640AbXCVDtN (ORCPT ); Wed, 21 Mar 2007 23:49:13 -0400 Received: from smtp.osdl.org ([65.172.181.24]:37592 "EHLO smtp.osdl.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965636AbXCVDtL (ORCPT ); Wed, 21 Mar 2007 23:49:11 -0400 Date: Wed, 21 Mar 2007 20:45:40 -0700 (PDT) From: Linus Torvalds To: Adrian Bunk cc: Andrew Morton , Linux Kernel Mailing List , Michal Piotrowski , Mariusz Kozlowski , Oliver Pinter , Sid Boyce , Nick Piggin , Jens Axboe Subject: Re: [1/6] 2.6.21-rc4: known regressions In-Reply-To: <20070318184908.GU752@stusta.de> Message-ID: References: <20070318184908.GU752@stusta.de> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org On Sun, 18 Mar 2007, Adrian Bunk wrote: > > Subject : weird system hangs > References : http://lkml.org/lkml/2007/3/16/288 > Submitter : Michal Piotrowski > Mariusz Kozlowski > Status : unknown According to the console log, it seems to be hung because a lot of processes are stuck in D state in various variations of this: Call Trace: [] start_this_handle+0x2d7/0x355 [] journal_start+0xb3/0xe1 [] ext3_journal_start_sb+0x48/0x4a [] ext3_create+0x47/0xe2 [] vfs_create+0xcd/0x13e [] open_namei+0x176/0x5b5 [] do_filp_open+0x26/0x3b [] do_sys_open+0x43/0xc2 [] sys_open+0x1c/0x1e [] syscall_call+0x7/0xb and then you have "kget" (whatever that is) which is doing Call Trace: [] schedule_timeout+0x70/0x8e [] schedule_timeout_uninterruptible+0x15/0x17 [] journal_stop+0xe2/0x1e6 [] journal_force_commit+0x1d/0x1f [] ext3_force_commit+0x22/0x24 [] ext3_write_inode+0x34/0x3a [] __writeback_single_inode+0x1c5/0x2cb [] sync_inode+0x1c/0x2e [] ext3_sync_file+0xab/0xc0 [] do_fsync+0x4b/0x98 [] __do_fsync+0x20/0x2f [] sys_fdatasync+0x10/0x12 [] syscall_call+0x7/0xb with kjournald in D sleep at [] journal_commit_transaction+0x15d/0x11d3 [] kjournald+0xab/0x1e8 [] kthread+0xb5/0xe0 [] kernel_thread_helper+0x7/0x10 which certainly looks like something is waiting for an IO to finish. In contrast, the hang reported by Mariusz Kozlowski has a slightly different feel to it, but there's a tantalizing pattern in there too: http://www.ussg.iu.edu/hypermail/linux/kernel/0703.0/1243.html Call Trace: [] io_schedule+0x42/0x59 [] sleep_on_buffer+0x8/0xc [] __wait_on_bit+0x47/0x6c [] out_of_line_wait_on_bit+0x5b/0x64 [] __wait_on_buffer+0x27/0x2d [] journal_commit_transaction+0x707/0x127f [] kjournald+0xac/0x1ed [] kthread+0xa2/0xc9 [] kernel_thread_helper+0x7/0x1c which certainly also looks like an IO never completed (or completed but never woke anything up). It also seems to be related to *buffers*. Maybe the whole bh layer thing is a fluke, but it's not waiting for normal data, it's very much waiting for those journal things that all use buffer heads.Which just makes me worry about those patches by Nick (which did come in through Andrew). I don't think it's the memorder one (it looks safe and shouldn't matter on x86 anyway!), but what about the fs: fix __block_write_full_page error case buffer submission locking change for example? Or that "fs: fix nobh data leak" thing with its fix? It uses "SetPageUptodate(page);" without waking up anybody who might wait for it (but the waiters here seem to wait on buffers, so that's probably not it).. Alternatively, maybe it really is an _io_ problem (and the buffer-head thing is just a red herring, and it could happen to other IO, it's just that metadata IO uses buffer heads), and it's the scheduler changes since 2.6.20.. Jens, Nick.. Could you take a look? Linus