From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S261261AbULWPz5 (ORCPT ); Thu, 23 Dec 2004 10:55:57 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S261262AbULWPz5 (ORCPT ); Thu, 23 Dec 2004 10:55:57 -0500 Received: from acheron.informatik.uni-muenchen.de ([129.187.214.135]:41933 "EHLO acheron.informatik.uni-muenchen.de") by vger.kernel.org with ESMTP id S261261AbULWPzQ (ORCPT ); Thu, 23 Dec 2004 10:55:16 -0500 Message-ID: <41CAEA62.4060903@bio.ifi.lmu.de> Date: Thu, 23 Dec 2004 16:55:14 +0100 From: Frank Steiner User-Agent: Mozilla Thunderbird 1.0 (X11/20041207) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Denis Vlasenko Cc: Linux Kernel Mailing List Subject: Re: kblockd/1: page allocation failure in 2.6.9 References: <41C7D32D.2010809@bio.ifi.lmu.de> <41CAAB61.3030704@bio.ifi.lmu.de> <200412231551.15767.vda@port.imtp.ilyichevsk.odessa.ua> In-Reply-To: <200412231551.15767.vda@port.imtp.ilyichevsk.odessa.ua> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Hi, Denis Vlasenko wrote > On Thursday 23 December 2004 11:26, Frank Steiner wrote: >>No one able to give me least some hints what this message could mean? >>If it is sth. like a kernel oops or sth. else? If it could point to >>a hardware error, or kernel bug, or kernel configuration problem or sth? >> >>As long as I don't even know that this message means or where it comes >>from, I cannot really debug things :-( > > Details? what exactly do you mean by details? The details of the kernel messages? They were in my first email, but I post them again below in case the first mail got lost! Or do you need more details about my system? I quote the complete first mail to the lkml below, because it contains some details about the host, too. cu, Frank Frank Steiner wrote > Hi, > > we got sth. that looks similar to but not exactly like a kernel oops on one > of our hosts saying "kblockd/1: page allocation failure. order:0, mode:0x21" > (the full output is below). The kernel is 2.6.9. > > After that happend, "free" told me that 1.7GB of the 2GB was used and nothing > was buffered (i.e., the "-/+ buffers" shopwed the same amount of used memory), > which was strange because there was no process running that did any real > work. A tomcat server and a mysql database are running on this machine, but > the database is still empty and the tomcat does not yet offer any services. > > Also, "ps -aux" showed only 2/3 of the processes then hang until I killed it. > I've never seen "ps -aux" hang! I had to reboot to get rid of this. > > I guess the problem occured after copying a large amount of data to the disks. > The host is a dual xeon 1.7GHz with an internal 147GB Raid-1 (ICP vortex controlled > with gdth modul) and two external 600GB IDE-Raid5, connected to an Adaptec 29160 > with the aic7xxx modul. > > Can someone tell me what this kblockd messages mean? What is kblockd? Could > that point to a hardware problem (CPU, memory, mainboard) or a driver problems > with one the scsi controllers or sth.? The log says sth about scsi_get_command. > And are these messages sth. like a kernel oops or not? I've never seen sth. > like this before so I'm a bit lost :-) > > Thanks for any hints! > Best regards, > Frank > > > Dec 20 13:17:21 turing kernel: kblockd/1: page allocation failure. order:0, mode:0x21 > Dec 20 13:17:21 turing kernel: [__alloc_pages+824/832] __alloc_pages+0x338/0x340 > Dec 20 13:17:21 turing kernel: [] __alloc_pages+0x338/0x340 > Dec 20 13:17:21 turing kernel: [__get_free_pages+24/48] __get_free_pages+0x18/0x30 > Dec 20 13:17:21 turing kernel: [] __get_free_pages+0x18/0x30 > Dec 20 13:17:21 turing kernel: [kmem_getpages+36/224] kmem_getpages+0x24/0xe0 > Dec 20 13:17:21 turing kernel: [] kmem_getpages+0x24/0xe0 > Dec 20 13:17:21 turing kernel: [cache_grow+168/352] cache_grow+0xa8/0x160 > Dec 20 13:17:21 turing kernel: [] cache_grow+0xa8/0x160 > Dec 20 13:17:21 turing kernel: [cache_alloc_refill+354/560] > cache_alloc_refill+0x162/0x230 > Dec 20 13:17:21 turing kernel: [] cache_alloc_refill+0x162/0x230 > Dec 20 13:17:21 turing kernel: [kmem_cache_alloc+52/64] kmem_cache_alloc+0x34/0x40 > Dec 20 13:17:22 turing kernel: [] kmem_cache_alloc+0x34/0x40 > Dec 20 13:17:22 turing kernel: [__scsi_get_command+21/96] __scsi_get_command+0x15/0x60 > Dec 20 13:17:22 turing kernel: [] __scsi_get_command+0x15/0x60 > Dec 20 13:17:22 turing kernel: [scsi_get_command+15/144] scsi_get_command+0xf/0x90 > Dec 20 13:17:22 turing kernel: [] scsi_get_command+0xf/0x90 > Dec 20 13:17:22 turing kernel: [scsi_prep_fn+227/448] scsi_prep_fn+0xe3/0x1c0 > Dec 20 13:17:22 turing kernel: [] scsi_prep_fn+0xe3/0x1c0 > Dec 20 13:17:22 turing kernel: [elv_next_request+85/224] elv_next_request+0x55/0xe0 > Dec 20 13:17:22 turing kernel: [] elv_next_request+0x55/0xe0 > Dec 20 13:17:22 turing kernel: [scsi_request_fn+501/960] scsi_request_fn+0x1f5/0x3c0 > Dec 20 13:17:22 turing kernel: [] scsi_request_fn+0x1f5/0x3c0 > Dec 20 13:17:22 turing kernel: [__generic_unplug_device+46/48] > __generic_unplug_device+0x2e/0x30 > Dec 20 13:17:22 turing kernel: [] __generic_unplug_device+0x2e/0x30 > Dec 20 13:17:22 turing kernel: [generic_unplug_device+21/48] > generic_unplug_device+0x15/0x30 > Dec 20 13:17:22 turing kernel: [] generic_unplug_device+0x15/0x30 > Dec 20 13:17:22 turing kernel: [blk_unplug_work+6/16] blk_unplug_work+0x6/0x10 > Dec 20 13:17:22 turing kernel: [] blk_unplug_work+0x6/0x10 > Dec 20 13:17:22 turing kernel: [worker_thread+424/560] worker_thread+0x1a8/0x230 > Dec 20 13:17:22 turing kernel: [] worker_thread+0x1a8/0x230 > Dec 20 13:17:22 turing kernel: [blk_unplug_work+0/16] blk_unplug_work+0x0/0x10 > Dec 20 13:17:22 turing kernel: [] blk_unplug_work+0x0/0x10 > Dec 20 13:17:22 turing kernel: [default_wake_function+0/16] > default_wake_function+0x0/0x10 > Dec 20 13:17:22 turing kernel: [] default_wake_function+0x0/0x10 > Dec 20 13:17:22 turing kernel: [default_wake_function+0/16] > default_wake_function+0x0/0x10 > Dec 20 13:17:22 turing kernel: [] default_wake_function+0x0/0x10 > Dec 20 13:17:22 turing kernel: [worker_thread+0/560] worker_thread+0x0/0x230 > Dec 20 13:17:22 turing kernel: [] worker_thread+0x0/0x230 > Dec 20 13:17:22 turing kernel: [kthread+134/176] kthread+0x86/0xb0 > Dec 20 13:17:22 turing kernel: [] kthread+0x86/0xb0 > Dec 20 13:17:22 turing kernel: [kthread+0/176] kthread+0x0/0xb0 > Dec 20 13:17:22 turing kernel: [] kthread+0x0/0xb0 > Dec 20 13:17:22 turing kernel: [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10 > Dec 20 13:17:22 turing kernel: [] kernel_thread_helper+0x5/0x10 > Dec 20 13:17:22 turing kernel: klogd: page allocation failure. order:0, mode:0x21 > Dec 20 13:17:22 turing kernel: [__alloc_pages+824/832] __alloc_pages+0x338/0x340 > Dec 20 13:17:22 turing kernel: [] __alloc_pages+0x338/0x340 > Dec 20 13:17:22 turing kernel: [__get_free_pages+24/48] __get_free_pages+0x18/0x30 > Dec 20 13:17:22 turing kernel: [] __get_free_pages+0x18/0x30 > Dec 20 13:17:22 turing kernel: [kmem_getpages+36/224] kmem_getpages+0x24/0xe0 > Dec 20 13:17:22 turing kernel: [] kmem_getpages+0x24/0xe0 > Dec 20 13:17:22 turing kernel: [cache_grow+168/352] cache_grow+0xa8/0x160 > Dec 20 13:17:22 turing kernel: [] cache_grow+0xa8/0x160 > Dec 20 13:17:22 turing kernel: [cache_alloc_refill+354/560] > cache_alloc_refill+0x162/0x230 > Dec 20 13:17:22 turing kernel: [] cache_alloc_refill+0x162/0x230 > Dec 20 13:17:22 turing kernel: [kmem_cache_alloc+52/64] kmem_cache_alloc+0x34/0x40 > Dec 20 13:17:22 turing kernel: [] kmem_cache_alloc+0x34/0x40 > Dec 20 13:17:22 turing kernel: [__scsi_get_command+21/96] __scsi_get_command+0x15/0x60 > Dec 20 13:17:22 turing kernel: [] __scsi_get_command+0x15/0x60 > Dec 20 13:17:22 turing kernel: [scsi_get_command+15/144] scsi_get_command+0xf/0x90 > Dec 20 13:17:22 turing kernel: [] scsi_get_command+0xf/0x90 > Dec 20 13:17:22 turing kernel: [scsi_prep_fn+227/448] scsi_prep_fn+0xe3/0x1c0 > Dec 20 13:17:22 turing kernel: [] scsi_prep_fn+0xe3/0x1c0 > Dec 20 13:17:22 turing kernel: [elv_next_request+85/224] elv_next_request+0x55/0xe0 > Dec 20 13:17:22 turing kernel: [] elv_next_request+0x55/0xe0 > Dec 20 13:17:22 turing kernel: [__generic_unplug_device+34/48] > __generic_unplug_device+0x22/0x30 > Dec 20 13:17:22 turing kernel: [] __generic_unplug_device+0x22/0x30 > Dec 20 13:17:22 turing kernel: [generic_unplug_device+21/48] > generic_unplug_device+0x15/0x30 > Dec 20 13:17:22 turing kernel: [] generic_unplug_device+0x15/0x30 > Dec 20 13:17:22 turing kernel: [blk_backing_dev_unplug+18/32] > blk_backing_dev_unplug+0x12/0x20 > Dec 20 13:17:22 turing kernel: [] blk_backing_dev_unplug+0x12/0x20 > Dec 20 13:17:22 turing kernel: [swap_unplug_io_fn+90/144] swap_unplug_io_fn+0x5a/0x90 > Dec 20 13:17:22 turing kernel: [] swap_unplug_io_fn+0x5a/0x90 > Dec 20 13:17:22 turing kernel: [swap_unplug_io_fn+0/144] swap_unplug_io_fn+0x0/0x90 > Dec 20 13:17:22 turing kernel: [] swap_unplug_io_fn+0x0/0x90 > Dec 20 13:17:22 turing kernel: [block_sync_page+68/80] block_sync_page+0x44/0x50 > Dec 20 13:17:22 turing kernel: [] block_sync_page+0x44/0x50 > Dec 20 13:17:22 turing kernel: [__lock_page+235/256] __lock_page+0xeb/0x100 > Dec 20 13:17:22 turing kernel: [] __lock_page+0xeb/0x100 > Dec 20 13:17:22 turing kernel: [page_wake_function+0/64] page_wake_function+0x0/0x40 > Dec 20 13:17:22 turing kernel: [] page_wake_function+0x0/0x40 > Dec 20 13:17:22 turing kernel: [page_wake_function+0/64] page_wake_function+0x0/0x40 > Dec 20 13:17:22 turing kernel: [] page_wake_function+0x0/0x40 > Dec 20 13:17:22 turing kernel: [grab_swap_token+158/176] grab_swap_token+0x9e/0xb0 > Dec 20 13:17:22 turing kernel: [] grab_swap_token+0x9e/0xb0 > Dec 20 13:17:22 turing kernel: [do_swap_page+327/656] do_swap_page+0x147/0x290 > Dec 20 13:17:22 turing kernel: [] do_swap_page+0x147/0x290 > Dec 20 13:17:22 turing kernel: [handle_mm_fault+244/336] handle_mm_fault+0xf4/0x150 > Dec 20 13:17:22 turing kernel: [] handle_mm_fault+0xf4/0x150 > Dec 20 13:17:22 turing kernel: [do_page_fault+443/1400] do_page_fault+0x1bb/0x578 > Dec 20 13:17:22 turing kernel: [] do_page_fault+0x1bb/0x578 > Dec 20 13:17:22 turing kernel: [avc_has_perm_noaudit+188/416] > avc_has_perm_noaudit+0xbc/0x1a0 > Dec 20 13:17:22 turing kernel: [] avc_has_perm_noaudit+0xbc/0x1a0 > Dec 20 13:17:22 turing kernel: [recalc_task_prio+330/480] recalc_task_prio+0x14a/0x1e0 > Dec 20 13:17:22 turing kernel: [] recalc_task_prio+0x14a/0x1e0 > Dec 20 13:17:22 turing kernel: [finish_task_switch+50/112] finish_task_switch+0x32/0x70 > Dec 20 13:17:22 turing kernel: [] finish_task_switch+0x32/0x70 > Dec 20 13:17:22 turing kernel: [schedule+848/2832] schedule+0x350/0xb10 > Dec 20 13:17:22 turing kernel: [] schedule+0x350/0xb10 > Dec 20 13:17:22 turing kernel: [do_page_fault+0/1400] do_page_fault+0x0/0x578 > Dec 20 13:17:22 turing kernel: [] do_page_fault+0x0/0x578 > Dec 20 13:17:22 turing kernel: [error_code+45/64] error_code+0x2d/0x40 > Dec 20 13:17:22 turing kernel: [] error_code+0x2d/0x40 > Dec 20 13:17:22 turing kernel: [do_syslog+779/992] do_syslog+0x30b/0x3e0 > Dec 20 13:17:22 turing kernel: [] do_syslog+0x30b/0x3e0 > Dec 20 13:17:22 turing kernel: [autoremove_wake_function+0/48] > autoremove_wake_function+0x0/0x30 > Dec 20 13:17:22 turing kernel: [] autoremove_wake_function+0x0/0x30 > Dec 20 13:17:22 turing kernel: [autoremove_wake_function+0/48] > autoremove_wake_function+0x0/0x30 > Dec 20 13:17:22 turing kernel: [] autoremove_wake_function+0x0/0x30 > Dec 20 13:17:22 turing kernel: [kmsg_read+0/64] kmsg_read+0x0/0x40 > Dec 20 13:17:22 turing kernel: [] kmsg_read+0x0/0x40 > Dec 20 13:17:22 turing kernel: [vfs_read+174/240] vfs_read+0xae/0xf0 > Dec 20 13:17:22 turing kernel: [] vfs_read+0xae/0xf0 > Dec 20 13:17:22 turing kernel: [sys_read+60/112] sys_read+0x3c/0x70 > Dec 20 13:17:22 turing kernel: [] sys_read+0x3c/0x70 > Dec 20 13:17:22 turing kernel: [syscall_call+7/11] syscall_call+0x7/0xb > Dec 20 13:17:22 turing kernel: [] syscall_call+0x7/0xb > Dec 20 13:17:22 turing kernel: klogd: page allocation failure. order:0, mode:0x21 > Dec 20 13:17:22 turing kernel: [__alloc_pages+824/832] __alloc_pages+0x338/0x340 > Dec 20 13:17:22 turing kernel: [] __alloc_pages+0x338/0x340 > Dec 20 13:17:22 turing kernel: [__get_free_pages+24/48] __get_free_pages+0x18/0x30 > Dec 20 13:17:22 turing kernel: [] __get_free_pages+0x18/0x30 > Dec 20 13:17:22 turing kernel: [kmem_getpages+36/224] kmem_getpages+0x24/0xe0 > Dec 20 13:17:22 turing kernel: [] kmem_getpages+0x24/0xe0 > Dec 20 13:17:22 turing kernel: [cache_grow+168/352] cache_grow+0xa8/0x160 > Dec 20 13:17:22 turing kernel: [] cache_grow+0xa8/0x160 > Dec 20 13:17:22 turing kernel: [cache_alloc_refill+354/560] > cache_alloc_refill+0x162/0x230 > Dec 20 13:17:22 turing kernel: [] cache_alloc_refill+0x162/0x230 > Dec 20 13:17:22 turing kernel: [kmem_cache_alloc+52/64] kmem_cache_alloc+0x34/0x40 > Dec 20 13:17:22 turing kernel: [] kmem_cache_alloc+0x34/0x40 > Dec 20 13:17:22 turing kernel: [__scsi_get_command+21/96] __scsi_get_command+0x15/0x60 > Dec 20 13:17:22 turing kernel: [] __scsi_get_command+0x15/0x60 > Dec 20 13:17:22 turing kernel: [scsi_get_command+15/144] scsi_get_command+0xf/0x90 > Dec 20 13:17:22 turing kernel: [] scsi_get_command+0xf/0x90 > Dec 20 13:17:22 turing kernel: [scsi_prep_fn+227/448] scsi_prep_fn+0xe3/0x1c0 > Dec 20 13:17:22 turing kernel: [] scsi_prep_fn+0xe3/0x1c0 > Dec 20 13:17:22 turing kernel: [elv_next_request+85/224] elv_next_request+0x55/0xe0 > Dec 20 13:17:22 turing kernel: [] elv_next_request+0x55/0xe0 > Dec 20 13:17:22 turing kernel: [__generic_unplug_device+34/48] > __generic_unplug_device+0x22/0x30 > Dec 20 13:17:22 turing kernel: [] __generic_unplug_device+0x22/0x30 > Dec 20 13:17:22 turing kernel: [generic_unplug_device+21/48] > generic_unplug_device+0x15/0x30 > Dec 20 13:17:22 turing kernel: [] generic_unplug_device+0x15/0x30 > Dec 20 13:17:22 turing kernel: [blk_backing_dev_unplug+0/32] > blk_backing_dev_unplug+0x0/0x20 > Dec 20 13:17:22 turing kernel: [] blk_backing_dev_unplug+0x0/0x20 > Dec 20 13:17:22 turing kernel: [blk_backing_dev_unplug+18/32] > blk_backing_dev_unplug+0x12/0x20 > Dec 20 13:17:22 turing kernel: [] blk_backing_dev_unplug+0x12/0x20 > Dec 20 13:17:22 turing kernel: [block_sync_page+68/80] block_sync_page+0x44/0x50 > Dec 20 13:17:22 turing kernel: [] block_sync_page+0x44/0x50 > Dec 20 13:17:22 turing kernel: [__lock_page+235/256] __lock_page+0xeb/0x100 > Dec 20 13:17:22 turing kernel: [] __lock_page+0xeb/0x100 > Dec 20 13:17:22 turing kernel: [page_wake_function+0/64] page_wake_function+0x0/0x40 > Dec 20 13:17:22 turing kernel: [] page_wake_function+0x0/0x40 > Dec 20 13:17:22 turing kernel: [page_wake_function+0/64] page_wake_function+0x0/0x40 > Dec 20 13:17:22 turing kernel: [] page_wake_function+0x0/0x40 > Dec 20 13:17:22 turing kernel: [find_get_page+57/80] find_get_page+0x39/0x50 > Dec 20 13:17:22 turing kernel: [] find_get_page+0x39/0x50 > Dec 20 13:17:22 turing kernel: [filemap_nopage+697/864] filemap_nopage+0x2b9/0x360 > Dec 20 13:17:22 turing kernel: [] filemap_nopage+0x2b9/0x360 > Dec 20 13:17:22 turing kernel: [do_no_page+180/688] do_no_page+0xb4/0x2b0 > Dec 20 13:17:22 turing kernel: [] do_no_page+0xb4/0x2b0 > Dec 20 13:17:22 turing kernel: [handle_mm_fault+272/336] handle_mm_fault+0x110/0x150 > Dec 20 13:17:22 turing kernel: [] handle_mm_fault+0x110/0x150 > Dec 20 13:17:22 turing kernel: [do_page_fault+443/1400] do_page_fault+0x1bb/0x578 > Dec 20 13:17:22 turing kernel: [] do_page_fault+0x1bb/0x578 > Dec 20 13:17:22 turing kernel: [do_syslog+722/992] do_syslog+0x2d2/0x3e0 > Dec 20 13:17:22 turing kernel: [] do_syslog+0x2d2/0x3e0 > Dec 20 13:17:22 turing kernel: [autoremove_wake_function+0/48] > autoremove_wake_function+0x0/0x30 > Dec 20 13:17:22 turing kernel: [] autoremove_wake_function+0x0/0x30 > Dec 20 13:17:22 turing kernel: [autoremove_wake_function+0/48] > autoremove_wake_function+0x0/0x30 > Dec 20 13:17:22 turing kernel: [] autoremove_wake_function+0x0/0x30 > Dec 20 13:17:22 turing kernel: [kmsg_read+0/64] kmsg_read+0x0/0x40 > Dec 20 13:17:22 turing kernel: [] kmsg_read+0x0/0x40 > Dec 20 13:17:22 turing kernel: [dnotify_parent+31/112] dnotify_parent+0x1f/0x70 > Dec 20 13:17:22 turing kernel: [] dnotify_parent+0x1f/0x70 > Dec 20 13:17:22 turing kernel: [vfs_read+194/240] vfs_read+0xc2/0xf0 > Dec 20 13:17:22 turing kernel: [] vfs_read+0xc2/0xf0 > Dec 20 13:17:22 turing kernel: [sys_read+60/112] sys_read+0x3c/0x70 > Dec 20 13:17:22 turing kernel: [] sys_read+0x3c/0x70 > Dec 20 13:17:22 turing kernel: [do_page_fault+0/1400] do_page_fault+0x0/0x578 > Dec 20 13:17:22 turing kernel: [] do_page_fault+0x0/0x578 > Dec 20 13:17:22 turing kernel: [error_code+45/64] error_code+0x2d/0x40 > Dec 20 13:17:22 turing kernel: [] error_code+0x2d/0x40 > -- Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/ Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/ LMU, Amalienstr. 17 Phone: +49 89 2180-4049 80333 Muenchen, Germany Fax: +49 89 2180-99-4049 * Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *