This serie of 8 Oops happens in a production server using vanilla 2.6.13.4 from kernel.org, config file attached. The workload was normal at this time, and 'cyrmaster' process don't show any unusual log before and just after the first double Oops. Uptime was more than 3 months at this time. The machine is a K7 processor with 512M of RAM and a 'md' mirror of two IDE disks with 6 raid1 autodetect partitions, one of them used as swap. The machine never show any glitch since many years and sucesfully pass memtest86+ test. It work well since it has been rebooted. I don't know if this event can be linked to something going wrong in the kernel. Jan 30 09:08:37 localhost kernel: Unable to handle kernel paging request at virtual address 00c87cf8 Jan 30 09:08:37 localhost kernel: printing eip: Jan 30 09:08:37 localhost kernel: c01245d5 Jan 30 09:08:37 localhost kernel: *pde = 00000000 Jan 30 09:08:37 localhost kernel: Oops: 0002 [#1] Jan 30 09:08:37 localhost kernel: Modules linked in: loop af_packet ipv6 evdev i2c_viapro i2c_core tulip de4x5 crc32 ide_cd cdrom genrtc ext3 jbd ide_disk ide_generic via82cxxx trm290 triflex slc90e66 sis5513 siimage serverworks sc1200 rz1000 piix pdc202xx_old opt i621 ns87415 hpt366 hpt34x generic cy82c693 cs5530 cs5520 cmd64x atiixp amd74xx alim15x3 aec62xx pdc202xx_new ide_core raid1 md_mod unix Jan 30 09:08:37 localhost kernel: CPU: 0 Jan 30 09:08:37 localhost kernel: EIP: 0060:[remove_wait_queue+14/38] Not tainted VLI Jan 30 09:08:37 localhost kernel: EFLAGS: 00010092 (2.6.13.4-3) Jan 30 09:08:37 localhost kernel: EIP is at remove_wait_queue+0xe/0x26 Jan 30 09:08:37 localhost kernel: eax: 00c87cf4 ebx: c9f45159 ecx: 80000000 edx: c9f4514d Jan 30 09:08:37 localhost kernel: esi: 00000292 edi: 00000000 ebp: 00000080 esp: ce7f9ee0 Jan 30 09:08:37 localhost kernel: ds: 007b es: 007b ss: 0068 Jan 30 09:08:37 localhost kernel: Process cyrmaster (pid: 18658, threadinfo=ce7f8000 task=df280a80) Jan 30 09:08:37 localhost kernel: Stack: c9f45149 c9f45000 c0151c4f c87cff60 00000002 c015203e ce7f9f44 00000000 Jan 30 09:08:37 localhost kernel: 00000000 0000004d 00000000 00000000 00000001 00000007 0000004d d7c56338 Jan 30 09:08:37 localhost kernel: d7c56330 d7c56328 d7c56350 d7c56348 d7c56340 00000000 00000027 00000000 Jan 30 09:08:37 localhost kernel: Call Trace: Jan 30 09:08:37 localhost kernel: [poll_freewait+30/62] poll_freewait+0x1e/0x3e Jan 30 09:08:37 localhost kernel: [do_select+619/640] do_select+0x26b/0x280 Jan 30 09:08:37 localhost kernel: [__pollwait+0/154] __pollwait+0x0/0x9a Jan 30 09:08:37 localhost kernel: [sys_select+648/935] sys_select+0x288/0x3a7 Jan 30 09:08:37 localhost kernel: [sys_time+15/49] sys_time+0xf/0x31 Jan 30 09:08:37 localhost kernel: [syscall_call+7/11] syscall_call+0x7/0xb Jan 30 09:08:37 localhost kernel: Code: 83 0a 01 53 9c 5e fa 89 42 0c 8b 58 04 8d 4a 0c 89 48 04 89 0b 89 59 04 56 9d 5b 5e c3 56 53 9c 5e fa 8b 42 0c 8d 5a 0c 8b 4b 04 <89> 48 04 89 01 c7 43 04 00 02 20 00 c7 42 0c 00 01 10 00 56 9d Jan 30 09:08:37 localhost kernel: <1>Unable to handle kernel paging request at virtual address 34000000 Jan 30 09:08:37 localhost kernel: printing eip: Jan 30 09:08:37 localhost kernel: 34000000 Jan 30 09:08:37 localhost kernel: *pde = 00000000 Jan 30 09:08:37 localhost kernel: Oops: 0000 [#2] Jan 30 09:08:37 localhost kernel: Modules linked in: loop af_packet ipv6 evdev i2c_viapro i2c_core tulip de4x5 crc32 ide_cd cdrom genrtc ext3 jbd ide_disk ide_generic via82cxxx trm290 triflex slc90e66 sis5513 siimage serverworks sc1200 rz1000 piix pdc202xx_old opt i621 ns87415 hpt366 hpt34x generic cy82c693 cs5530 cs5520 cmd64x atiixp amd74xx alim15x3 aec62xx pdc202xx_new ide_core raid1 md_mod unix Jan 30 09:08:37 localhost kernel: CPU: 0 Jan 30 09:08:37 localhost kernel: EIP: 0060:[phys_startup_32+871366656/-1073741824] Not tainted VLI Jan 30 09:08:37 localhost kernel: EFLAGS: 00010002 (2.6.13.4-3) Jan 30 09:08:37 localhost kernel: EIP is at 0x34000000 Jan 30 09:08:37 localhost kernel: eax: d51aa1f5 ebx: 00000001 ecx: 00000000 edx: d51aa201 Jan 30 09:08:37 localhost kernel: esi: 00000000 edi: d51aa200 ebp: ce7f9cf0 esp: ce7f9ccc Jan 30 09:08:37 localhost kernel: ds: 007b es: 007b ss: 0068 Jan 30 09:08:37 localhost kernel: Process cyrmaster (pid: 18658, threadinfo=ce7f8000 task=df280a80) Jan 30 09:08:37 localhost kernel: Stack: c011300d d51aa1f5 00000001 00000000 00000000 34c9f450 00000202 d53c5880 Jan 30 09:08:37 localhost kernel: c15e5318 ce7f9d10 c0113044 d51aa200 00000001 00000001 00000000 00000000 Jan 30 09:08:37 localhost kernel: c15e5318 dffe0440 c014cb83 00000000 00000008 c014ccef c15e5318 00000000 Jan 30 09:08:37 localhost kernel: Call Trace: Jan 30 09:08:37 localhost kernel: [__wake_up_common+43/78] __wake_up_common+0x2b/0x4e Jan 30 09:08:37 localhost kernel: [__wake_up+20/30] __wake_up+0x14/0x1e Jan 30 09:08:37 localhost kernel: [pipe_release+96/160] pipe_release+0x60/0xa0 Jan 30 09:08:37 localhost kernel: [pipe_write_release+26/30] pipe_write_release+0x1a/0x1e Jan 30 09:08:37 localhost kernel: [__fput+132/290] __fput+0x84/0x122 Jan 30 09:08:37 localhost kernel: [filp_close+76/85] filp_close+0x4c/0x55 Jan 30 09:08:37 localhost kernel: [put_files_struct+84/171] put_files_struct+0x54/0xab Jan 30 09:08:37 localhost kernel: [do_exit+364/721] do_exit+0x16c/0x2d1 Jan 30 09:08:37 localhost kernel: [do_trap+0/193] do_trap+0x0/0xc1 Jan 30 09:08:37 localhost kernel: [do_page_fault+855/1281] do_page_fault+0x357/0x501 Jan 30 09:08:37 localhost kernel: [do_page_fault+940/1281] do_page_fault+0x3ac/0x501 Jan 30 09:08:37 localhost kernel: [mempool_free+75/80] mempool_free+0x4b/0x50 Jan 30 09:08:37 localhost kernel: [prep_new_page+67/71] prep_new_page+0x43/0x47 Jan 30 09:08:37 localhost kernel: [prep_new_page+67/71] prep_new_page+0x43/0x47 Jan 30 09:08:37 localhost kernel: [buffered_rmqueue+246/334] buffered_rmqueue+0xf6/0x14e Jan 30 09:08:37 localhost kernel: [__alloc_pages+216/894] __alloc_pages+0xd8/0x37e Jan 30 09:08:37 localhost kernel: [__alloc_pages+882/894] __alloc_pages+0x372/0x37e Jan 30 09:08:37 localhost kernel: [deactivate_task+21/33] deactivate_task+0x15/0x21 Jan 30 09:08:37 localhost kernel: [schedule+1113/1215] schedule+0x459/0x4bf Jan 30 09:08:37 localhost kernel: [do_page_fault+0/1281] do_page_fault+0x0/0x501 Jan 30 09:08:37 localhost kernel: [error_code+79/84] error_code+0x4f/0x54 Jan 30 09:08:37 localhost kernel: [remove_wait_queue+14/38] remove_wait_queue+0xe/0x26 Jan 30 09:08:37 localhost kernel: [poll_freewait+30/62] poll_freewait+0x1e/0x3e Jan 30 09:08:37 localhost kernel: [do_select+619/640] do_select+0x26b/0x280 Jan 30 09:08:37 localhost kernel: [__pollwait+0/154] __pollwait+0x0/0x9a Jan 30 09:08:37 localhost kernel: [sys_select+648/935] sys_select+0x288/0x3a7 Jan 30 09:08:37 localhost kernel: [sys_time+15/49] sys_time+0xf/0x31 Jan 30 09:08:37 localhost kernel: [syscall_call+7/11] syscall_call+0x7/0xb Jan 30 09:08:37 localhost kernel: Code: Bad EIP value. Jan 30 09:08:37 localhost kernel: <1>Fixing recursive fault but reboot is needed! Soon after others Oops happens with 'imapd': Jan 30 09:09:55 localhost kernel: Unable to handle kernel paging request at virtual address a4000000 Jan 30 09:09:55 localhost kernel: printing eip: Jan 30 09:09:55 localhost kernel: a4000000 Jan 30 09:09:55 localhost kernel: *pde = 00000000 Jan 30 09:09:55 localhost kernel: Oops: 0000 [#3] Jan 30 09:09:55 localhost kernel: Modules linked in: loop af_packet ipv6 evdev i2c_viapro i2c_core tulip de4x5 crc32 ide_cd cdrom genrtc ext3 jbd ide_disk ide_generic via82cxxx trm290 triflex slc90e66 sis5513 siimage serverworks sc1200 rz1000 piix pdc202xx_old opt i621 ns87415 hpt366 hpt34x generic cy82c693 cs5530 cs5520 cmd64x atiixp amd74xx alim15x3 aec62xx pdc202xx_new ide_core raid1 md_mod unix Jan 30 09:09:55 localhost kernel: CPU: 0 Jan 30 09:09:55 localhost kernel: EIP: 0060:[phys_startup_32+-1544552448/-1073741824] Not tainted VLI Jan 30 09:09:55 localhost kernel: EFLAGS: 00010002 (2.6.13.4-3) Jan 30 09:09:55 localhost kernel: EIP is at 0xa4000000 Jan 30 09:09:55 localhost kernel: eax: defb09f5 ebx: 00000001 ecx: 00000000 edx: defb0a01 Jan 30 09:09:55 localhost kernel: esi: 00000000 edi: defb0a00 ebp: cb4e5edc esp: cb4e5eb8 Jan 30 09:09:55 localhost kernel: ds: 007b es: 007b ss: 0068 Jan 30 09:09:55 localhost kernel: Process imapd (pid: 24236, threadinfo=cb4e4000 task=d5e87a80) Jan 30 09:09:55 localhost kernel: Stack: c011300d defb09f5 00000001 00000000 00000000 a4c9f450 00000202 defb0a20 Jan 30 09:09:55 localhost kernel: 00000008 cb4e5efc c0113044 defb0a00 00000001 00000001 00000000 00000000 Jan 30 09:09:55 localhost kernel: defb0a00 00000000 c014c9c1 00000000 cb4e5f48 c0136aaa cb4e5f48 b7efd000 Jan 30 09:09:55 localhost kernel: Call Trace: Jan 30 09:09:55 localhost kernel: [__wake_up_common+43/78] __wake_up_common+0x2b/0x4e Jan 30 09:09:55 localhost kernel: [__wake_up+20/30] __wake_up+0x14/0x1e Jan 30 09:09:55 localhost kernel: [pipe_writev+1038/1101] pipe_writev+0x40e/0x44d Jan 30 09:09:55 localhost kernel: [free_pgtables+92/106] free_pgtables+0x5c/0x6a Jan 30 09:09:55 localhost kernel: [pipe_write+37/41] pipe_write+0x25/0x29 Jan 30 09:09:55 localhost kernel: [vfs_write+164/315] vfs_write+0xa4/0x13b Jan 30 09:09:55 localhost kernel: [sys_write+59/99] sys_write+0x3b/0x63 Jan 30 09:09:55 localhost kernel: [syscall_call+7/11] syscall_call+0x7/0xb Jan 30 09:09:55 localhost kernel: Code: Bad EIP value. Jan 30 09:11:21 localhost kernel: <1>Unable to handle kernel paging request at virtual address a4000000 Jan 30 09:11:21 localhost kernel: printing eip: Jan 30 09:11:21 localhost kernel: a4000000 Jan 30 09:11:21 localhost kernel: *pde = 00000000 Jan 30 09:11:21 localhost kernel: Oops: 0000 [#4] Jan 30 09:11:21 localhost kernel: Modules linked in: loop af_packet ipv6 evdev i2c_viapro i2c_core tulip de4x5 crc32 ide_cd cdrom genrtc ext3 jbd ide_disk ide_generic via82cxxx trm290 triflex slc90e66 sis5513 siimage serverworks sc1200 rz1000 piix pdc202xx_old opt i621 ns87415 hpt366 hpt34x generic cy82c693 cs5530 cs5520 cmd64x atiixp amd74xx alim15x3 aec62xx pdc202xx_new ide_core raid1 md_mod unix Jan 30 09:11:21 localhost kernel: CPU: 0 Jan 30 09:11:21 localhost kernel: EIP: 0060:[phys_startup_32+-1544552448/-1073741824] Not tainted VLI Jan 30 09:11:21 localhost kernel: EFLAGS: 00010002 (2.6.13.4-3) Jan 30 09:11:21 localhost kernel: EIP is at 0xa4000000 Jan 30 09:11:21 localhost kernel: eax: defb09f5 ebx: 00000001 ecx: 00000000 edx: defb0a01 Jan 30 09:11:21 localhost kernel: esi: 00000000 edi: defb0a00 ebp: c5903edc esp: c5903eb8 Jan 30 09:11:21 localhost kernel: ds: 007b es: 007b ss: 0068 Jan 30 09:11:21 localhost kernel: Process imapd (pid: 24598, threadinfo=c5902000 task=deeb4530) Jan 30 09:11:21 localhost kernel: Stack: c011300d defb09f5 00000001 00000000 00000000 a4c9f450 00000202 defb0a20 Jan 30 09:11:21 localhost kernel: 00000008 c5903efc c0113044 defb0a00 00000001 00000001 00000000 00000000 Jan 30 09:11:21 localhost kernel: defb0a00 00000000 c014c9c1 00000000 ca928480 00000000 dffe04a0 c01447a1 Jan 30 09:11:21 localhost kernel: Call Trace: Jan 30 09:11:21 localhost kernel: [__wake_up_common+43/78] __wake_up_common+0x2b/0x4e Jan 30 09:11:21 localhost kernel: [__wake_up+20/30] __wake_up+0x14/0x1e Jan 30 09:11:21 localhost kernel: [pipe_writev+1038/1101] pipe_writev+0x40e/0x44d Jan 30 09:11:21 localhost kernel: [invalidate_inode_buffers+12/55] invalidate_inode_buffers+0xc/0x37 Jan 30 09:11:21 localhost kernel: [pipe_write+37/41] pipe_write+0x25/0x29 Jan 30 09:11:21 localhost kernel: [vfs_write+164/315] vfs_write+0xa4/0x13b Jan 30 09:11:21 localhost kernel: [sys_write+59/99] sys_write+0x3b/0x63 Jan 30 09:11:21 localhost kernel: [syscall_call+7/11] syscall_call+0x7/0xb Jan 30 09:11:21 localhost kernel: Code: Bad EIP value. Then with 'mysqld' 11 minutes after: Jan 30 09:32:21 localhost kernel: <1>Unable to handle kernel NULL pointer dereference at virtual address 00000000 Jan 30 09:32:21 localhost kernel: printing eip: Jan 30 09:32:21 localhost kernel: c0155630 Jan 30 09:32:21 localhost kernel: *pde = 00000000 Jan 30 09:32:21 localhost kernel: Oops: 0000 [#5] Jan 30 09:32:21 localhost kernel: Modules linked in: loop af_packet ipv6 evdev i2c_viapro i2c_core tulip de4x5 crc32 ide_cd cdrom genrtc ext3 jbd ide_disk ide_generic via82cxxx trm290 triflex slc90e66 sis5513 siimage serverworks sc1200 rz1000 piix pdc202xx_old opt i621 ns87415 hpt366 hpt34x generic cy82c693 cs5530 cs5520 cmd64x atiixp amd74xx alim15x3 aec62xx pdc202xx_new ide_core raid1 md_mod unix Jan 30 09:32:21 localhost kernel: CPU: 0 Jan 30 09:32:21 localhost kernel: EIP: 0060:[__d_find_alias+26/160] Not tainted VLI Jan 30 09:32:21 localhost kernel: EFLAGS: 00010217 (2.6.13.4-3) Jan 30 09:32:21 localhost kernel: EIP is at __d_find_alias+0x1a/0xa0 Jan 30 09:32:21 localhost kernel: eax: 00008000 ebx: 00000000 ecx: d9f442d4 edx: 00000000 Jan 30 09:32:21 localhost kernel: esi: d9f44239 edi: 00000000 ebp: d9f442ec esp: d671fe24 Jan 30 09:32:21 localhost kernel: ds: 007b es: 007b ss: 0068 Jan 30 09:32:21 localhost kernel: Process mysqld (pid: 31400, threadinfo=d671e000 task=deeb4a40) Jan 30 09:32:21 localhost kernel: Stack: d9f442d4 dd5376b4 00000000 d671fed0 c0155fa1 d9f442d4 00000001 d9f442d4 Jan 30 09:32:21 localhost kernel: d9f442d4 dd5376b4 e09042cd d9f442d4 dd5376b4 c59233c4 fffffff4 dd5376b4 Jan 30 09:32:21 localhost kernel: caf5e644 c014d54c caf5e644 dd5376b4 d671ff6c 00000000 d671ff6c d671fec8 Jan 30 09:32:21 localhost kernel: Call Trace: Jan 30 09:32:21 localhost kernel: [d_splice_alias+25/160] d_splice_alias+0x19/0xa0 Jan 30 09:32:21 localhost kernel: [pg0+542937805/1070347264] ext3_lookup+0x77/0x93 [ext3] Jan 30 09:32:21 localhost kernel: [real_lookup+74/169] real_lookup+0x4a/0xa9 Jan 30 09:32:21 localhost kernel: [do_lookup+74/123] do_lookup+0x4a/0x7b Jan 30 09:32:21 localhost kernel: [__link_path_walk+1755/2718] __link_path_walk+0x6db/0xa9e Jan 30 09:32:21 localhost kernel: [link_path_walk+60/171] link_path_walk+0x3c/0xab Jan 30 09:32:21 localhost kernel: [sys_read+59/99] sys_read+0x3b/0x63 Jan 30 09:32:21 localhost kernel: [sys_read+91/99] sys_read+0x5b/0x63 Jan 30 09:32:21 localhost kernel: [path_lookup+234/242] path_lookup+0xea/0xf2 Jan 30 09:32:21 localhost kernel: [__user_walk+35/58] __user_walk+0x23/0x3a Jan 30 09:32:21 localhost kernel: [sys_readlink+32/151] sys_readlink+0x20/0x97 Jan 30 09:32:21 localhost kernel: [sys_read+59/99] sys_read+0x3b/0x63 Jan 30 09:32:21 localhost kernel: [sys_read+91/99] sys_read+0x5b/0x63 Jan 30 09:32:21 localhost kernel: [syscall_call+7/11] syscall_call+0x7/0xb Jan 30 09:32:21 localhost kernel: Code: 41 1c 89 50 04 89 02 89 5b 04 89 59 1c 5b 89 c8 c3 55 57 31 ff 56 53 8b 4c 24 14 8b 51 18 8d 69 18 39 ea 0f 84 81 00 00 00 89 d3 <8b> 12 8d 44 20 00 0f b7 41 28 8d 73 cc 25 00 f0 00 00 3d 00 40 Jan 30 09:32:28 localhost kernel: <0>Bad page state at prep_new_page (in process 'apache', page c108a4a0) Jan 30 09:32:28 localhost kernel: flags:0x40000004 mapping:00000000 mapcount:-1 count:0 Jan 30 09:32:28 localhost kernel: Backtrace: Jan 30 09:32:28 localhost kernel: [bad_page+80/129] bad_page+0x50/0x81 Jan 30 09:32:28 localhost kernel: [prep_new_page+42/71] prep_new_page+0x2a/0x47 Jan 30 09:32:28 localhost kernel: [buffered_rmqueue+246/334] buffered_rmqueue+0xf6/0x14e Jan 30 09:32:28 localhost kernel: [__alloc_pages+216/894] __alloc_pages+0xd8/0x37e Jan 30 09:32:28 localhost kernel: [do_anonymous_page+72/239] do_anonymous_page+0x48/0xef Jan 30 09:32:28 localhost kernel: [do_no_page+69/586] do_no_page+0x45/0x24a Jan 30 09:32:28 localhost kernel: [__handle_mm_fault+105/225] __handle_mm_fault+0x69/0xe1 Jan 30 09:32:28 localhost kernel: [do_page_fault+390/1281] do_page_fault+0x186/0x501 Jan 30 09:32:28 localhost kernel: [vma_link+52/148] vma_link+0x34/0x94 Jan 30 09:32:28 localhost kernel: [autoremove_wake_function+0/58] autoremove_wake_function+0x0/0x3a Jan 30 09:32:28 localhost kernel: [do_mmap_pgoff+1319/1560] do_mmap_pgoff+0x527/0x618 Jan 30 09:32:28 localhost kernel: [vfs_read+256/315] vfs_read+0x100/0x13b Jan 30 09:32:28 localhost kernel: [sys_read+59/99] sys_read+0x3b/0x63 Jan 30 09:32:28 localhost kernel: [do_page_fault+0/1281] do_page_fault+0x0/0x501 Jan 30 09:32:28 localhost kernel: [error_code+79/84] error_code+0x4f/0x54 Jan 30 09:32:28 localhost kernel: Trying to fix it up, but a reboot is needed 9 minutes after this happens with 'kswapd0': Jan 30 09:41:42 localhost kernel: Unable to handle kernel paging request at virtual address ffffff7e Jan 30 09:41:42 localhost kernel: printing eip: Jan 30 09:41:42 localhost kernel: e0906edd Jan 30 09:41:42 localhost kernel: *pde = 00002067 Jan 30 09:41:42 localhost kernel: *pte = 00000000 Jan 30 09:41:42 localhost kernel: Oops: 0002 [#6] Jan 30 09:41:42 localhost kernel: Modules linked in: loop af_packet ipv6 evdev i2c_viapro i2c_core tulip de4x5 crc32 ide_cd cdrom genrtc ext3 jbd ide_disk ide_generic via82cxxx trm290 triflex slc90e66 sis5513 siimage serverworks sc1200 rz1000 piix pdc202xx_old opt i621 ns87415 hpt366 hpt34x generic cy82c693 cs5530 cs5520 cmd64x atiixp amd74xx alim15x3 aec62xx pdc202xx_new ide_core raid1 md_mod unix Jan 30 09:41:42 localhost kernel: CPU: 0 Jan 30 09:41:42 localhost kernel: EIP: 0060:[pg0+542949085/1070347264] Tainted: G B VLI Jan 30 09:41:42 localhost kernel: EFLAGS: 00010286 (2.6.13.4-3) Jan 30 09:41:42 localhost kernel: EIP is at ext3_clear_inode+0x46/0x76 [ext3] Jan 30 09:41:42 localhost kernel: eax: d9f64d70 ebx: d9f64e04 ecx: dfc18c60 edx: ffffff7e Jan 30 09:41:42 localhost kernel: esi: 00000000 edi: dfd01ec0 ebp: 00000011 esp: dfd01e84 Jan 30 09:41:42 localhost kernel: ds: 007b es: 007b ss: 0068 Jan 30 09:41:42 localhost kernel: Process kswapd0 (pid: 78, threadinfo=dfd00000 task=dff53aa0) Jan 30 09:41:42 localhost kernel: Stack: d9f64e04 d9f64f30 c0156c93 d9f64e04 d9f64e0c d9f64e04 c0156d11 d9f64e04 Jan 30 09:41:42 localhost kernel: d972646c d9726474 0000003f 0000007f c0156fd5 dfd01ec0 00000080 c7b0f184 Jan 30 09:41:42 localhost kernel: c54e035c 0000076c 00000000 00000080 dffea9e0 c0157021 00000080 c01346a7 Jan 30 09:41:42 localhost kernel: Call Trace: Jan 30 09:41:42 localhost kernel: [clear_inode+164/210] clear_inode+0xa4/0xd2 Jan 30 09:41:42 localhost kernel: [dispose_list+80/170] dispose_list+0x50/0xaa Jan 30 09:41:42 localhost kernel: [prune_icache+287/339] prune_icache+0x11f/0x153 Jan 30 09:41:42 localhost kernel: [shrink_icache_memory+24/48] shrink_icache_memory+0x18/0x30 Jan 30 09:41:42 localhost kernel: [shrink_slab+277/385] shrink_slab+0x115/0x181 Jan 30 09:41:42 localhost kernel: [balance_pgdat+508/786] balance_pgdat+0x1fc/0x312 Jan 30 09:41:42 localhost kernel: [kswapd+226/231] kswapd+0xe2/0xe7 Jan 30 09:41:42 localhost kernel: [autoremove_wake_function+0/58] autoremove_wake_function+0x0/0x3a Jan 30 09:41:42 localhost kernel: [autoremove_wake_function+0/58] autoremove_wake_function+0x0/0x3a Jan 30 09:41:42 localhost kernel: [kswapd+0/231] kswapd+0x0/0xe7 Jan 30 09:41:42 localhost kernel: [kernel_thread_helper+5/11] kernel_thread_helper+0x5/0xb Jan 30 09:41:42 localhost kernel: Code: 85 d2 74 10 ff 0a 0f 94 c0 84 c0 74 07 52 e8 f4 c2 82 df 58 c7 43 d4 ff ff ff ff 8b 53 d8 85 d2 74 20 83 fa ff 74 1b 85 d2 74 10 0a 0f 94 c0 84 c0 74 07 52 e8 cd c2 82 df 58 c7 43 d8 ff ff And finaly with 'apache' a few minutes after: Jan 30 09:45:11 localhost kernel: <1>Unable to handle kernel NULL pointer dereference at virtual address 00000000 Jan 30 09:45:11 localhost kernel: printing eip: Jan 30 09:45:11 localhost kernel: c0146710 Jan 30 09:45:11 localhost kernel: *pde = 00000000 Jan 30 09:45:11 localhost kernel: Oops: 0000 [#7] Jan 30 09:45:11 localhost kernel: Modules linked in: loop af_packet ipv6 evdev i2c_viapro i2c_core tulip de4x5 crc32 ide_cd cdrom genrtc ext3 jbd ide_disk ide_generic via82cxxx trm290 triflex slc90e66 sis5513 siimage serverworks sc1200 rz1000 piix pdc202xx_old opt i621 ns87415 hpt366 hpt34x generic cy82c693 cs5530 cs5520 cmd64x atiixp amd74xx alim15x3 aec62xx pdc202xx_new ide_core raid1 md_mod unix Jan 30 09:45:11 localhost kernel: CPU: 0 Jan 30 09:45:11 localhost kernel: EIP: 0060:[drop_buffers+23/121] Tainted: G B VLI Jan 30 09:45:11 localhost kernel: EFLAGS: 00010213 (2.6.13.4-3) Jan 30 09:45:11 localhost kernel: EIP is at drop_buffers+0x17/0x79 Jan 30 09:45:11 localhost kernel: eax: 00000000 ebx: 00000000 ecx: 00000000 edx: 00000000 Jan 30 09:45:11 localhost kernel: esi: c19f3687 edi: dffdcbc4 ebp: c1315860 esp: dd219c8c Jan 30 09:45:11 localhost kernel: ds: 007b es: 007b ss: 0068 Jan 30 09:45:11 localhost kernel: Process apache (pid: 32717, threadinfo=dd218000 task=d839c020) Jan 30 09:45:11 localhost kernel: Stack: c1315860 c1315860 dffdcbc4 dd219df0 c01467ae c1315860 dd219ca8 00000000 Jan 30 09:45:11 localhost kernel: dffdcbc4 c1315860 c0134b10 c1315860 000000d2 00000001 00000001 0000001b Jan 30 09:45:11 localhost kernel: 00000000 dd219cd0 dd219cd0 0000000d 00000001 c1330ac0 c1335b40 c11e2200 Jan 30 09:45:11 localhost kernel: Call Trace: Jan 30 09:45:11 localhost kernel: [try_to_free_buffers+60/107] try_to_free_buffers+0x3c/0x6b Jan 30 09:45:11 localhost kernel: [shrink_list+603/860] shrink_list+0x25b/0x35c Jan 30 09:45:11 localhost kernel: [shrink_cache+231/522] shrink_cache+0xe7/0x20a Jan 30 09:45:11 localhost kernel: [shrink_zone+164/181] shrink_zone+0xa4/0xb5 Jan 30 09:45:11 localhost kernel: [shrink_caches+87/108] shrink_caches+0x57/0x6c Jan 30 09:45:11 localhost kernel: [try_to_free_pages+191/383] try_to_free_pages+0xbf/0x17f Jan 30 09:45:11 localhost kernel: [__alloc_pages+522/894] __alloc_pages+0x20a/0x37e Jan 30 09:45:11 localhost kernel: [do_wp_page+243/594] do_wp_page+0xf3/0x252 Jan 30 09:45:11 localhost kernel: [do_swap_page+399/465] do_swap_page+0x18f/0x1d1 Jan 30 09:45:11 localhost kernel: [__handle_mm_fault+143/225] __handle_mm_fault+0x8f/0xe1 Jan 30 09:45:11 localhost kernel: [do_page_fault+390/1281] do_page_fault+0x186/0x501 Jan 30 09:45:11 localhost kernel: [prep_new_page+67/71] prep_new_page+0x43/0x47 Jan 30 09:45:11 localhost kernel: [buffered_rmqueue+246/334] buffered_rmqueue+0xf6/0x14e Jan 30 09:45:11 localhost kernel: [__alloc_pages+326/894] __alloc_pages+0x146/0x37e Jan 30 09:45:11 localhost kernel: [__alloc_pages+882/894] __alloc_pages+0x372/0x37e Jan 30 09:45:11 localhost kernel: [free_hot_cold_page+33/173] free_hot_cold_page+0x21/0xad Jan 30 09:45:11 localhost kernel: [sys_getcwd+329/340] sys_getcwd+0x149/0x154 Jan 30 09:45:11 localhost kernel: [do_page_fault+0/1281] do_page_fault+0x0/0x501 Jan 30 09:45:11 localhost kernel: [error_code+79/84] error_code+0x4f/0x54 Jan 30 09:45:11 localhost kernel: Code: 0e be fb ff ff ff eb 07 89 d8 e8 1c d6 ff ff 5b 89 f0 5e c3 55 57 56 53 8b 6c 24 14 8b 45 00 f6 c4 08 75 02 0f 0b 8b 75 0c 89 f1 <8b> 01 f6 c4 08 74 0c 8b 45 10 85 c0 74 05 0f ba 68 34 14 8b 11 The same minute the machine hanged with the following backtrace on the console from the process 'bounce' (transcripted from mobile phone photo): __find_get_block+0x86/0xa5 __getblk+0x1d/0x35 ext3_getblk+0xa8/0x1d2 [ext3] ext3_do_update_inode+0x310/0x33b [ext3] ext3_bread+0x1c/0x74 [ext3] ext3_add_entry+0xc5/0x10f [ext3] ext3_add_mondir+0x18/0x47 [ext3] ext3_create+0xab/0xe2 [ext3] vfs_create+0xca/0x124 open_namel+0x2c/0x49 sys_open+0x3c/0xb2 syscall_call+0x7/0xb -- Jean-Christian de Rivaz