From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S261404AbVFCVYo (ORCPT ); Fri, 3 Jun 2005 17:24:44 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S261418AbVFCVYo (ORCPT ); Fri, 3 Jun 2005 17:24:44 -0400 Received: from tassadar.physics.auth.gr ([155.207.123.25]:11417 "EHLO tassadar.physics.auth.gr") by vger.kernel.org with ESMTP id S261404AbVFCVXz (ORCPT ); Fri, 3 Jun 2005 17:23:55 -0400 Date: Sat, 4 Jun 2005 00:23:46 +0300 (EEST) From: Dimitris Zilaskos To: openafs-info@openafs.org cc: linux-kernel@vger.kernel.org Subject: oopses with openafs 1.3.83 and linux 2.4.31 Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-AntiVirus: checked by AntiVir Milter (version: 1.1.0-4; AVE: 6.30.0.15; VDF: 6.30.0.224; host: tassadar) Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org (resending due to size , sorry mods) Hi , After moving to linux 2.4.31 and openafs 1.3.83 , I still get oopses once per day. Usually this causes the system to either freeze totally (only replies to ping) or all processes segfault. Exacly the same happened with 2.4.29 and 2.4.30 with openafs 1.3.82. With openafs 1.3.78 the oopses occur usually after 10+ days uptime (dunno with 2.4.31 yet though,running it now) My cell consists of one 1.2.x server with the openafs database and one 1.3.x server , for large files support. The volumes I access are on the 1.3.x server. All servers are running gentoo linux , the 1.3.x 2.6.11.6 kernels and the 1.2.x 2.4.27. My system is a dual p3 @ 600 Mhz with 768 Mb ram. I have just memtested the system , in completed several passes and no errors were detected. The cpu temperature is ok , I also checked thedisks. Without openafs running the system is stable in all stress tests.Running slackware 10.1. The problems always occur after 4 am , sometimes exacly at 4 am sometimes one hour later. At 4 am my afs servers restart themselves.At the same moment usually there are tenths of concurrent read/write operations on my volumes. However I have not succeeded in reproducing the oopses while manually performing fileserver restarts with processes writing to afs. First , the ooposes with 2.4.31 SMP and openafs 1.3.83 ksymoops 2.4.11 on i686 2.4.31. Options used -V (default) -k /proc/ksyms (default) -l /proc/modules (default) -o /lib/modules/2.4.31/ (default) -m /usr/src/linux/System.map (default) Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options. Warning (compare_maps): libafs-2.4.31.mp symbol sys_chdir not found in /usr/local/lib/openafs/libafs-2.4.31.mp.o. Ignoring /usr/local/lib/openafs/libafs-2.4.31.mp.o entry Warning (compare_maps): libafs-2.4.31.mp symbol sys_exit not found in /usr/local/lib/openafs/libafs-2.4.31.mp.o. Ignoring /usr/local/lib/openafs/libafs-2.4.31.mp.o entry Warning (compare_maps): libafs-2.4.31.mp symbol sys_ioctl not found in /usr/local/lib/openafs/libafs-2.4.31.mp.o. Ignoring /usr/local/lib/openafs/libafs-2.4.31.mp.o entry Warning (compare_maps): libafs-2.4.31.mp symbol sys_open not found in /usr/local/lib/openafs/libafs-2.4.31.mp.o. Ignoring /usr/local/lib/openafs/libafs-2.4.31.mp.o entry Warning (compare_maps): libafs-2.4.31.mp symbol sys_wait4 not found in /usr/local/lib/openafs/libafs-2.4.31.mp.o. Ignoring /usr/local/lib/openafs/libafs-2.4.31.mp.o entry Warning (compare_maps): libafs-2.4.31.mp symbol sys_write not found in /usr/local/lib/openafs/libafs-2.4.31.mp.o. Ignoring /usr/local/lib/openafs/libafs-2.4.31.mp.o entry Jun 2 05:03:24 system kernel: Unable to handle kernel paging request at virtual address a7bd037a Jun 2 05:03:24 system kernel: c015a6fb Jun 2 05:03:24 system kernel: *pde = 00000000 Jun 2 05:03:24 system kernel: Oops: 0000 Jun 2 05:03:24 system kernel: CPU: 1 Jun 2 05:03:24 system kernel: EIP: 0010:[] Tainted: P Using defaults from ksymoops -t elf32-i386 -a i386 Jun 2 05:03:24 system kernel: EFLAGS: 00010286 Jun 2 05:03:24 system kernel: eax: 00000000 ebx: c9e41d60 ecx: c9e41d70 edx: c9e41d70 Jun 2 05:03:24 system kernel: esi: a7bd035a edi: 00000000 ebp: 0001bec2 esp: effddf04 Jun 2 05:03:24 system kernel: ds: 0018 es: 0018 ss: 0018 Jun 2 05:03:24 system kernel: Process kswapd (pid: 5, stackpage=effdd000) Jun 2 05:03:24 system kernel: Stack: ee2f4140 c01575d1 eb4deee0 d70f5938 d70f5920 c9e41d60 c0157a2e c9e41d60 Jun 2 05:03:24 system kernel: d70f59a0 00000006 c17e3558 c03a3b1c 000007d7 c0157e04 00021f7b c0138ee0 Jun 2 05:03:24 system kernel: 00000006 000001d0 ffffffff 000001d0 00000006 00000013 000001d0 c03a3b1c Jun 2 05:03:24 system kernel: Call Trace: [] [] [] [] [] Jun 2 05:03:24 system kernel: [] [] [] [] [] [] Jun 2 05:03:24 system kernel: [] [] Jun 2 05:03:24 system kernel: Code: 8b 46 20 85 c0 74 0e 89 c7 8b 40 18 85 c0 0f 85 c6 02 00 00 >>EIP; c015a6fb <===== >>ebx; c9e41d60 <_end+99d9920/304f8c20> >>ecx; c9e41d70 <_end+99d9930/304f8c20> >>edx; c9e41d70 <_end+99d9930/304f8c20> >>esp; effddf04 <_end+2fb75ac4/304f8c20> Trace; c01575d1 Trace; c0157a2e Trace; c0157e04 Trace; c0138ee0 Trace; c013934a Trace; c01393c2 Trace; c0119881 Trace; c0139586 Trace; c01395f8 Trace; c0139738 Trace; c0105000 <_stext+0/0> Trace; c010740e Trace; c01396a0 Code; c015a6fb 00000000 <_EIP>: Code; c015a6fb <===== 0: 8b 46 20 mov 0x20(%esi),%eax <===== Code; c015a6fe 3: 85 c0 test %eax,%eax Code; c015a700 5: 74 0e je 15 <_EIP+0x15> Code; c015a702 7: 89 c7 mov %eax,%edi Code; c015a704 9: 8b 40 18 mov 0x18(%eax),%eax Code; c015a707 c: 85 c0 test %eax,%eax Code; c015a709 e: 0f 85 c6 02 00 00 jne 2da <_EIP+0x2da> Jun 2 05:03:24 system kernel: <1>Unable to handle kernel paging request at virtual address 345fc3e1 Jun 2 05:03:24 system kernel: c015a6fb Jun 2 05:03:24 system kernel: *pde = 00000000 Jun 2 05:03:24 system kernel: Oops: 0000 Jun 2 05:03:24 system kernel: CPU: 0 Jun 2 05:03:24 system kernel: EIP: 0010:[] Tainted: P Jun 2 05:03:24 system kernel: EFLAGS: 00010202 Jun 2 05:03:24 system kernel: eax: 00000000 ebx: c9e41b60 ecx: c9e41b70 edx: c9e41b70 Jun 2 05:03:24 system kernel: esi: 345fc3c1 edi: 00000000 ebp: 00000003 esp: ebe39c44 Jun 2 05:03:24 system kernel: ds: 0018 es: 0018 ss: 0018 Jun 2 05:03:24 system kernel: Process httpd (pid: 239, stackpage=ebe39000) Jun 2 05:03:24 system kernel: Stack: f09c210f f0add800 00000000 d70f58b8 d70f58a0 c9e41b60 c0157a2e c9e41b60 Jun 2 05:03:24 system kernel: 00000003 d4f6fb40 d4f6fb50 d4f6fb40 f0b31b34 c0157dd4 00000003 d4f6fb40 Jun 2 05:03:24 system kernel: c01577c3 d4f6fb40 f0b31a00 f0b31a10 f09bd41b d4f6fb40 00000000 00000000 Jun 2 05:03:24 system kernel: Call Trace: [] [] [] [] [] Jun 2 05:03:24 system kernel: [] [] [] [] [] [] Jun 2 05:03:24 system kernel: [] [] [] [] [] [] Jun 2 05:03:24 system kernel: [] [] [] [] [] [] Jun 2 05:03:24 system kernel: [] [] [] [] Jun 2 05:03:24 system kernel: Code: 8b 46 20 85 c0 74 0e 89 c7 8b 40 18 85 c0 0f 85 c6 02 00 00 >>EIP; c015a6fb <===== >>ebx; c9e41b60 <_end+99d9720/304f8c20> >>ecx; c9e41b70 <_end+99d9730/304f8c20> >>edx; c9e41b70 <_end+99d9730/304f8c20> >>esp; ebe39c44 <_end+2b9d1804/304f8c20> Trace; f09c210f <[libafs-2.4.31.mp].text.lock.KBUILD_BASENAME+127/3c8> Trace; c0157a2e Trace; c0157dd4 Trace; c01577c3 Trace; f09bd41b <[libafs-2.4.31.mp]afs_syscall_call+10cb/1910> Trace; f09c9427 <[libafs-2.4.31.mp]PCallBackAddr+3d7/405> Trace; f09ab280 <[libafs-2.4.31.mp]AllocPacketBufs+0/180> Trace; f09a549e <[libafs-2.4.31.mp]rxi_Start+6ee/790> Trace; f09a55b0 <[libafs-2.4.31.mp]rxi_Send+70/c0> Trace; f09d380f <[libafs-2.4.31.mp].rodata.end+9914/e505> Trace; f09af7b8 <[libafs-2.4.31.mp]_RXAFSCB_WhoAreYou+98/120> Trace; f09f466e Trace; f09af87c <[libafs-2.4.31.mp]_RXAFSCB_InitCallBackState3+3c/110> Trace; f09af5cd <[libafs-2.4.31.mp]_RXAFSCB_GetXStats+27d/2b0> Trace; f09ca472 <[libafs-2.4.31.mp].rodata.end+577/e505> Trace; f09c1d69 <[libafs-2.4.31.mp]afs_icl_EnumerateSets+b9/150> Trace; c01575d1 Trace; f0a15a60 Trace; f09f7206 Trace; c0157e41 Trace; c014d032 Trace; c014d9ec Trace; c014dec9 Trace; c014e229 <__user_walk+49/60> Trace; c014a03f Trace; c01411e9 Trace; c0108ebb Code; c015a6fb 00000000 <_EIP>: Code; c015a6fb <===== 0: 8b 46 20 mov 0x20(%esi),%eax <===== Code; c015a6fe 3: 85 c0 test %eax,%eax Code; c015a700 5: 74 0e je 15 <_EIP+0x15> Code; c015a702 7: 89 c7 mov %eax,%edi Code; c015a704 9: 8b 40 18 mov 0x18(%eax),%eax Code; c015a707 c: 85 c0 test %eax,%eax Code; c015a709 e: 0f 85 c6 02 00 00 jne 2da <_EIP+0x2da> The complete log can be found at http://tassadar.physics.auth.gr/~dzila/oops1 oops with non-smp 2.4.31 and openafs 1.3.82 follows : ksymoops 2.4.11 on i686 2.4.31. Options used -V (default) -k /proc/ksyms (default) -l /proc/modules (default) -o /lib/modules/2.4.31/ (default) -m /usr/src/linux/System.map (default) Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options. Jun 3 04:56:55 system kernel: kernel BUG at inode.c:1204! Jun 3 04:56:55 system kernel: invalid operand: 0000 Jun 3 04:56:55 system kernel: CPU: 0 Jun 3 04:56:55 system kernel: EIP: 0010:[] Tainted: P Using defaults from ksymoops -t elf32-i386 -a i386 Jun 3 04:56:55 system kernel: EFLAGS: 00010246 Jun 3 04:56:55 system kernel: eax: 00000000 ebx: e28b7c20 ecx: e28b7c30 edx: e28b7c30 Jun 3 04:56:55 system kernel: esi: ef86fc00 edi: 00000000 ebp: 00019072 esp: effe3f04 Jun 3 04:56:55 system kernel: ds: 0018 es: 0018 ss: 0018 Jun 3 04:56:55 system kernel: Process kswapd (pid: 4, stackpage=effe3000) Jun 3 04:56:55 system kernel: Stack: e2190a20 00000010 00000282 e88a0378 e88a0360 e28b7c20 c014b5e3 e28b7c20 Jun 3 04:56:55 system kernel: d67bc240 00000002 c1762b7c c0374638 000003ea c014b934 0002293f c0130796 Jun 3 04:56:55 system kernel: 00000006 000001d0 ffffffff 000001d0 00000002 00000020 000001d0 c0374638 Jun 3 04:56:55 system kernel: Call Trace: [] [] [] [] [] Jun 3 04:56:55 system kernel: [] [] [] [] [] [] Jun 3 04:56:55 system kernel: Code: 0f 0b b4 04 66 12 32 c0 e9 93 fd ff ff c7 04 24 31 00 00 00 >>EIP; c014de40 <===== >>ebx; e28b7c20 <_end+224930d4/3046f514> >>ecx; e28b7c30 <_end+224930e4/3046f514> >>edx; e28b7c30 <_end+224930e4/3046f514> >>esi; ef86fc00 <_end+2f44b0b4/3046f514> >>esp; effe3f04 <_end+2fbbf3b8/3046f514> Trace; c014b5e3 Trace; c014b934 Trace; c0130796 Trace; c0130b5d Trace; c0130be2 Trace; c0130da6 Trace; c0130e18 Trace; c0130f58 Trace; c0105000 <_stext+0/0> Trace; c01072be Trace; c0130ec0 Code; c014de40 00000000 <_EIP>: Code; c014de40 <===== 0: 0f 0b ud2a <===== Code; c014de42 2: b4 04 mov $0x4,%ah Code; c014de44 4: 66 data16 Code; c014de45 5: 12 32 adc (%edx),%dh Code; c014de47 7: c0 e9 93 shr $0x93,%cl Code; c014de4a a: fd std Code; c014de4b b: ff (bad) Code; c014de4c c: ff c7 inc %edi Code; c014de4e e: 04 24 add $0x24,%al Code; c014de50 10: 31 00 xor %eax,(%eax) Jun 3 04:57:01 system kernel: kernel BUG at inode.c:1204! Jun 3 04:57:01 system kernel: invalid operand: 0000 Jun 3 04:57:01 system kernel: CPU: 0 Jun 3 04:57:01 system kernel: EIP: 0010:[] Tainted: P Jun 3 04:57:01 system kernel: EFLAGS: 00010246 Jun 3 04:57:01 system kernel: eax: 00000000 ebx: e28b7e00 ecx: e28b7e10 edx: e28b7e10 Jun 3 04:57:01 system kernel: esi: ef86fc00 edi: 00000000 ebp: 00000005 esp: e9a73cd8 Jun 3 04:57:01 system kernel: ds: 0018 es: 0018 ss: 0018 Jun 3 04:57:01 system kernel: Process lftp (pid: 3126, stackpage=e9a73000) Jun 3 04:57:01 system kernel: Stack: f095a5b0 e9a73d2c 0015249f e88a03f8 e88a03e0 e28b7e00 c014b5e3 e28b7e00 Jun 3 04:57:01 system kernel: 00000005 c7e37ca0 c7e37cb0 c7e37ca0 f0a85abc c014b904 00000005 c7e37ca0 Jun 3 04:57:01 system kernel: c014b3d9 c7e37ca0 f0a8599c f0a859ac f0972349 c7e37ca0 00000001 c013911f Jun 3 04:57:01 system kernel: Call Trace: [] [] [] [] [] Jun 3 04:57:01 system kernel: [] [] [] [] [] [] Jun 3 04:57:01 system kernel: [] [] [] [] [] [] Jun 3 04:57:01 system kernel: [] [] [] [] [] [] Jun 3 04:57:01 system kernel: [] [] Jun 3 04:57:01 system kernel: Code: 0f 0b b4 04 66 12 32 c0 e9 93 fd ff ff c7 04 24 31 00 00 00 >>EIP; c014de40 <===== >>ebx; e28b7e00 <_end+224932b4/3046f514> >>ecx; e28b7e10 <_end+224932c4/3046f514> >>edx; e28b7e10 <_end+224932c4/3046f514> >>esi; ef86fc00 <_end+2f44b0b4/3046f514> >>esp; e9a73cd8 <_end+2964f18c/3046f514> Trace; f095a5b0 Trace; c014b5e3 Trace; c014b904 Trace; c014b3d9 Trace; f0972349 Trace; c013911f Trace; f0975dc7 Trace; c0137ded Trace; f0974044 Trace; f0964788 Trace; f096484c Trace; f096459d Trace; f097f296 Trace; f0976c99 Trace; c0205c3e Trace; f09c9fe0 Trace; f09abaf5 Trace; c014b971 Trace; c014230d Trace; c014278b Trace; c0142fa9 Trace; c0143259 <__user_walk+49/60> Trace; c013f81f Trace; c010a56b Trace; c0108bab Code; c014de40 00000000 <_EIP>: Code; c014de40 <===== 0: 0f 0b ud2a <===== Code; c014de42 2: b4 04 mov $0x4,%ah Code; c014de44 4: 66 data16 Code; c014de45 5: 12 32 adc (%edx),%dh Code; c014de47 7: c0 e9 93 shr $0x93,%cl Code; c014de4a a: fd std Code; c014de4b b: ff (bad) Code; c014de4c c: ff c7 inc %edi Code; c014de4e e: 04 24 add $0x24,%al Code; c014de50 10: 31 00 xor %eax,(%eax) The complete log of that can be found at http://tassadar.physics.auth.gr/~dzila/oops2 Any hints are greatly appreciated. Best regards , -- ============================================================================= Dimitris Zilaskos Department of Physics @ Aristotle University of Thessaloniki , Greece PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc MD5sum : de2bd8f73d545f0e4caf3096894ad83f pgp_public_key.asc =============================================================================