RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[Dimitris Zilaskos]

 	Unfortunately , it seems our conclusions were premature:

ksymoops 2.4.11 on i686 2.4.30.  Options used
      -V (default)
      -k /proc/ksyms (default)
      -l /proc/modules (default)
      -o /lib/modules/2.4.30/ (default)
      -m /usr/src/linux/System.map (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Warning (compare_maps): libafs-2.4.30.mp symbol kallsyms_address_to_symbol not found in /usr/local/lib/openafs/libafs-2.4.30.mp.o.  Ignoring /usr/local/lib/openafs/libafs-2.4.30.mp.o entry
Warning (compare_maps): libafs-2.4.30.mp symbol kallsyms_symbol_to_address not found in /usr/local/lib/openafs/libafs-2.4.30.mp.o.  Ignoring /usr/local/lib/openafs/libafs-2.4.30.mp.o entry
Warning (compare_maps): libafs-2.4.30.mp symbol sys_chdir not found in /usr/local/lib/openafs/libafs-2.4.30.mp.o.  Ignoring /usr/local/lib/openafs/libafs-2.4.30.mp.o entry
Warning (compare_maps): libafs-2.4.30.mp symbol sys_exit not found in /usr/local/lib/openafs/libafs-2.4.30.mp.o.  Ignoring /usr/local/lib/openafs/libafs-2.4.30.mp.o entry
Warning (compare_maps): libafs-2.4.30.mp symbol sys_ioctl not found in /usr/local/lib/openafs/libafs-2.4.30.mp.o.  Ignoring /usr/local/lib/openafs/libafs-2.4.30.mp.o entry
Warning (compare_maps): libafs-2.4.30.mp symbol sys_open not found in /usr/local/lib/openafs/libafs-2.4.30.mp.o.  Ignoring /usr/local/lib/openafs/libafs-2.4.30.mp.o entry
Warning (compare_maps): libafs-2.4.30.mp symbol sys_wait4 not found in /usr/local/lib/openafs/libafs-2.4.30.mp.o.  Ignoring /usr/local/lib/openafs/libafs-2.4.30.mp.o entry
Warning (compare_maps): libafs-2.4.30.mp symbol sys_write not found in /usr/local/lib/openafs/libafs-2.4.30.mp.o.  Ignoring /usr/local/lib/openafs/libafs-2.4.30.mp.o entry
May 20 15:31:15 system kernel: dcache hc<1>Unable to handle kernel paging request at virtual address ffffffff
May 20 15:31:15 system kernel: f09de170
May 20 15:31:15 system kernel: *pde = 00004063
May 20 15:31:15 system kernel: Oops: 0002
May 20 15:31:15 system kernel: CPU:    1
May 20 15:31:15 system kernel: EIP:    0010:[<f09de170>]    Tainted: P
Using defaults from ksymoops -t elf32-i386 -a i386
May 20 15:31:15 system kernel: EFLAGS: 00010282
May 20 15:31:15 system kernel: eax: 00000009   ebx: 000a93f4   ecx: 00000082   edx: ef0c7f84
May 20 15:31:15 system kernel: esi: f1e0f5f8   edi: f0ae2000   ebp: 00000001   esp: c8d15dd4
May 20 15:31:15 system kernel: ds: 0018   es: 0018   ss: 0018
May 20 15:31:15 system kernel: Process afs_cachetrim (pid: 938, stackpage=c8d15000)
May 20 15:31:15 system kernel: Stack: f0a03fe1 f099fc43 00000010 f1e0f66c f1e0f5f8 0000000f 00000001 f099fc0f
May 20 15:31:15 system kernel:        f0a03fe1 f099fc43 00000010 f1e0f66c 00528f57 f1e0f5f8 0000000f f099f897
May 20 15:31:15 system kernel:        f1e0f5f8 00000000 00000000 000003aa ecf4a7f0 00000000 00000000 c8d14000
May 20 15:31:15 system kernel: Call Trace:    [<f0a03fe1>] [<f099fc43>] [<f099fc0f>] [<f0a03fe1>] [<f099fc43>]
May 20 15:31:15 system kernel:   [<f099f897>] [<f099ef5e>] [<f0a0caa0>] [<f0a0caa0>] [<f09efe39>] [<f0a0509c>]
May 20 15:31:15 system kernel:   [<c010740e>] [<f09efad0>]
May 20 15:31:15 system kernel: Code: c6 05 ff ff ff ff 2a 83 c4 1c c3 90 8d 74 26 00 b8 f9 4a a0


>>EIP; f09de170 <[libafs-2.4.30.mp]osi_Panic+20/40>   <=====

>>edx; ef0c7f84 <_end+2ec5fb44/3042bc20>
>>esp; c8d15dd4 <_end+88ad994/3042bc20>

Trace; f0a03fe1 <[libafs-2.4.30.mp].rodata.end+60e6/e545>
Trace; f099fc43 <[libafs-2.4.30.mp]afs_HashOutDCache+b3/120>
Trace; f099fc0f <[libafs-2.4.30.mp]afs_HashOutDCache+7f/120>
Trace; f0a03fe1 <[libafs-2.4.30.mp].rodata.end+60e6/e545>
Trace; f099fc43 <[libafs-2.4.30.mp]afs_HashOutDCache+b3/120>
Trace; f099f897 <[libafs-2.4.30.mp]afs_GetDownD+567/860>
Trace; f099ef5e <[libafs-2.4.30.mp]afs_CacheTruncateDaemon+10e/460>
Trace; f0a0caa0 <[libafs-2.4.30.mp]afs_global_lock+0/20>
Trace; f0a0caa0 <[libafs-2.4.30.mp]afs_global_lock+0/20>
Trace; f09efe39 <[libafs-2.4.30.mp]afsd_thread+369/5d0>
Trace; f0a0509c <[libafs-2.4.30.mp].rodata.end+71a1/e545>
Trace; c010740e <arch_kernel_thread+2e/40>
Trace; f09efad0 <[libafs-2.4.30.mp]afsd_thread+0/5d0>

Code;  f09de170 <[libafs-2.4.30.mp]osi_Panic+20/40>
00000000 <_EIP>:
Code;  f09de170 <[libafs-2.4.30.mp]osi_Panic+20/40>   <=====
    0:   c6 05 ff ff ff ff 2a      movb   $0x2a,0xffffffff   <=====
Code;  f09de177 <[libafs-2.4.30.mp]osi_Panic+27/40>
    7:   83 c4 1c                  add    $0x1c,%esp
Code;  f09de17a <[libafs-2.4.30.mp]osi_Panic+2a/40>
    a:   c3                        ret 
Code;  f09de17b <[libafs-2.4.30.mp]osi_Panic+2b/40>
    b:   90                        nop 
Code;  f09de17c <[libafs-2.4.30.mp]osi_Panic+2c/40>
    c:   8d 74 26 00               lea    0x0(%esi),%esi
Code;  f09de180 <[libafs-2.4.30.mp]osi_Panic+30/40>
   10:   b8 f9 4a a0 00            mov    $0xa04af9,%eax


9 warnings issued.  Results may not be reliable.


--
=============================================================================

Dimitris Zilaskos

Department of Physics @ Aristotle University of Thessaloniki , Greece
PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc
 	  http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc
MD5sum  : de2bd8f73d545f0e4caf3096894ad83f  pgp_public_key.asc
=============================================================================

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[ted creedon]

What processor chip are you running on? A 686 has one different instruction
than a 586 does.

Running a 2.6 kernel here.
tedc

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[ted creedon]

If a new kernel was built, what was it named?

Looks like the existing kernel was overwritten. Normally the
CONIFG_LOCALVERSION is changed to differentiate from a working kernel. (I.e.
if the new kernel fails one can reboot with an existing stable kernel).
tedc

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[Dimitris Zilaskos]

> What processor chip are you running on? A 686 has one different instruction
> than a 586 does.

dual Pentium III (Coppermine) @ 600 Mhz


--
=============================================================================

Dimitris Zilaskos

Department of Physics @ Aristotle University of Thessaloniki , Greece
PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc
 	  http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc
MD5sum  : de2bd8f73d545f0e4caf3096894ad83f  pgp_public_key.asc
=============================================================================

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[ted creedon]

Gcc -dumpmachine  #should prints out i586-suse-linux for a P III here.

I'd try a fresh single processor machine and force a 2.6 kernel, module and
afs recompile for a i586.

SuSE 9.3 costs $90 and it solved a similar problem noted in the mailings. In
fact the YasT installed openafs binaries ran fine.

The ksymoops man page has a script to tail -f /var/log/messages|ksymoops1
explained.

Are you sure there isn't a memory problem? I'm running out of ideas.
tedc

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[ted creedon]

/proc/cpuinfo shows that the cmov instruction is there for Durons, PIII's
and P4's.

Its not for the Via C3 processors which you don't have.

tedc

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[Jeffrey Hutzelman]

On Friday, May 20, 2005 12:12:05 PM -0700 ted creedon 
<tcreedon@easystreet.com> wrote:

> Gcc -dumpmachine  #should prints out i586-suse-linux for a P III here.
>
> I'd try a fresh single processor machine and force a 2.6 kernel, module
> and afs recompile for a i586.
>
> SuSE 9.3 costs $90 and it solved a similar problem noted in the mailings.
> In fact the YasT installed openafs binaries ran fine.
>
> The ksymoops man page has a script to tail -f /var/log/messages|ksymoops1
> explained.
>
> Are you sure there isn't a memory problem? I'm running out of ideas.
> tedc

You're really trying too hard.  The oops in question clearly shows EIP in 
osi_Panic(), which is _supposed_ to result in a crash.  So, stop trying to 
analyze the mechanism by which osi_Panic forces a fault, and instead go 
look at your logs and tell us what the panic string was.

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[Dimitris Zilaskos]

> If a new kernel was built, what was it named?

Linux system 2.4.30 #2 SMP Sat May 7 14:54:35 EEST 2005 i686 unknown 
unknown GNU/Linux

> Looks like the existing kernel was overwritten. Normally the
> CONIFG_LOCALVERSION is changed to differentiate from a working kernel. (I.e.
> if the new kernel fails one can reboot with an existing stable kernel).
> tedc
>

that does not seem to exist in my 2.4 kernel...



--
=============================================================================

Dimitris Zilaskos

Department of Physics @ Aristotle University of Thessaloniki , Greece
PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc
 	  http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc
MD5sum  : de2bd8f73d545f0e4caf3096894ad83f  pgp_public_key.asc
=============================================================================

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[Dimitris Zilaskos]

> Gcc -dumpmachine  #should prints out i586-suse-linux for a P III here.

here it prints: i486-slackware-linux
>
> I'd try a fresh single processor machine and force a 2.6 kernel, module and
> afs recompile for a i586.
>
> SuSE 9.3 costs $90 and it solved a similar problem noted in the mailings. In
> fact the YasT installed openafs binaries ran fine.

 	Do you have any pointers to Suse's solution and source code?I 
could check what they did and try it...

> The ksymoops man page has a script to tail -f /var/log/messages|ksymoops1
> explained.

 	I will try that , thnx :)

> Are you sure there isn't a memory problem? I'm running out of ideas.
> tedc

 	It's been a while since I memtested the box , I will try that also 
when I return at the office ( in about a week).



--
=============================================================================

Dimitris Zilaskos

Department of Physics @ Aristotle University of Thessaloniki , Greece
PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc
 	  http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc
MD5sum  : de2bd8f73d545f0e4caf3096894ad83f  pgp_public_key.asc
=============================================================================

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[Dimitris Zilaskos]

> Dimitris,
>
> Glad to be of help. Send a response to the mailing list after a week or so
> of operation.
>
> This is a very important step and not documented.
>
> ted
>
> -----Original Message-----
> From: Dimitris Zilaskos [mailto:dzila@tassadar.physics.auth.gr]
> Sent: Wednesday, May 11, 2005 10:07 PM
> To: ted creedon
> Cc: openafs-info@openafs.org; linux-kernel@vger.kernel.org
> Subject: RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same
> for 2.4.30 and openafs 1.3.82
>
> On Mon, 9 May 2005, ted creedon wrote:
>
>> Looks like a compile problem if there's a symbol table error.
>>
>> To eliminate that as a cause:
>> Make bzImage;make modules;make modules_install;make install; Reboot
>> into the new image Run regen.sh then ./configure and built a new
>> openafs system; install ane test it.
>>
>> I think there may be small differences in the m4 macros between
>> various operating systems.
>>
>> This is the only way I can get reliable compiles. I have had one
>> server crash with 1.3.81 but I suspect the software raid filesystem.
>>

 	It's been 9 days now of heavy usage , and not a single oops 
occured again. Problem solved:)


Best regards ,


--
=============================================================================

Dimitris Zilaskos

Department of Physics @ Aristotle University of Thessaloniki , Greece
PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc
 	  http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc
MD5sum  : de2bd8f73d545f0e4caf3096894ad83f  pgp_public_key.asc
=============================================================================

RE: [OpenAFS] Re: Openafs 1.3.78 and kernel 2.4.29 oopses , same for 2.4.30 and openafs 1.3.82

[Dimitris Zilaskos]

On Mon, 9 May 2005, ted creedon wrote:

> Looks like a compile problem if there's a symbol table error.
>
> To eliminate that as a cause:
> Make bzImage;make modules;make modules_install;make install;
> Reboot into the new image
> Run regen.sh then ./configure and built a new openafs system; install ane
> test it.
>
> I think there may be small differences in the m4 macros between various
> operating systems.
>
> This is the only way I can get reliable compiles. I have had one server
> crash with 1.3.81 but I suspect the software raid filesystem.
>


 	Hello ted ,

I folowed your advice , and now after 60 hours of continuous  usage I had 
no oops. I will keep stressing the system to see how it goes. Using 2.4.30 
and openafs 1.3.78 at the moment.

 	Thanks to everyone who replied :)


--
=============================================================================

Dimitris Zilaskos

Department of Physics @ Aristotle University of Thessaloniki , Greece
PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc
 	  http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc
MD5sum  : de2bd8f73d545f0e4caf3096894ad83f  pgp_public_key.asc
=============================================================================