oops on SUSE LES9-SP2-smp on dual EM64T processor system

oops on SUSE LES9-SP2-smp on dual EM64T processor system

[Emmett Lazich]

Hello.  Hope you find this useful.
Posting to you after reading Documentation/oops-tracing.txt
I am still looking into how to report this fault to SUSE.
OS upgraded by SUSE SP2 (latest) CDROM. Cannot get online
update to work to see if newer patches available.

System has 2 x Intel P4/Xeon EM64T cpus. Hyper threading so seen as 4
cpus.  Production server in core network for a Telco in Australia. Not
looking good since I lobbied hard to put in Linux instead of Solaris.

After this oops, the machine kept running (phew!), however these
processes hang: ps, top, w.  However vmstat, iostat and uptime still
work ok. uptime says machine has a load average of around 20, but vmstat
and iostat say machine is near idle. We will reboot it with nosmp until
this fault is better understood.

dmesg output:
general protection fault: 0000 [1] SMP 
CPU 3 
Pid: 26432, comm: ps Not tainted (2.6.5-7.191-smp
SLES9_SP2_BRANCH-200506281458560000)
RIP: 0010:[<ffffffff80177e9b>] <ffffffff80177e9b>{get_user_pages+267}
RSP: 0018:000001000d843d58  EFLAGS: 00010202
RAX: 00009a4ef0009ff8 RBX: 00000000ffffe000 RCX: 0000010000000000
RDX: 0000994ef0009ff8 RSI: 000ffffffffff000 RDI: ffffffff803d4f00
RBP: 000001007c014400 R08: 0000000000000000 R09: 0000000000000001
R10: 00000000006e6f6d R11: 6c7265702f6e6962 R12: 0000000000000000
R13: 0000000000000000 R14: 000001007a6d73a0 R15: 0000000000000001
FS:  0000002a9588e6e0(0000) GS:ffffffff80563000(0000)
knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000040a7d0 CR3: 0000000037e4c000 CR4: 00000000000006e0
Process ps (pid: 26432, threadinfo 000001000d842000, task
00000100007fc250)
Stack: 0000010079541ca0 ffffffff8019c49b 000001000d843e98
0000000000000000 
       000fffffeff80106 0000000000000106 000000100d843de8
00000000ffffe000 
       0000000000000001 000001000d843e18 
Call Trace:<ffffffff8019c49b>{real_lookup+123}
<ffffffff80145423>{access_process_vm+179} 
       <ffffffff801c4d02>{proc_pid_cmdline+146}
<ffffffff801c418f>{proc_info_read+111} 
       <ffffffff8018d184>{vfs_read+244} <ffffffff8018d3dd>{sys_read+157}
       <ffffffff80189dd7>{sys_open+231}
<ffffffff80110794>{system_call+124} 
       

Code: 48 8b 00 48 c1 eb 09 81 e3 f8 0f 00 00 48 21 f0 48 01 d8 48 
RIP <ffffffff80177e9b>{get_user_pages+267} RSP <000001000d843d58>


--------------------
/proc/version
Linux version 2.6.5-7.191-smp (geeko@buildhost) (gcc version 3.3.3 (SuSE
Linux)) #1 SMP Tue Jun 28 14:58:56 UTC 2005

/proc/cmdline
root=/dev/sda5 vga=0x317 selinux=0 console=tty0 resume=/dev/sda3
elevator=cfq pci=nommconf splash=silent
---------------
iostat -x
avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.05    0.00    0.05    0.05   99.85

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await  svctm  %util
sda          0.00   2.00  0.00  1.20    0.00   25.60     0.00   
12.80    21.33     0.00    0.17   0.17   0.02
sdb          0.00   1.00  0.00  0.80    0.00   14.40     0.00    
7.20    18.00     0.00    0.25   0.25   0.02
sdc          0.00   3.40  0.40  4.60    3.20   64.00     1.60   
32.00    13.44     0.00    0.80   0.52   0.26
sdd          0.00   1.20  0.00  0.60    0.00   14.40     0.00    
7.20    24.00     0.00    0.00   0.00   0.00
fd0          0.00   0.00  0.00  0.00    0.00    0.00     0.00    
0.00     0.00     0.00    0.00   0.00   0.00
-------------------------------------
cpuinfo:
cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 3
model name      :                   Intel(R) Xeon(TM) CPU 3.20GHz
stepping        : 4
cpu MHz         : 3192.097
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall
lm 
pni monitor ds_cpl cid
bogomips        : 6340.60
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 15
model           : 3
model name      :                   Intel(R) Xeon(TM) CPU 3.20GHz
stepping        : 4
cpu MHz         : 3192.097
cache size      : 1024 KB
physical id     : 3
siblings        : 2
core id         : 3
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall
lm 
pni monitor ds_cpl cid
bogomips        : 6373.37
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 15
model           : 3
model name      :                   Intel(R) Xeon(TM) CPU 3.20GHz
stepping        : 4
cpu MHz         : 3192.097
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall
lm 
pni monitor ds_cpl cid
bogomips        : 6373.37
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 15
model           : 3
model name      :                   Intel(R) Xeon(TM) CPU 3.20GHz
stepping        : 4
cpu MHz         : 3192.097
cache size      : 1024 KB
physical id     : 3
siblings        : 2
core id         : 3
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall
lm 
pni monitor ds_cpl cid
bogomips        : 6373.37
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual

Re: oops on SUSE LES9-SP2-smp on dual EM64T processor system

[Jesper Juhl]

On 10/21/05, Emmett Lazich <elazich@hutchison.com.au> wrote:
> Posting to you after reading Documentation/oops-tracing.txt
> I am still looking into how to report this fault to SUSE.

Call your SuSE representative I'd say would be a pretty good first step.
SuSE/Novell have plenty of support options listed on their website,
shouldn't be hard to find the right contact info.


> After this oops, the machine kept running (phew!), however these

Don't put too much trust in the machine after the kernel has Oopsed.
It may appear to run, but most likely something is badly wrong in
kernel land at this point, so anything could happen - I'd say reboot
that machine as soon as you are able to.

> processes hang: ps, top, w.  However vmstat, iostat and uptime still
> work ok. uptime says machine has a load average of around 20, but vmstat
> and iostat say machine is near idle. We will reboot it with nosmp until
> this fault is better understood.
>
> dmesg output:
> general protection fault: 0000 [1] SMP
> CPU 3
> Pid: 26432, comm: ps Not tainted (2.6.5-7.191-smp

2.6.5 is a pretty darn old kernel by now. First thing I would
personally do would be to go to kernel.org, grab a 2.6.13.4 kernel
source, build that, then try to reproduce the problem. An enourmous
amount of fixes have happened since 2.6.5.


--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: oops on SUSE LES9-SP2-smp on dual EM64T processor system

[Jesper Juhl]

On 10/22/05, Emmett Lazich <elazich@hutchison.com.au> wrote:
>  Thanks for your thoughts Jesper.  Things are now in progress with
[snip]
>
>  I also thought the age of the kernel version in SUSE Enterprise 9 was a bit
> old. I assume that SuSE maintain their own branch of the 2.6 kernel for E9
> and they will port their fixes into the mainstream kernel at some time
> before they pick a newer kernel as the base source for their next major
> release. A lot of work, but I guess they need control in order to offer paid
> support.
>
I don't run SuSE nor use their kernels, so I can't really say anything
about that, but I assume you are right that they maintain their own
tree and backport fixes.


>  Can you explain for me: With these kernel opps messages, we can see
> register contents and a stack call trace, but it does not (seem) to state
> what actually went wrong.

My Oops reading skills are not the best you can find. Often you can
tell from the oops if it was a null pointer deref or something else
equally obvious - from your oops message I'm only able to see the call
chain and what function eventually caused the oops - you'll neem
someone more skilled than me to actually decipher what exactely went
wrong - sorry.


>Am I right?  If yes, then how can someone resolve
> the fault - particularly after the machine had to be rebooted?
>
I'm pretty sure someone more knowledgable than me can tell more
specifically what went wrong based on that oops.


>  As you might guess, I am a little bit nervous about ever getting a fix for
> this one. Believe me I considered running the newest generic kernel on this
> machine. I was going to use /proc/config.gz then compile from source.  But

That's definately what I'd do.
SuSE/Novell support aside, reproducing the oops with a recent kernel
(taking /proc/config.gz as a base for "make oldconfig") will surely
result in much better response from LKML since very few people care
about problems with as old a kernel as 2.6.5 - but if the bug is
reproducable with the latest stable kernel or the most recent -git
snapshot (or similar) then it's sure to raise eyebrows and get some
attention - and in the end such a bug report (and eventual fix) helps
all of us.


> knowing how SuSE 9E holds together I figured I might be asking for trouble,
> so I shall try it on the model (dev/test) machine first. Even if it works,
> we will then have no support from Novell.

But at least you'll have a working kernel.

Be aware that you may need to upgrade other stuff besides the kernel.
Things change and stuff regularly need updating to keep up. Take a
look at Documentation/Changes in a recent kernel source (the one you
build) to see the minimum requirements for the most basic tools. A new
kernel with out-of-date, known to not work, tools won't win you any
bug reporting prices.


>So I do not know what to think.
> Management as usual need to pay someone for support and to take
> responsibility. But probably get better stability with "community support".
>
I'd say
 a) try the most recent stable kernel.org kernel, see what results you get
 b) see what answer Novell/Suse come up with
Then deside what to do.

Son't be a slave to "only doing it the vendors way" - if the most
recent kernel.org kernel solves the problem but Novell/Suse do not,
then personally I know what I'd use.
If Novell/Suse come up with a fix that makes your support contract
still be valid, then that's probably what you want...

In the end it's your call.


--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html