Hardware problem detection? Linux 2.4.3-pre4 lockups.

Hardware problem detection? Linux 2.4.3-pre4 lockups.

[Nathan Black]

[-- Attachment #1: Type: text/plain, Size: 1824 bytes --]


I am at a total loss, But I have found some interesting anomalies with my
hardware. 

My Current Setup:
Supermicro S370DE6 (Serverworks Chipset)
Dual PIII 866
2 x 256 MB PC133 ECC SDRAM
onboard AIC 7899 SCSI Controller.
36G,73GB Seagate Cheetah Drive.
Voodoo4 4500 AGP video,
Fore PCA 200e ATM 


Problem, I have a program the can read a file(large, or small) it will then
transmit the data over atm, ethernet, localhost,or write to a file.

I have noticed that the machine will consistently crash(hard lockup) when I
do a read loop of the File. It never locks up at the same place, and I have
changed it so that it never actually does anything with the data after it
reads. Still, same result.

Things that have "fixed" the problem. Setting the FSB to 100(jumpered) will
allow me to run forever.
Also, Setting the L1 Cache to Write-through instead of write back will allow
me to run forever at 133, but the performance hit is worse than setting the
FSB to 100. 

Another note. When I have attempted to compile the kernel for Uni processor.
I started getting segmentation faults with gcc.
Now this tells me it might be the processor. But I have nothing overclocked,
so I would think that it might be some kind of timing issue in the kernel.

I have two machines set up this way. One is much more stable. But I do
observe the occasional crash.(hard lockup) 

I have also seen fsck crash as well. when the system was not shut down
correctly. ( as a hard lockup happens very frequently.)

Here are some things that I have tried, but Have not fixed it.
1) SMP Kernel with "noapic" at lilo prompt ( and without the noapic)
2) Uni Kernel w/ & w/out apic

I am at a total loss. 
Is there anything I can do(other than run at 100 FSB)?

Nathan

P.S. I have enclosed the dmesg output for my Uniprocessor kernel
 <<dmesg.out.uni>> 



[-- Attachment #2: dmesg.out.uni --]
[-- Type: application/octet-stream, Size: 6698 bytes --]

Linux version 2.4.3-pre4 (root@aacrecorder2.aacisd.com) (gcc version 2.95.3 19991030 (prerelease)) #5 Thu Mar 15 09:01:35 EST 2001
BIOS-provided physical RAM map:
 BIOS-e820: 000000000009fc00 @ 0000000000000000 (usable)
 BIOS-e820: 0000000000000400 @ 000000000009fc00 (reserved)
 BIOS-e820: 0000000000020000 @ 00000000000e0000 (reserved)
 BIOS-e820: 000000001ff00000 @ 0000000000100000 (usable)
 BIOS-e820: 0000000000001000 @ 00000000fec00000 (reserved)
 BIOS-e820: 0000000000001000 @ 00000000fec01000 (reserved)
 BIOS-e820: 0000000000001000 @ 00000000fee00000 (reserved)
 BIOS-e820: 0000000000080000 @ 00000000fff80000 (reserved)
On node 0 totalpages: 131072
zone(0): 4096 pages.
zone(1): 126976 pages.
zone(2): 0 pages.
Kernel command line: auto BOOT_IMAGE=uni-2.4.3 ro root=807
Initializing CPU#0
Detected 866.303 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 1730.15 BogoMIPS
Memory: 513216k/524288k available (1115k kernel code, 10684k reserved, 400k data, 220k init, 0k highmem)
Dentry-cache hash table entries: 65536 (order: 7, 524288 bytes)
Buffer-cache hash table entries: 32768 (order: 5, 131072 bytes)
Page-cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
CPU: Before vendor init, caps: 0383fbff 00000000 00000000, vendor = 0
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 256K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: After vendor init, caps: 0383fbff 00000000 00000000 00000000
CPU: After generic, caps: 0383fbff 00000000 00000000 00000000
CPU: Common caps: 0383fbff 00000000 00000000 00000000
CPU: Intel Pentium III (Coppermine) stepping 06
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
mtrr: v1.37 (20001109) Richard Gooch (rgooch@atnf.csiro.au)
mtrr: detected mtrr type: Intel
PCI: PCI BIOS revision 2.10 entry at 0xfdbb1, last bus=2
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI: Discovered primary peer bus 02 [IRQ]
PCI: Using IRQ router ServerWorks [1166/0200] at 00:0f.0
PCI: Cannot allocate resource region 1 of device 00:00.0
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Starting kswapd v1.8
pty: 256 Unix98 ptys configured
block: queued sectors max/low 341114kB/210042kB, 1024 slots per queue
Uniform Multi-Platform E-IDE driver Revision: 6.31
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ServerWorks OSB4: IDE controller on PCI bus 00 dev 79
ServerWorks OSB4: chipset revision 0
ServerWorks OSB4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
fore200e: FORE Systems 200E-series driver - version 0.2g
PCI: Found IRQ 11 for device 00:04.0
fore200e: device PCA-200E found at 0xfe600000, IRQ 11
fore200e: device PCA-200E-0 self-test passed
fore200e: device PCA-200E-0 firmware started
fore200e: device PCA-200E-0 initialized
fore200e: device PCA-200E-0, rev. A, S/N: 28273, ESI: 00:20:48:64:6e:71
fore200e: IRQ 11 reserved for device PCA-200E-0
NTFS version 000607
udf: registering filesystem
Serial driver version 5.05 (2000-12-13) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled
ttyS00 at 0x03f8 (irq = 4) is a 16550A
ttyS01 at 0x02f8 (irq = 3) is a 16550A
Real Time Clock Driver v1.10d
eepro100.c:v1.09j-t 9/29/99 Donald Becker http://cesdis.gsfc.nasa.gov/linux/drivers/eepro100.html
eepro100.c: $Revision: 1.36 $ 2000/11/17 Modified by Andrey V. Savochkin <saw@saw.sw.com.sg> and others
PCI: Found IRQ 9 for device 00:06.0
eth0: Intel Corporation 82557 [Ethernet Pro 100], 00:30:48:10:95:39, IRQ 9.
  Receiver lock-up bug exists -- enabling work-around.
  Board assembly 000000-000, Physical connectors present: RJ45
  Primary interface chip i82555 PHY #1.
  General self-test: passed.
  Serial sub-system self-test: passed.
  Internal registers self-test: passed.
  ROM checksum self-test: passed (0x04f4518b).
  Receiver lock-up workaround activated.
[drm] Initialized tdfx 1.0.0 20000928 on minor 63
SCSI subsystem driver Revision: 1.00
PCI: Found IRQ 10 for device 00:05.1
PCI: Found IRQ 5 for device 00:05.0
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.1.5
        <Adaptec aic7899 Ultra160 SCSI adapter>
        aic7899: Wide Channel A, SCSI Id=7, 32/255 SCBs

scsi1 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.1.5
        <Adaptec aic7899 Ultra160 SCSI adapter>
        aic7899: Wide Channel B, SCSI Id=7, 32/255 SCBs

  Vendor: SEAGATE   Model: ST336704LWV       Rev: 4301
  Type:   Direct-Access                      ANSI SCSI revision: 03
Detected scsi disk sda at scsi0, channel 0, id 0, lun 0
  Vendor: SEAGATE   Model: ST173404LW        Rev: 0004
  Type:   Direct-Access                      ANSI SCSI revision: 03
Detected scsi disk sdb at scsi0, channel 0, id 2, lun 0
scsi0:0:0:0: Tagged Queuing enabled.  Depth 253
scsi0:0:2:0: Tagged Queuing enabled.  Depth 253
  Vendor: PLEXTOR   Model: CD-R   PX-W1210S  Rev: 1.00
  Type:   CD-ROM                             ANSI SCSI revision: 02
Detected scsi CD-ROM sr0 at scsi1, channel 0, id 4, lun 0
(scsi1:A:4): 20.000MB/s transfers (20.000MHz, offset 16)
sr0: scsi3-mmc drive: 32x/32x writer cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.12
(scsi0:A:0): 160.000MB/s transfers (80.000MHz DT, offset 63, 16bit)
SCSI device sda: 71687369 512-byte hdwr sectors (36704 MB)
Partition check:
 sda: sda1 sda2 < sda5 sda6 sda7 sda8 sda9 >
(scsi0:A:2): 160.000MB/s transfers (80.000MHz DT, offset 63, 16bit)
SCSI device sdb: 143374738 512-byte hdwr sectors (73408 MB)
 sdb: sdb1 sdb2 < sdb5 sdb6 sdb7 sdb8 >
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 4096 buckets, 32Kbytes
TCP: Hash tables configured (established 32768 bind 32768)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 220k freed
Adding Swap: 1044184k swap-space (priority -1)
mtrr: type mismatch for e0000000,4000000 old: uncachable new: write-combining
mtrr: type mismatch for e0000000,4000000 old: uncachable new: write-combining
mtrr: type mismatch for e0000000,4000000 old: uncachable new: write-combining
mtrr: type mismatch for e0000000,4000000 old: uncachable new: write-combining

Re: Hardware problem detection? Linux 2.4.3-pre4 lockups.

[Chris Meadors]

On Thu, 15 Mar 2001, Nathan Black wrote:

> I am at a total loss, But I have found some interesting anomalies with my
> hardware.

That is about how I was feeling when I had similar problems.

> My Current Setup:
> Supermicro S370DE6 (Serverworks Chipset)
> Dual PIII 866
> 2 x 256 MB PC133 ECC SDRAM
> onboard AIC 7899 SCSI Controller.
> 36G,73GB Seagate Cheetah Drive.
> Voodoo4 4500 AGP video,
> Fore PCA 200e ATM

My setup was (is):
Tyan Thunder 2500 (Serverworks)
Dual PIII 667s
2 x 512 MB PC133 ECC SDRAM
Onboard dual SYM53C896 controller
5 18.2 GB Seagate Cheetahs
nVideo Vanta
Onboard Intel 10/100

> Problem, I have a program the can read a file(large, or small) it will then
> transmit the data over atm, ethernet, localhost,or write to a file.

I could move a lot of network traffic as long as I wasn't hitting the disk
too hard.

> I have noticed that the machine will consistently crash(hard lockup) when I
> do a read loop of the File. It never locks up at the same place, and I have
> changed it so that it never actually does anything with the data after it
> reads. Still, same result.

Any time I pushed the disk subsystem hard I would get a lockup.  Sometimes
the kernel would oops, the program writing to the disk would segfault, but
always the machine locked hard.

> Things that have "fixed" the problem. Setting the FSB to 100(jumpered) will
> allow me to run forever.
> Also, Setting the L1 Cache to Write-through instead of write back will allow
> me to run forever at 133, but the performance hit is worse than setting the
> FSB to 100.

If I forced things to run slower I could run longer, like changing the
cache setting, never tried the FSB setting.  But even with the machine
slowed down I could eventually lock it up if I pushed the disks hard
enough (12 bonnies at the same time would always do it).

> Another note. When I have attempted to compile the kernel for Uni processor.
> I started getting segmentation faults with gcc.
> Now this tells me it might be the processor. But I have nothing overclocked,
> so I would think that it might be some kind of timing issue in the kernel.

I saw so much strange stuff I couldn't pin it down to one thing, except
perhaps the processor.

> I have two machines set up this way. One is much more stable. But I do
> observe the occasional crash.(hard lockup)

I too had two identical machines.  I was doing all my work on one, and was
planning on copying the finished product over to the second when I was
done.  After I started suspecting the hardware, I started up the other
machine.  It ran perfectly.  I could push it as hard as I wanted with no
trouble at all.

> I have also seen fsck crash as well. when the system was not shut down
> correctly. ( as a hard lockup happens very frequently.)
>
> Here are some things that I have tried, but Have not fixed it.
> 1) SMP Kernel with "noapic" at lilo prompt ( and without the noapic)
> 2) Uni Kernel w/ & w/out apic
>
> I am at a total loss.
> Is there anything I can do(other than run at 100 FSB)?
>
> Nathan
>
> P.S. I have enclosed the dmesg output for my Uniprocessor kernel
>  <<dmesg.out.uni>>

In the end I started swapping processors between the two machines.  I
found the problem followed 1 of my processors.  I called Intel and after 2
days of convincing they RMAed my old processor and sent me a replacement.
Both machines have been running perfectly since then.  If you have any
more processors I'd try swapping them around.  But since you are seeing
problems with 2 similar machines, I wouldn't get my hopes up as to this
being the solution.

-Chris
-- 
Two penguins were walking on an iceberg.  The first penguin said to the
second, "you look like you are wearing a tuxedo."  The second penguin
said, "I might be..."                         --David Lynch, Twin Peaks