kernel cache mem bug(?)

kernel cache mem bug(?)

[kernel]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 8375 bytes --]

Hello,

don't know where to properly send this, so I'm hereby sending this to the
address specified in the "REPORTING-BUGS" document.

[1.] One line summary of the problem:

  Memory cached files get corrupt

[2.] Full description of the problem/report:

  When doing multiple "md5sum" on the same, cached, files, over and over -
  the result varies. The same file(s) get different checksums in some
  cases.

  Doing a memory (RAM) diagnostic shows everything is ok.

  Same version of md5sum (md5sum (coreutils) 5.2.1) on other, similar
  machines does not show this problem.

[3.] Keywords (i.e., modules, networking, kernel):

  kernel, file caching, memory caching

[4.] Kernel version (from /proc/version):

  Linux version 2.6.15.6 (root@burk) (gcc version 3.3.6) #4
    SMP PREEMPT Thu Mar 16 17:10:13 CET 2006

[5.] Most recent kernel version which did not have the bug:

  unknown

[6.] Output of Oops.. message (if applicable) with symbolic information
     resolved (see Documentation/oops-tracing.txt)

  N/A

[7.] A small shell script or example program which triggers the
     problem (if possible)

------
#!/usr/bin/bash
# This script will create 35 20MB-files containing only nulls,
# load all of it into the ram file cache (using md5sum), and then
# keep md5sum:ing the same files a hundred times.
# If everything works fine, all files will have the same checksum.
# If not, sometimes, some files will return a different checksum!
PID=$$
MAXFILES=35
MD5LOOPS=100

echo "[1/4] Removing old cache files..."

rm -f cache.*-*-* 2>/dev/null

echo "[2/4] Creating files to cache..."
while [ 0$MAXFILES -gt 0 ]; do
  echo -ne "$MAXFILES left    \r"
  dd if=/dev/zero of=cache.$RANDOM-$RANDOM-$RANDOM \
    bs=1024 count=20000 2>/dev/null
  MAXFILES=`expr 0$MAXFILES - 1`
done

echo "                   "

echo "[3/4] Starting background looping md5sum... ($MD5LOOPS runs)"

while [ 0$MD5LOOPS -gt 0 ]; do
  md5sum cache.*-*-* >> md5log.$PID.lis
  MD5LOOPS=`expr 0$MD5LOOPS - 1`
done &

CPID=$!

echo "[4/4] Checking for bad checksums..."

while ps -p $CPID >/dev/null 2>&1; do
  AMNT=`awk '$1!="91b82dcc83230890dbcdfc6b80571ddd"' md5log.$PID.lis | \
    wc -l`
  echo MD5 discrepancies so far: $AMNT
  sleep 5
done
------

Example output:

[1/4] Removing old cache files...
[2/4] Creating files to cache...

[3/4] Starting background looping md5sum... (100 runs)
[4/4] Checking for bad checksums...
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 0
MD5 discrepancies so far: 1
MD5 discrepancies so far: 1
MD5 discrepancies so far: 1
MD5 discrepancies so far: 1
[...]

------


[8.] Environment
[8.1.] Software (add the output of the ver_linux script here)

Linux burk 2.6.15.6 #4 SMP PREEMPT Thu Mar 16 17:10:13 CET
2006 i686 unknown unknown GNU/Linux

Gnu C                  3.3.6
Gnu make               3.80
binutils               2.15.92.0.2
util-linux             2.12p
mount                  2.12p
module-init-tools      3.1
e2fsprogs              1.38
jfsutils               1.1.8
reiserfsprogs          3.6.19
reiser4progs           line
xfsprogs               2.6.13
pcmcia-cs              3.2.8
quota-tools            3.12.
PPP                    2.4.4b1
nfs-utils              1.0.7
Linux C Library        2.3.5
Dynamic linker (ldd)   2.3.5
Linux C++ Library      5.0.7
Procps                 3.2.5
Net-tools              1.60
Kbd                    1.12
Sh-utils               5.2.1
udev                   064
Modules Loaded         i2c_i801 tg3 snd_emu10k1 snd_rawmidi snd_util_mem
snd_hwdep ext2


[8.2.] Processor information (from /proc/cpuinfo):

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      : Intel(R) Pentium(R) 4 CPU 3.40GHz
stepping        : 1
cpu MHz         : 3392.246
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 3
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx pni
monitor ds_cpl cid xtpr
bogomips        : 6793.25

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      : Intel(R) Pentium(R) 4 CPU 3.40GHz
stepping        : 1
cpu MHz         : 3392.246
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 3
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx pni
monitor ds_cpl cid xtpr
bogomips        : 6783.63

[8.3.] Module information (from /proc/modules):

i2c_i801 8076 0 - Live 0xf8869000
tg3 100484 0 - Live 0xf9aea000
snd_emu10k1 115620 0 - Live 0xf9a39000
snd_rawmidi 20512 1 snd_emu10k1, Live 0xf9a10000
snd_util_mem 3712 1 snd_emu10k1, Live 0xf8867000
snd_hwdep 7584 1 snd_emu10k1, Live 0xf8864000
ext2 44292 1 - Live 0xf886e000

[8.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)

/proc/ioports:
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-006f : keyboard
0070-0077 : rtc
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
01f0-01f7 : ide0
03c0-03df : vga+
03f6-03f6 : ide0
03f8-03ff : serial
0800-0803 : PM1a_EVT_BLK
0804-0805 : PM1a_CNT_BLK
0808-080b : PM_TMR
0810-0815 : ACPI CPU throttle
0828-082f : GPE0_BLK
0860-08ff : pnp 00:00
0c00-0c7f : pnp 00:00
c000-cfff : PCI Bus #04
  ccb8-ccbf : 0000:04:01.1
  ccc0-ccff : 0000:04:01.0
    ccc0-ccff : EMU10K1
d000-dfff : PCI Bus #01
  dc00-dcff : 0000:01:00.0
ece0-ecff : 0000:00:1f.3
  ece0-ece7 : i801_smbus
fe00-fe07 : 0000:00:1f.2
  fe00-fe07 : libata
fe10-fe13 : 0000:00:1f.2
  fe10-fe13 : libata
fe20-fe27 : 0000:00:1f.2
  fe20-fe27 : libata
fe30-fe33 : 0000:00:1f.2
  fe30-fe33 : libata
fea0-feaf : 0000:00:1f.2
  fea0-feaf : libata
ff20-ff3f : 0000:00:1d.3
  ff20-ff3f : uhci_hcd
ff40-ff5f : 0000:00:1d.2
  ff40-ff5f : uhci_hcd
ff60-ff7f : 0000:00:1d.1
  ff60-ff7f : uhci_hcd
ff80-ff9f : 0000:00:1d.0
  ff80-ff9f : uhci_hcd
ffa0-ffaf : 0000:00:1f.1
  ffa0-ffa7 : ide0

/proc/iomem:
00000000-0009ffff : System RAM
000a0000-000bffff : Video RAM area
000c0000-000cd7ff : Video ROM
000cd800-000cffff : Adapter ROM
000f0000-000fffff : System ROM
00100000-3fe8cbff : System RAM
  00100000-003b7f18 : Kernel code
  003b7f19-004a30d3 : Kernel data
3fe8cc00-3fe8ebff : ACPI Non-volatile Storage
3fe8ec00-3fe90bff : ACPI Tables
3fe90c00-3fffffff : reserved
c0000000-cfffffff : PCI Bus #01
  c0000000-cfffffff : 0000:01:00.0
    c0000000-cfffffff : radeonfb framebuffer
dfa00000-dfafffff : PCI Bus #04
  dfafb800-dfafbfff : 0000:04:01.2
  dfafc000-dfafffff : 0000:04:01.2
dfb00000-dfbfffff : PCI Bus #03
dfc00000-dfcfffff : PCI Bus #02
  dfcf0000-dfcfffff : 0000:02:00.0
    dfcf0000-dfcfffff : tg3
dfd00000-dfefffff : PCI Bus #01
  dfde0000-dfdeffff : 0000:01:00.0
    dfde0000-dfdeffff : radeonfb mmio
  dfdf0000-dfdfffff : 0000:01:00.1
  dfe00000-dfe1ffff : 0000:01:00.0
dffffc00-dfffffff : 0000:00:1f.2
  dffffc00-dfffffff : libata
e0000000-efffffff : reserved
fec00000-fed003ff : reserved
fed20000-fed9ffff : reserved
fee00000-feefffff : reserved
ffa80800-ffa80bff : 0000:00:1d.7
  ffa80800-ffa80bff : ehci_hcd
ffb00000-ffffffff : reserved

[8.5.] PCI information ('lspci -vvv' as root)

  See attached file lspci-vvv.txt

[8.6.] SCSI information (from /proc/scsi/scsi)

  N/A

[8.7.] Other information that might be relevant to the problem
       (please look in /proc and include all information that you
       think to be relevant):

  N/A

[X.] Other notes, patches, fixes, workarounds:

     Workaround: When we disable HyperThreading in BIOS, this
     problem goes away. We re-enabling HT, it comes back...



TIA,
//D

[-- Attachment #2: Type: TEXT/PLAIN, Size: 14369 bytes --]

00:00.0 Host bridge: Intel Corporation 925X/XE Memory Controller Hub (rev 04)
        Subsystem: Dell: Unknown device 0176
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
        Latency: 0
        Capabilities: [e0] #09 [2109]

00:01.0 PCI bridge: Intel Corporation 925X/XE PCI Express Root Port (rev 04) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, cache line size 10
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 0000d000-0000dfff
        Memory behind bridge: dfd00000-dfefffff
        Prefetchable memory behind bridge: c0000000-cfffffff
        BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
        Capabilities: [88] #0d [0000]
        Capabilities: [80] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-
                Address: 00000000  Data: 0000
        Capabilities: [a0] #10 [0141]

00:1c.0 PCI bridge: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) PCI Express Port 1 (rev 03) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, cache line size 10
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: dfc00000-dfcfffff
        Prefetchable memory behind bridge: 00000000fff00000-0000000000000000
        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
        Capabilities: [40] #10 [0141]
        Capabilities: [80] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-
                Address: 00000000  Data: 0000
        Capabilities: [90] #0d [0000]
        Capabilities: [a0] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

00:1c.1 PCI bridge: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) PCI Express Port 2 (rev 03) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, cache line size 10
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: dfb00000-dfbfffff
        Prefetchable memory behind bridge: 00000000fff00000-0000000000000000
        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
        Capabilities: [40] #10 [0141]
        Capabilities: [80] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-
                Address: 00000000  Data: 0000
        Capabilities: [90] #0d [0000]
        Capabilities: [a0] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

00:1d.0 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) USB UHCI #1 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Dell: Unknown device 0176
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0
        Interrupt: pin A routed to IRQ 19
        Region 4: I/O ports at ff80 [size=32]

00:1d.1 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) USB UHCI #2 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Dell: Unknown device 0176
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0
        Interrupt: pin B routed to IRQ 20
        Region 4: I/O ports at ff60 [size=32]

00:1d.2 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) USB UHCI #3 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Dell: Unknown device 0176
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0
        Interrupt: pin C routed to IRQ 21
        Region 4: I/O ports at ff40 [size=32]

00:1d.3 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) USB UHCI #4 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Dell: Unknown device 0176
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0
        Interrupt: pin D routed to IRQ 22
        Region 4: I/O ports at ff20 [size=32]

00:1d.7 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) USB2 EHCI Controller (rev 03) (prog-if 20 [EHCI])
        Subsystem: Dell: Unknown device 0176
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0
        Interrupt: pin A routed to IRQ 19
        Region 0: Memory at ffa80800 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] #0a [20a0]

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d3) (prog-if 01 [Subtractive decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0
        Bus: primary=00, secondary=04, subordinate=04, sec-latency=32
        I/O behind bridge: 0000c000-0000cfff
        Memory behind bridge: dfa00000-dfafffff
        Prefetchable memory behind bridge: 00000000fff00000-0000000000000000
        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
        Capabilities: [50] #0d [0000]

00:1f.0 ISA bridge: Intel Corporation 82801FB/FR (ICH6/ICH6R) LPC Interface Bridge (rev 03)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0

00:1f.1 IDE interface: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) IDE Controller (rev 03) (prog-if 8a [Master SecP PriP])
        Subsystem: Dell: Unknown device 0176
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: I/O ports at <ignored>
        Region 1: I/O ports at <ignored>
        Region 2: I/O ports at <ignored>
        Region 3: I/O ports at <ignored>
        Region 4: I/O ports at ffa0 [size=16]

00:1f.2 Class 0106: Intel Corporation 82801FR/FRW (ICH6R/ICH6RW) SATA Controller (rev 03) (prog-if 01)
        Subsystem: Dell: Unknown device 0176
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0
        Interrupt: pin C routed to IRQ 18
        Region 0: I/O ports at fe00 [size=8]
        Region 1: I/O ports at fe10 [size=4]
        Region 2: I/O ports at fe20 [size=8]
        Region 3: I/O ports at fe30 [size=4]
        Region 4: I/O ports at fea0 [size=16]
        Region 5: Memory at dffffc00 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [70] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

00:1f.3 SMBus: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) SMBus Controller (rev 03)
        Subsystem: Dell: Unknown device 0176
        Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Interrupt: pin B routed to IRQ 10
        Region 4: I/O ports at ece0 [size=32]

01:00.0 VGA compatible controller: ATI Technologies Inc R423 5F57 [Radeon X800XT (PCIE)] (prog-if 00 [VGA])
        Subsystem: ATI Technologies Inc: Unknown device 0302
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, cache line size 10
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at c0000000 (32-bit, prefetchable) [size=256M]
        Region 1: I/O ports at dc00 [size=256]
        Region 2: Memory at dfde0000 (32-bit, non-prefetchable) [size=64K]
        Expansion ROM at dfe00000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] #10 [0001]
        Capabilities: [80] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
                Address: 0000000000000000  Data: 0000

01:00.1 Display controller: ATI Technologies Inc R423 5F57 [Radeon X800XT (PCIE)] (Secondary)
        Subsystem: ATI Technologies Inc: Unknown device 0303
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, cache line size 10
        Region 0: Memory at dfdf0000 (32-bit, non-prefetchable) [size=64K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] #10 [0001]

02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 01)
        Subsystem: Dell: Unknown device 0176
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, cache line size 10
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at dfcf0000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [48] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] Vital Product Data
        Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
                Address: fc7fbe069d20ffec  Data: fb03
        Capabilities: [d0] #10 [0001]

04:01.0 Multimedia audio controller: Creative Labs SB Audigy (rev 04)
        Subsystem: Creative Labs: Unknown device 2006
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 64 (500ns min, 5000ns max)
        Interrupt: pin A routed to IRQ 17
        Region 0: I/O ports at ccc0 [size=64]
        Capabilities: [dc] Power Management version 2
                Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

04:01.1 Input device controller: Creative Labs SB Audigy MIDI/Game port (rev 04)
        Subsystem: Creative Labs SB Audigy MIDI/Game Port
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 64
        Region 0: I/O ports at ccb8 [size=8]
        Capabilities: [dc] Power Management version 2
                Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

04:01.2 FireWire (IEEE 1394): Creative Labs SB Audigy FireWire Port (rev 04) (prog-if 10 [OHCI])
        Subsystem: Creative Labs SB Audigy FireWire Port
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 64 (500ns min, 1000ns max), cache line size 10
        Interrupt: pin B routed to IRQ 3
        Region 0: Memory at dfafb800 (32-bit, non-prefetchable) [size=2K]
        Region 1: Memory at dfafc000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [44] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

Re: kernel cache mem bug(?)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

On Thu, 16 Mar 2006 18:41:02 +0100, kernel@ministry.se said:
> [X.] Other notes, patches, fixes, workarounds:

>     Workaround: When we disable HyperThreading in BIOS, this
>     problem goes away. We re-enabling HT, it comes back...

Have you ruled out marginal memory, or overclocking/overheating?

I'm guessing something is barely within tolerance when one CPU is
beating up on it, and it falls over when the HT adds to the mix.

For that matter, this looks racy:

while [ 0$MD5LOOPS -gt 0 ]; do
  md5sum cache.*-*-* >> md5log.$PID.lis
  MD5LOOPS=`expr 0$MD5LOOPS - 1`
done &

 AMNT=`awk '$1!="91b82dcc83230890dbcdfc6b80571ddd"' md5log.$PID.lis | wc -l`

If md5sum uses stdio to write the output, and it writes more than 4K or so,
it's possible you can get a partial line right at the buffer boundary,
which will then come up as a mismatch according to the awk.  You might want
to actually output the mismatched line in its entirety, and make sure you're
looking at a complete line, and not a partial....

[-- Attachment #2: Type: application/pgp-signature, Size: 228 bytes --]

Re: kernel cache mem bug(?)

[kernel]

> On Thu, 16 Mar 2006 18:41:02 +0100, kernel@ministry.se said:
> > [X.] Other notes, patches, fixes, workarounds:
>
> >     Workaround: When we disable HyperThreading in BIOS, this
> >     problem goes away. We re-enabling HT, it comes back...
>
> Have you ruled out marginal memory, or overclocking/overheating?

No overclocking done, no overheating happening. All RAM memory checks
turn out just fine, with and without HT.

> For that matter, this looks racy:
>
> while [ 0$MD5LOOPS -gt 0 ]; do
>   md5sum cache.*-*-* >> md5log.$PID.lis
>   MD5LOOPS=`expr 0$MD5LOOPS - 1`
> done &
>
>  AMNT=`awk '$1!="91b82dcc83230890dbcdfc6b80571ddd"' md5log.$PID.lis | wc -l`
>
> If md5sum uses stdio to write the output, and it writes more than 4K or so,
> it's possible you can get a partial line right at the buffer boundary,
> which will then come up as a mismatch according to the awk.  You might want
> to actually output the mismatched line in its entirety, and make sure you're
> looking at a complete line, and not a partial....

Well, the script itself was not how we noticied the problem. It was
created to see if the problem was reproducable. It was, during mentioned
conditions.

Regarding a possible race condition, the log file (md5log.*.lis) is left
for manual inspection if needed.

Following line will fix this (possible) event:
  AMNT=`awk 'length($1)==32 && $1!="91b82dcc83230890dbcdfc6b80571ddd"' \
    md5log.$PID.lis | wc -l`

//D

Re: kernel cache mem bug(?)

[Jesper Juhl]

On 3/17/06, kernel@ministry.se <kernel@ministry.se> wrote:
> > On Thu, 16 Mar 2006 18:41:02 +0100, kernel@ministry.se said:
> > > [X.] Other notes, patches, fixes, workarounds:
> >
> > >     Workaround: When we disable HyperThreading in BIOS, this
> > >     problem goes away. We re-enabling HT, it comes back...
> >
> > Have you ruled out marginal memory, or overclocking/overheating?
>
> No overclocking done, no overheating happening. All RAM memory checks
> turn out just fine, with and without HT.
>

When you say "All RAM memory checks turn out just fine" do you mean
the BIOS memory checks or something else?

For a proper memory test run memtest86 (http://www.memtest86.com/)
and/or memtest86+ (http://www.memtest.org/) overnight (some 8-12hours
or more) with all tests enabled.. If the machine survives without
errors that then you can be reasonably sure memory is OK... The BIOS
tests are more or less worthless.


--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: kernel cache mem bug(?)

[kernel]

> > > >     Workaround: When we disable HyperThreading in BIOS, this
> > > >     problem goes away. We re-enabling HT, it comes back...
> > >
> > > Have you ruled out marginal memory, or overclocking/overheating?
> >
> > No overclocking done, no overheating happening. All RAM memory checks
> > turn out just fine, with and without HT.
> >
>
> When you say "All RAM memory checks turn out just fine" do you mean
> the BIOS memory checks or something else?
>
> For a proper memory test run memtest86 (http://www.memtest86.com/)
> and/or memtest86+ (http://www.memtest.org/) overnight (some 8-12hours
> or more) with all tests enabled.. If the machine survives without
> errors that then you can be reasonably sure memory is OK... The BIOS
> tests are more or less worthless.

The RAM is 100% OK, both according to the initial BIOS memory check and
memtest86 running for more than four (4) hours straight.

One more thing that we just noticed: it seems the cache corruption (or
whatever it is)  only occurs when X is running.

//D

Re: kernel cache mem bug(?)

[Alan Cox]

On Llu, 2006-03-20 at 09:09 +0100, kernel@ministry.se wrote:
> The RAM is 100% OK, both according to the initial BIOS memory check and
> memtest86 running for more than four (4) hours straight.
> 
> One more thing that we just noticed: it seems the cache corruption (or
> whatever it is)  only occurs when X is running.

X is special in a couple of ways - it accesses hardware directly and it
uses AGP. See if setting the NoAccel X option makes any difference to
the failures - it'll make your desktop slower but should help validate X
is the problem. If that does help then try disabling DRI (3D) support if
your box has it, and see what that does. If it works with noaccel and
fails with DRI then finally try with no AGP support built into your
kernel.  That should identify the problem as X, DRI, AGP or other.

Alan

Re: kernel cache mem bug(?)

[kernel]

> > The RAM is 100% OK, both according to the initial BIOS memory check and
> > memtest86 running for more than four (4) hours straight.
> >
> > One more thing that we just noticed: it seems the cache corruption (or
> > whatever it is)  only occurs when X is running.
>
> X is special in a couple of ways - it accesses hardware directly and it
> uses AGP. See if setting the NoAccel X option makes any difference to
> the failures - it'll make your desktop slower but should help validate X
> is the problem. If that does help then try disabling DRI (3D) support if
> your box has it, and see what that does. If it works with noaccel and
> fails with DRI then finally try with no AGP support built into your
> kernel.  That should identify the problem as X, DRI, AGP or other.

I gave it a try, running the same reboot & test script for every change:

  X with "noaccel"         - no change (i.e. corruption still occur)
  X w/o kernel fb support  - no change
  X without "dri"          - no change
  w/o AGP support in kernel  - no change
  X without "fgl"          - no change
  X without "glx"          - no change
  X without "fglrx"        - no change (ATI:s driver, used vesa instead)
  X without "dbe"          - no change
  X with only 1 screen     - no change
  X w/o Dev PCI:1...       - no change
  X without "dri"          - no change
  w/o kernel DRM support   - no change
  w/PCI acess=direct       - no change
  w/o kernel SMP HT support - no change

  and finally...

  w/o kernel SMP support   - 100% ok!

  (No corruption at all after 12 x 100 x 35 x 20 megabyte writing and
   reading to ram cache)

I.e, the only thing that seem to fix this is by removing SMP (and by
doing that also HT) support :(

The processor is an Intel P4 3.4GHz w/1MB L2 cache
The graphics card is an ATI X800 using the PCI Express slot
(see my first post for more detailed info).

Perhaps worth noting: when doing repeated md5sum runs on the cached files
and discrepancies are found, it is "always" different files. I.e., the
same file will never show a bad checksum twice. A subsequent run will show
a good checksum for a file which just seconds ago had a bad one.

Of course no other programs were accessing the files during these tests.

Even though disabling kernel SMP support entirely will make this problem
go away, I seriously do not like that kind of workarounds...

Please let me know if there is more system/hardware information I could
provide to help sort this out.

Regards,
//D