-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello. Since 2.6-preX series we are using XFS filesystem in the Spanish Debian mirror. The machine is under load most of the time, and I think that our periodic crashes (most of them occur only after some hours of operation, some in a couple of days) could be related to the main data filesystem. For example, today I rebooted the server. All is working, well, with FTP and HTTP concurrent access, and then I run the periodic script to sync up with official Debian mirrors. It is a recursive rsync over the whole tree, over 96 GB of size. Now it's running 2.6.6 vanilla. The crash is: May 12 15:12:00 ulises kernel: b44: eth0: Link is down. May 12 15:12:06 ulises kernel: b44: eth0: Link is up at 100 Mbps, full duplex. May 12 15:12:06 ulises kernel: b44: eth0: Flow control is on for TX and on for RX. May 12 15:14:18 ulises kernel: b44: eth0: Link is down. May 12 15:14:21 ulises kernel: b44: eth0: Link is up at 100 Mbps, full duplex. May 12 15:14:21 ulises kernel: b44: eth0: Flow control is on for TX and on for RX. May 12 15:18:01 ulises kernel: b44: eth0: Link is down. May 12 15:18:04 ulises kernel: b44: eth0: Link is up at 100 Mbps, full duplex. May 12 15:18:04 ulises kernel: b44: eth0: Flow control is on for TX and on for RX. May 12 15:18:42 ulises kernel: b44: eth0: Link is down. May 12 15:18:45 ulises kernel: b44: eth0: Link is up at 100 Mbps, full duplex. May 12 15:18:45 ulises kernel: b44: eth0: Flow control is on for TX and on for RX. May 12 15:19:17 ulises kernel: b44: eth0: Link is down. May 12 15:19:19 ulises kernel: b44: eth0: Link is up at 100 Mbps, full duplex. May 12 15:19:19 ulises kernel: b44: eth0: Flow control is on for TX and on for RX. May 12 16:32:48 ulises kernel: printing eip: May 12 16:32:49 ulises kernel: c0216e8f May 12 15:21:16 ulises kernel: Oops: 0000 [#1] May 12 15:21:16 ulises kernel: CPU: 0 May 12 15:21:17 ulises kernel: EIP: 0060:[xfs_count_page_state+49/138] Not tainted May 12 15:21:17 ulises kernel: EFLAGS: 00010206 (2.6.6) May 12 15:21:17 ulises kernel: EIP is at xfs_count_page_state+0x31/0x8a May 12 15:21:17 ulises kernel: eax: 736ef8dd ebx: c15f1ce8 ecx: ce048460 edx: e6de4d26 May 12 15:21:18 ulises kernel: esi: c15f1ce4 edi: c15f1ce0 ebp: c15f1ddc esp: c15f1cc0 May 12 15:21:18 ulises kernel: ds: 007b es: 007b ss: 0068 May 12 15:21:18 ulises kernel: Process kswapd0 (pid: 9, threadinfo=c15f1000 task=c15c4630) May 12 15:21:18 ulises kernel: Stack: c10444e0 d921609c 00000001 c0217007 c10444e0 c15f1ce0 c15f1ce4 c15f1ce8 May 12 15:21:18 ulises kernel: 00000000 00000000 00000001 00000000 000000d0 c01408b2 c10444e0 000000d0 May 12 15:21:18 ulises kernel: d9216134 c10444e0 c012eee2 c10444e0 000000d0 d2f46b00 00000080 00000008 May 12 15:21:18 ulises kernel: Call Trace: May 12 15:21:18 ulises kernel: [linvfs_release_page+52/156] linvfs_release_page+0x34/0x9c May 12 15:21:18 ulises kernel: [try_to_release_page+81/107] try_to_release_page+0x51/0x6b May 12 15:21:18 ulises kernel: [shrink_list+797/1005] shrink_list+0x31d/0x3ed May 12 15:21:18 ulises kernel: [vn_purge+264/278] vn_purge+0x108/0x116 May 12 15:21:18 ulises kernel: [schedule+749/1272] schedule+0x2ed/0x4f8 May 12 15:21:18 ulises kernel: [shrink_cache+334/787] shrink_cache+0x14e/0x313 May 12 15:21:18 ulises kernel: [blk_congestion_wait+133/142] blk_congestion_wait+0x85/0x8e May 12 15:21:18 ulises kernel: [shrink_slab+132/365] shrink_slab+0x84/0x16d May 12 15:21:18 ulises kernel: [balance_pgdat+466/603] balance_pgdat+0x1d2/0x25b May 12 15:21:18 ulises kernel: [kswapd+274/290] kswapd+0x112/0x122 May 12 15:21:18 ulises kernel: [autoremove_wake_function+0/79] autoremove_wake_function+0x0/0x4f May 12 15:21:18 ulises kernel: [autoremove_wake_function+0/79] autoremove_wake_function+0x0/0x4f May 12 15:21:18 ulises kernel: [kswapd+0/290] kswapd+0x0/0x122 May 12 15:21:18 ulises kernel: [kernel_thread_helper+5/11] kernel_thread_helper+0x5/0xb May 12 15:21:18 ulises kernel: May 12 15:21:18 ulises kernel: Code: 8b 02 a8 01 74 17 8b 02 a8 10 75 11 c7 06 01 00 00 00 8b 52 After that, the XFS fs is unmountable. The only remarkable thing is that the fs is on top of a RAID0 across two 180 GB IDE disks. I checked the filesystem with xfs_check 2.6.5 without errors. What I fear is that the kernel dies under XFS pressure. But I don't have relevant info against that, mainly because the FTP/HTTP mirror is the only source of pressure (i.e. concurrent accesses) in the machine. The data access pattern is ~200 users downloading big (~650 MB) files (in fact, almost all the users try to download the same 3 main images). Other data: Intel P4 2.5 GHz 512 MB RAM mount: ulises:~# mount /dev/hda5 on / type ext3 (rw,errors=remount-ro) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) usbfs on /proc/bus/usb type usbfs (rw) /dev/hda1 on /boot type ext3 (rw) /dev/hda6 on /var type ext3 (rw) /dev/hda8 on /home type ext3 (rw) /dev/md0 on /mirror type xfs (rw) raidtab: ulises:~# cat /etc/raidtab raiddev /dev/md0 raid-level 0 nr-raid-disks 2 persistent-superblock 1 chunk-size 16 device /dev/hde1 raid-disk 0 device /dev/hdg1 raid-disk 1 Interesting excerpts from dmesg: [...] ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx ICH4: IDE controller at PCI slot 0000:00:1f.1 ICH4: chipset revision 2 ICH4: not 100% native mode: will probe irqs later ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:pio, hdd:pio hda: ST320423A, ATA DISK drive hdb: HL-DT-STDVD-ROM GDR8161B, ATAPI CD/DVD-ROM drive Using anticipatory io scheduler ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 SiI3112 Serial ATA: IDE controller at PCI slot 0000:02:0e.0 SiI3112 Serial ATA: chipset revision 2 SiI3112 Serial ATA: 100% native mode on irq 18 ide2: MMIO-DMA , BIOS settings: hde:pio, hdf:pio ide3: MMIO-DMA , BIOS settings: hdg:pio, hdh:pio hde: ST3120026AS, ATA DISK drive ide2 at 0xe0820080-0xe0820087,0xe082008a on irq 18 hdg: ST3120026AS, ATA DISK drive ide3 at 0xe08200c0-0xe08200c7,0xe08200ca on irq 18 hda: max request size: 128KiB hda: 40011300 sectors (20485 MB) w/512KiB Cache, CHS=39693/16/63, UDMA(66) hda: hda1 hda2 < hda5 hda6 hda7 hda8 > hde: max request size: 64KiB hde: 234441648 sectors (120034 MB) w/8192KiB Cache, CHS=16383/255/63 hde: hde1 hdg: max request size: 64KiB hdg: 234441648 sectors (120034 MB) w/8192KiB Cache, CHS=16383/255/63 hdg: hdg1 hdb: ATAPI 48X DVD-ROM drive, 256kB Cache, UDMA(33) [...] lspci: 0000:00:00.0 Host bridge: Intel Corp. 82845G/GL[Brookdale-G]/GE/PE DRAM Controller/Host-Hub Interface (rev 02) 0000:00:01.0 PCI bridge: Intel Corp. 82845G/GL[Brookdale-G]/GE/PE Host-to-AGP Bridge (rev 02) 0000:00:1d.0 USB Controller: Intel Corp. 82801DB (ICH4) USB UHCI #1 (rev 02) 0000:00:1d.1 USB Controller: Intel Corp. 82801DB (ICH4) USB UHCI #2 (rev 02) 0000:00:1d.2 USB Controller: Intel Corp. 82801DB (ICH4) USB UHCI #3 (rev 02) 0000:00:1d.7 USB Controller: Intel Corp. 82801DB (ICH4) USB2 EHCI Controller (rev 02) 0000:00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI Bridge (rev 82) 0000:00:1f.0 ISA bridge: Intel Corp. 82801DB (ICH4) LPC Bridge (rev 02) 0000:00:1f.1 IDE interface: Intel Corp. 82801DB (ICH4) Ultra ATA 100 Storage Controller (rev 02) 0000:00:1f.3 SMBus: Intel Corp. 82801DB/DBM (ICH4) SMBus Controller (rev 02) 0000:00:1f.5 Multimedia audio controller: Intel Corp. 82801DB (ICH4) AC'97 Audio Controller (rev 02) 0000:01:00.0 VGA compatible controller: nVidia Corporation NV5M64 [RIVA TNT2 Model 64/Model 64 Pro] (rev 15) 0000:02:00.0 Ethernet controller: Broadcom Corporation BCM4401 100Base-T (rev 01) 0000:02:0e.0 RAID bus controller: CMD Technology Inc Silicon Image SiI 3112 SATARaid Controller (rev 02) I attach my kernel config. Am I overlooking anything? The box ran a complete memtest86 test last month without errors... Many thanks in advance, Ender. - -- To the infinity, and beyond! -- Buzz Lightyear (Toy Story). - -- Servicios de red - Network services RedIRIS - Spanish Academic Network for Research and Development Red.es - Madrid (Spain) Tlf (+34) 91.212.76.25 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFAozznWs/EhA1iABsRAon5AJ9bDC3A1Ke03CJSBBEWazBWKtXCNgCfXzJF N3k7sgLDHyFRp1iPzo5OoNw= =GwQL -----END PGP SIGNATURE-----