Hi, A week ago I wrote to linux-raid mailing list about my software raid5 problem (post included below), but I encoutered some new information about Western Digital WD3200JD-60KLB0 disks and maybe its "powersaving" is the real problem. What do you think? ------------- Ehlo, Here is a particularly interesting problem. I have 1TB RAID5 production arrary and a lot of stress ;) It's gonna be a very long mail and I think a good case study but unfortunately still without a solution. I've checked almost everything and need your support, suggestions etc. Feedback: 4x WD320G SATA Software RAID managed by mdadm - v2.3 - 2 February 2006 System Debian GNU/Linux: Linux raidmaster 2.6.14.2-1 #40 Tue Apr 18 10:11:15 CEST 2006 i686 GNU/Linux RAID1: /dev/md0 on / type reiserfs (rw) : /dev/sda1 /dev/sdb1 RAID1: /dev/md1 on /var type reiserfs (rw) : /dev/sdc1 /dev/sdd1 RAID5: /dev/md2 on /home type reiserfs (rw,usrquota) : /dev/sda3 /dev/sdb3 /dev/sdd2 /dev/sdc2 Device Boot Start End Blocks Id System /dev/sda1 1 122 979933+ 82 Linux swap / Solaris /dev/sda2 * 123 365 1951897+ fd Linux raid autodetect /dev/sda3 366 38913 309636810 fd Linux raid autodetect Disk /dev/sdb: 320.0 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 122 979933+ 82 Linux swap / Solaris /dev/sdb2 * 123 365 1951897+ fd Linux raid autodetect /dev/sdb3 366 38913 309636810 fd Linux raid autodetect Disk /dev/sdc: 320.0 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 1 365 2931831 fd Linux raid autodetect /dev/sdc2 366 38913 309636810 fd Linux raid autodetect Disk /dev/sdd: 320.0 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdd1 1 365 2931831 fd Linux raid autodetect /dev/sdd2 366 38913 309636810 fd Linux raid autodetect 0000:00:00.0 Host bridge: Intel Corp. 915G/P/GV Processor to I/O Controller (rev 04) 0000:00:01.0 PCI bridge: Intel Corp. 915G/P/GV PCI Express Root Port (rev 04) 0000:00:1c.0 PCI bridge: Intel Corp. 82801FB/FBM/FR/FW/FRW (ICH6 Family) PCI Express Port 1 (rev 03) 0000:00:1f.2 IDE interface: Intel Corp. 82801FB/FW (ICH6/ICH6W) SATA Controller (rev 03) Example of error: ------------ ATA: abnormal status 0x80 on port 0xE807 ATA: abnormal status 0x80 on port 0xE807 ATA: abnormal status 0x80 on port 0xE807 ata1: command 0x35 timeout, stat 0x51 host_stat 0x61 ata1: status=0x51 { DriveReady SeekComplete Error } ata1: error=0x04 { DriveStatusError } sd 0:0:0:0: SCSI error: return code = 0x8000002 sda: Current: sense key=0xb ASC=0x0 ASCQ=0x0 end_request: I/O error, dev sda, sector 625137197 raid5: Disk failure on sda3, disabling device. Operation continuing on 3 devices ------------- Another error: ------------- ata2: command 0x35 timeout, stat 0x51 host_stat 0x61 ata2: status=0x51 { DriveReady SeekComplete Error } ata2: error=0x04 { DriveStatusError } sd 1:0:0:0: SCSI error: return code = 0x8000002 sdc: Current: sense key=0xb ASC=0x0 ASCQ=0x0 end_request: I/O error, dev sdc, sector 5863487 raid1: Disk failure on sdc1, disabling device. Operation continuing on 1 devices -------------- Every night I make a 200G backup through the network (samba + scp) and next day morning RAID fails. Recently I had 1Gbit NIC and had to off TCP Segmentation Offload (TSO) using ethtool because of NIC reseting and errors. Now there is a brand new motherboard with 100Mbit NIC. The most surprising thing is that mostly partition from RAID5 fails but sometimes also partitions from RAID1. What was done: 1. All SATA cables were changed 2. Disks were checked with TuffTest Pro software: surface and seek tests. 3. RAM was checked with memtest86 4. SATA controler was changed to Silicon Image SIS. 5. Power Supply was changed: 400W ChiefTec 6. Mother Board was changed [sic!]. The First was Intel Desktop Board D955XBK + Celeron 2.8GHz, 1GB RAM. The second one Intel Desktop Board D915GAG + Pentium4 3Ghz, 1GB RAM 7. I thought tha ACPI and disk suspend may cause problems. I didn't have ACPI support compiled and the BIOS might took control over power management. 8. I add additional coolers to cool harddrives. 9. BIOS was upgraded. 10. I found cash for Adaptec HW RAID controler ;) but I believe in power and stability of Linux Software RAID. In attachment you will find my kernel config. Additional RAID information: /dev/md0: Version : 00.90.02 Creation Time : Mon Mar 20 19:14:04 2006 Raid Level : raid1 Array Size : 1951808 (1906.38 MiB 1998.65 MB) Device Size : 1951808 (1906.38 MiB 1998.65 MB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Thu Apr 20 13:48:17 2006 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : cbbf5e2c:6708ab7c:70e6e865:cde7da1e Events : 0.153302 Number Major Minor RaidDevice State 0 8 2 0 active sync /dev/sda2 1 8 18 1 active sync /dev/sdb2 /dev/md1: Version : 00.90.02 Creation Time : Mon Mar 20 19:14:24 2006 Raid Level : raid1 Array Size : 2931712 (2.80 GiB 3.00 GB) Device Size : 2931712 (2.80 GiB 3.00 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 1 Persistence : Superblock is persistent Update Time : Thu Apr 20 13:49:22 2006 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : 03450c7b:a054486e:92c8a7f8:9a30a85c Events : 0.1579556 Number Major Minor RaidDevice State 0 8 49 0 active sync /dev/sdd1 1 8 33 1 active sync /dev/sdc1 /dev/md2: Version : 00.90.02 Creation Time : Mon Mar 20 19:14:40 2006 Raid Level : raid5 Array Size : 928910208 (885.88 GiB 951.20 GB) Device Size : 309636736 (295.29 GiB 317.07 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 2 Persistence : Superblock is persistent Update Time : Thu Apr 20 13:46:57 2006 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K UUID : 3bd09e5e:f2e20e49:92bb4575:3784041c Events : 0.525277 Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 1 8 19 1 active sync /dev/sdb3 2 8 50 2 active sync /dev/sdd2 3 8 34 3 active sync /dev/sdc2 Regards. -- Marcin Hlybin, marcin.hlybin@swmind.com Sys/Net Administrator, tel. +48 12 2523 402 SoftwareMind, www.softwaremind.pl | Where quality meets the future