raid5:md3: read error corrected , followed by , Machine Check Exception: .

raid5:md3: read error corrected , followed by , Machine Check Exception: .

[Mr. James W. Laferriere]

 	Hello All ,  I was under the impression that a 'machine check' would be 
caused by some near to the CPU hardware failure ,  Not a bad disk ?
 	I was also under the impression that software raid s/b a little more 
resilient than this .
 	But then maybe one or the other of the subsystems walked all over the 
other one trying to correct itself .  Who knows .
 	The below is ALL I'll be able to get .  A log of a previous boot is 
available at ...

http://www.baby-dragons.com/test-2.6.21.5-mptscsi-4.00.10.00-2007006161326.log

 		Tia ,  JimL

filesrv2 login: Jul 14 22:58:48 filesrv2 -- MARK --
Jul 14 23:00:26 filesrv2 kernel: sd 2:0:1:0: SCSI error: return code = 0x08000002
Jul 14 23:00:26 filesrv2 kernel: sdd: Current: sense key: Medium Error
Jul 14 23:00:26 filesrv2 kernel:     Additional sense: Read retries exhausted
Jul 14 23:00:26 filesrv2 kernel: Info fld=0x2f9c0f8
Jul 14 23:00:26 filesrv2 kernel: end_request: I/O error, dev sdd, sector 49922296
Jul 14 23:00:27 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 49922296 on sdd)
Jul 14 23:17:06 filesrv2 kernel: sd 2:0:1:0: SCSI error: return code = 0x08000002
Jul 14 23:17:06 filesrv2 kernel: sdd: Current: sense key: Medium Error
Jul 14 23:17:06 filesrv2 kernel:     Additional sense: Read retries exhausted
Jul 14 23:17:06 filesrv2 kernel: Info fld=0x44ebd82
Jul 14 23:17:06 filesrv2 kernel: end_request: I/O error, dev sdd, sector 72269186
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269184 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269192 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269200 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269208 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269216 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269224 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269232 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269240 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269248 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269256 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269264 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269272 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269280 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269288 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269296 on sdd)
Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269304 on sdd)
Jul 14 23:20:11 filesrv2 kernel: sd 2:0:1:0: SCSI error: return code = 0x08000002
Jul 14 23:20:11 filesrv2 kernel: sdd: Current: sense key: Medium Error
Jul 14 23:20:11 filesrv2 kernel:     Additional sense: Read retries exhausted
Jul 14 23:20:11 filesrv2 kernel: Info fld=0x4895ac1
Jul 14 23:20:11 filesrv2 kernel: end_request: I/O error, dev sdd, sector 76110529
Jul 14 23:20:12 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76110528 on sdd)
Jul 14 23:20:12 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76110536 on sdd)
Jul 14 23:20:12 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76110544 on sdd)
Jul 14 23:20:12 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76110552 on sdd)
Jul 14 23:20:12 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76110560 on sdd)
Jul 14 23:20:12 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76110568 on sdd)
Jul 14 23:20:12 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76110576 on sdd)
Jul 14 23:20:12 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76110584 on sdd)
Jul 14 23:20:28 filesrv2 kernel: sd 2:0:1:0: SCSI error: return code = 0x08000002
Jul 14 23:20:28 filesrv2 kernel: sdd: Current: sense key: Medium Error
Jul 14 23:20:28 filesrv2 kernel:     Additional sense: Read retries exhausted
Jul 14 23:20:28 filesrv2 kernel: Info fld=0x48d97ac
Jul 14 23:20:28 filesrv2 kernel: end_request: I/O error, dev sdd, sector 76388268
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388264 on sdd)
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388272 on sdd)
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388280 on sdd)
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388288 on sdd)
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388296 on sdd)
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388304 on sdd)
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388312 on sdd)
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388320 on sdd)
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388328 on sdd)
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388336 on sdd)
Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388344 on sdd)
Jul 14 23:38:48 filesrv2 -- MARK --
CPU 5: Machine Check Exception: 0000000000000004
CPU 4: Machine Check Exception: 0000000000000004
Kernel panic - not syncing: Unable to continue



-- 
+-----------------------------------------------------------------+
| James   W.   Laferriere | System   Techniques | Give me VMS     |
| Network        Engineer | 663  Beaumont  Blvd |  Give me Linux  |
| babydr@baby-dragons.com | Pacifica, CA. 94044 |   only  on  AXP |
+-----------------------------------------------------------------+

Re: raid5:md3: read error corrected , followed by , Machine Check Exception: .

[Justin Piszcz]

On Sat, 14 Jul 2007, Mr. James W. Laferriere wrote:

> 	Hello All ,  I was under the impression that a 'machine check' would 
> be caused by some near to the CPU hardware failure ,  Not a bad disk ?
> 	I was also under the impression that software raid s/b a little more 
> resilient than this .
> 	But then maybe one or the other of the subsystems walked all over the 
> other one trying to correct itself .  Who knows .
> 	The below is ALL I'll be able to get .  A log of a previous boot is 
> available at ...
>
> http://www.baby-dragons.com/test-2.6.21.5-mptscsi-4.00.10.00-2007006161326.log

Looks like /dev/sdd is on its way out, repace the drive?  I'm not sure
however why you'd get those MCEs at the end unless you really have bad
CPUs or memory in the system?

Justin.

Re: raid5:md3: read error corrected , followed by , Machine Check Exception: .

[Alan Cox]

On Sat, 14 Jul 2007 17:08:27 -0700 (PDT)
"Mr. James W. Laferriere" <babydr@baby-dragons.com> wrote:

>  	Hello All ,  I was under the impression that a 'machine check' would be 
> caused by some near to the CPU hardware failure ,  Not a bad disk ?

It indicates a hardware failure

> Jul 14 23:00:26 filesrv2 kernel: sd 2:0:1:0: SCSI error: return code = 0x08000002
> Jul 14 23:00:26 filesrv2 kernel: sdd: Current: sense key: Medium Error
> Jul 14 23:00:26 filesrv2 kernel:     Additional sense: Read retries exhausted

So your disk throws a fit

> Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269184 on sdd)
> Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269192 on sdd)

Raid happily recovers the mess

> Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388336 on sdd)
> Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388344 on sdd)
> Jul 14 23:38:48 filesrv2 -- MARK --
> CPU 5: Machine Check Exception: 0000000000000004
> CPU 4: Machine Check Exception: 0000000000000004

And at some point at least 18 minutes after the raid incident you log
CPU problems.

Are you sure the box or the room it is in didn't get excessively hot ?

Re: raid5:md3: read error corrected , followed by , Machine Check Exception: .

[Mr. James W. Laferriere]

 	Hello Alan (& Justin) ,

On Sun, 15 Jul 2007, Alan Cox wrote:
> On Sat, 14 Jul 2007 17:08:27 -0700 (PDT)
> "Mr. James W. Laferriere" <babydr@baby-dragons.com> wrote:
>
>>  	Hello All ,  I was under the impression that a 'machine check' would be
>> caused by some near to the CPU hardware failure ,  Not a bad disk ?
>
> It indicates a hardware failure

 	OK ,  Be funny :-} .  Tho it ain't .  I still think I am correct in 
stating that a "'machine check' would be caused by some near to the CPU hardware 
failure" .  Ie: like physically shoved into the motherboard .  Not a 
indirectly connected device .  Such as a disk drive .


>> Jul 14 23:00:26 filesrv2 kernel: sd 2:0:1:0: SCSI error: return code = 0x08000002
>> Jul 14 23:00:26 filesrv2 kernel: sdd: Current: sense key: Medium Error
>> Jul 14 23:00:26 filesrv2 kernel:     Additional sense: Read retries exhausted
>
> So your disk throws a fit

 	Actually it's brand new .  Infant mortallity ?  I at least have a cold 
spare available .  So Yes I am replacing that puppy .  I'll drop it into another 
system & give it the format command & see how much the user bad block table 
grows .  I'll bet I'll get a table full overflow on it .


>> Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269184 on sdd)
>> Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 72269192 on sdd)
>
> Raid happily recovers the mess

 	That they did .  scsi & raid .


>> Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388336 on sdd)
>> Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors at 76388344 on sdd)
>> Jul 14 23:38:48 filesrv2 -- MARK --
>> CPU 5: Machine Check Exception: 0000000000000004
>> CPU 4: Machine Check Exception: 0000000000000004
>
> And at some point at least 18 minutes after the raid incident you log
> CPU problems.

 	I didn't notice the 18 Minute differance .  Drats .
 	The 'MCE's have been ongoing for sometime .  I have replaced every item 
in the system except the chassis & scsi backplane & power supply(750Watts) .
 	Everything .  MB,cpu,memory,scsi controllers, ...
 	These MCE's only happen when I am trying to build or bonnie++ test the 
md3 .  It consists of (now 7+1spare) 146GB drives in the SuperMicro 
SYS-6035B-8B's backplane attached to a LSI22320 .

> Are you sure the box or the room it is in didn't get excessively hot ?

 	The room before and at the time of the MCE was never over 71F .  Now the 
memory temp might be a tad hot but I've got a small jet engine running in that 
chassis as noisy as it is with fans so I don't see how anything in there could 
be too hot .
 	Temp this time ,  71.2F .  everytime I try that array it MCE's .
 	I have another array in a JBOD box of 14 disks & I have no problems 
with this system as long as I disconnect all the drives in the 8 drive Backplane .
 	I guess I'll just have to do without the extra 143,564,800K of data 
space .
 		Twyl ,  JimL

ps: system face diagram ...
   +-------------------------+
   |0|1|2|3|4|6|7|-----------| 
+-| | | | | | | |-----------|
| | | | | | | | |-----------|
| +-------------------------+
|  \           /
|   +---md3---+
|
| +-------------------------+
| |0|1|2|3|4|6|0|1|2|3|4|5|6|
+-| | | | | | | | | | | | | |
   | | | | | | | | | | | | | |
   +-------------------------+
    \           /\          /
     +---md4---+  +---md4--+
-- 
+-----------------------------------------------------------------+
| James   W.   Laferriere | System   Techniques | Give me VMS     |
| Network        Engineer | 663  Beaumont  Blvd |  Give me Linux  |
| babydr@baby-dragons.com | Pacifica, CA. 94044 |   only  on  AXP |
+-----------------------------------------------------------------+

Re: raid5:md3: read error corrected , followed by , Machine Check Exception: .

[Rogier Wolff]

On Sat, Jul 14, 2007 at 07:32:38PM -0700, Mr. James W. Laferriere wrote:
> >So your disk throws a fit
> 
> 	Actually it's brand new .  Infant mortallity ?  I at least have a 
> 	cold spare available .  So Yes I am replacing that puppy .  I'll drop it 
> into another system & give it the format command & see how much the user 
> bad block table grows .  I'll bet I'll get a table full overflow on it .

Manually keep both tables under supervision. I'd guess that if you
send it a format, it will update the factory table (and move the user
bad block table there). 

But most will retry writing to the bad sectors. And with a fully-fresh
copy of the data, it will still be readable, and the blocks will be
marked as ready-for-use, because that's better for performance....

> >And at some point at least 18 minutes after the raid incident you log
> >CPU problems.
> 
> 	I didn't notice the 18 Minute differance .  Drats .

I'm not sure how this happened, but the disk errror messages seem to have
been logged by syslog, and the MCEs seem to have been copied from the 
console: They don't have the date attached....?

	Roger.
-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ