XFS filesystem corruption: 2.6.0. Massive failure. With raid5

XFS filesystem corruption: 2.6.0. Massive failure. With raid5

[Jerry Haltom]

This has happened twice now. Massive XFS file system corruption. The
system is running on a 3ware card in Raid5 config. / is XFS. Cannot
mount:

XFS: log has mismatchd uuid - can't recover
XFS: failed to find log head
XFS: log mount/recovery/failed
...

xfs_repair lets me know a lot of stuff, and:

* ERROR: mismathced uuid in log
* SB: some long number
* log: a slightly different long number

It doesn't work.

xfs_logprint -t /dev/sda4 produces a lot of illegal type errors and ends
up with Segmentation fault (uh oh).

I am willing to give an XFS developer access to the machine to poke
around (and fix it!!!) It's just my desktop at home, not a big deal. But
I have a ton of large files i'd like to recover. :)

I could fix it by forcing hte logs clean, that is what I did the first
time this happened. However, I lost a lot of files last time, and this
shouldn't happen. So here it is for you guys. I am hanging out in #xfs
on irc.freenode.net if anybody wants to check it out.

Jerry Haltom
Feedback Plus, Inc.

Re: XFS filesystem corruption: 2.6.0. Massive failure. With raid5

[Miquel van Smoorenburg]

In article <1072425031.737.9.camel@osaka>,
Jerry Haltom  <jhaltom@feedbackplusinc.com> wrote:
>This has happened twice now. Massive XFS file system corruption. The
>system is running on a 3ware card in Raid5 config. / is XFS. Cannot
>mount:
>
>XFS: log has mismatchd uuid - can't recover
>XFS: failed to find log head
>XFS: log mount/recovery/failed
>...
>
>xfs_repair lets me know a lot of stuff, and:
>
>* ERROR: mismathced uuid in log
>* SB: some long number
>* log: a slightly different long number
>
>It doesn't work.
>
>xfs_logprint -t /dev/sda4 produces a lot of illegal type errors and ends
>up with Segmentation fault (uh oh).

I'm testing the same setup here right now, and you got me a bit worried..
If you know of a way to reproduce the bug, I can see if I can
replicate it on my hardware.

Mike.
-- 
When life hands you lemons, grab the salt and pass the tequila.

Re: XFS filesystem corruption: 2.6.0. Massive failure. With raid5

[Eric Sandeen]

On Fri, 26 Dec 2003, Jerry Haltom wrote:

> This has happened twice now. Massive XFS file system corruption. The
> system is running on a 3ware card in Raid5 config. / is XFS. Cannot
> mount:
> 
> XFS: log has mismatchd uuid - can't recover
> XFS: failed to find log head
> XFS: log mount/recovery/failed

Either a corrupt log, or a misstated external log, it seems.

Ah, on irc you said only the first part of the uuid was off,
that's a bit odd.  Knowing the actual numbers would be very
helpful.  Perhaps the raid swizzled some bits?

> xfs_repair lets me know a lot of stuff, and:
> 
> * ERROR: mismathced uuid in log
> * SB: some long number
> * log: a slightly different long number

Those numbers may be important, without the real output this isn't very
helpful.

> It doesn't work.
> 
> xfs_logprint -t /dev/sda4 produces a lot of illegal type errors and ends
> up with Segmentation fault (uh oh).

Hm, so the log was in very bad shape.  I don't know how it got there;
it may or may not be xfs's fault, but it shouldn't segfault.  If you have
the core file maybe we can take a look.

> I could fix it by forcing hte logs clean, that is what I did the first
> time this happened. However, I lost a lot of files last time, and this
> shouldn't happen. So here it is for you guys. I am hanging out in #xfs
> on irc.freenode.net if anybody wants to check it out.

Sounds like you've already repaired it and whacked the log, so no use
now.  It would also be interesting to know if any I/O or other errors
occurred in the system before the problems.  You might also run without
the nvidia driver* for a while and see if things go better.

-Eric

*info gleaned from irc