Spurious Filesystem corruption with ext3 + large (<400GB) hw RAID partition + SMP

Spurious Filesystem corruption with ext3 + large (<400GB) hw RAID partition + SMP

[Manfred Wassmann]

Hello,

I'm experiencing spurious filesystem corruption problems with ext3 on
a partition larger than 400 GB on a hardware SATA RAID.

The problems first occured on an Intel dual core system with a custom
installation DVD which uses a 32 bit 2.6.26 kernel with SMP enabled
and a 4GB memory limit. The problems occured only randomly but it was
possible to reproduce them reliably with a standard Ubunu installation
disk using a 2.6.27-7 kernel (apparently also 32bit SMP kernel w 4GB
limit).
When installing Ubuntu a message appears in the kernel log stating:
"EXT3-fs error (device sda6): ext3_vlid_block_bitmap: Invalid block
bitmap - block_group = 3076, block = <some large number>"
This message is the first indication of any problem, the filesystem is
then remounted readonly and the installation stops understandably.

The problem does only occur with an ext3 filesystem, installation
using an xfs filesystem completes without error.

Furthermore with the installation from my custom installation DVD
succeeded, which installs a prebuild Debian etch system featuring a 64
bit monolithic 2.6.26 kernel, i was able to stress the system to its
limits using the stress and dbench programs without any sign of
filesystem corruption.

As XFS does substantially outperform ext3 under the given
circumstances I'm going to use that but if someone wants to
investigate the problem and needs further details I'll do my very best
;-)

regards Manfred Wassmann

Re: Spurious Filesystem corruption with ext3 + large (<400GB) hw RAID partition + SMP

[Henrique de Moraes Holschuh]

On Sat, 31 Jan 2009, Manfred Wassmann wrote:
> I'm experiencing spurious filesystem corruption problems with ext3 on
> a partition larger than 400 GB on a hardware SATA RAID.

Please describe the SATA RAID hardware in a bit more detail.  Which
SATA controller does it use?

Also, have you made sure it is not caused by bad memory?

> The problems first occured on an Intel dual core system with a custom
> installation DVD which uses a 32 bit 2.6.26 kernel with SMP enabled
> and a 4GB memory limit. The problems occured only randomly but it was
> possible to reproduce them reliably with a standard Ubunu installation
> disk using a 2.6.27-7 kernel (apparently also 32bit SMP kernel w 4GB
> limit).
> When installing Ubuntu a message appears in the kernel log stating:
> "EXT3-fs error (device sda6): ext3_vlid_block_bitmap: Invalid block
> bitmap - block_group = 3076, block = <some large number>"
> This message is the first indication of any problem, the filesystem is
> then remounted readonly and the installation stops understandably.
> 
> The problem does only occur with an ext3 filesystem, installation
> using an xfs filesystem completes without error.
> 
> Furthermore with the installation from my custom installation DVD
> succeeded, which installs a prebuild Debian etch system featuring a 64
> bit monolithic 2.6.26 kernel, i was able to stress the system to its
> limits using the stress and dbench programs without any sign of
> filesystem corruption.
> 
> As XFS does substantially outperform ext3 under the given
> circumstances I'm going to use that but if someone wants to
> investigate the problem and needs further details I'll do my very best
> ;-)

Be careful.  XFS might not trigger the problem as readly, but it could
still be there and you can suffer data loss.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Spurious Filesystem corruption with ext3 + large (<400GB) hw RAID partition + SMP

[Manfred Wassmann]

On Sat, Jan 31, 2009 at 2:46 PM, Henrique de Moraes Holschuh
<hmh@hmh.eng.br> wrote:
> On Sat, 31 Jan 2009, Manfred Wassmann wrote:
>> I'm experiencing spurious filesystem corruption problems with ext3 on
>> a partition larger than 400 GB on a hardware SATA RAID.
>
> Please describe the SATA RAID hardware in a bit more detail.  Which
> SATA controller does it use?

It is a 3ware 9650SE initially supplied with a firmware version FE9X
3.08.00.029 but the problem remained after an upgrade to the current
version 4.06.00.004

> Also, have you made sure it is not caused by bad memory?

I didn't think it could be the memory because the problem occurs with
the 32bit kernel alone but what do you suggest? I only have remote
access to the system over the weekend.

-- 
Unix "Birthday" on 2009-02-13 23:31:30 UTC
...it's 1234567890 seconds since the epoch.

Re: Spurious Filesystem corruption with ext3 + large (<400GB) hw RAID partition + SMP

[Henrique de Moraes Holschuh]

On Sat, 31 Jan 2009, Manfred Wassmann wrote:
> It is a 3ware 9650SE initially supplied with a firmware version FE9X
> 3.08.00.029 but the problem remained after an upgrade to the current
> version 4.06.00.004

Search the list archives, there are other problem reports for that
3ware card, you probably want to check those.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: Spurious Filesystem corruption with ext3 + large (<400GB) hw RAID partition + SMP

[Manfred Wassmann]

On 2/1/09, Henrique de Moraes Holschuh <hmh@hmh.eng.br> wrote:
[...]
> Search the list archives, there are other problem reports for that
> 3ware card, you probably want to check those.

Thank you very much, I checked these but I could not find much about
filesystem corruption problems. The most similar problem was mentioned
in http://lkml.org/lkml/2008/6/23/230 but was related to xen which
isn't in use here.

Regarding the memory test, memtest86+ ran for almost 70 hours without
detecting any error.

I trying to reproduce the problem using my installation disk and
running stress/dbench but wasn't able to do so. The only reliable way
to reproduce it is to install Ubuntu, which isn't very helpful :-(

As already said I'm no longer using ext3 for the systems in question,
so if switching filesystems really does solve my problem, this will be
my last post in this regard. In any case thanks for your  help.

regards Manfred
-- 
Unix "Birthday" on 2009-02-13 23:31:30 UTC
...it's 1234567890 seconds since the epoch.

Re: Spurious Filesystem corruption with ext3 + large (<400GB) hw RAID partition + SMP

[Ray Lee]

On Fri, Feb 6, 2009 at 10:08 AM, Manfred Wassmann
<tux.wassmann@googlemail.com> wrote:
> On 2/1/09, Henrique de Moraes Holschuh <hmh@hmh.eng.br> wrote:
> [...]
>> Search the list archives, there are other problem reports for that
>> 3ware card, you probably want to check those.
>
> Thank you very much, I checked these but I could not find much about
> filesystem corruption problems. The most similar problem was mentioned
> in http://lkml.org/lkml/2008/6/23/230 but was related to xen which
> isn't in use here.
>
> Regarding the memory test, memtest86+ ran for almost 70 hours without
> detecting any error.
>
> I trying to reproduce the problem using my installation disk and
> running stress/dbench but wasn't able to do so. The only reliable way
> to reproduce it is to install Ubuntu, which isn't very helpful :-(

Huh. It may be related to whatever kernel version Ubuntu uses to boot
up the install media. Try a different version of Ubuntu (older, newer
if a newer one exists) to see if it has the same problem.

Re: Spurious Filesystem corruption with ext3 + large (<400GB) hw RAID partition + SMP

[Manfred Wassmann]

On Fri, Feb 6, 2009 at 19:50, Ray Lee <ray-lk@madrabbit.org> wrote:
> Huh. It may be related to whatever kernel version Ubuntu uses to boot
> up the install media. Try a different version of Ubuntu (older, newer
> if a newer one exists) to see if it has the same problem

Apparently google mail junks any references headers when replying to a
mail with a modified subject, how bad is that!
In short for those who didn't get it, the problem was neither an ext3
nor a Ubuntu problem but apparently an obscure hardware problem,
details can be found in
<6bd94bae0902130829r15e6eacag25220487457027ad@mail.gmail.com>

Regards Manfred

Re: Spurious Filesystem corruption with ext3 + large (<400GB) hw RAID partition + SMP

[Manfred Wassmann]

On Sat, Jan 31, 2009 at 4:19 PM, Henrique de Moraes Holschuh
<hmh@hmh.eng.br> wrote:
> Please reply to the list... and it is probably useful to add which kind of SATA chip the 3Ware card uses.
> I only asked for some information I knew would be required, but probably I won't be able to help much.

Thank you very much and sorry, my reply was intended to go to the list
but it's the first time I use google mail with a list and it is much
different.

Well I have neither hardware nor documents around here and the dmesg
output doesn't tell me anything more than I already stated :(

>> I didn't think it could be the memory because the problem occurs with
>> the 32bit kernel alone but what do you suggest? I only have remote
>> access to the system over the weekend.
>
> Try one of the memtest utils that don't need to run from the boot over the weekend. If nothing fails, run the real thing on Monday.

OK memtest is running but there is one other thing I forgot to
mention, we had the same problem on two different machines.

Furthermore it always affects the ext3 filesystem and not the ext2
partition which is much smaller with only 1GB and mounted r/o on the
final system, but that is where most of the software gets installed.
And the problem is triggered at installation time so don't you think
it should more likely hit the ext2 partition if it is not an ext3
problem?

regards Manfred Wassmann
-- 
Unix "Birthday" on 2009-02-13 23:31:30 UTC
...it's 1234567890 seconds since the epoch.